betaFeedback

WorksComputer Science

Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

Abstract

Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training faster at long context. We propose two variants, Distilled-LSA (D-LSA), which can be applied to pre-trained models via distillation, and Native-LSA (N-LSA), which is trained natively on the next-token loss alone. On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks. N-LSA matches the accuracy of full attention with a training speedup of 1.6x per layer at 128k tokens. LSA offers practical and efficient attention for models to benefit from ultra-long context.

Introduction

Large language models have moved from answering single requests to agentic workflows in long-horizon tasks, such as software engineering and open-ended scientific research [33] (Lu et al. 2026). These tasks keep growing in length [38], and models with longer context windows tend to score higher on them [97]. In line with this trend, recent models extend their context windows to 1M tokens and beyond [21, 55, 15, 64].

At this context size, attention dominates the cost, as its work grows with context [78, 11]. An established technique to reduce such costs is sparse attention [23], in which each query attends to only a subset of the keys, chosen by a fixed pattern [9, 5, 89], by their content [36, 68], or by a dedicated indexer [87, 14]. Recent frontier models are increasingly adopting sparse

attention, in which keys are selected using a dedicated indexer ([14, 64]).

However, current indexers still score every preceding key, or block of keys, for each query. This remains expensive (quadratic in prefill, linear in decode), and becomes a bottleneck at long context ([91] (Xu et al. 2026)). For example, at 10M tokens, 98% of the prefill time of a sparse attention layer is spent on indexing (Figure 2). The cost of sparse attention at ultra-long context is therefore dominated by the cost of finding the keys.

Figure 2. Traditional sparse attention at ultra-long context. One global attention layer of GLM-5.3-FLASH (512k to 10M tokens) on one H200, prefilling the last chunk of 32,768 queries and decoding one token. Speedup of sparse attention over full attention (blue), share of the indexer (%), and potential speedup over sparse attention if the indexer were ‘free’ (green).

To address this, we introduce Latent Sparse Attention (LSA), which clusters keys in a learned latent space and lets each query search the clusters instead of scoring every key. This can be done using only one level of clustering (Flat), or with a hierarchy of clusters (Tree), which makes the prefill subquadratic. Furthermore, we propose two trainable designs: D-LSA, which adds LSA to pre-trained models and only trains small projection matrices per layer (contribution #1), and N-LSA, which trains LSA natively with the Next-Token Prediction (NTP) loss alone (contribution #2). We evaluate state-of-the-art models (QWEN-3.8-27B and GLM-5.3-FLASH) beyond their advertised context, up to 10M tokens (contribution #3), on RULER ([26]), which is an established long-context benchmark, and PaperTrail, which we developed to test long-context scientific understanding by finding and connecting facts across unseen scientific papers (contribution #4). Finally, we integrate LSA into SGLang ([96])2 with context parallelism, provide GPU kernels for H200 and B300, and release the code3 with all training and inference details (contribution #5). Figure 1 outlines our contributions.

Figure 1. D-LSA and N-LSA overview. Top: one attention layer. The keys are pooled and clustered into buckets. A query (🔴) scores the bucket centroids (🔺) and attends only to the selected long-range keys (plus sink and window). Bottom: accuracy and performance (P: prefill, D: decode, T: training) for both D-LSA and N-LSA. D-LSA is shown with flat search. Its subquadratic variant, D-LSA (Tree), follows a similar trend (Figure 5).

Sparse Attention

In full attention, every query attends to every preceding key. Prefilling a context of TT tokens therefore costs O(T2)O(T^2), and decoding one token costs O(T)O(T). In sparse attention, the work is split between an indexer and the attention. For each query, the indexer selects which keys the attention attends to4. With a fixed budget of kk keys, the attention then costs O(Tk)O(Tk) to prefill and O(k)O(k) per decoded token. If the indexer selects the most relevant keys, this is sufficient to stay close to the accuracy of full attention (Appendix § 9).

However, the indexer has a cost. Since it must score all preceding keys, it typically remains O(T2)O(T^2) in prefill and O(T)O(T) in decode, though with reduced constant factors. This makes ultra-long context still prohibitively expensive. For example, at 10M tokens the indexer takes 98% of the prefill time of traditional sparse attention (Figure 2), and a ‘free’ indexer would make sparse attention a further 58× faster in prefill and 8.9× in decode.

Design Choices

Sparse attention methods can be categorised by their indexer designs and how they are trained.

Token Sparse Attention. For each query, a lightweight indexer (fewer heads, smaller dimension, lower precision) scores every preceding key and selects which to attend to.

Block Sparse Attention. Consecutive keys are grouped into blocks, and each query scores blocks instead of every key. All keys of the selected blocks are attended to.

Semantic Sparse Attention. Keys are clustered by similarity rather than position, and each query scores one centroid per cluster. All keys of the selected clusters are attended to.

For the last two categories, a query can search the groups in three ways. Flat search scores every group directly, which is O(T)O(T) work per query. Fixed search adds a shallow hierarchy and refines from coarse to fine groups, which is also O(T)O(T) per query. Tree search descends a deeper hierarchy and expands only the groups it keeps at each level, which is O(log⁡T)O(\log T) per query.

An indexer can be training-free, distilled from a pre-trained model ([71]), or trained natively on the NTP loss alone.

Table 1. Sparse attention methods compared. Rows are grouped by the unit the indexer scores: single keys, blocks of consecutive keys, or clusters of similar keys. Design column: compression of the sequence the indexer scans by compressing consecutive keys (C. Seq.), or clustering keys by content (C. Sem.), and its search: flat (●●●), a fixed number of levels (♟), or levels efficiently growing as log⁡T\log T (♟♟). Evaluation: the largest context at which accuracy (Acc.) and performance (Perf.) are reported. ✓ supported, ◐ partial support, × not supported.

Related Work

Table 1 provides an overview of the existing work on sparse attention, as categorised above. We focus on methods that keep the whole KV cache and select keys per query, unlike fixed sparse patterns (Child et al., [9]; Beltagy et al., [5]; Zaheer et al., [89]) and KV cache eviction techniques (Zhang et al., [94]; Li et al., 2024; Xiao et al., [83]).

NameTraining FreeTraining DistilledTraining NativeSpeedup PrefillSpeedup DecodeDesign C. Seq.Design C. Sem.Design SearchEvaluation Acc.Evaluation Perf.
DSA (DeepSeek-AI, [14])×✓×✓✓××●●●128k128k
MiniMax SA (Lai et al., [39])×✓×✓✓××●●●128k1M
Block Sparse Attention
CSA (DeepSeek-AI, [15])×✓×✓✓✓×●●●1M1M
QSA (Qwen Team, [64])×✓×✓✓✓×●●●1M1M
GLM-5.3-FLASH SA (Z.ai, [90])××✓✓✓✓×●●●1M1M
NSA (Yuan et al., [87])××✓✓✓✓×●●●64k64k
HSA-UltraLong (Hu et al., 2026b)××✓✓✓✓×●●●16M256k
MoBA (Lu et al., 2025)××✓✓✓✓×●●●1M10M
SeerAttention (Gao et al., [19, 20])×✓×✓✓××♟128k128k
LongCat LSA (Zan et al., [91])✓××✓✓✓×♟1M1M
HISA (Xu et al., 2026)✓××✓✓✓×♟128k128k
HiP (Lee et al., [40])✓××✓✓×✓♟128k512k
PISA (Tang et al., [74])××✓✓✓××♟16k256k
Semantic Sparse Attention
ClusterKV (Liu et al., [48])✓×××✓×✓●●●32k32k
RetroInfer (Chen et al., [8])✓×××✓×✓●●●1M1M
Squeezed Attention (Hooper et al., [25])✓××◐✓×✓♟32k512k
Multipole Attention (Hooper et al., [24])✓×××✓×✓♟128k128k
LycheeCluster (Li et al., [43])✓×××✓✓✓♟128k72k
IceCache (Mao et al., [54])✓×××✓✓✓♟250k300k
LSA✓✓✓✓✓✓✓♟10M10M

Table 1.

Among distillation methods, DSA (DeepSeek-AI, [14]) and MiniMax SA (Lai et al., [39]) first distil the indexer and then continue training the full model under sparse attention. Alternatively, SeerAttention (Gao et al., [19, 20]) distils a gate over pooled blocks and HashAttention (Desai et al., [17]) learns binary codes of queries and keys, both with the model frozen. As the context grows, the indexer itself becomes the bottleneck (Xu et al., 2026; Zan et al., [91]; Bai et al., 2026). To reduce its cost, CSA (DeepSeek-AI, [15]), QSA (Qwen Team, [64]), and GLM-5.3-FLASH SA (Z.ai, [90]) compress every four keys into one before scoring them. HISA (Xu et al., 2026) and LongCat LSA (Zan et al., [91]) score blocks first and then only the keys inside the selected blocks. IndexCache (Bai et al., 2026), LongCat LSA and DeepSeek-V4.1 (DeepSeek-AI, [16]) reuse the selection of one layer in the following layers. Each improvement divides the cost of the indexer by a constant, and its work per query therefore still grows linearly with the context.

NSA (Yuan et al., [87]) and MoBA (Lu et al., 2025) instead pre-train the model with sparse attention from scratch, such that the indexer learns from the NTP loss alone. NSA scores each block by a learned compressed key, and MoBA by the mean of its keys. InfLLM-V2 (Zhao et al., [95]) and SSA (Mao et al., [53]) bring such block selection to existing dense models by fine-tuning. HSA (Hu et al., 2025; Leng et al., [42]; Hu et al., 2026a) and HSA-UltraLong (Hu et al., 2026b) also weigh the output of each block by its score, such that the loss trains the scores directly. All of these methods still scan every block. PISA (Tang et al., [74]) replaces this scan with a tree, but evaluates accuracy only at small scale (up to 16k tokens).

At the other end, Quest (Tang et al., [75]), InfLLM (Xiao et al., [82]) and MInference (Jiang et al., [32]) select blocks based on simple summaries of their keys without any training. HiP (Lee et al., [40]) searches the blocks with a tree, and its successor InfiniteHiP (Lee et al., [41]) with three fixed stages. BLASST (Yuan et al., [88]) instead skips blocks directly inside the attention kernel, without a dedicated indexer. Most of these methods summarise blocks of consecutive keys, which may hide a single relevant key (Xiao et al., [82]; Hu et al., 2026a).

Closest to LSA are methods that organise the keys by semantic content rather than by position. A few early works train at short context, with Reformer (Kitaev et al., [36]) hashing keys into buckets, Routing Transformer (Roy et al., [68]) learning k-means centroids, and Treeformer (Madaan et al., [52]) routing each query down a learned tree. Most other works here only operate at inference time: ClusterKV (Liu et al., [48]) and RetroInfer (Chen et al., [8]) cluster the keys during prefill and score every centroid at decode (flat search), Squeezed Attention (Hooper et al., [25]), Multipole Attention (Hooper et al., [24]) and LycheeCluster (Li et al., [43]) search two levels (fixed search), and IceCache (Mao et al., [54]) descends a tree of keys (tree search), while RetrievalAttention (Liu et al., [47]) links the keys in a nearest-neighbour graph. All of these inference methods are training-free, only Squeezed Attention speeds up prefill, and only partially, and none are evaluated at ultra-long context (above 1M tokens).

Latent Sparse Attention (LSA)

LSA is a sparse attention layer for ultra-long context. It clusters keys by content, in a space that is learned: through index projections in D-LSA, and through the attention’s own keys in N-LSA. LSA can replace the attention of a pretrained layer by distillation (D-LSA) or be trained natively with the model (N-LSA).

In both settings, the indices for the attention are selected from the keys and queries by the same process. We first describe this process, numbered as in Figure 3, which consists of semantic clustering of the index keys kj(idx)k_j^{(\mathrm{idx})}, and the search with the index queries qi(idx)q_i^{(\mathrm{idx})} so that each query selects its keys. We then describe the specifics of D-LSA and N-LSA.

Figure 3. D-LSA design. Left: D-LSA (Flat). Indexer keys: ❶ projection and L2 norm, ❷ pooling (4:1), ❸ bottom-up agglomerative clustering (Ward’s method), ❹ bucket cache (centroids and index table). Indexer queries: A sum over query heads, projection and L2 norm, B scoring the cached centroids and selecting the top-pp buckets per query, C expanding the buckets into indices (sink + top-kk long-range + window), D attention over those indices (one index list per query shared by all query heads). Right: D-LSA (Tree) groups the buckets into a hierarchy of clusters, a query descends using beam search (close in the latent space but scattered through the KV cache).

Semantic Clustering

❶ Key normalisation. Each index key kj(idx)k_j^{(\mathrm{idx})} is first L2 normalised: kj(idx)←kj(idx)∥kj(idx)∥k_j^{(\mathrm{idx})} \leftarrow\frac{k_j^{(\mathrm{idx})}}{\lVert k_j^{(\mathrm{idx})}\rVert}

❷ Pooling. Consecutive keys are then compressed, as in CSA (DeepSeek-AI, [15]), QSA (Qwen Team, [64]Z.ai, 2026`), to improve computational efficiency. Here, every npooln_{\mathrm{pool}} consecutive index keys are summed into one pool key kpool(idx)k_{\mathrm{pool}}^{(\mathrm{idx})}, so that clustering runs over T/npoolT/n_{\mathrm{pool}} pool keys rather than TT keys.

❸ Clustering. Keys are clustered in cluster blocks of nclustern_{\mathrm{cluster}} consecutive keys. Once a cluster block contains all its keys, its pool keys are clustered bottom up with Ward’s method (Ward, [81]) into mm buckets (Algorithm 1 in Appendix § 10). Each pool key starts as a bucket of its own, and each bucket keeps the sum of its keys, whose normalisation is its centroid cbc_b. In every round, all pairs of buckets that are each other’s cheapest merge are merged in parallel. The cost of a merge is how much it increases the keys’ total squared

distance from their centroid (Ward’s criterion). Merging stops once there are m=ncluster/nˉbm = n_{\mathrm{cluster}/\bar{n}_b} buckets, where nˉb\bar{n}_b is the average number of keys per bucket, such that every cluster block holds the same number of buckets. A merge is skipped if the merged bucket would hold more than nbmax⁡n_b^{\max} keys.

❹ Bucket cache. The bucket cache Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})} stores the centroid cbc_b of every bucket and an index table of the positions of its keys. This is so later selected buckets can ‘expand’ into their key indices. With one centroid per nˉb\bar{n}_b keys, it corresponds to a small fraction of the KV cache.

Once a cluster block is clustered, its buckets never change. In the Tree variant, this clustering process is repeated to group the buckets into multiple levels of a hierarchy.

Semantic Search

We now describe how the top-kk key indices are extracted for each query, and then submitted to attention. We first describe the process for Flat, in which there is only one level of buckets, and then discuss Tree.

A Query normalisation. Each index query qi(idx)q_i^{(\mathrm{idx})} is first L2 normalised: qi(idx)←qi(idx)∥qi(idx)∥q_i^{(\mathrm{idx})} \leftarrow\frac{q_i^{(\mathrm{idx})}}{\lVert q_i^{(\mathrm{idx})}\rVert}

B Scoring. The index query scores the unit-norm centroid of every bucket in cluster blocks before its own via si,b=cb⋅qi(idx)s_{i,b}=c_b\cdot q_i^{(\mathrm{idx})}, which compares buckets of any size on one scale, and selects the top-pp buckets. Its own cluster block is not clustered yet, and its keys are covered by the window instead.

C Expansion. The selected pp buckets are expanded into their keys, best first, until kk ‘long-range’ keys are selected. As every bucket holds at least npooln_{\mathrm{pool}} keys, pp is chosen such that p⋅npool≥kp \cdot n_{\mathrm{pool}} \ge k, and so expansion always fills kk keys.

Besides the kk long-range keys, every query attends to nsinkn_{\mathrm{sink}} sink keys [83] and the nwn_w most recent keys (window). A key of a selected bucket that falls in the sink or the window is already attended to, and the expansion moves on to the next key instead. We set the window to one cluster block, nw=nclustern_w = n_{\mathrm{cluster}}, so that the window covers every key not yet clustered. This gives each query ii a set JiJ_i of at most nsink+k+nwn_{\mathrm{sink}} + k + n_w key indices.

D Attention. Attention is then computed only over the keys indexed by JiJ_i. A single JiJ_i is shared across all query heads.

Tree search arranges the buckets into a hierarchy (Figure 3, right). Level 0 holds the buckets, and each higher level ℓ≥1\ell\ge1 holds buckets we term ‘node buckets’, each made of nb,ℓn_{b,\ell} buckets of the level below, with a centroid over all the keys it contains. As at level 0, buckets are merged one cluster block at a time. Buckets that are not merged yet are defined as open. Once a level holds ncluster,ℓn_{\mathrm{cluster},\ell} open buckets, they are merged into node buckets as in Algorithm 1.

Beam search is used to select the buckets through the hierarchy (Algorithm 2 in Appendix § 10). On each level, the query is scored against the centroids of the candidate node buckets, and the best pℓp_\ell are kept. The candidates on the level below are the children of the kept node buckets and the open buckets of that level. At level 0, the top-pp buckets are selected, as in the flat search. With a constant beam per level, a query scores O(log⁡T)O(\log T) centroids, which makes the prefill of the indexer subquadratic, O(Tlog⁡T)O(T \log T), where the flat search is quadratic.

D-LSA

D-LSA (Figure 3) follows the process outlined above, but since it distils from a pre-trained model which already has key and query projections (kj(attn)k_j^{(\mathrm{attn})} and qi(attn)q_i^{(\mathrm{attn})}), we additionally learn separate key and query projections for LSA. In particular, each key kj(attn)k_j^{(\mathrm{attn})} and query qi(attn)q_i^{(\mathrm{attn})} is projected to a smaller dimension d(idx)d^{(\mathrm{idx})} to keep the scoring, clustering, and bucket cache cheap:

kj(idx)=kj(attn)WK(idx)qi(idx)=∑hqi,h(attn)WQ(idx)(1)\begin{aligned} k_j^{(\mathrm{idx})} &= k_j^{(\mathrm{attn})} W_K^{(\mathrm{idx})} \\ q_i^{(\mathrm{idx})} &= \sum_h q_{i,h}^{(\mathrm{attn})} W_Q^{(\mathrm{idx})} \tag*{(1)} \end{aligned}

The query heads are summed such that one index query serves every head, which yields the single index list JiJ_i per query. On QWEN-3.8-27B, whose keys and queries carry RoPE (Su et al., 2024) on a quarter of their dimensions, the projections read only the RoPE-free part. These keys and queries then go into the normalisation process as before. Trainable projections can also be replaced by the leading d(idx)d^{(\mathrm{idx})} principal directions of the keys in the layer, which project both keys and queries into one index space with no training. We call this variant D-LSA (Training-Free).

The base model’s attention is otherwise unchanged: its own queries attend only to the keys and values indexed by JiJ_i.

Distillation. D-LSA adds LSA to a pre-trained model by a short distillation that trains only WQ,K(idx)W_{Q,K}^{(\mathrm{idx})} and a temperature τ\tau per layer, with the base model frozen. This mirrors the dense warm-up of DSA [14], which also trains only the indexer against the head-aggregated attention of the frozen model. The distillation scores every key directly, without clustering. For a query ii of a layer, let RiR_i be the keys it can select (all preceding keys outside the sink and the window). The target AiA_i is the attention of that frozen layer over RiR_i, the softmax of each head’s logits over RiR_i averaged over its nhn_h heads of dimension dhd_h, and A^i\hat{A}_i is the softmax of the D-LSA scores over the same keys. The distillation minimises the forward KL divergence between the two:

Li=KL⁡(Ai∥A^i)\mathcal{L}_i = \operatorname{KL}(A_i \parallel\hat{A}_i)
Ai,j=1nh∑hsoftmax⁡j∈Ri(qi,h(attn)⋅kj(attn)/dh)(2)A_{i,j} = \frac{1}{n_h} \sum_h \operatorname{softmax}_{j \in R_i}\left(q_{i,h}^{(\mathrm{attn})} \cdot k_j^{(\mathrm{attn})} / \sqrt{d_h}\right) \tag*{(2)}
A^i,j=softmax⁡j∈Ri(qi(idx)⋅kj(idx)/τ)\hat{A}_{i,j} = \operatorname{softmax}_{j \in R_i}\left(q_i^{(\mathrm{idx})} \cdot k_j^{(\mathrm{idx})} / \tau\right)

The temperature τ\tau sharpens the softmax of the D-LSA scores to match the peaked attention of the model, and is used only in this loss. Both projections are initialised with these principal directions.

Kernels. We write GPU kernels for the clustering, scoring, selection and expansion of the indexer, in CuTe DSL for H200 and B300, with a Triton version of each, which runs on any GPU. For the attention, we reuse existing attention kernels, such as FlashAttention-3 [72], which read the index list JiJ_i as a page table.

Parallelism. We integrate LSA into SGLang with support for tensor, expert, and pipeline parallelism, and for context parallelism in both prefill and decode. Context parallelism splits the KV cache across ranks by token, but LSA replicates its bucket cache on every rank. Every rank thus builds the same index and selects the same index list. The ranks only exchange the index keys of the newest tokens. This design choice trades memory for communication. At 10M tokens on GLM-5.3-FLASH with context parallelism over 8 GPUs, each rank holds a 1.25 GiB share of the 10 GiB KV cache of a layer and the full 246 MiB bucket cache.

We provide further details on kernels in Appendix § 14.§ 1 and on parallelism in Appendix § 14.§ 2.

N-LSA

N-LSA (Figure 4) trains LSA natively with the model, using only the NTP loss. As the model learns with the selection from the start, we cluster the keys of the attention itself, without dedicated index projections. We use a single key-value head, such that one key per token serves every query head, and score each bucket by the best similarity any query head gives its centroid:

Figure 4. N-LSA design. The attention layer of N-LSA, which can be trained from scratch on the next-token loss alone.

si,b=max⁡hq^i,h⋅cb(3)s_{i,b} = \max_h \hat{q}_{i,h} \cdot c_b \tag*{(3)}

where q^i,h\hat{q}_{i,h} is the normalised query of head hh. We stop the gradient through the selection, and the NTP loss trains the attention projections only through the keys each query attends to, the selected keys plus the sink and the window. N-LSA therefore adds no parameters to the model and needs no teacher, no index projections and no auxiliary loss.

The attention layers of N-LSA use no positional encoding, as is becoming more common in hybrid models, and RoPE could be supported by clustering only the RoPE-free part of the keys, as D-LSA does on QWEN-3.8-27B.

Evaluation

In this section, we describe the models, the methods we compare, and their settings. Then, we evaluate the accuracy of D-LSA up to 10M tokens, and its performance per attention layer and end to end. Finally, we show that N-LSA can be trained natively at an accuracy close to full attention.

Setup

Models. D-LSA aims to keep the accuracy of a model while making its attention significantly cheaper. We compare each model with D-LSA against its baseline attention, in accuracy and in performance. We demonstrate D-LSA on both a model with full attention (QWEN-3.8-27B ([65 c])) and one with sparse attention (GLM-5.3-FLASH ([90])). On QWEN-3.8-27B, we also replace the attention with block sparse attention, inspired by QSA ([64]), as a second sparse baseline. We also compare flat to tree search where possible.

Methods. We compare the following methods:

  • ▪ Full Attention: every query attends to every preceding key, the baseline attention of QWEN-3.8-27B.

  • ▪ Block Sparse Attention (Flat): sparse attention whose indexer scores every block of four consecutive keys, as in QSA ([64]) and GLM-5.3-FLASH ([90]).

  • ▪ Block Sparse Attention (Tree): sparse attention whose indexer searches a tree over blocks of the keys of the attention itself, as in HiP ([40]) and PISA ([74]). To represent these methods, we search a tree over blocks of keys directly on the KV cache, without training, which uses the full keys and favours accuracy over performance. We use blocks of eight keys, the most accurate of the sizes we compare (Appendix § 13.§ 7).

  • ▪ D-LSA (Flat) and D-LSA (Tree): D-LSA with a flat search over every bucket centroid and with a tree search over the hierarchy of buckets. In decode, each request processes one token per step, and the tree search runs its levels one after another, which pays off only when many requests decode together. As we evaluate one request at a time, D-LSA (Tree) decodes with the flat search (Appendix § 14.§ 1).

  • ▪ D-LSA (Training-Free): D-LSA (Flat) with the principal directions of the keys as both projections instead of trained ones.

Every trained indexer we compare is distilled in the same way to ensure fairness.

Design parameters. Table 2 lists the parameters of every method per model. For D-LSA, we found that a budget of k=4096k = 4096 long-range keys holds accuracy past 1M tokens better than smaller budgets, and these extra keys cost little next to the scan of the indexer they replace. On GLM-5.3-FLASH, D-LSA (Flat) selects k=2048k = 2048 long-range keys for queries before position 1M and k=4096k = 4096 beyond. We found that the smaller budget keeps accuracy below 1M tokens while making the attention faster. Tuning the budget further, for example per layer or per request, is left to future work. A small index space of d(idx)=64d^{(\mathrm{idx})} = 64 dimensions keeps the scoring and the clustering cheap, and we found wider spaces to add little accuracy.

QWEN-3.8-27BGLM-5.3-FLASH
Unitblockblockbucketbucketblockblockbucketbucket
Search•••♣•••♣•••♣•••♣
kk2,0484,6084,0964,0962,0484,6082,048/4,0964,096
pp5125761,0241,024512576512/1,0241,024
nsink,nwn_{\mathrm{sink}}, n_w-, -8, 164, 5124, 512-, -8, 164, 5124, 512
didxd_{\mathrm{idx}}12825664641285126464
npooln_{\mathrm{pool}}48444844
nclustern_{\mathrm{cluster}}--512512--512512
nˉb,nbmax⁡\bar{n}_b, n_b^{\max}--8, 168, 16--8, 168, 16
nb,ℓn_{b,\ell}-2-4-2-4
ncluster,ℓn_{\mathrm{cluster},\ell}---4,096---4,096
pℓp_\ell-576-2,048-576-2,048

Table 2. Parameters. Block Sparse Attention (Flat), Block Sparse Attention (Tree), D-LSA (Flat) and D-LSA (Tree) per model (symbols in Table 5). The search is flat (•••) or tree (♣), over blocks of npooln_{\mathrm{pool}} consecutive keys or over buckets. Block Sparse Attention (Tree) always keeps the first, previous and current blocks among its pp blocks. On GLM-5.3-FLASH, D-LSA (Flat) uses the first kk and pp below 1M tokens and the second beyond.

Indexer distillation. We train on 352M tokens of documents of 16k to 1M tokens from seven domains, for 48k steps, each on 128 queries sampled from one document, with a learning rate of 3⋅10−43 \cdot10^{-4}. This is a fraction of the 2.1B tokens of the dense warm-up of DSA. In Equation (2), the target AiA_i keeps only the 2,048 keys with the most attention weight per query, renormalised over them, while A^i\hat{A}_i still spans all of RiR_i, and τ\tau is calibrated per layer before the first step. Unlike DSA, we do not follow the distillation by training the whole model under sparse attention (sparse training), a step D-LSA also supports but we omit here due to its cost. We provide further details about the composition and construction of the dataset in Appendix § 11.

Benchmarks. We evaluate accuracy in three categories, each with a representative set of benchmarks. For knowledge, we use MMLU-Pro [80] (a stratified sample of 2,000 questions) and GPQA-Diamond [67]. For reasoning, we use AIME25 (four repeats) and MATH500 [46] (100 problems). For long context, we use LongBench v2 [3] (60 questions) and the 13 tasks of RULER [26] from 4k to 10M tokens (100 prompts per task up to 1M and 20 beyond). Evaluating each benchmark in full at ultra-long context is prohibitively expensive, and we therefore use the subsets given in parentheses.

PaperTrail. To better understand model capabilities in a less synthetic ultra-long context setting, we extend RULER with a set of new tasks called PaperTrail. The tasks are constructed by embedding a trail of fabricated facts in a large collection of scientific papers. All papers are recent, almost all published after the release date of the models, such that no model could have seen them in training. A question can only be answered by finding the facts of its trail and connecting them, as in a literature search. Samples are built in three domains: exoplanets, long-context machine learning and quantum error correction. The questions come in three tiers of increasing difficulty, where tier 1 recalls a fact from one paper, tier 2 links facts across two papers and tier 3 traces a trail across three or more. We provide further details about the construction of PaperTrail in Appendix § 12.

Model sampling. We configure each model’s sampling as recommended by its model card with generation up to 163,840 tokens (Appendix E). QWEN-3.8-27B reaches 10M tokens with YaRN [63] at factor 40 (42 for PaperTrail at 10M), while GLM-5.3-FLASH needs no extension as it has no explicit positional encoding.

Hardware and serving. We serve all models in SGLang [96] on H200 and B300 GPUs. To show that D-LSA can be integrated with the main parallelism strategies, we evaluate it with tensor parallelism (TP8) and with context parallelism in both prefill and decode (CP8). We show more details in Appendix § 14.§ 2. All accuracy measurements use context parallelism wherever the KV cache of a request outgrows the memory of a single GPU.

Accuracy

Figure 5 shows that on knowledge, reasoning and long-context tasks, D-LSA stays close to the accuracy of baseline Full Attention on QWEN-3.8-27B and of baseline sparse attention on GLM-5.3-FLASH. PaperTrail further shows how well the models find and connect facts across large collections of scientific papers.

Figure 5. D-LSA accuracy is on par with full attention and sparse attention baselines on downstream tasks. Shown on knowledge, reasoning and long-context tasks up to 10M tokens. Baseline scores are given with their 95% confidence interval.

Knowledge and reasoning. On knowledge, reasoning and LongBench v2, D-LSA stays within the 95% confidence interval of both baselines, even with reasoning growing to tens of thousands of tokens (Appendix E). We further show that D-LSA does not measurably affect the length of the reasoning.

Long context. On QWEN-3.8-27B, D-LSA (Flat) stays within 5 points of baseline Full Attention up to 1M tokens and exceeds it at 10M tokens, at 63.9 against 59.9 on RULER and 29.1 against 19.8 on PaperTrail. It also stays above Block Sparse Attention (Flat) on RULER from 128k tokens on, in a fair comparison, as both indexers are distilled the same way.

On GLM-5.3-FLASH, D-LSA (Flat) stays within 3 points of baseline sparse attention up to 1M tokens, trails it by up to 8 points at 2M to 5M tokens, and is within 4 points on RULER at 10M tokens. This comparison favours the baseline, which was trained together with

D-LSA (Flat) is distilled from attention that is already sparse (with no finetuning of the base model itself). On both models, MultiKey 2, MultiKey 3 and CWE fall the most for every method, the baselines included. We break down RULER by task and PaperTrail by tier in Appendix § 13.

Design comparison. D-LSA (Tree) follows D-LSA (Flat) on QWEN-3.8-27B, but loses accuracy past 1M tokens on GLM-5.3-FLASH, at 55.8 against 66.8 on RULER at 10M tokens. Block Sparse Attention (Tree) falls behind from 512k tokens, to about 63 on RULER at 1M tokens on both models, against 78 and 88 for D-LSA (Tree), and we therefore do not evaluate it beyond 1M tokens. Large blocks fall earlier (Appendix § 13.§ 7). D-LSA (Training-Free), without any training, stays close to D-LSA (Flat) on QWEN-3.8-27B, but trails it on GLM-5.3-FLASH.

D-LSA follows the accuracy of the baselines, showing that our semantic clustering can preserve accuracy up to 10M tokens. However, we do observe a steep drop in accuracy beyond 512k tokens for Block Sparse Attention (Tree).

Performance

Next, we evaluate the performance of D-LSA. We first measure a single attention layer and then the whole model in production serving.

Methodology. For one layer, we time the whole attention module, including its projections, the indexer and the attention, on a single GPU. In prefill, we time the last chunk of 32,768 queries before the given context. In decode, we time one step of one request with CUDA graph enabled (clustering is amortised across the steps). For the whole model, we serve each method in SGLang with prefill chunks of 32,768 tokens, and measure the time to first token (TTFT) and the time per output token (TPOT) of one request at a time, the median of three requests of 256 output tokens.

Figure 6 shows that the prefill speedup of D-LSA grows with the context. On GLM-5.3-FLASH, it rises from 3.5× at 1M to 22.7× at 10M tokens over baseline sparse attention, and on QWEN-3.8-27B to 72.4× at 10M tokens over baseline Full Attention. Below 1M tokens, the gain is smaller, at 2.1× at 512k tokens, as the scan of every block is still cheap. Extrapolated beyond 10M tokens, the prefill of D-LSA (Tree) grows subquadratically and overtakes the flat search, and at 100M tokens it is predicted to be more than 90× faster than baseline sparse attention.

Figure 6. D-LSA speedup of one global attention layer. Prefill and decode speedup over the baseline attention (log scale), Full Attention for QWEN-3.8-27B and Block Sparse Attention (Flat) for GLM-5.3-FLASH, required memory at 10M with one index key per block of four keys for Block Sparse Attention (Flat), and prefill scaling up to 100M (log-log), extrapolated beyond 10M (shaded).

Decode follows the same trend, with a speedup up to 4.6× over baseline sparse attention and 42.5× over baseline Full Attention at 10M tokens, with D-LSA (Tree) decoding with the flat search. On B300, D-LSA reaches 12.5× and 42.9× in prefill and 5.2× and 18.4× in decode at 10M tokens.

D-LSA also requires less memory for its indexer. Baseline sparse attention caches an index key for every block of four keys, while D-LSA caches one centroid in its smaller index space for every bucket of about eight keys, together with the id of each key. At 10M tokens, the index of D-LSA (Flat) takes 246 MiB per layer against 331 MiB in GLM-5.3-FLASH and 642 MiB in QSA. Larger buckets would shrink the index of D-LSA further.

End-to-end. Table 3 shows that the speedup of one layer carries over to the whole model at ultra-long context. On GLM-5.3-FLASH, D-LSA (Flat) is on par with baseline sparse attention up to 1M tokens with CP8 and already 1.5× faster to first token at 1M with TP8, and at 10M tokens reduces the TTFT by 2.2× and the TPOT by 2.1× with CP8, and by 5.8× and 1.9× with TP8. On QWEN-3.8-27B, it reduces them by 9.2× and 2.9× at 10M tokens against baseline Full Attention. On B300, D-LSA (Flat) reduces the TTFT and the TPOT at 10M tokens by 1.6× and 2.2× with CP8 and by 4.8× and 2.6× with TP8 on GLM-5.3-FLASH, and by 5.7× and 1.5× on QWEN-3.8-27B. The speedup of the whole model is expected to be smaller than that of one layer, as only the global attention layers become faster, while the other layers keep their cost.

SettingMethodTTFT (s) 512kTTFT (s) 1MTTFT (s) 10MTPOT (ms) 512kTPOT (ms) 1MTPOT (ms) 10M
H200; QWEN-3.8-27B; TP8 DCP8Full Attention21.765.34582.27.89.132.1
H200; QWEN-3.8-27B; TP8 DCP8Block Sparse Attention (Flat)1.61×2.30×8.02×0.89×0.91×0.89×
H200; QWEN-3.8-27B; TP8 DCP8D-LSA (Flat)1.03×1.51×9.21×0.87×1.00×2.87×
H200; QWEN-3.8-27B; TP8 DCP8D-LSA (Tree)0.96×1.41×8.83×0.87×1.00×2.87×
H200; GLM-5.3-FLASH; CP8 DCP8Full Attention8.518.1451.57.68.217.9
H200; GLM-5.3-FLASH; CP8 DCP8Block Sparse Attention (Flat)1.03×1.09×2.24×1.01×1.06×2.10×
H200; GLM-5.3-FLASH; CP8 DCP8D-LSA (Flat)0.93×0.98×2.14×0.97×1.03×2.10×
H200; GLM-5.3-FLASH; TP8Full Attention17.945.11300.95.86.318.9
H200; GLM-5.3-FLASH; TP8Block Sparse Attention (Flat)1.22×1.51×5.83×0.99×1.06×1.91×
H200; GLM-5.3-FLASH; TP8D-LSA (Flat)0.98×1.21×5.62×0.95×1.03×1.89×
B300; QWEN-3.8-27B; TP8Full Attention10.631.62172.24.96.027.3
B300; QWEN-3.8-27B; TP8Block Sparse Attention (Flat)0.79×0.82×0.84×0.82×0.90×1.33×
B300; QWEN-3.8-27B; TP8D-LSA (Flat)0.68×0.96×5.73×0.63×0.73×1.46×
B300; QWEN-3.8-27B; TP8D-LSA (Tree)0.63×0.91×5.40×0.63×0.73×1.46×
B300; GLM-5.3-FLASH; CP8 DCP8Full Attention6.513.5273.16.97.417.3
B300; GLM-5.3-FLASH; CP8 DCP8Block Sparse Attention (Flat)0.96×0.99×1.56×0.97×1.03×2.21×
B300; GLM-5.3-FLASH; CP8 DCP8D-LSA (Flat)0.78×0.81×1.48×0.96×1.03×2.22×
B300; GLM-5.3-FLASH; TP8Full Attention11.127.21186.45.25.715.6
B300; GLM-5.3-FLASH; TP8Block Sparse Attention (Flat)1.15×1.39×4.76×0.96×1.03×2.56×
B300; GLM-5.3-FLASH; TP8D-LSA (Flat)0.91×1.09×4.28×0.95×1.03×2.56×

Table 3. D-LSA speedup in production setting. Time to first token (TTFT) and time per output token (TPOT) of one request under SGLang. Full Attention, Block Sparse Attention (Flat), D-LSA (Flat) and D-LSA (Tree). Each block’s first row gives the baseline’s times and the others their speedup over it, with D-LSA at 10M in bold. QWEN-3.8-27B is served with tensor parallelism (TP8) and decode context parallelism (DCP8) on H200 and with TP8 on B300, GLM-5.3-FLASH with either context parallelism (CP8, with DCP8 and expert parallelism EP8) or TP8. Without DCP, Block Sparse Attention’s prefill indexer runs whole on every TP rank. All models run on a single node, except TP8 for GLM-5.3-FLASH at 10M on H200 (2 nodes with PP2); a B300 node holds it on one.

Native Training

Finally, we evaluate N-LSA, which is trained natively on the NTP loss alone.

Setup. We train a hybrid model of about 0.5B parameters with 24 layers, in which every fourth layer is an attention layer and the others are Gated DeltaNet layers (Yang et al., [86]), a form of linear attention (Schmidhuber, [70]; Schlag et al., [69]). Each attention layer has 8 query heads over a single key-value head of 128 dimensions, without positional encoding. We pre-train with a standard Warmup-Stable-Decay (WSD) learning rate schedule (Hu et al., 2024), on 100B tokens of FineWeb-Edu (Penedo et al., [62]) at a context of 4k tokens in the Warmup and Stable phases, followed by 12B tokens of LongDocs in the Decay phase, at 128k tokens (Appendix § 11). We compare N-LSA with Full Attention, Sliding Window Attention (SWA), and the native forms of block sparse attention, NSA (Yuan et al., [87]) for Block Sparse Attention (Flat) and PISA (Tang et al., [74]) for Block Sparse Attention (Tree), all in the same model architecture and on the same data. Every sparse method attends to about 2,560 keys per query. N-LSA selects k=2048k = 2048 long-range keys with flat search, next to a window of 512 keys and 4 sink keys. SWA attends to a window of 2,560 keys and 4 sink keys, NSA attends to 32 selected blocks of 64 keys and a window of 512 keys, and PISA attends to 40 blocks of 64 keys, which always include the first block and the two most recent blocks (window). We train N-LSA with flat search only, and leave native training with tree search to future work.

Figure 7 shows that N-LSA reaches the validation loss of Full Attention throughout the decay phase at 128k token context, at 1.085 against 1.085, while PISA, NSA and SWA trail at 1.100, 1.117 and 1.154. During pre-training at 4k tokens, all methods stay within 0.004 of each other, as their keys cover most of the context. Table 4 confirms this on downstream tasks. On commonsense reasoning ([10, 92]), all models are on par. On recall ([2]), N-LSA stays close to Full Attention and ahead of the other sparse methods. On RULER, N-LSA stays within 3.5 points of Full Attention up to 128k tokens, at 32.2 against 34.7 at 128k tokens, while NSA and PISA fall to 19.2 and 14.6, and SWA to 3.0 once the context outgrows its window.

N-LSA training speedup

Figure 7. N-LSA training speedup. Models trained natively on the NTP loss with Full Attention, SWA, NSA, PISA and N-LSA. Left: one attention layer, forward and backward, on one H200. Right: validation loss over the training time of the whole model in the decay phase (all on the same data and number of tokens).

CommonsenseRecallRULER
ARC-eARC-cHellaSwagSWDEFDA4k8k16k32k64k128k
Full Attention50.030.045.667.879.353.946.042.538.135.634.7
SWA53.931.445.264.669.626.715.610.15.93.03.0
NSA53.830.145.862.272.239.434.631.625.218.919.2
PISA52.228.745.563.778.656.248.440.426.619.114.6
N-LSA54.630.845.566.479.252.242.541.538.432.932.2

Table 4. Native training accuracy. Accuracy (%) of models trained from scratch with Full Attention, SWA, NSA, PISA and N-LSA. Every sparse method attends to about 2,560 keys per query.

N-LSA also trains faster than Full Attention at long context. For one attention layer on a single GPU, the forward and backward pass of N-LSA is 1.1× faster than Full Attention with FlashAttention-3 ([72]) at 64k tokens and 1.6× faster at 128k tokens (Figure 7, left). End to end, the whole model completes its decay phase 1.24× faster at the same validation loss (Figure 7, right), as only every fourth layer is an attention layer. At shorter contexts, the selection of the keys costs more than it saves. Its peak memory is higher, at 6.8 GiB against 5.8 GiB per layer at 128k tokens,

partly because N-LSA keeps the indices of the selected keys of every query for the backward pass, which grow linearly with the context.

Overall, N-LSA trains natively at the accuracy of Full Attention and becomes faster as the context grows.

Discussion

LSA reduces the cost of finding the keys, but not of storing them, and the KV cache still grows linearly with the context. It is orthogonal to methods that compress the KV cache (DeepSeek-AI, [15]), share the index across layers (Bai et al., (2026); DeepSeek-AI [16]), or predict several tokens per step (Gloeckner et al., (2024); DeepSeek-AI [13]), and could be combined with each.

We observed tree search to lose accuracy faster than flat search past 1M tokens on GLM-5.3-FLASH, while it overtakes flat search in decode only from about 5M tokens with one request, and is predicted to do so in prefill only beyond 10M tokens. It remains open whether sparse training would close this gap beyond indexer distillation.

We train N-LSA at a small scale, with about 0.5B parameters, up to 128k tokens and with flat search only. Scaling native training to larger models, longer contexts and tree search remains open.

Conclusion

At ultra-long context, the cost of sparse attention lies in finding which keys to attend to. We introduce LSA, which clusters the keys in a learned latent space and replaces the scan of every key with a search over the clusters, flat or along a tree. D-LSA distils LSA into pre-trained models and stays close to the accuracy of their baseline attention up to 10M tokens. At 10M tokens, one attention layer of D-LSA prefills 22.7× and decodes 4.6× faster than sparse attention in GLM-5.3-FLASH, and 72.4× and 42.5× faster than Full Attention in QWEN-3.8-27B. End to end, this reduces the time to first token by 2.2× and 9.2×, respectively. N-LSA trains LSA natively on the next-token loss alone, at the accuracy of Full Attention and 1.6× faster per layer at 128k tokens. With tree search, the cost of finding the keys grows subquadratically, which makes LSA a step towards attention that scales well beyond 10M tokens. LSA thus provides a practical way forward for ultra-long context attention, both for distilling pre-trained models and for native training.

Acknowledgements

We thank Tri Dao for his valuable feedback and support. We also thank Daniel Woods, Roman Iurkov, Cyril Kondratenko, Manuel Gomez, and the rest of the Nebius team for excellent support with the compute platform.

Attention Sparsity

To test how many keys a query needs to keep accuracy, we let each query of QWEN-3.8-27B attend only to the sink, the window and its true top-kk keys (Figure 8). At 128k tokens, Top-kk Oracle Attention keeps 98% of the RULER accuracy of Full Attention down to k=512k = 512, while each query attends to 0.8% of the keys. Single-needle retrieval holds down to k=16k = 16, while MultiKey 2, MultiKey 3 and CWE fall first in accuracy. Therefore, sparse attention loses little if it finds the most relevant keys, which is what LSA is designed to do without scoring every key.

Figure 8. Top-kk Oracle Attention on RULER. QWEN-3.8-27B at 128k tokens. Each query attends to a 4-token sink, a 512-token window and the top-kk long-range keys, one list shared by all heads. Process: sum over all heads, pick top-kk, renormalise per head over the shared selection.

Method Details

Table 5 lists the symbols of LSA. Algorithm 1 clusters one cluster block into buckets, and Algorithm 2 is the beam search of D-LSA (Tree).

SymbolDescription
TTcontext size
ii, jj, hhquery position, key position, head
WQ,K,V(attn)W^{(\mathrm{attn})}_{Q,K,V}attention projections
qi,h(attn)q^{(\mathrm{attn})}_{i,h}, kj(attn)k^{(\mathrm{attn})}_{j}attention query (head hh) and key
nhn_h, dhd_hquery heads, head dimension
WQ,K(idx)W^{(\mathrm{idx})}_{Q,K}, d(idx)d^{(\mathrm{idx})}index projections, index dimension
qi(idx)q^{(\mathrm{idx})}_{i}, kj(idx)k^{(\mathrm{idx})}_{j}index query and key, unit norm
RiR_ilong-range keys of query ii (without sink and window)
Ai,jA_{i,j}, A^i,j\hat{A}_{i,j}attention weight of the model / indexer for query ii and key jj
τ\tautemperature of the indexer softmax
npooln_{\mathrm{pool}}, kpool(idx)k^{(\mathrm{idx})}_{\mathrm{pool}}keys summed per pool key, pool key
nclustern_{\mathrm{cluster}}, mmkeys per cluster block, buckets per block
nˉb\bar{n}_b, nbmax⁡n_b^{\max}average and max number of keys per bucket
kbsumk_b^{\mathrm{sum}}, ∣b∣\lvert b\rvertbucket key sum, keys in bucket bb
aba_b, Δ(b,b′)\Delta(b,b')alignment, merge cost
cbc_b, Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})}bucket centroid (unit norm), bucket cache
ℓ\ell, nb,ℓn_{b,\ell}, ncluster,ℓn_{\mathrm{cluster},\ell}level, buckets per node, buckets per node block
pℓp_\ell, nstartn_{\mathrm{start}}node buckets kept per level, at most at the start level
si,bs_{i,b}, ppbucket score, buckets selected per query
kk, nsinkn_{\mathrm{sink}}, nwn_wlong-range, sink and window keys
JiJ_ikey positions query ii attends to

Table 5. Notation. D-LSA symbols and their description.

Training Data

In this section, we report additional details on the training dataset we use to train D-LSA under indexer distillation, and also train N-LSA in the decay phase: LongDocs.

We use the following terminology:

  • Source: a collection of documents, such as Project Gutenberg.

  • Domain: a group of sources: books, academic, code, agentic, government, parliament and discussion.

  • Document: one item of a source, such as a book, a paper, a code repository or an agent trace.

  • Sequence: a contiguous part of one document, at a particular sequence length from 16k to 1M tokens.

Sources. LongDocs is a collection of long documents from publicly available sources and in-house agent traces. Eight sources are publicly available: Project Gutenberg, DOAB, arXiv, USGPO, UK Hansard and Wikimedia from the filtered releases of the Common Pile (Kandpal et al., [35]), The Stack (Kocetkov et al., [37]), of which we take the

Algorithm 1 Ward clustering of one cluster block

Require: index keys kj(idx)k_j^{(\mathrm{idx})} of a complete cluster block; sizes ncluster,npool,m,nbmax⁡n_{\mathrm{cluster}}, n_{\mathrm{pool}}, m, n_b^{\max}

Ensure: Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})} with mm centroids cbc_b and their key positions

Pooling

1: for all pool keys kpool(idx)k_{\mathrm{pool}}^{(\mathrm{idx})}, in parallel do

2: kpool(idx)←k_{\mathrm{pool}}^{(\mathrm{idx})} \leftarrow the sum of its npooln_{\mathrm{pool}} consecutive kj(idx)k_j^{(\mathrm{idx})}

3: end for

4: start one bucket bb per pool key, kbsum←kpool(idx)k_b^{\mathrm{sum}} \leftarrow k_{\mathrm{pool}}^{(\mathrm{idx})}, ∣b∣←npool\lvert b\rvert\leftarrow n_{\mathrm{pool}}

Merging

5: while more than mm buckets remain do

every pair of buckets, in parallel

6: G←[kbsum⋅kb′sum]G \leftarrow[k_b^{\mathrm{sum}} \cdot k_{b'}^{\mathrm{sum}}] ▷ Gram matrix

7: ab←Gbba_b \leftarrow\sqrt{G_{bb}}, ab∪b′←Gbb+Gb′b′+2Gbb′a_{b\cup b'} \leftarrow\sqrt{G_{bb}+G_{b'b'}+2G_{bb'}}

8: Δ(b,b′)←ab+ab′−ab∪b′\Delta(b,b') \leftarrow a_b+a_{b'}-a_{b\cup b'} ▷ merge cost

9: π(b)←arg⁡min⁡b′ Δ(b,b′)\pi(b) \leftarrow\underset{b'}{\arg\min}\ \Delta(b,b'), ∣b∣+∣b′∣≤nbmax⁡\lvert b\rvert+\lvert b'\rvert\le n_b^{\max}

the cheapest mutual pairs π(b)=b′\pi(b)=b', π(b′)=b\pi(b')=b at most as many as buckets above mm, in parallel

10: kbsum←kbsum+kb′sumk_b^{\mathrm{sum}}\leftarrow k_b^{\mathrm{sum}}+k_{b'}^{\mathrm{sum}}, ∣b∣←∣b∣+∣b′∣\lvert b\rvert\leftarrow\lvert b\rvert+\lvert b'\rvert, remove b′b'

11: end while

Bucket cache

12: for all buckets bb, in parallel do

13: cb←kbsum/∥kbsum∥c_b\leftarrow k_b^{\mathrm{sum}}/\lVert k_b^{\mathrm{sum}}\rVert

14: store cbc_b and the positions of the keys of bb in Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})}

15: end for

D-LSA (Tree) only

16: merge blocks of ncluster,ℓn_{\mathrm{cluster},\ell} buckets into node buckets

Algorithm 2 Beam search of one query in D-LSA (Tree)

Require: index query qi(idx)q_i^{(\mathrm{idx})}, levels ℓ=0,…,ℓmax⁡\ell=0,\ldots,\ell^{\max} of Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})}; sizes p,pℓ,nstartp,p_\ell,n_{\mathrm{start}}

Ensure: the top-pp buckets for query ii

1: C←C\leftarrow all node buckets of the lowest level ℓ\ell with ≤nstart\le n_{\mathrm{start}}

Descent

2: while ℓ>0\ell>0 do

every candidate in CC, in parallel

3: si,b←qi(idx)⋅cbs_{i,b}\leftarrow q_i^{(\mathrm{idx})}\cdot c_b ▷ node score

4: P←P\leftarrow the best pℓp_\ell candidates of CC

5: C←C\leftarrow the children of P∪P\cup the open nodes of level ℓ−1\ell-1

6: ℓ←ℓ−1\ell\leftarrow\ell-1

7: end while

Selection

every bucket in CC, in parallel

8: si,b←qi(idx)⋅cbs_{i,b}\leftarrow q_i^{(\mathrm{idx})}\cdot c_b ▷ bucket score

9: return the top-pp buckets of CC

Python and Java repositories, and the software engineering traces of Open-SWE-Traces (NVIDIA, 2026e). The in-house agent traces are long sessions of agents that play games or solve long-context tasks, and they make up 13.1% of the training tokens. We use every public source as released.

Sequences. Each sequence comes from one document and never joins two documents. A code document is one repository, its files joined in the order of their paths, and an agent trace is one session rendered as text. We hold out 2% of the documents for validation.

Distillation training. We distil every indexer on a mix of 352M tokens drawn from LongDocs (Figure 9). The mix is uniformly sampled across domain and sequence length, with a bias towards longer context windows at 512k and 1M. In particular, we give each sequence length 32M tokens, and twice and four times as many to 512k and 1M. Sequences of 1M tokens thus hold about a third of the training tokens. We experimentally determine this choice to be beneficial, with +0.036 accuracy on RULER at 1M (averaged over 3 seeds, notable lifts in MULTIKEY_2, MULTIKEY_3, +0.35 and +0.15 respectively), while accuracy from 4k to 512k stays roughly identical. Within each length, every domain receives an equal share of the tokens. A domain with too few documents of that length gives its share to the others (parliament stops at 256k tokens and discussion at 512k).

Figure 9. LongDocs. Tokens of the distillation mix by domain and source (left) and by sequence length (right), with the in-house agent traces hatched.

Native training. The decay training stage of N-LSA reads the whole collection rather than the mix, without the long-context agent sessions. The data is tokenised (18.9B tokens over 134,740 documents), and cut into sequence lengths the training requires.

PaperTrail

As part of this work, we introduce PaperTrail, an extension of the RULER (Hsieh et al., [26]) benchmark that tests models at ultra-long context in a more realistic setting, inspired by real-life scientific literature review (Figure 10). Instead of finding a needle in synthetic text, a model finds facts spread across real scientific papers and connects them, as in a literature search. We use the following terminology:

Figure 10. PaperTrail. ❶ A sample, its fabricated, host and background papers. ❷ One trail, introduced, revised and renamed, and still quoted in its old form after the revision. ❸ Questions by tier, and one whose answer is not in the papers. ❹ Context sizes, each holding every paper of the size below. ❺ Controls.

  • Sample: a collection of real arXiv preprints of 2026 from one domain.

  • Trail: a chain of papers that introduces a result, cites it, and later revises and renames it.

  • Fabricated paper: a paper we write for a trail.

  • Host paper: a real paper into which we insert passages that cite or use facts from a fabricated paper.

  • Insertion: a passage we add to a host paper, which only adds sentences and never changes the text of the authors.

  • Background paper: any other paper of a sample, real and unchanged.

❶ Context. PaperTrail has three samples, on exoplanets, long-context machine learning and quantum error correction, with 1,246 papers in total. Each sample hides two trails (Trail A and Trail B). A trail has two fabricated papers and 8 to 9 insertions in 6 to 7 host papers, and background papers fill the rest of the context. An insertion places its references where an author would place them, such that a host paper reads naturally as before. Papers in the context

are not presented in chronological order, and a host often comes before the paper it cites.

❷ Trails. The first fabricated paper of a trail introduces a result, for example AlloKV, a KV cache eviction method whose budget grows as the 0.43 power of the context length. Host papers cite and use it. Two to three weeks later, the second fabricated paper revises the value and renames the method, here to SuBA with an exponent of 0.61. Host papers posted after the revision either cite the new name or still quote the old value, as real literature often does.

❸ Questions. Each sample asks 30 scored questions, 10 per tier. A tier 1 question recalls a fact from one paper, a tier 2 question links two papers, and a tier 3 question links three or more papers, often the whole trail. Each sample also asks 6 questions whose answer is not in the papers, about an absent entity, a false premise or a near miss, which we report apart from the score (Table 8). The model ends its reply with the papers it used and its answer. A number counts within a relative tolerance of 2% by default, and a set of papers is scored by its F1 against the reference set.

ExoplanetsLong-context MLQECAverage
Not in corpus, QWEN-3.8-27B
Full Attention100.0100.0100.0100.0
Block Sparse (Flat)100.0100.083.394.4
Block Sparse (Tree)100.0100.088.996.3
D-LSA (Flat)100.0100.088.996.3
D-LSA (Tree)100.094.494.496.3
D-LSA (Training-Free)100.094.4100.098.1
Not in corpus, GLM-5.3-FLASH
Block Sparse (Flat)100.088.9100.096.3
Block Sparse (Tree)100.088.994.494.4
D-LSA (Flat)100.0100.0100.0100.0
D-LSA (Tree)100.088.9100.096.3
D-LSA (Training-Free)100.094.4100.098.1
Controls, GLM-5.3-FLASH
Question alone0.00.00.00.0
Hosts without insertions0.00.00.20.1
Fabricated + insertions100.0100.0100.0100.0
Background alone0.00.00.00.0

Table 8. PaperTrail not-in-corpus questions and controls. Score (%) on the questions whose answer is not in the papers at 512k tokens, for Full Attention, Block Sparse Attention (Flat) and (Tree), and D-LSA (Flat), (Tree) and (Training-Free). The controls ask the scored questions over parts of a sample, with GLM-5.3-FLASH and D-LSA (Training-Free).

❹ Context sizes. We build each sample at 512k, 1M, 2M, 5M and 10M tokens, and every size holds every paper of the size below. The fabricated and host papers, at most 18 per sample, are the same at every size. The background papers grow the context from 25 to 30 papers at 512k to 386 to 465 papers at 10M, and spread the trails over the whole context. The trails are fabricated, and every paper up to 5M was posted after the release of both QWEN-3.8-27B and GLM-5.3-FLASH, guaranteeing that no model has seen these papers in training. The 10M contexts hold papers posted at most six weeks before its release.

❺ Controls. Four controls ask the same questions over parts of a sample: the question alone, the host papers without their insertions, the fabricated papers with the insertions alone, and the background papers alone. By construction, only the fabricated papers and the insertions hold the answers. We validate the controls with GLM-5.3-FLASH and D-LSA (Training-Free) in Table 8.

Accuracy

In this section, we show details of the evaluation setup, compare every method with its baseline on knowledge and reasoning, examine whether D-LSA changes the length of the reasoning, and break down the long-context results, RULER by task and PaperTrail by tier.

Evaluation Setup

Table 6 lists the setup of every benchmark. We sample QWEN-3.8-27B at temperature 1.0 with top-p 0.95, top-k 20, min-p 0 and no presence penalty, and GLM-5.3-FLASH at temperature 1.0 with top-p 0.95, as their model cards recommend. On RULER, both models decode greedily with reasoning disabled. On every benchmark with reasoning, at most 4.6% of the replies reach the output cap of 163,840 tokens for any model and method.

QuestionsRepeatsReasoningMax outputMetric
MMLU-Pro2,0001on163,840Exact match
GPQA-Diamond1981on163,840Exact match
AIME25304on163,840Math-Verify
MATH5001001on163,840Math-Verify
LongBench v2601on163,840Accuracy
RULER100 (≤\leq1M) / 20 (>>1M)1off4,096String match
PaperTrail903on163,840F1

Table 6. Evaluation setup. Questions, repeats, generation and metric of every benchmark, with RULER questions per task.

Knowledge and Reasoning

Table 7 compares every method with the baselines on knowledge, reasoning and LongBench v2. Every score lies within the 95% confidence interval of the respective baselines.

MMLU-ProGPQA-DAIME25MATH500LongBench v2
QWEN-3.8-27B
Full Attention83.288.498.398.070.0
Block Sparse (Flat)+0.3−2.0-2.0+0.8+1.00.0
Block Sparse (Tree)+0.2−2.0-2.0−0.8-0.8+2.0−1.7-1.7
D-LSA (Flat)−0.3-0.3−0.5-0.5−0.8-0.8+1.0−5.0-5.0
D-LSA (Tree)+0.3−1.5-1.5+1.7+2.0−8.3-8.3
D-LSA (Training-Free)−0.6-0.6−3.0-3.0−1.7-1.7+1.0−3.3-3.3
95% CI±1.6\pm1.6±4.5\pm4.5±2.3\pm2.3±2.7\pm2.7±11.6\pm11.6
GLM-5.3-FLASH
Block Sparse (Flat)85.485.997.598.070.0
Block Sparse (Tree)+0.2+2.0−0.8-0.80.0−1.7-1.7
D-LSA (Flat)+0.3+1.0−1.7-1.70.0−1.7-1.7
D-LSA (Tree)+0.3+0.5+1.7+1.00.0
D-LSA (Training-Free)+0.4+2.0−2.5-2.5+1.0+1.7
95% CI±1.5\pm1.5±4.9\pm4.9±2.8\pm2.8±2.7\pm2.7±11.6\pm11.6

Table 7. Accuracy compared to the baseline. Baseline scores are in bold. Other rows show the difference to the baseline in points. Differences smaller than the 95% confidence interval (CI) of the baseline are not significant.

Reasoning Length

Figure 12 compares the reasoning length of each method with the baseline of its model, question by question. For each question, we average the reasoning tokens over its repeats and divide them by those of the baseline on the same question, and each marker is the geometric mean of these ratios over the benchmark. The grey band of a benchmark is the median width of the 95% bootstrap intervals of its methods, and a marker inside it shows no measurable change. Every D-LSA design stays inside its band, except D-LSA (Tree) on GPQA-Diamond with QWEN-3.8-27B, which reasons 11% shorter.

Reasoning tokens against baseline (same paired questions) for Qwen-3.8-27B and GLM-5.3-Flash

Figure 12. Reasoning tokens against baselines. Reasoning tokens of each method relative to the baseline of its model, on the same questions. Grey bands mark the expected variation, and markers inside them show no measurable change in reasoning length.

RULER

Figure 11 shows the 13 tasks of RULER from 4k to 10M tokens, grouped into retrieval, tracing, aggregation and question answering. Single-needle retrieval stays at 95 or above for every method up to 5M tokens, except Block Sparse Attention (Tree), which falls to 88 and 71 at 1M tokens. MultiKey 2, MultiKey 3 and CWE fall to at most 5 at 10M tokens for every method, including the baselines, and bound the RULER average of every method. On QWEN-3.8-27B, D-LSA (Flat) exceeds baseline Full Attention at 10M tokens mostly on MultiQuery and HotpotQA, at 92 and 35 against 72 and 20. On GLM-5.3-FLASH, D-LSA (Tree) falls to 60, 56 and 55 on MultiKey 1, MultiValue and VT at 10M tokens, against 100 for baseline sparse attention, and these three tasks account for most of its gap to D-LSA (Flat).

Figure 11. D-LSA accuracy on RULER individual tasks. The 13 tasks of RULER from 4k to 10M tokens on QWEN-3.8-27B and GLM-5.3-FLASH, by task group, with their average.

PaperTrail

Figure 13 shows the three tiers of PaperTrail and their average from 512k to 10M tokens. The harder the tier, the earlier it falls. Tier 3 questions fall to at most 19.0 at 10M tokens for every method on both models, while tier 1 questions stay above 90 on GLM-5.3-FLASH. On QWEN-3.8-27B, D-LSA exceeds baseline Full Attention from 5M tokens on tier 1 and tier 2 questions, and D-LSA (Flat) reaches 53.3 and 32.2 against 32.2 and 22.2 at 10M tokens. On GLM-5.3-FLASH, D-LSA trails baseline sparse attention mostly

Figure 13. D-LSA accuracy on PaperTrail per tier. Tier 1, tier 2 and tier 3 questions, each averaged over the three domains and three repeats.

on tier 3 questions from 2M tokens, and D-LSA (Flat) also on tier 2 questions at 10M tokens, at 56.7 against 76.7.

Table 8 further reports the questions whose answer is not in the papers, at 512k tokens. Every D-LSA design recognises these questions within 3.7 points of the baseline of its model, at 96.3 to 100.0 against 100.0 for QWEN-3.8-27B and 96.3 for GLM-5.3-FLASH.

Block Size

Figure 14 compares Block Sparse Attention (Tree) with blocks of 8, 16 and 64 keys, all selecting the same number of keys. At short context, every block size matches the baseline. As the context grows, accuracy falls earlier and further the larger the block, on both benchmarks and both models, as a larger block summarises more keys in one score and can hide a single relevant key. Even with blocks of eight keys, the size we use in the main results, the tree baseline trails D-LSA (Flat) at long context.

Figure 14. Block size of Block Sparse Attention (Tree). RULER and PaperTrail up to 1M tokens with blocks of 8, 16 and 64 keys (npooln_{\mathrm{pool}}), each selecting the same 4,608 keys, against the baseline of each model and D-LSA (Flat).

Performance

We first introduce the hardware, describe the kernels of the indexer (Appendix § 14.§ 1), and how D-LSA integrates with different parallelism strategies (Appendix § 14.§ 2): Tensor Parallelism (TP) and Context Parallelism (CP).

Hardware. We run on NVIDIA H200 (Hopper) and B300 (Blackwell Ultra) GPUs, eight per node (NVLink domain), with the details presented in Table 9.

H200B300
ArchitectureHopper (sm_90)Blackwell Ultra (sm_103)
SMs132148
Shared memory per SM228 KiB228 KiB
Matrix instructionsWGMMAtcgen05
Memory (HBM3e)141 GB288 GB
HBM bandwidth4.8 TB/s8 TB/s
NVLink per GPU900 GB/s1.8 TB/s
bf16 tensor989 Tflop/s2,250 Tflop/s
fp3267 Tflop/s75 Tflop/s
Ridge point (bf16)206 flop/B281 flop/B

Table 9. Hardware specs. Specifications of the GPUs we use, as given by NVIDIA [58 d;a;b].

Precision. The KV cache, the index keys and the bucket centroids are in bf16. For the bucket centroids, which every decode step of the flat search reads in full, bf16 halves the bytes of fp32 at little cost to the selection. In an offline study on GLM-5.3-FLASH, the bf16 indexer selects roughly 20 of

the 4,096 long-range keys of the flat search differently from an fp32 reference, and changes the attention mass that the selected keys cover by less than 0.0001. Bucket centroids in fp8 would halve their bytes again, but would select about six times as many keys differently, so we keep them in bf16. We do not observe similar sensitivity for the node centroids, which we set to fp8 in D-LSA (Tree).

Conventions. We run prefill over chunks CC of ∣C∣=32768|C| = 32768 consecutive queries, and decode over steps of one new query per request. All sizes are per layer, and per chunk in prefill or per step in decode. We run D-LSA with index keys and queries of d(idx)=64d^{(\mathrm{idx})} = 64 dimensions, cluster blocks of ncluster=512n_{\mathrm{cluster}} = 512 keys, buckets of nˉb=8\bar{n}_{b} = 8 keys on average and at least npool=4n_{\mathrm{pool}} = 4. Each query selects p=1024p = 1024 buckets, so that even buckets of npooln_{\mathrm{pool}} keys should hold its k=4096k = 4096 long-range keys, and the expansion fills kk from about the best half of them on average. It also attends to nsink=4n_{\mathrm{sink}} = 4 sink keys and a window of nw=nclustern_{w} = n_{\mathrm{cluster}} keys. On GLM-5.3-FLASH, D-LSA (Flat) uses k=2048k = 2048 and p=512p = 512 for the queries in the first 1M positions of the context (Table 2), but for queries of a chunk at 10M tokens, the sizes above apply. Therefore, at a context of T=10×220≈10T = 10 \times2^{20} \approx10M tokens, a layer holds T/nˉb=1310720T / \bar{n}_{b} = 1310720 buckets, whose centroids take 2d(idx)T/nˉb=1602d^{(\mathrm{idx})}T/\bar{n}_{b} = 160 MiB.

Kernels

We refer to the steps of D-LSA by their badges in Figure 3, and Figure 15 shows how we implement each of them. We write kernels for three steps of the indexer: ❸ clustering, B scoring and selection of the buckets, and C expansion of the selected buckets into the index lists JiJ_i. These kernels are written in CuTe DSL, the Python-based domain-specific language of CUTLASS [76], to target H200 and B300 specifically. We also keep a Triton [77] version of each kernel, which is not tied to these two GPUs. We give the shapes and times of the kernels on H200, and note where B300 differs. For D attention, we rely on pre-existing attention kernels, that attend over the index lists JiJ_i unchanged. The only exception is CP (Appendix § 14.§ 2), for which we do add an attention kernel.

Implementation of the steps of D-LSA

Figure 15. Implementation of the steps of D-LSA. Badges as in Figure 3.

Clustering

The clustering does relatively little work: one Gram matrix per cluster block, over its pooled keys, takes about 0.13 Gflop for a whole chunk, against 5.5 Tflop for the bucket scan at 10M tokens. Ward’s method, however, merges buckets in rounds that run one after the other. Each of these rounds scores every pair of buckets and merges the cheapest mutual pairs (Algorithm 1, lines 5-11), until mm buckets remain. The overall runtime of the clustering step is therefore set by the latency of these rounds. Our kernel cuts the time of each round by keeping its data on chip and avoiding round trips to the host between rounds. More precisely, the kernel keeps the state of a cluster block in shared memory for all its rounds, and spreads its search for partners over a thread-block cluster of up to eight streaming multiprocessors (SMs), which exchange their results through distributed shared memory. In D-LSA (Tree), each pairing that forms the node buckets (line 16) runs all its rounds in a single launch, whose thread blocks synchronise through counters in high-bandwidth memory (HBM), so the host neither launches nor reads anything between rounds.

B C Selection by threshold. D-LSA scores an index query qi(idx)q_i^{(\mathrm{idx})} against the centroids cbc_b of the buckets Bi\mathcal{B}_i of the cluster blocks before its own, and keeps its top-pp buckets. Over the index queries of a chunk, the scores si,b=qi(idx)⋅cbs_{i,b}=q_i^{(\mathrm{idx})}\cdot c_b form a matrix with one row per query, so we call the scores of query ii its row, in decode as well. The row holds Ni=∣Bi∣N_i=|\mathcal{B}_i| scores, which grows with the position of the query, to about T/nˉbT/\bar n_b at the end of a context of TT tokens. In prefill and in decode alike, many thread blocks compute the scores of a row, each over its own span of the row, so no thread block sees the whole row, and the selection cannot take its top-pp directly. Instead, inspired by Floyd & Rivest [18] (who select from only the elements between two bounds estimated from a sample), before the scan, we set a threshold θi\theta_i for each query, and a thread block compares each of its scores with θi\theta_i. Then, it appends only the buckets whose score is at or above it to the list LiL_i of the query: one comparison per score, without seeing the rest of the row. Only the buckets of LiL_i, a small multiple γp\gamma p of them (see below), are then

ranked.

We also evaluate an alternative inspired by the k-selection of FAISS (Johnson et al., [34]), which keeps the top-pp buckets of each span and merges them. With this design, every thread block has to rank its span, rather than compare each score once. Furthermore, the merge receives pp buckets from every span, rather than about γp\gamma p in all. Overall, at 10M tokens in prefill, a row is split into 320 spans, which list 327,680 buckets, against about 1,536 in LiL_i.

We aim for about γp\gamma p buckets of the row to score at or above θi\theta_i, with γ>1\gamma>1: enough to hold the top-pp with room for error, and few enough to keep LiL_i short. A first pass, the threshold estimate, sets θi\theta_i from a sample of nsn_s buckets of the row (Figure 16):

Threshold estimate

Figure 16. Threshold estimate. For one query ii. Top: the sampled scores, with θi\theta_i at or just below the nθn_{\theta}-th best. Bottom: the row, of which about γp\gamma p buckets score at or above θi\theta_i and are listed, the top-pp among them.

θi≤the nθ-th best sampled si,b≈the γp-th best si,b of the rownθ=γ⋅p⋅ns/Ni(4)\begin{aligned} \theta_i &\leq\text{the }n_{\theta}\text{-th best sampled }s_{i,b}\\ &\approx\text{the }\gamma p\text{-th best }s_{i,b}\text{ of the row}\\ n_{\theta} &= \gamma\cdot p\cdot n_s/N_i \tag*{(4)} \end{aligned}

The sample takes about one bucket in every Ni/nsN_i/n_s of the row, so its nθn_{\theta}-th best score ranks about nθNi/ns=γpn_{\theta}N_i/n_s=\gamma p-th in the row, and θi\theta_i is placed at or just below it. This estimate is noisy: for independent samples, the count of buckets above θi\theta_i would typically differ from γp\gamma p by about γp/nθ\gamma p/\sqrt{n_{\theta}}. However, since neighbouring buckets are correlated, this is only a heuristic. Prefill fixes nθn_{\theta}, so that its sample grows with the row, and decode fixes nsn_s, and derives nθn_{\theta} for each request.

The noise affects only the cost of the selection. The threshold sets how many buckets are listed, but not which are selected: when LiL_i holds every bucket with si,b≥θis_{i,b}\geq\theta_i, and at least pp of them, the top-pp buckets of the row all reach θi\theta_i, and are thus all in LiL_i. The estimate misses in two ways: θi\theta_i comes out too high, and fewer than pp buckets score at or above it, or too low, and more score at or above it than LiL_i has room for. γ\gamma leaves room for the noise on both sides, γp−p\gamma p-p below and the rest of LiL_i above. A query that the estimate misses anyway falls back to its whole row. In prefill, it scores all its buckets again and takes their top-pp, the work the threshold avoids, but for one query rather than a whole chunk. In decode, it takes the top-pp of the scores the scan wrote. The fallback thus costs little as long as few queries take it.

Each bucket of LiL_i is packed with its score into one 64-bit integer σi,b\sigma_{i,b}: the bits of si,bs_{i,b}, in a form that keeps their order, in its upper 32 bits, and the bucket id, inverted, in its lower 32. A larger σi,b\sigma_{i,b} is thus a better score, or, at equal scores, a lower bucket id. The C expansion finds, by radix select ([1, 93]) over the σi,b\sigma_{i,b}, the cut σi⋆\sigma_i^\star at which the buckets of LiL_i, taken best first, reach kk keys or pp buckets. It writes into JiJ_i the keys of the buckets above σi⋆\sigma_i^\star, in the order of LiL_i, then the first keys of the bucket at σi⋆\sigma_i^\star until JiJ_i holds kk. If these buckets hold fewer than kk keys outside the sink and the window, JiJ_i holds fewer. These are the keys that an expansion of the buckets best first would select, in another order. The attention does not depend on the order, so LiL_i is never sorted. Prefill and decode differ in where the scores go, how θi\theta_i is set, and what a fallback costs.

B C Flat prefill. In prefill, the flat search selects the buckets of all the queries of a chunk at once, in one matrix product on tensor cores, whose kernel compares each score with θi\theta_i as soon as the score is computed, without writing it to memory. We first show why the scores are not written, then how the product is spread over the GPU, and then the steps of Algorithm 3.

With the ∣C∣|C| index queries of a chunk CC as the rows of QC(idx)Q_C^{(\mathrm{idx})}, and the T/nˉbT/\bar{n}_b bucket centroids cbc_b as the rows of CC, the rows of the chunk form one score matrix

S=QC(idx)CT,Si,b=qi(idx)⋅cb(5)S = Q_C^{(\mathrm{idx})} C^{\mathsf{T}}, \qquad S_{i,b} = q_i^{(\mathrm{idx})} \cdot c_b \tag*{(5)}

At 10M tokens, computing SS takes FpreF^{\mathrm{pre}} floating-point operations (flop) on at least BreadpreB_{\mathrm{read}}^{\mathrm{pre}} bytes of bf16 queries and centroids, and writing it takes BscorespreB_{\mathrm{scores}}^{\mathrm{pre}} bytes in fp32:

Fpre=2∣C∣d(idx)T/nˉb≈5.5 TflopBreadpre=2d(idx)(T/nˉb+∣C∣)≈164 MiBBscorespre=4∣C∣T/nˉb=160 GiB(6)\begin{aligned} F^{\mathrm{pre}} &= 2|C|d^{(\mathrm{idx})}T/\bar{n}_b \approx5.5\ \mathrm{Tflop}\\ B_{\mathrm{read}}^{\mathrm{pre}} &= 2d^{(\mathrm{idx})}(T/\bar{n}_b + |C|) \approx164\ \mathrm{MiB}\\ B_{\mathrm{scores}}^{\mathrm{pre}} &= 4|C|T/\bar{n}_b = 160\ \mathrm{GiB} \tag*{(6)} \end{aligned}

On an H200, at the peak of its tensor cores, computing SS would take at least Fpre/989 Tflop/s=5.6 msF^{\mathrm{pre}}/989\ \mathrm{Tflop/s} = 5.6\ \mathrm{ms}. If every query and centroid were read from HBM only once, the products would have an arithmetic intensity of Fpre/Breadpre≈32000 flop/BF^{\mathrm{pre}}/B_{\mathrm{read}}^{\mathrm{pre}} \approx32000\ \mathrm{flop/B}, two orders of magnitude above the ridge point of the H200. Under this ideal reuse, they can be bound by compute.

Writing SS, however, is bound by memory, and would take at least Bscorespre/4.8 TB/s=36 msB_{\mathrm{scores}}^{\mathrm{pre}}/4.8\ \mathrm{TB/s} = 36\ \mathrm{ms}. Both are lower bounds from the peaks of the GPU, and the second is more than six times the first at any context, as both grow with TT. At 9M tokens, SS alone would already exceed the 141 GB of HBM of an H200. At long context, we therefore never write SS, and the selection writes only the lists LiL_i.

As in a matrix multiplication (GEMM), the product is tiled (Figure 17). A thread block holds a tile of tq=128t_q = 128 index queries in shared memory, and streams the centroids through it in tiles of tb=64t_b = 64 (128 on B300). It computes one tq×tbt_q \times t_b tile StileS^{\mathrm{tile}} of SS at a time, on tensor cores, and keeps it on chip. Each thread block takes such a tile of queries and a span of up to 64 tiles of centroids. At 10M tokens, a row is thus split into 320 spans, and the chunk into 256×320=81920256 \times320 = 81920 thread blocks, which, on H200 runs 528 at a time, 4 on each of its 132 SMs, which is enough to keep every SM busy. At shorter contexts, the spans shrink, so that there are still enough thread blocks.

Figure 17. B Bucket scan on H200. Left: the thread blocks (colours) over tiles of tqt_q index queries and spans of centroid tiles, and below, candidate scoring and C expansion for one query. Right: one thread block (Algorithm 3, lines 3–9).

The bucket scan differs from a GEMM only in its epilogue. Instead of writing each entry Si,bS_{i,b} of StileS^{\mathrm{tile}}, the epilogue compares it with the threshold (lines 6–7), and compacts the buckets at or above it in shared memory, so that each span appends them to LiL_i in HBM with one atomic per query (line 9). With d(idx)=64d^{(\mathrm{idx})} = 64, a tile takes few products, and this

Algorithm 3 Flat prefill selection of one chunk

Require: index queries qi(idx)q_i^{(\mathrm{idx})}, i∈Ci \in C, and centroids cbc_b, in HBM buckets BiB_i before the cluster block of each ii sizes pp, kk, nsinkn_{\mathrm{sink}}, nwn_w, nθn_\theta, margin δ\delta, tile sizes tqt_q, tbt_b

Ensure: index lists JiJ_i, rows of nsink+k+nwn_{\mathrm{sink}} + k + n_w KV-cache slots

Threshold estimate

every tile of tqt_q queries ×\times part of the sample, in parallel

  1. score a strided sample of the cbc_b of BiB_i, on tensor cores

  2. θi←\theta_i \leftarrow a bin edge at or below the nθn_\theta-th best sampled score

Bucket scan

every tile of tqt_q queries ×\times span of centroid tiles, in parallel

  1. load the tile’s qi(idx)q_i^{(\mathrm{idx})} from HBM into shared memory

  2. for each tile of tbt_b centroids of the span do

  3. load its cbc_b from HBM into shared memory

  4. Si,b←qi(idx)⋅cbS_{i,b} \leftarrow q_i^{(\mathrm{idx})} \cdot c_b, on tensor cores ▷ score tile

  5. compact its bb with Si,b≥θi−δS_{i,b} \ge\theta_i - \delta in shared memory

  6. end for

  7. Li←Li∪{b:Si,b≥θi−δ}L_i \leftarrow L_i \cup\{b : S_{i,b} \ge\theta_i - \delta\}, for the tile’s ii, in HBM

Candidate scoring

every query ii, in parallel

  1. si,b←qi(idx)⋅cbs_{i,b} \leftarrow q_i^{(\mathrm{idx})} \cdot c_b in fp32, for b∈Lib \in L_i

  2. σi,b←(si,b,b)\sigma_{i,b} \leftarrow(s_{i,b}, b) packed in one int64 ▷ ties by bucket

  3. if LiL_i overflowed, or fewer than pp of its si,b≥θis_{i,b} \ge\theta_i then Li←BiL_i \leftarrow B_i; redo lines 10–11

Expansion

every query ii, in parallel

  1. σi⋆←σi,b\sigma_i^\star\leftarrow\sigma_{i,b} where LiL_i, best first, has kk keys or pp buckets

  2. Ji←J_i \leftarrow keys in RiR_i of {b∈Li:σi,b>σi⋆}\{b \in L_i : \sigma_{i,b} > \sigma_i^\star\}, in listed order

  3. Ji←Ji∪J_i \leftarrow J_i \cup first k−∣Ji∣k - |J_i| keys in RiR_i of the bucket at σi⋆\sigma_i^\star

  4. Ji←Ji∪J_i \leftarrow J_i \cup the sink and the window; pad JiJ_i with −1-1

The threshold estimate and bucket scan epilogue costs about as much as they do, so the two must overlap. On H200, the thread block splits its warps by role: one warp loads the tiles of centroids into shared memory with the Tensor Memory Accelerator (TMA), in a pipeline of two stages, while a warpgroup multiplies them on tensor cores (WGMMA) into two accumulators in registers, and compares one with the threshold while the tensor cores fill the other. On B300, the products accumulate in the tensor memory of the SM instead, from which the comparing threads load them. Triton emits the same instructions, but chooses this schedule itself and gives no control of shared memory, which is why we write the scan in CuTe DSL.

At short context, below 32,768 buckets (about 256k tokens), the selection costs more than writing SS, which then takes at most 4 GiB, and we instead write SS with one matrix product in PyTorch and take the top-pp buckets of each row.

In prefill, we fix nθ=64n_\theta= 64, so that the count above θi\theta_i is off by about γp/8\gamma p/8, and γ=1.5\gamma= 1.5, which sets the sample: every (γ⋅p/nθ)=24(\gamma\cdot p/n_\theta) = 24th bucket of the row, about 4% of its scores at any context. Algorithm 3 then selects in four steps. The threshold estimate scores the sample twice on tensor cores, once for the mean and the best of its scores and once to count them in 64 bins between the two, binning them straight from the accumulators so that the scores of the sample are never written, and sets θi\theta_i to the lower edge of the highest bin with at least nθn_\theta samples at or above it (lines 1-2). The bucket scan computes SS tile by tile, and lists the buckets whose score Si,bS_{i,b} is at or above θi\theta_i minus a margin δ\delta (lines 3-9):

Li={b∈Bi:Si,b≥θi−δ}(7)L_i = \{b \in B_i : S_{i,b} \ge\theta_i - \delta\} \tag*{(7)}

Candidate scoring then re-scores the buckets of LiL_i, as vector search re-scores a shortlist [31]: it computes their score si,bs_{i,b} again in fp32, one dot product per bucket, and packs it into σi,b\sigma_{i,b} (lines 10-11).

The margin is there because the scan and candidate scoring, which multiply the same bf16 operands and both accumulate in fp32, accumulate in different orders, so Si,bS_{i,b} and si,bs_{i,b} can differ: with unit-norm index queries and centroids and d(idx)=64d^{(\mathrm{idx})} = 64, by less than 10−510^{-5}. The margin δ=10−4\delta= 10^{-4} is ten times larger, so LiL_i holds every bucket with si,b≥θis_{i,b} \ge\theta_i. The list LiL_i has room for 3p=30723p = 3072 buckets, twice the target γp\gamma p. A query whose list overflows, or holds fewer than pp buckets at or above θi\theta_i, falls back: it scores all its buckets again, and takes the top-pp (line 12). The expansion writes JiJ_i (lines 13-16).

CUDA graphs

A prefill chunk takes milliseconds, against which the launches of its kernels from the host cost little, and its selection reads the sizes of its lists back on the host, so we serve prefill without graphs. A decode step of the selection, however, takes only tens of microseconds, about as long as its launches, so SGLang captures each decode step, our kernels included, in a CUDA graph, and replays it with one launch from the host. A CUDA graph fixes the shapes of its kernels, so a graph sized for the longest context the server admits would launch the thread blocks of 10M tokens at every step, whatever the contexts of the batch. We therefore capture one graph per range of context lengths, up to 1.125⋅2e1.125 \cdot2^e tokens for e=17,…,23e = 17, \ldots, 23 (from 144k to 9M), and one graph for the longest context, and each step replays the narrowest graph that covers the longest request of the batch. The threshold estimate of the flat search samples across the width of the graph, so a request far shorter than its range would see too few samples and fall back to its full row. The factor 1.125 avoids this for the common case: a request of a power-of-two length, with its answer, sits at about 89% of its range. The clustering of a complete cluster block runs outside the graph, before the replay, once every ncluster=512n_{\mathrm{cluster}} = 512 decoded tokens, and the decode times of § 5 spread its cost over those tokens.

Flat decode

In decode, each request adds one new query per step, whose buckets the flat search selects. It does so in a single pass, which reads the centroids of the request Algorithm 4 Flat decode selection of one request

Require: index query qi(idx)q_i^{(\mathrm{idx})} and centroids cbc_b of BiB_i, in HBM the request’s pages of cbc_b sizes pp, kk, nsinkn_{\mathrm{sink}}, nwn_w, nsn_s, nθn_\theta

Ensure: index list JiJ_i, as in Algorithm 3

Threshold estimate

every part of the sample, in parallel

  1. score a strided sample of the cbc_b, on CUDA cores

  2. θi←\theta_i \leftarrow the nθn_\theta-th best sampled score, rounded down

Bucket scan

every span of the request’s buckets, in parallel

  1. read each cbc_b once from HBM, through the request’s pages

  2. si,b←qi(idx)⋅cbs_{i,b} \leftarrow q_i^{(\mathrm{idx})} \cdot c_b in fp32, into shared memory and HBM

  3. wait for θi\theta_i

  4. Li←Li∪{σi,b:si,b≥θi}L_i \leftarrow L_i \cup\{\sigma_{i,b}:s_{i,b}\ge\theta_i\}, for bb in the span, in HBM

Listing

  1. gather the spans of LiL_i, in bucket order

  2. if LiL_i overflowed, or fewer than pp of its si,b≥θis_{i,b}\ge\theta_i then Li←L_i \leftarrow top-pp of the written si,bs_{i,b}, as σi,b\sigma_{i,b}

Expansion

  1. Ji←J_i \leftarrow lines 13–16 of Algorithm 3 on LiL_i

once, scores them, and lists LiL_i as it goes. We first show why this one read sets its time, then how the pass is spread over the GPU, and then the steps of Algorithm 4.

In decode, SS is a single row: the one index query qi(idx)q_i^{(\mathrm{idx})} of a request scores each of its T/nˉbT/\bar n_b centroids cbc_b once per step. With ∣C∣=1|C|=1 in Equation (6), a step at 10M tokens takes FdecF^{\mathrm{dec}} flop on at least BreaddecB_{\mathrm{read}}^{\mathrm{dec}} bytes, and writing its row takes BscoresdecB_{\mathrm{scores}}^{\mathrm{dec}} bytes in fp32:

Fdec=2d(idx)T/nˉb≈168 MflopF^{\mathrm{dec}} = 2d^{(\mathrm{idx})}T/\bar n_b \approx168\,\mathrm{Mflop}
Breaddec=2d(idx)(T/nˉb+1)≈160 MiBB_{\mathrm{read}}^{\mathrm{dec}} = 2d^{(\mathrm{idx})}(T/\bar n_b+1) \approx160\,\mathrm{MiB}
Bscoresdec=4T/nˉb=5 MiB(8)B_{\mathrm{scores}}^{\mathrm{dec}} = 4T/\bar n_b = 5\,\mathrm{MiB} \tag*{(8)}

Its arithmetic intensity Fdec/Breaddec≈1F^{\mathrm{dec}}/B_{\mathrm{read}}^{\mathrm{dec}}\approx1 flop/B is far below the ridge point: the selection is bound by memory, and reading the centroids once would take at least Breaddec/4.8 TB/s=35 μsB_{\mathrm{read}}^{\mathrm{dec}}/4.8\,\mathrm{TB/s}=35\,\mu\mathrm{s} on H200. Prefill, bound by compute, can afford to read the centroids more than once: every tile of queries reads them all, and candidate scoring reads those of LiL_i again. In decode, a second read would about double the bytes the scan reads, so Algorithm 4 selects within a single read of the centroids, besides the small sample of its threshold estimate. It can, however, afford to write every si,bs_{i,b} to HBM, which adds only 3% to the bytes it moves, so that its fallback takes the top-pp of the written scores, without reading the centroids again. The threshold is thus not strictly needed in decode, but it saves a selection over all NiN_i written scores of each request: the expansion ranks only the buckets of LiL_i.

With a single query, only the row is split: the scan takes spans of 256 buckets, one per thread block. The centroids of a request lie in pages of Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})}, as its keys lie in pages of the KV cache, and each thread block reads the cbc_b of its span through the page table of the request. The threshold estimate and the scan score on the CUDA cores, with the same dot product: at 1 flop/B the scan is bound by HBM, not by arithmetic, and gains nothing from tensor cores. A request whose row fits in its list takes every bucket.

Each CUDA graph samples a fixed number nsn_s of buckets, from 2,048 to 16,384, in runs of 8 neighbouring buckets of a page, spread evenly over the row. A graph that covers longer contexts thus samples a smaller share of the row, with a smaller nθn_\theta and a noisier estimate, so the target γp\gamma p grows with the graph, from 1.5p1.5p to 3p3p. We set γ\gamma for each graph by hand, so that LiL_i still holds at least pp buckets, and fits in its buffer, when the count misses γp\gamma p by four times the deviation above, 4γp/nθ4\gamma p/\sqrt{n_\theta}. Algorithm 4 runs in four launches, followed by the expansion, and has no candidate scoring. The threshold estimate takes two launches: one scores the sample, and one finds its nθn_\theta-th best score by radix select over the bits of the scores, and rounds it down (lines 1–2). The bucket scan writes the si,bs_{i,b} of each span, and appends the buckets with si,b≥θis_{i,b}\ge\theta_i to LiL_i, already packed as σi,b\sigma_{i,b} (lines 3–6). It compares the very scores that rank the buckets, so it needs no margin. The listing gathers into LiL_i the spans that the thread blocks appended, in bucket order (line 7), and the expansion is that of Algorithm 3 (line 9). The list LiL_i has room for 2,048 to 8,192 buckets, by graph, and a request whose list overflows, or holds fewer than pp buckets, falls back to the top-pp of the scores the scan wrote (line 8). Programmatic dependent launch ([61]), a feature of Hopper and Blackwell GPUs, lets a kernel start while the one before it still runs, and wait for its results only where it reads them (line 5). The scan thus reads and scores the centroids while θi\theta_i is still being computed.

Tree prefill. In prefill, the beam search (Algorithm 2) starts at the lowest level with at most nstart=32768n_{\mathrm{start}}=32768 node buckets, a value we chose by a sweep of accuracy against time. It keeps a beam of pℓ=2048p_\ell=2048 node buckets on every level. The descent starts from every node bucket of this level (line 1), which one matrix product in cuBLAS scores in full. Each level of the descent (lines 2–7) then runs in one kernel (five or six launches in Triton), besides a matrix product in cuBLAS that scores the open node buckets of the level. In this kernel, a thread block holds the scores of one query in shared memory, one 32-bit word per candidate, up to the 227 KiB a thread block can take. It scores the children of the node buckets kept on the level above (line 3). It then finds the pℓp_\ell-th score by a radix select of four passes, each of which counts one byte of the words in a histogram of 256 bins (line 4). Finally, it writes the children of the node buckets it keeps as the candidates CC of the next level (line 5). Above about 24k node buckets, keeping the beam of the start level this way would leave a single thread block per SM, so it takes two launches instead: a listing of the node buckets above a sampled threshold, as in the flat search, and the keep of the beam. At level 0 (lines 8-9), the thread block writes the top-pp buckets as packed scores σi,b\sigma_{i,b}. The expansion of Algorithm 3 (lines 13-16) reads them as it reads LiL_i.

Tree decode. In decode, the new queries of all the requests of a batch descend together, through the beam kernels of the prefill, but over a copy of the hierarchy at fixed addresses, and at shapes set by the longest context, so that a CUDA graph can replay the step. Each request adds one query per step, but the descent still runs its levels one after another, so its time barely changes with the number of requests, while the flat search scans every bucket once per request. Every query scores the levels from the top level of the longest context down, and holds no candidates above its own start level. A step takes 7 to 13 kernels from 128k to 10M tokens, as the number of levels grows with the context.

Other steps. The remaining steps of D-LSA are the projections and the L2 normalisation of the index keys kj(idx)k_j^{(\mathrm{idx})} and queries qi(idx)q_i^{(\mathrm{idx})}, the pooling, and the tables of Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})}. They consist of matrix products, reductions and a sort, which run as standard PyTorch operations, with the products through cuBLAS. These steps cost little next to the scoring. On GLM-5.3-FLASH, for example, each of the two projections of a chunk of 32,768 tokens from the 512 dimensions of the latent takes 2⋅32768⋅512⋅642 \cdot32768 \cdot512 \cdot64 flop, 4.3 Gflop for both. This is under 1% of the scoring of the chunk at contexts of 1M tokens and beyond (Equation (6)).

Attention kernels. D-LSA attends through existing kernels, which only need to read the index lists in a layout of their own. The expansion writes the index list JiJ_i of every query ii as a row of nsink+k+nw=4612n_{\mathrm{sink}} + k + n_w = 4612 int32 entries. Each entry is the slot in the KV cache of one key j∈Jij \in J_i, and the row is padded with −1-1 when the query has fewer keys to attend to. The index lists of a chunk CC thus form one matrix of ∣C∣\lvert C\rvert rows. On GLM-5.3-FLASH, JiJ_i takes the place of the index list of the model’s own indexer, and the sparse attention of SGLang attends over it unchanged. On QWEN-3.8-27B, we pass every query ii to a paged attention kernel as a sequence of its own: its row is the page table, each page holds one slot, and the sequence length is ∣Ji∣\lvert J_i\rvert. Table 10 lists the attention kernels we use, from FlashAttention-3 (FA3) (Shah et al., [72]) and FA4, FlashMLA (Li et al., [44]), the TRT-LLM kernels of FlashInfer (Ye et al., 2025) and Triton. Each is the default of SGLang (Zheng et al., [96]) for its model and GPU, except where that default cannot run. For instance, on B300, D-LSA on QWEN-3.8-27B attends through the Triton kernel of SGLang, as FlashAttention supports pages of a single slot only on Hopper. With CP8, GLM-5.3-FLASH decodes through a split-KV sparse kernel that we add to SGLang

H200B300
ModelPhaseFull AttentionBlock Sparse Attention (Flat)D-LSA (Flat and Tree)
GLM-5.3-FLASHPrefillFA3FA3FA3TRT-LLMTRT-LLM
GLM-5.3-FLASHDecodeFlashMLAFA3FA3TritonTRT-LLM
QWEN-3.8-27BPrefillFA3TritonFA3TRT-LLMTriton
QWEN-3.8-27BDecodeFA3Triton + FA4FA3TRT-LLMTriton

Table 10. Attention kernels. Kernel through which each method attends, by model, phase and GPU, at TP8 or on one GPU. Full Attention, Block Sparse Attention (Flat), and D-LSA (Flat and Tree). Kernels in italics are non-default SGLang options.

(Appendix F.2), and on B300 it prefills through a TileLang kernel (Wang et al., [79]). With decode context parallelism (DCP8, Appendix § 14.§ 2) on H200, Full Attention and Block Sparse Attention (Flat) on QWEN-3.8-27B decode through Triton kernels.

N-LSA training. The kernels above serve D-LSA at inference. N-LSA also needs kernels for training, for its forward and its backward pass, with which a layer is up to 1.6×1.6\times faster than with Full Attention at 128k tokens (§ 5.1).

A few changes stem from the architectural differences of N-LSA compared to D-LSA. A training step processes whole sequences, of up to 128k tokens in our runs, with no KV cache and no decode, so every query of a sequence selects at once, as in a prefill chunk as long as the sequence. The keys change at every step as the attention projections learn, so every step clusters all keys again, where D-LSA clusters each cluster block once and keeps its buckets in the cache. The selection clusters and scores the attention’s own normalised keys and queries, of dimension 128, and takes the best score of the 8 query heads (Equation (3)). At 128k tokens, a row holds at most 16,384 buckets, from which a query selects p=512p = 512 for k=2048k = 2048 keys, next to a window of 512 keys and 4 sink keys. Finally, every step runs a backward pass, which D-LSA never does at long context.

Two of these differences shape the kernels. As the rows are short, the selection places θi\theta_i from the whole row rather than from a sample: a first pass over the row counts its scores in a histogram of 256 bins, and a second lists the buckets from the bin of the pp-th score up, both on tensor cores, and the top-pp of the list are then taken by their scores. The backward pass is the main addition. As N-LSA stops the gradient through the selection, the clustering and the scoring, the centroids need no backward pass, and the only new backward kernel is that of the sparse attention. For one query head, with logits zi,j=qi⋅kj/dhz_{i,j} = q_i \cdot k_j / \sqrt{d_h}, its forward pass, as in any attention kernel, runs over the queries, each query ii attending to the keys it selected and writing only its own output:

oi=∑j∈Jiαi,jvj,αi,j=softmax⁡j∈Ji(zi,j)(9)o_i = \sum_{j \in J_i} \alpha_{i,j} v_j,\qquad\alpha_{i,j} = \operatorname{softmax}_{j \in J_i}(z_{i,j}) \tag*{(9)}

Its backward pass cannot run over the queries in the same way, as each key jj and its value receive a gradient of the NTP loss L\mathcal{L} from every query that selected the key:

∂L∂kj=∑i∈Ij∂L∂zi,jqidh,∂L∂vj=∑i∈Ijαi,j∂L∂oi(10)\frac{\partial\mathcal{L}}{\partial k_j} = \sum_{i \in I_j} \frac{\partial\mathcal{L}}{\partial z_{i,j}} \frac{q_i}{\sqrt{d_h}},\qquad\frac{\partial\mathcal{L}}{\partial v_j} = \sum_{i \in I_j} \alpha_{i,j} \frac{\partial\mathcal{L}}{\partial o_i} \tag*{(10)}

where Ij={i:j∈Ji}I_j = \{i : j \in J_i\}, and the queries of IjI_j lie anywhere in the sequence at or after jj. As the 8 query heads share one key-value head, ∂L/∂kj\partial\mathcal{L}/\partial k_j and ∂L/∂vj\partial\mathcal{L}/\partial v_j also sum over the heads. Run over the queries, the backward pass would have many thread blocks add into the gradient of the same key, with an atomic addition for each share. The backward kernels instead run over the keys, as the backward pass of FlashAttention Dao et al., [11] does for dense attention (Algorithm 5). With a sparse selection, this first requires inverting the selection, listing the queries of each group of keys (lines 1-2). A thread block then holds the gradients of a group of keys in registers, adds the shares of these queries into them (lines 4-9), and writes them with few atomic additions (line 11). The gradient of each query sums over its own JiJ_i, so a separate pass gathers it as in the forward pass, computing ∂L/∂zi,j\partial\mathcal{L}/\partial z_{i,j} again (line 12). To invert it, the backward pass needs the selection of every query, the kk int32 long-range indices that the forward pass keeps, as the sink and the window follow from the positions: 1 GiB at 128k tokens, about the gap between the peak memory of N-LSA and that of Full Attention. Evaluating and implementing a port of these kernels to CuTe DSL, as for D-LSA, is left to future work.

With these kernels, § 5.1 times one attention layer of N-LSA on one H200, at batch 1 in bf16, against the same layer with causal attention in FA3. Each time covers the forward and the backward pass of the whole layer, from its input projections to its output projection, with the clustering and the selection.

Measurements. Table 11 times each step on one H200 with Nsight Compute, in a single profile with the clocks unlocked, with its CuTe DSL and its Triton kernels. The inputs are synthetic, at the shapes we serve: a prefill chunk of 32,768 queries at 10M tokens, a beam of pℓ=2048p_\ell= 2048 over node buckets of nb,ℓ=4n_{b,\ell} = 4, decode with one request at 10M tokens, the Ward clustering of the 64 cluster blocks of one prefill chunk, and the clustering of one node block of ncluster,ℓ=4096n_{\mathrm{cluster},\ell} = 4096 buckets into node buckets. The times are those of the kernels alone, without the PyTorch launches between them. The regime of a step is where its main CuTe DSL kernel sits: bound by neither as a radix select over packed integer keys (selection), or limited by its serial rounds (latency). The bucket scan, the largest step of the flat prefill, reaches 49% of the bf16 peak and uses 1% of the HBM bandwidth, so the tiles of queries reread the centroids mostly from the L2 cache. On real inputs, the index queries of 12 held-out documents of 1M tokens and 3 books of 10M tokens on GLM-5.3-FLASH, over all its sparse layers, LiL_i holds 1,679 buckets on average at 1M tokens and 1,737 at 10M, never overflows, and falls back, with fewer than pp buckets above θi\theta_i, for 5×10−55 \times10^{-5} of the queries at 1M and 10−310^{-3} at 10M.

StepCuTe DSL (ms)Triton (ms)SpeedupRegime
Flat prefill
Threshold estimate2.028.854.38×compute (28%)
Bucket scan11.3924.822.18×compute (49%)
Candidate scoring1.631.661.02×memory (85%)
Expansion2.374.141.75×selection
Total17.4139.482.27×
Tree prefill
Start level5.285.801.10×selection
One beam level6.257.851.26×selection
Level 0 (top-pp)9.139.591.05×memory (66%)
Clustering
Ward (buckets)0.0430.0561.30×latency
Node buckets0.4051.3873.42×latency
Flat decode
Decode select0.0710.1532.15×memory (75%)

Table 11. Indexer kernels. Time of each step of the indexer of D-LSA on one H200, with its CuTe DSL and its Triton kernels, and the regime of its main CuTe DSL kernel.

Algorithm 5 Sparse attention backward of N-LSA, for one query head

Require: qiq_i, kjk_j, vjv_j, oio_i and doido_i, in HBM log-sum-exp λi\lambda_i and the kk long-range indices of JiJ_i, kept by the forward pass

Ensure: dqidq_i, dkjdk_j, dvjdv_j, writing dxdx for ∂L/∂x\partial\mathcal{L}/\partial x

Inversion

  1. count, for each group GG of keys, the queries ii with Ji∩G≠∅J_i \cap G \ne\varnothing

  2. IG←⋃j∈GIjI_G \gets\bigcup_{j \in G} I_j, laid out by a prefix sum over the counts

Key and value gradients

every query ii, in parallel

  1. Di←doi⋅oiD_i \gets do_i \cdot o_i

every group GG of keys, in parallel

  1. hold kjk_j, vjv_j, j∈Gj \in G, in registers; dkjdk_j, dvj←0dv_j \gets0

  2. for each query i∈IGi \in I_G and j∈Ji∩Gj \in J_i \cap G do

  3.  zi,j←qi⋅kj/dh; αi,j←exp⁡(zi,j−λi)\ z_{i,j} \gets q_i \cdot k_j / \sqrt{d_h};\ \alpha_{i,j} \gets\exp(z_{i,j} - \lambda_i)

  4.  dvj←dvj+αi,j doi\ dv_j \gets dv_j + \alpha_{i,j}\,do_i

  5.  dzi,j←αi,j(doi⋅vj−Di)\ dz_{i,j} \gets\alpha_{i,j}(do_i \cdot v_j - D_i)

  6.  dkj←dkj+dzi,j qi/dh\ dk_j \gets dk_j + dz_{i,j}\,q_i/\sqrt{d_h}

  7. end for

  8. add dkjdk_j, dvjdv_j into zeroed HBM, with few atomic additions

Query gradients

every query ii, in parallel

  1. dqi←∑j∈Jidzi,j kj/dhdq_i \gets\sum_{j \in J_i} dz_{i,j}\,k_j/\sqrt{d_h}   ▷ dzi,jdz_{i,j} computed again

In decode, 3×10−43 \times10^{-4} of the steps fall back at 1M, mostly as their lists overflow, and none at 10M. The decode scan reads at 3.58 TB/s, 75% of the HBM bandwidth. At 10M tokens, the CuTe DSL kernels are faster than the Triton ones in every step, from 1.02× for candidate scoring, which is bound by the HBM bandwidth in both languages, to 4.4× for the threshold estimate. On B300, they are faster by up to 4.3× in every step but level 0 of the tree search, where the two languages tie. Of the 22 ms that CuTe DSL saves on a prefill chunk at 10M tokens, 13.4 ms come from the bucket scan, mostly from overlapping its epilogue with its products, and 6.8 ms from the threshold estimate, whose Triton version writes the scores of its sample to HBM and reads them twice. The Triton versions of other steps take more launches or more passes: the node-bucket clustering takes three launches per round and reads two flags on the host every eight rounds (3.4×), and the expansion finds its cut one bit per pass rather than one byte (1.75×). Table 11 sums the kernel times of the four launches of the decode select (the sample, the threshold, the scan and the listing), 71 µs at 10M tokens. Replayed from a CUDA graph, where programmatic dependent launch overlaps them, they take 14.7 µs at 1M tokens and 60.7 µs at 10M, against 36.4 µs and 146.6 µs in Triton. With the expansion and the other kernels of the selection, a flat decode step selects in 35 µs at 1M tokens and 86 µs at 10M, and Table 12 compares it with the tree search. With one request, the flat search is faster up to about 5M tokens. With 4 and 16 requests, the tree search is faster from about 1M and 256k tokens, and with 16 requests it is 5.6× faster at 10M tokens. As we evaluate one request at a time, every result of D-LSA (Tree) decodes with the flat search, and only its prefill uses the tree. At 1M tokens, the four steps of the flat prefill take 5.2 ms in CuTe DSL against 9.4 ms in Triton. For a whole layer of GLM-5.3-FLASH on H200, CuTe DSL makes a prefill chunk at 10M tokens 1.35× faster than Triton.

Requests128k Flat128k Tree1M Flat1M Tree10M Flat10M Tree
1303835498668
43240565421181
16424710982700125

Table 12. Tree search in decode. Time of the selection of one layer for one decode step, from the scoring of the centroids to the index lists, in µs, with the flat and the tree search, on one H200, replayed from a CUDA graph. The faster search is in bold.

Parallelism

We describe our integration of D-LSA with tensor (TP) and context parallelism (CP) in SGLang, for multi-head latent attention (MLA), as in GLM-5.3-FLASH, and for grouped-query attention (GQA), as in QWEN-3.8-27B. NN denotes the number of ranks of a parallel group, and rr one of them. We study setups with N=8N = 8 (size of our NVLink domain).

LSA is orthogonal to expert and pipeline parallelism: the former splits only the experts, and the latter keeps the index of a layer co-located with its attention on the same rank, so in this section we do not describe these two mesh configurations, but focus on TP and CP instead.

For both TP and CP, every rank holds the whole bucket cache Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})}, which trades memory for communication (Section 3). Table 13 summarises the communication that the indexer adds for the two mesh configurations. We discuss these in more detail below.

PhaseCollectiveSizeFrequency
TP, MLAbothALLREDUCE of query head sums2 KiB / tokenevery chunk and step
TP, GQAbothALLREDUCE of partial index queries and keys512 B / tokenevery chunk and step
PCP, MLAprefillALLREDUCE of index keys——
DCPdecodeALLREDUCE of index keys64 KiB / blockevery nclustern_{\mathrm{cluster}} steps

Table 13. Communication added by the indexer. Collectives that the indexer of D-LSA adds to one layer for TP alone, PCP and DCP, with MLA (GLM-5.3-FLASH) and GQA (QWEN-3.8-27B), beyond those of the attention. Sizes are per rank, and decode frequencies per request.

Tensor parallelism. We first describe TP alone, without CP, as illustrated in Figure 18. For TP, work gets split by head: rank rr holds a set QrQ_r of nh/Nn_h/N query heads, and projects them into a partial index query qi,r(idx)q_{i,r}^{(\mathrm{idx})}:

Figure 18. D-LSA with tensor parallelism (TP8). Top: MLA (GLM-5.3-FLASH). Bottom: GQA (QWEN-3.8-27B). Indexer keys: 1 projection (from the replicated latents with MLA, a partial per KV head with GQA), 2 bucket cache (the same on every rank). Indexer queries: A sum over the rank’s query heads and projection (a partial, summed across ranks by one ALLREDUCE, before the projection with MLA), B selecting the index list (the same on every rank), C attention over the rank’s own query heads.

qi,r(idx)=∑h∈Qrqi,h(attn)WQ(idx)(11)q_{i,r}^{(\mathrm{idx})} = \sum_{h \in Q_r} q_{i,h}^{(\mathrm{attn})} W_Q^{(\mathrm{idx})} \tag*{(11)}

With MLA, the ranks sum the query heads qi(attn)q_i^{(\mathrm{attn})} on the latent instead, and apply WQ(idx)W_Q^{(\mathrm{idx})} to the sum after the ALLREDUCE below, which is equivalent by linearity. ❶ With MLA, the latent kj(attn)k_j^{(\mathrm{attn})} is shared by all heads, thus replicated on every rank. With GQA, the KV heads are split across ranks too, and an index key kj,g(idx)k_{j,g}^{(\mathrm{idx})} is projected from all of them, so rank rr projects the keys kj,g(attn)k_{j,g}^{(\mathrm{attn})} of the set Kr\mathcal{K}_r of KV heads it contributes into a partial index key kj,r(idx)k_{j,r}^{(\mathrm{idx})}:

kj,r(idx)=∑g∈Krkj,g(attn)WK,g(idx)(12)k_{j,r}^{(\mathrm{idx})} = \sum_{g \in\mathcal{K}_r} k_{j,g}^{(\mathrm{attn})} W_{K,g}^{(\mathrm{idx})} \tag*{(12)}

where WK,g(idx)W_{K,g}^{(\mathrm{idx})} are the rows of WK(idx)W_K^{(\mathrm{idx})} that project KV head gg. Likewise, query head hh uses only the rows WQ,g(h)(idx)W_{Q,g(h)}^{(\mathrm{idx})} of the KV head g(h)g(h) it attends, in place of WQ(idx)W_Q^{(\mathrm{idx})} in (11).

For TP8, the four KV heads of QWEN-3.8-27B are each replicated on two ranks: both contribute their own query heads, but only one counts the KV head in its Kr\mathcal{K}_r, and the other contributes kj,r(idx)=0k_{j,r}^{(\mathrm{idx})}=0. One ALLREDUCE sums the partials, in fp32, into the projections of (1), before their normalisation:

qi(idx)=∑rqi,r(idx),kj(idx)=∑rkj,r(idx)(13)q_i^{(\mathrm{idx})}=\sum_r q_{i,r}^{(\mathrm{idx})},\qquad k_j^{(\mathrm{idx})}=\sum_r k_{j,r}^{(\mathrm{idx})} \tag*{(13)}

B Every rank thus holds the same index query qi(idx)q_i^{(\mathrm{idx})} and selects the same JiJ_i from its copy of Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})}. C It attends over JiJ_i with its own query heads QrQ_r. 2 The clustering takes the index keys kj(idx)k_j^{(\mathrm{idx})} of a cluster block only once all its nclustern_{\mathrm{cluster}} keys have arrived, over several chunks or decode steps. With MLA, a rank recomputes them from the latents kj(attn)k_j^{(\mathrm{attn})} in its KV cache, so they are never stored. With GQA, a rank holds only some KV heads, so it cannot recompute them alone. Instead, it stores kj(idx)k_j^{(\mathrm{idx})} once summed: the ALLREDUCE of (13) carries the partial key kj,r(idx)k_{j,r}^{(\mathrm{idx})} of a token together with the token’s own partial query, so storing it costs memory, 2d(idx)=128B2d^{(\mathrm{idx})}=128\mathrm{B} per token, but no communication. Recomputing it would save that memory, at the cost of a second ALLREDUCE of the fp32 partial keys,

256 B per token, once each block is complete.

With MLA, the ALLREDUCE carries the 512 dimensions of the latent, 2 KiB. Since the KV cache is replicated, Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})} adds the same 24.5 B per token to the 1 KiB of KV cache as without parallelism.

With GQA, the ALLREDUCE carries the partial index query and key, 512 B. A rank of QWEN-3.8-27B holds one of its four KV heads, 1 KiB, to which the indexer adds 24.5 B of bucket cache and 2d(idx)=128B2d^{(\mathrm{idx})}=128\mathrm{B} of stored index key.

Context parallelism

CP splits work by token in two ways (Liu et al., [49]; Yang et al. [85]), which we describe in the following, and illustrate in Figure 19.

Figure 19. D-LSA with context parallelism (CP8). Top: prefill of GLM-5.3-FLASH (PCP8 + DCP8), each rank holding a contiguous span of the query rows. Indexer keys: ❶ projection of the whole chunk (on every rank), ❷ bucket cache (the same on every rank). Indexer queries: A projection of the rank’s own rows, B selecting their index lists (other ranks’ rows lighter), C marking the selected prefix rows (a 1 B mask, merged across ranks), D gathering those rows from the ranks that hold them (padded), E attention of the rank’s own rows. Bottom: decode of GLM-5.3-FLASH (PCP8 + DCP8) and QWEN-3.8-27B (TP8 + DCP8). Indexer keys: ❶ projection of a complete cluster block (summed across ranks, once per block), ❷ bucket cache. Indexer queries: A projection, B selecting the index list (the same on every rank), C attention over the entries the rank holds, D merging the partial outputs of the ranks (by log-sum-exp).

Decode context parallelism (DCP), also called KV parallelism (Bhatia et al., [6]), splits the KV cache across the ranks, in both prefill and decode, so that each rank stores only 1/N1/N of it, which lets a context of millions of tokens fit in HBM, and reads only its share in each decode step. SGLang deals the tokens out round-robin: token jj goes to rank rjr_j, its slot in the KV cache modulo NN, and that rank holds key kjk_j and thus computes index key kj(idx)k_j^{(\mathrm{idx})}.

Prefill context parallelism (PCP) also splits the query rows of a chunk CC, giving rank rr the rows CrC_r, so that it selects the index lists JiJ_i of its own rows i∈Cri \in C_r only. The index query is not split: with PCP, every rank holds in prefill all nhn_h query heads qi,h(attn)q_{i,h}^{(\mathrm{attn})} of its rows, and in decode (and with GQA also in prefill), the ranks gather them, as the attention does anyway, so every rank computes qi(idx)q_i^{(\mathrm{idx})} itself.

We extend SGLang with PCP for GLM-5.3-FLASH, which leaves us free to choose its layout, for both the base model and LSA. We make each CrC_r one contiguous span of CC, which lets the linear-attention layers pass their recurrent state along it. LSA does not depend on this split, but it is a natural fit: as every query attends a fixed budget of keys, a contiguous split already balances the sparse attention across ranks, whereas dense causal attention commonly uses a zigzag or striped split to balance its causal work [7, 85].

With MLA, we use both PCP and DCP (Figure 19, top), with no TP in the prefill attention, since SGLang was already developing PCP for GLM-5.3-FLASH, while it had none for QWEN-3.8-27B. This also suits LSA: PCP splits the prefill by query rows rather than by heads. A Every rank thus holds all query heads of its rows and computes their index queries qi(idx)q_i^{(\mathrm{idx})}, i∈Cri \in C_r, alone, without the ALLREDUCE of Equation (13) that TP needs. For PCP, the attention already gathers the latents KC(attn)K_C^{(\mathrm{attn})} of the whole chunk. 1 Every rank thus projects the index keys KC(idx)K_C^{(\mathrm{idx})} of the whole chunk itself, and 2 clusters them into the same Bcache(idx)B_{\mathrm{cache}}^{(\mathrm{idx})}. The indexer thus adds no communication, as long as each chunk starts on a cluster block, as chunks of 32768 = 6464 cluster tokens do. B Each rank selects JiJ_i for its rows i∈Cri \in C_r, but DCP splits the KV cache of the prefix across ranks, so most of the selected keys j∈Jij \in J_i live on other ranks, rj≠rr_j \ne r. sparse attention, of the base model as of LSA, gathers only those rows. C Every rank marks the prefix rows that its lists JiJ_i, i∈Cri \in C_r, select in a mask JCrJ_{C_r} of one byte per prefix row, and one ALLREDUCE with the maximum gives every rank their union JCJ_C over the chunk, less the chunk itself, whose rows every rank already holds:

JC=⋃i∈CJi∖C(14)J_C = \bigcup_{i \in C} J_i \setminus C \tag*{(14)}

D Each rank then contributes the latents KJC,r(attn)K^{(\mathrm{attn})}_{J_C,r} of the rows of JCJ_C it holds, padded to the largest count over the ranks, to one ALLGATHER, which gives every rank all of JCJ_C. Every rank lays them out with the rows of the chunk in one buffer, KJC∪C(attn)K^{(\mathrm{attn})}_{J_C \cup C}, and remaps JiJ_i, i∈Cri \in C_r, onto it. E The attention kernel then runs unchanged on the remapped lists. Without the mask, every rank would gather the whole prefix, 1 KiB per row, or 10 GiB per layer at a prefix of T=10MT = 10\mathrm{M} tokens, while the mask costs 1 B per row. Empirically, in our runs of GLM-5.3-FLASH on a 10M-token prompt, the union JCJ_C of the index lists of all 32,768 queries of a late chunk covers only about 30% of the prefix, as nearby queries select largely the same keys, so the ALLGATHER moves, and every rank holds, about 30% of the prefix rather than all of it, up to the padding.

With GQA, we run attention with TP8 and add DCP8, without PCP: TP splits the prefill by head and computes the keys of the whole chunk on every rank, and DCP splits the KV cache, which is what lets a context of 10M tokens fit on one node. DCP8 replaces the split of the KV heads: every rank holds all four KV heads of the tokens it owns, rj=rr_j = r, and gathers the query heads of the other ranks before the indexer and the attention, in prefill and decode. It thus needs neither the partial sums of Equation (13) nor a stored kj(idx)k_j^{(\mathrm{idx})} like MLA, it recomputes the index keys of its own tokens from its KV cache.

In decode (Figure 19, bottom), each new token jj is written to its rank rjr_j, so the keys of a cluster block are spread across all ranks. 1 Once its nclustern_{\mathrm{cluster}} keys are complete, every rank needs all their index keys kj(idx)k_j^{(\mathrm{idx})}, but can compute only those with rj=rr_j = r. Each rank writes them into an ncluster×d(idx)n_{\mathrm{cluster}} \times d^{(\mathrm{idx})} buffer, one row per token of the block in order, with zeros in the rows of the other ranks, and one ALLREDUCE sums the buffers, 64 KiB per block in bf16, or 128 B per token. As every row is non-zero on one rank only, the sum gathers the keys of the block in order, where an ALLGATHER would group them by rank. 2 Every rank then clusters the same keys into the same buckets of Bcache(idx)B^{(\mathrm{idx})}_{\mathrm{cache}}. Until then, the keys of the block are covered by the window of the nwn_w most recent keys, which is selected by position, so they need no index key. This ALLREDUCE runs before the CUDA graph of the decode step is replayed. In decode the attention splits the query heads across the ranks, by TP with GQA and, with MLA, by a split of the MLA heads that we add to SGLang, so every rank gathers all nhn_h query heads qi,h(attn)q^{(\mathrm{attn})}_{i,h} of every request, as the attention does anyway. A Every rank thus projects the same index query qi(idx)q_i^{(\mathrm{idx})}, B selects the same JiJ_i, and C attends over the entries of JiJ_i it holds,

Ji,r={j∈Ji:rj=r}(15)J_{i,r} = \{j \in J_i : r_j = r\} \tag*{(15)}

into a partial output oi,ro_{i,r}, softmax-normalised over Ji,rJ_{i,r}, with λi,r\lambda_{i,r} the log-sum-exp of its logits. D The ranks merge the partial outputs of each query head by their log-sum-exp (Dao et al., [12]):

oi=∑reλi,r−λioi,r,λi=log⁡∑reλi,r(16)o_i = \sum_r e^{\lambda_{i,r}-\lambda_i} o_{i,r}, \qquad\lambda_i = \log\sum_r e^{\lambda_{i,r}} \tag*{(16)}

As the Ji,rJ_{i,r} partition JiJ_i, oio_i is the attention over JiJ_i, up to floating-point rounding.

Memory. Replicating Bcache(idx)B^{(\mathrm{idx})}_{\mathrm{cache}} trades memory for communication. With βKV\beta_{\mathrm{KV}} and βB\beta_B the bytes per token of the KV cache and of the bucket cache, the persistent storage of the KV cache and the flat index, per layer and rank, is approximately

T(βKVN+βB)(17)T \left(\frac{\beta_{\mathrm{KV}}}{N} + \beta_B\right) \tag*{(17)}

where βB≈24.5\beta_B \approx24.5B, and βKV=1024\beta_{\mathrm{KV}} = 1024B for GLM-5.3-FLASH and 4,096 B for Qwen-3.8-27B. At T=10T = 10M with DCP8, the bucket cache adds 246 MiB as measured, TβBT\beta_B plus padding rows, to the TβKV/N=1.25T\beta_{\mathrm{KV}}/N = 1.25 GiB share of the KV cache of a layer of GLM-5.3-FLASH (+19%), and 5% on Qwen-3.8-27B. An index per rank over its own keys would avoid the ALLREDUCE of the keys, but its buckets and its JiJ_i would differ from those of a single GPU.

References

  1. [1]Alabi, T., Blanchard, J. D., Gordon, B., and Steinbach, R. Fast k-selection algorithms for graphics processing units. ACM Journal of Experimental Algorithmics, 17, 2012. doi: 10.1145/2133803.2345676.DOI
  2. [2]Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Zinsley, D., Zou, J., Rudra, A., and Ré, C. Simple linear attention language models balance the recall-throughput tradeoff. In Proceedings of the 41st International Conference on Machine Learning, 2024.arxiv.org/abs/2402.18668
  3. [3]Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., Tang, J., and Li, J. Long-Bench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025.arxiv.org/abs/2412.15204
  4. [5]Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.arxiv.org/abs/2004.05150
  5. [6]Bhatia, N., More, A., Borkar, R., Mitra, T., Matas, R., Zhao, R., Golub, M., Mudigere, D., Pharris, B., and Darvish Rouhani, B. Helix parallelism: Rethinking sharding strategies for interactive multi-million-token LLM decoding. arXiv preprint arXiv:2507.07120, 2025.arxiv.org/abs/2507.07120
  6. [7]Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J. Striped attention: Faster ring attention for causal transformers. arXiv preprint arXiv:2311.09431, 2023.arxiv.org/abs/2311.09431
  7. [8]Chen, Y., Zhang, J., Lu, B., Zhang, Q., Zhang, C., Liu, J., Luo, J., Liu, D., Jiang, H., Chen, Q., Ding, B., Yan, X., Jiang, J., Chen, C., Zhang, M., Li, C., Yang, Y., Yang, F., and Yang, M. RetroInfer: A vector storage engine for scalable long-context LLM inference. Proceedings of the VLDB Endowment, 19(5):1016–1031, 2026. doi: 10.14778/3796195.3796212.DOI
  8. [9]Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.arxiv.org/abs/1904.10509
  9. [10]Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  10. [11]Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, volume 35, pp. 16344–16359. Curran Associates, Inc., 2022. doi: 10.52202/068431-1189.DOI
  11. [12]Dao, T., Haziza, D., Massa, F., and Sizov, G. Flash-Decoding for long-context inference, 2023. URL https://pytorch.org/blog/flash-decoding/.
  12. [13]DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
  13. [14]DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.arxiv.org/abs/2512.02556
  14. [15]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026a.arxiv.org/abs/2606.19348
  15. [16]DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv preprint arXiv:2609.19969, 2026b.
  16. [17]Desai, A., Yang, S., Cuadron, A., Zaharia, M., Gonzalez, J. E., and Stoica, I. HashAttention: Semantic sparsity for faster inference. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 13402–13418. PMLR, 2025.arxiv.org/abs/2412.14468
  17. [18]Floyd, R. W. and Rivest, R. L. Expected time bounds for selection. Communications of the ACM, 18(3):165–172, 1975. doi: 10.1145/360680.360691.DOI
  18. [19]Gao, Y., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H., Cao, T., Yang, F., and Yang, M. SeerAttention: Self-distilled attention gating for efficient long-context prefilling. In Advances in Neural Information Processing Systems, volume 38, pp. 62230–62253. Curran Associates, Inc., 2025. doi: 10.52202/085713-1869.DOI
  19. [20]Gao, Y., Guo, S., Cao, S., Xia, Y., Cheng, Y., Wang, L., Ma, L., Sun, Y., Ye, T., Dong, L., So, H. K-H., Hua, Y., Cao, T., Yang, F., and Yang, M. Sparse attention adaptation for long reasoning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=c5BOchM6J8.arxiv.org/abs/2506.08889
  20. [21]Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
  21. [23]Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., and Peste, A. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1–124, 2021. URL http://jmlr.org/papers/v22/21-0366.html.arxiv.org/abs/2102.00554
  22. [24]Hooper, C., Zhao, S., Manolache, L., Kim, S., Mahoney, M., Shao, S., Keutzer, K., and Gholami, A. Multipole attention for efficient long context reasoning. In Advances in Neural Information Processing Systems, volume 38, pp. 41954–41977. Curran Associates, Inc., 2025a. doi: 10.52202/085713-1255.DOI
  23. [25]Hooper, C. R. C., Kim, S., Mohammadzadeh, H., Mah-eswaran, M., Zhao, S., Paik, J., Mahoney, M. W., Keutzer, K., and Gholami, A. Squeezed attention: Accelerating long context length LLM inference. In Proceedings of the 63rd Annual Meeting of the Association of Computational Linguistics (Volume 1: Long Papers), pp. 32631–32652, Vienna, Austria, 2025b. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.1568.DOI
  24. [26]Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024.arxiv.org/abs/2404.06654
  25. [31]Jégou, H., Tavenard, R., Douze, M., and Amsaleg, L. Searching in one billion vectors: Re-rank with source coding. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 861–864. IEEE, 2011. doi: 10.1109/ICASSP.2011.5946540.DOI
  26. [32]Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems, volume 37, pp. 52481–52515. Curran Associates, Inc., 2024. doi: 10.52202/079017-1663.DOI
  27. [33]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VF8yNQM66.arxiv.org/abs/2310.06770
  28. [34]Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2021. doi: 10.1109/TBDATA.2019.2921572.
  29. [35]Kandpal, N., Lester, B., Raffel, C., Majstorovic, S., Biderman, S., Abbasi, B., Soldaini, L., Shippole, E., Cooper, A. F., Skowron, A., Kirchenbauer, J., Longpre, S., Swatiwaka, L., Albalak, A., Xu, Z., Penedo, G., Ben Al-lal, L., Bakouch, E., Pressman, J. D., Fan, H., Stander, D., Song, G., Gokaslan, A., Goldstein, T., Bartoldson, B. R., Kailkhura, B., and Murray, T. The Common Pile v0.1: An 8TB dataset of public domain and openly licensed text. arXiv preprint arXiv:2506.05209, 2025.arxiv.org/abs/2506.05209
  30. [36]Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. In The Eighth International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkgNKkHtvB.arxiv.org/abs/2001.04451
  31. [37]Kocetkov, D., Li, R., Ben Allal, L., Li, J., Mou, C., Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, M., Muñoz Ferrandis, C., Jernite, Y., Mitchell, M., Hughes, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Wolfe, T. The Stack: 3 TB of permissively licensed source code. Transactions on Machine Learning Research, 2023.arxiv.org/abs/2211.15533
  32. [38]Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., Bloom, R., Broadley, T., Du, H., Goodrich, B., Jurkovic, N., Miles, L., Nix, S., Lin, T., Parikh, N., Rein, D., Koba Sato, L. J., Wijk, H., Ziegler, D., Barnes, E., and Chan, L. Measuring AI ability to complete long software tasks. In Advances in Neural Information Processing Systems, volume 38, pp. 102382–102435. Curran Associates, Inc., 2025. doi: 10.52202/085713-3086.DOI
  33. [39]Lai, X., Xu, W., Yang, Y., Chen, Q., Xu, Y., Zeng, L., Li, X., Sun, H., Zhu, H., Zhang, V., Hu, J., Li, J., Gao, R., Li, Z., Zhu, S., Zhou, J., and Zhao, P. MiniMax sparse attention. arXiv preprint arXiv:2606.13392, 2026.arxiv.org/abs/2606.13392
  34. [40]Lee, H., Park, G., Lee, Y., Suh, J., Kim, J., Jeong, W., Kim, B., Lee, H., Jeon, M., and Hwang, S. J. A training-free sub-quadratic cost transformer model serving framework with hierarchically pruned attention. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id=PTcMzQqKmn.arxiv.org/abs/2406.09827
  35. [41]Lee, H., Park, G., Suh, J., and Hwang, S. J. InfiniteHiP: Extending language model context up to 3 million tokens on a single GPU. arXiv preprint arXiv:2502.08910, 2025b.arxiv.org/abs/2502.08910
  36. [42]Leng, J., Hu, X., Wang, J., Li, J., Wu, W., and Lu, Y. Understanding and improving length generalization in hierarchical sparse attention models. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=HqdSQk6q6qc.arxiv.org/abs/2510.17196
  37. [43]Li, D., Liu, Z., Lin, G., Hu, B., and Zhang, M. LycheeCluster: Efficient long-context inference with structure-aware chunking and hierarchical KV indexing. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 7607–7623, San Diego, California, United States, 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.376.DOI
  38. [44]Li, J., Liu, S., and Sun, Y. FlashMLA: Efficient multi-head latent attention kernels, 2025. URL https://github.com/deepseek-ai/FlashMLA.
  39. [46]Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2305.20050
  40. [47]Liu, D., Chen, M., Lu, B., Jiang, H., Han, Z., Zhang, Q., Chen, Q., Zhang, C., Ding, B., Zhang, K., Chen, C., Yang, F., Yang, Y., and Qiu, L. RetrievalAttention: Accelerating long-context LLM inference via vector retrieval. In Advances in Neural Information Processing Systems, volume 38, pp. 60539–60566. Curran Associates, Inc., 2025a. doi: 10.52202/085713-1816.DOI
  41. [48]Liu, G., Li, C., Zhao, J., Zhang, C., and Guo, M. ClusterKV: Manipulating LLM KV cache in semantic space for recallable compression. In 2025 62nd ACM/IEEE Design Automation Conference (DAC), pp. 1–7. IEEE, 2025b. doi: 10.1109/DAC63849.2025.11132479.DOI
  42. [49]Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.01889
  43. [52]Madaan, L., Bhojanapalli, S., Jain, H., and Jain, P. Treeformer: Dense gradient trees for efficient attention computation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=DWn1Eb2fK.arxiv.org/abs/2208.09015
  44. [53]Mao, Y., Li, M. Y., and Fox, E. B. Simplified sparse attention via gist tokens. arXiv preprint arXiv:2604.20920, 2026a.arxiv.org/abs/2604.20920
  45. [54]Mao, Y., Wang, Q., Ester, M., and Li, K. IceCache: Memory-efficient KV-cache management for long-sequence LLMs. In The Fourteenth International Conference on Learning Representations, 2026b. URL https://openreview.net/forum?id=yHxSKM9kdr.arxiv.org/abs/2604.10539
  46. [55]MiniMax. MiniMax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313, 2025.arxiv.org/abs/2501.08313
  47. [58]NVIDIA. NVIDIA H200 tensor core GPU. https://www.nvidia.com/en-us/data-center/h200/, 2026c. Accessed 2026-10-28.
  48. [61]NVIDIA. CUDA programming guide, section 4.5: Programmatic dependent launch and synchronization. https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/programmatic-dependent-launch.html, 2026f. Accessed 2026-10-30.
  49. [62]Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024.arxiv.org/abs/2406.17557
  50. [63]Peng, B., Quesnelle, J., Fan, H., and Shippole, E. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2309.00071
  51. [64]Qwen Team. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026a.
  52. [65]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026b. URL https://qwen.ai/blog?id=qwen3.8.
  53. [67]Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
  54. [68]Roy, A., Saffar, M., Vaswani, A., and Grangier, D. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53–68, 2021. doi: 10.1162/tacl_a_00353.DOI
  55. [69]Schlag, I., Irie, K., and Schmidhuber, J. Linear transformers are secretly fast weight programmers. In Proceedings of the 38th International Conference on Machine Learning, pp. 9355–9366, 2021.arxiv.org/abs/2102.11174
  56. [70]Schmidhuber, J. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992a. doi: 10.1162/neco.1992.4.1.131.DOI
  57. [71]Schmidhuber, J. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234–242, 1992b. doi: 10.1162/neco.1992.4.2.234.DOI
  58. [72]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems, volume 37, pp. 68658–68685. Curran Associates, Inc., 2024. doi: 10.52202/079017-2193.
  59. [74]Tang, B., Qin, Z., Pan, Y., Li, Z., and Liu, P. Block sparse attention with log-linear complexity. arXiv preprint arXiv:2609.31093, 2026.arxiv.org/abs/2609.31093
  60. [75]Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasicki, B., and Han, S. QUEST: Query-aware sparsity for efficient long-context LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47901–47911. PMLR, 2024.arxiv.org/abs/2406.10774
  61. [76]Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Atluri, A., Qiao, F., Majcher, P., Springer, P., Hohenbarch, M., Wang, J., and Gupta, M. CUTLASS, 2023. URL https://github.com/NVIDIA/cutlass.
  62. [77]Tillet, P., Kung, H. T., and Cox, D. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19. Association for Computing Machinery, 2019. doi: 10.1145/3315508.3329973. Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang, S., Chen, T., Kasicki, B., Grover, V., Krishnamurthy, A., and Ceze, L. FlashInfer: Efficient and customizable attention engine for LLM inference serving. In Proceedings of Machine Learning and Systems, volume 7, 2025.
  63. [78]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.arxiv.org/abs/1706.03762
  64. [79]Wang, L., Cheng, Y., Shi, Y., Tang, Z., Mo, Z., Xie, W., Ma, L., Xia, Y., Xue, J., Yang, F., and Yang, Z. TileLang: A composable tiled programming model for AI systems. arXiv preprint arXiv:2504.17577, 2025.arxiv.org/abs/2504.17577
  65. [80]Wang, L., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024.arxiv.org/abs/2406.01574
  66. [81]Ward, Jr., J. H. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301):236–244, 1963. doi: 10.1080/01621459.1963.10500845.DOI
  67. [82]Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., and Sun, M. InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory. In Advances in Neural Information Processing Systems, volume 37, pp. 119638–119661. Curran Associates, Inc., 2024a. doi: 10.52202/079017-3801.DOI
  68. [83]Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=NG7sS51zVF.arxiv.org/abs/2309.17453
  69. [85]Yang, A., Yang, J., Ibrahim, A., Xie, X., Tang, B., Sizov, G., Park, J., and Huang, J. Context parallelism for scalable million-token inference. In Proceedings of Machine Learning and Systems, volume 7. MLSys, 2025a.arxiv.org/abs/2411.01783
  70. [86]Yang, S., Kautz, J., and Hatamizadeh, A. Gated delta networks: Improving Mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025b.arxiv.org/abs/2412.06464
  71. [87]Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.1126.DOI
  72. [88]Yuan, J., Shinn, C., Xu, K., Cui, J., Klimiashvili, G., Xiao, G., Zheng, P., Li, B., Zhou, Y., Ye, Z., You, W., Zheng, T., Brown, D., Wang, P., Hoehenricher, M., Cai, R., Demouth, J., Owens, J. D., Hu, X., Han, S., Liu, T., and Mao, H. BLASST: Dynamic BLocked attention sparsity via softmax thresholding. In Proceedings of Machine Learning and Systems, volume 8, pp. 843–859. MLSys, 2026.arxiv.org/abs/2512.12087
  73. [89]Zaheer, M., Guruganes h, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pp. 17283–17297. Curran Associates, Inc., 2020.arxiv.org/abs/2007.14062
  74. [90]Z.ai. GLM-5.3-Flash, 2026. URL https://z.ai/blog/glm-5.3-flash.
  75. [91]Zan, W., Zhang, J., Tan, J., Liu, H., Wang, C., Li, X., Ma, D., Wu, G., Lu, Y., Li, F., Sun, Y., Pei, P., Xie, Y., and Cai, X. LongCat sparse attention: Taming the lightning via streaming-aware hierarchical cross-layer indexing. arXiv preprint arXiv:2608.01662, 2026.arxiv.org/abs/2608.01662
  76. [92]Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019.arxiv.org/abs/1905.07830
  77. [93]Zhang, J., Naruse, A., Li, X., and Wang, Y. Parallel top-K algorithms on GPU: A comprehensive study and new methods. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–13. ACM, 2023a. doi: 10.1145/3581784.3607062.DOI
  78. [94]Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, volume 36, pp. 34661–34710. Curran Associates, Inc., 2023b. doi: 10.52202/075280-1506.DOI
  79. [95]Zhao, W., Zhou, Z., Su, Z., Xiao, C., Li, Y., Li, Y., Zhang, Y., Zhao, W., Li, Z., Huang, Y., Sun, A., Han, X., and Liu, Z. InfLLM-V2: Dense-sparse switchable attention for seamless short-to-long adaptation. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=z2F9V0H6Vi.arxiv.org/abs/2509.24663
  80. [96]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozakyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2312.07104
  81. [97]Zhu, D., Zhou, X., Qin, S., Zhu, X., Ding, H., Zhong, S., Wen, Z., Xie, Z., Gou, C., Ren, L., Wang, Y., Zhong, J., Liu, R., Gao, T., Lin, Y., Zhang, J., Song, M., Qi, X., Wu, J., Zhang, C., Piao, Y., Niu, Z., Lin, H., Meng, L., Tang, P., Tang, C., Wu, S., Zheng, H., Liu, Y., Zhu, L., Wang, H., Ding, M., Wan, Z., Liu, H., Wang, S., Zhu, H., Zhang, X., Chai, N., Liu, Y., Lai, P., Yuan, S., Su, Z., Zhang, G., Zhou, W., Du, Y., Huang, W., and Shi, G. EdgeBench: Unveiling scaling laws of learning from real-world environments. arXiv preprint arXiv:2607.05155, 2026.arxiv.org/abs/2607.05155

Paper details

vlnc:W_g5y8sw3c
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents