WorksComputer Science
Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
Abstract
Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training faster at long context. We propose two variants, Distilled-LSA (D-LSA), which can be applied to pre-trained models via distillation, and Native-LSA (N-LSA), which is trained natively on the next-token loss alone. On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks. N-LSA matches the accuracy of full attention with a training speedup of 1.6x per layer at 128k tokens. LSA offers practical and efficient attention for models to benefit from ultra-long context.
Introduction
Large language models have moved from answering single requests to agentic workflows in long-horizon tasks, such as software engineering and open-ended scientific research [33] (Lu et al. 2026). These tasks keep growing in length [38], and models with longer context windows tend to score higher on them [97]. In line with this trend, recent models extend their context windows to 1M tokens and beyond [21, 55, 15, 64].
At this context size, attention dominates the cost, as its work grows with context [78, 11]. An established technique to reduce such costs is sparse attention [23], in which each query attends to only a subset of the keys, chosen by a fixed pattern [9, 5, 89], by their content [36, 68], or by a dedicated indexer [87, 14]. Recent frontier models are increasingly adopting sparse
attention, in which keys are selected using a dedicated indexer ([14, 64]).
However, current indexers still score every preceding key, or block of keys, for each query. This remains expensive (quadratic in prefill, linear in decode), and becomes a bottleneck at long context ([91] (Xu et al. 2026)). For example, at 10M tokens, 98% of the prefill time of a sparse attention layer is spent on indexing (Figure 2). The cost of sparse attention at ultra-long context is therefore dominated by the cost of finding the keys.
Figure 2. Traditional sparse attention at ultra-long context. One global attention layer of GLM-5.3-FLASH (512k to 10M tokens) on one H200, prefilling the last chunk of 32,768 queries and decoding one token. Speedup of sparse attention over full attention (blue), share of the indexer (%), and potential speedup over sparse attention if the indexer were ‘free’ (green).
To address this, we introduce Latent Sparse Attention (LSA), which clusters keys in a learned latent space and lets each query search the clusters instead of scoring every key. This can be done using only one level of clustering (Flat), or with a hierarchy of clusters (Tree), which makes the prefill subquadratic. Furthermore, we propose two trainable designs: D-LSA, which adds LSA to pre-trained models and only trains small projection matrices per layer (contribution #1), and N-LSA, which trains LSA natively with the Next-Token Prediction (NTP) loss alone (contribution #2). We evaluate state-of-the-art models (QWEN-3.8-27B and GLM-5.3-FLASH) beyond their advertised context, up to 10M tokens (contribution #3), on RULER ([26]), which is an established long-context benchmark, and PaperTrail, which we developed to test long-context scientific understanding by finding and connecting facts across unseen scientific papers (contribution #4). Finally, we integrate LSA into SGLang ([96])2 with context parallelism, provide GPU kernels for H200 and B300, and release the code3 with all training and inference details (contribution #5). Figure 1 outlines our contributions.
Figure 1. D-LSA and N-LSA overview. Top: one attention layer. The keys are pooled and clustered into buckets. A query (🔴) scores the bucket centroids (🔺) and attends only to the selected long-range keys (plus sink and window). Bottom: accuracy and performance (P: prefill, D: decode, T: training) for both D-LSA and N-LSA. D-LSA is shown with flat search. Its subquadratic variant, D-LSA (Tree), follows a similar trend (Figure 5).
Sparse Attention
In full attention, every query attends to every preceding key. Prefilling a context of tokens therefore costs , and decoding one token costs . In sparse attention, the work is split between an indexer and the attention. For each query, the indexer selects which keys the attention attends to4. With a fixed budget of keys, the attention then costs to prefill and per decoded token. If the indexer selects the most relevant keys, this is sufficient to stay close to the accuracy of full attention (Appendix § 9).
However, the indexer has a cost. Since it must score all preceding keys, it typically remains in prefill and in decode, though with reduced constant factors. This makes ultra-long context still prohibitively expensive. For example, at 10M tokens the indexer takes 98% of the prefill time of traditional sparse attention (Figure 2), and a ‘free’ indexer would make sparse attention a further 58× faster in prefill and 8.9× in decode.
Design Choices
Sparse attention methods can be categorised by their indexer designs and how they are trained.
Token Sparse Attention. For each query, a lightweight indexer (fewer heads, smaller dimension, lower precision) scores every preceding key and selects which to attend to.
Block Sparse Attention. Consecutive keys are grouped into blocks, and each query scores blocks instead of every key. All keys of the selected blocks are attended to.
Semantic Sparse Attention. Keys are clustered by similarity rather than position, and each query scores one centroid per cluster. All keys of the selected clusters are attended to.
For the last two categories, a query can search the groups in three ways. Flat search scores every group directly, which is work per query. Fixed search adds a shallow hierarchy and refines from coarse to fine groups, which is also per query. Tree search descends a deeper hierarchy and expands only the groups it keeps at each level, which is per query.
An indexer can be training-free, distilled from a pre-trained model ([71]), or trained natively on the NTP loss alone.
Table 1. Sparse attention methods compared. Rows are grouped by the unit the indexer scores: single keys, blocks of consecutive keys, or clusters of similar keys. Design column: compression of the sequence the indexer scans by compressing consecutive keys (C. Seq.), or clustering keys by content (C. Sem.), and its search: flat (●●●), a fixed number of levels (♟), or levels efficiently growing as (♟♟). Evaluation: the largest context at which accuracy (Acc.) and performance (Perf.) are reported. ✓ supported, ◐ partial support, × not supported.
Related Work
Table 1 provides an overview of the existing work on sparse attention, as categorised above. We focus on methods that keep the whole KV cache and select keys per query, unlike fixed sparse patterns (Child et al., [9]; Beltagy et al., [5]; Zaheer et al., [89]) and KV cache eviction techniques (Zhang et al., [94]; Li et al., 2024; Xiao et al., [83]).
| Name | Training Free | Training Distilled | Training Native | Speedup Prefill | Speedup Decode | Design C. Seq. | Design C. Sem. | Design Search | Evaluation Acc. | Evaluation Perf. |
| DSA (DeepSeek-AI, [14]) | × | ✓ | × | ✓ | ✓ | × | × | ●●● | 128k | 128k |
| MiniMax SA (Lai et al., [39]) | × | ✓ | × | ✓ | ✓ | × | × | ●●● | 128k | 1M |
| Block Sparse Attention | ||||||||||
| CSA (DeepSeek-AI, [15]) | × | ✓ | × | ✓ | ✓ | ✓ | × | ●●● | 1M | 1M |
| QSA (Qwen Team, [64]) | × | ✓ | × | ✓ | ✓ | ✓ | × | ●●● | 1M | 1M |
| GLM-5.3-FLASH SA (Z.ai, [90]) | × | × | ✓ | ✓ | ✓ | ✓ | × | ●●● | 1M | 1M |
| NSA (Yuan et al., [87]) | × | × | ✓ | ✓ | ✓ | ✓ | × | ●●● | 64k | 64k |
| HSA-UltraLong (Hu et al., 2026b) | × | × | ✓ | ✓ | ✓ | ✓ | × | ●●● | 16M | 256k |
| MoBA (Lu et al., 2025) | × | × | ✓ | ✓ | ✓ | ✓ | × | ●●● | 1M | 10M |
| SeerAttention (Gao et al., [19, 20]) | × | ✓ | × | ✓ | ✓ | × | × | ♟ | 128k | 128k |
| LongCat LSA (Zan et al., [91]) | ✓ | × | × | ✓ | ✓ | ✓ | × | ♟ | 1M | 1M |
| HISA (Xu et al., 2026) | ✓ | × | × | ✓ | ✓ | ✓ | × | ♟ | 128k | 128k |
| HiP (Lee et al., [40]) | ✓ | × | × | ✓ | ✓ | × | ✓ | ♟ | 128k | 512k |
| PISA (Tang et al., [74]) | × | × | ✓ | ✓ | ✓ | × | × | ♟ | 16k | 256k |
| Semantic Sparse Attention | ||||||||||
| ClusterKV (Liu et al., [48]) | ✓ | × | × | × | ✓ | × | ✓ | ●●● | 32k | 32k |
| RetroInfer (Chen et al., [8]) | ✓ | × | × | × | ✓ | × | ✓ | ●●● | 1M | 1M |
| Squeezed Attention (Hooper et al., [25]) | ✓ | × | × | ◐ | ✓ | × | ✓ | ♟ | 32k | 512k |
| Multipole Attention (Hooper et al., [24]) | ✓ | × | × | × | ✓ | × | ✓ | ♟ | 128k | 128k |
| LycheeCluster (Li et al., [43]) | ✓ | × | × | × | ✓ | ✓ | ✓ | ♟ | 128k | 72k |
| IceCache (Mao et al., [54]) | ✓ | × | × | × | ✓ | ✓ | ✓ | ♟ | 250k | 300k |
| LSA | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ♟ | 10M | 10M |
Table 1.
Among distillation methods, DSA (DeepSeek-AI, [14]) and MiniMax SA (Lai et al., [39]) first distil the indexer and then continue training the full model under sparse attention. Alternatively, SeerAttention (Gao et al., [19, 20]) distils a gate over pooled blocks and HashAttention (Desai et al., [17]) learns binary codes of queries and keys, both with the model frozen. As the context grows, the indexer itself becomes the bottleneck (Xu et al., 2026; Zan et al., [91]; Bai et al., 2026). To reduce its cost, CSA (DeepSeek-AI, [15]), QSA (Qwen Team, [64]), and GLM-5.3-FLASH SA (Z.ai, [90]) compress every four keys into one before scoring them. HISA (Xu et al., 2026) and LongCat LSA (Zan et al., [91]) score blocks first and then only the keys inside the selected blocks. IndexCache (Bai et al., 2026), LongCat LSA and DeepSeek-V4.1 (DeepSeek-AI, [16]) reuse the selection of one layer in the following layers. Each improvement divides the cost of the indexer by a constant, and its work per query therefore still grows linearly with the context.
NSA (Yuan et al., [87]) and MoBA (Lu et al., 2025) instead pre-train the model with sparse attention from scratch, such that the indexer learns from the NTP loss alone. NSA scores each block by a learned compressed key, and MoBA by the mean of its keys. InfLLM-V2 (Zhao et al., [95]) and SSA (Mao et al., [53]) bring such block selection to existing dense models by fine-tuning. HSA (Hu et al., 2025; Leng et al., [42]; Hu et al., 2026a) and HSA-UltraLong (Hu et al., 2026b) also weigh the output of each block by its score, such that the loss trains the scores directly. All of these methods still scan every block. PISA (Tang et al., [74]) replaces this scan with a tree, but evaluates accuracy only at small scale (up to 16k tokens).
At the other end, Quest (Tang et al., [75]), InfLLM (Xiao et al., [82]) and MInference (Jiang et al., [32]) select blocks based on simple summaries of their keys without any training. HiP (Lee et al., [40]) searches the blocks with a tree, and its successor InfiniteHiP (Lee et al., [41]) with three fixed stages. BLASST (Yuan et al., [88]) instead skips blocks directly inside the attention kernel, without a dedicated indexer. Most of these methods summarise blocks of consecutive keys, which may hide a single relevant key (Xiao et al., [82]; Hu et al., 2026a).
Closest to LSA are methods that organise the keys by semantic content rather than by position. A few early works train at short context, with Reformer (Kitaev et al., [36]) hashing keys into buckets, Routing Transformer (Roy et al., [68]) learning k-means centroids, and Treeformer (Madaan et al., [52]) routing each query down a learned tree. Most other works here only operate at inference time: ClusterKV (Liu et al., [48]) and RetroInfer (Chen et al., [8]) cluster the keys during prefill and score every centroid at decode (flat search), Squeezed Attention (Hooper et al., [25]), Multipole Attention (Hooper et al., [24]) and LycheeCluster (Li et al., [43]) search two levels (fixed search), and IceCache (Mao et al., [54]) descends a tree of keys (tree search), while RetrievalAttention (Liu et al., [47]) links the keys in a nearest-neighbour graph. All of these inference methods are training-free, only Squeezed Attention speeds up prefill, and only partially, and none are evaluated at ultra-long context (above 1M tokens).
Latent Sparse Attention (LSA)
LSA is a sparse attention layer for ultra-long context. It clusters keys by content, in a space that is learned: through index projections in D-LSA, and through the attention’s own keys in N-LSA. LSA can replace the attention of a pretrained layer by distillation (D-LSA) or be trained natively with the model (N-LSA).
In both settings, the indices for the attention are selected from the keys and queries by the same process. We first describe this process, numbered as in Figure 3, which consists of semantic clustering of the index keys , and the search with the index queries so that each query selects its keys. We then describe the specifics of D-LSA and N-LSA.
Figure 3. D-LSA design. Left: D-LSA (Flat). Indexer keys: ❶ projection and L2 norm, ❷ pooling (4:1), ❸ bottom-up agglomerative clustering (Ward’s method), ❹ bucket cache (centroids and index table). Indexer queries: A sum over query heads, projection and L2 norm, B scoring the cached centroids and selecting the top- buckets per query, C expanding the buckets into indices (sink + top- long-range + window), D attention over those indices (one index list per query shared by all query heads). Right: D-LSA (Tree) groups the buckets into a hierarchy of clusters, a query descends using beam search (close in the latent space but scattered through the KV cache).
Semantic Clustering
❶ Key normalisation. Each index key is first L2 normalised:
❷ Pooling. Consecutive keys are then compressed, as in CSA (DeepSeek-AI, [15]), QSA (Qwen Team, [64]Z.ai, 2026`), to improve computational efficiency. Here, every consecutive index keys are summed into one pool key , so that clustering runs over pool keys rather than keys.
❸ Clustering. Keys are clustered in cluster blocks of consecutive keys. Once a cluster block contains all its keys, its pool keys are clustered bottom up with Ward’s method (Ward, [81]) into buckets (Algorithm 1 in Appendix § 10). Each pool key starts as a bucket of its own, and each bucket keeps the sum of its keys, whose normalisation is its centroid . In every round, all pairs of buckets that are each other’s cheapest merge are merged in parallel. The cost of a merge is how much it increases the keys’ total squared
distance from their centroid (Ward’s criterion). Merging stops once there are buckets, where is the average number of keys per bucket, such that every cluster block holds the same number of buckets. A merge is skipped if the merged bucket would hold more than keys.
❹ Bucket cache. The bucket cache stores the centroid of every bucket and an index table of the positions of its keys. This is so later selected buckets can ‘expand’ into their key indices. With one centroid per keys, it corresponds to a small fraction of the KV cache.
Once a cluster block is clustered, its buckets never change. In the Tree variant, this clustering process is repeated to group the buckets into multiple levels of a hierarchy.
Semantic Search
We now describe how the top- key indices are extracted for each query, and then submitted to attention. We first describe the process for Flat, in which there is only one level of buckets, and then discuss Tree.
A Query normalisation. Each index query is first L2 normalised:
B Scoring. The index query scores the unit-norm centroid of every bucket in cluster blocks before its own via , which compares buckets of any size on one scale, and selects the top- buckets. Its own cluster block is not clustered yet, and its keys are covered by the window instead.
C Expansion. The selected buckets are expanded into their keys, best first, until ‘long-range’ keys are selected. As every bucket holds at least keys, is chosen such that , and so expansion always fills keys.
Besides the long-range keys, every query attends to sink keys [83] and the most recent keys (window). A key of a selected bucket that falls in the sink or the window is already attended to, and the expansion moves on to the next key instead. We set the window to one cluster block, , so that the window covers every key not yet clustered. This gives each query a set of at most key indices.
D Attention. Attention is then computed only over the keys indexed by . A single is shared across all query heads.
Tree search arranges the buckets into a hierarchy (Figure 3, right). Level 0 holds the buckets, and each higher level holds buckets we term ‘node buckets’, each made of buckets of the level below, with a centroid over all the keys it contains. As at level 0, buckets are merged one cluster block at a time. Buckets that are not merged yet are defined as open. Once a level holds open buckets, they are merged into node buckets as in Algorithm 1.
Beam search is used to select the buckets through the hierarchy (Algorithm 2 in Appendix § 10). On each level, the query is scored against the centroids of the candidate node buckets, and the best are kept. The candidates on the level below are the children of the kept node buckets and the open buckets of that level. At level 0, the top- buckets are selected, as in the flat search. With a constant beam per level, a query scores centroids, which makes the prefill of the indexer subquadratic, , where the flat search is quadratic.
D-LSA
D-LSA (Figure 3) follows the process outlined above, but since it distils from a pre-trained model which already has key and query projections ( and ), we additionally learn separate key and query projections for LSA. In particular, each key and query is projected to a smaller dimension to keep the scoring, clustering, and bucket cache cheap:
The query heads are summed such that one index query serves every head, which yields the single index list per query. On QWEN-3.8-27B, whose keys and queries carry RoPE (Su et al., 2024) on a quarter of their dimensions, the projections read only the RoPE-free part. These keys and queries then go into the normalisation process as before. Trainable projections can also be replaced by the leading principal directions of the keys in the layer, which project both keys and queries into one index space with no training. We call this variant D-LSA (Training-Free).
The base model’s attention is otherwise unchanged: its own queries attend only to the keys and values indexed by .
Distillation. D-LSA adds LSA to a pre-trained model by a short distillation that trains only and a temperature per layer, with the base model frozen. This mirrors the dense warm-up of DSA [14], which also trains only the indexer against the head-aggregated attention of the frozen model. The distillation scores every key directly, without clustering. For a query of a layer, let be the keys it can select (all preceding keys outside the sink and the window). The target is the attention of that frozen layer over , the softmax of each head’s logits over averaged over its heads of dimension , and is the softmax of the D-LSA scores over the same keys. The distillation minimises the forward KL divergence between the two:
The temperature sharpens the softmax of the D-LSA scores to match the peaked attention of the model, and is used only in this loss. Both projections are initialised with these principal directions.
Kernels. We write GPU kernels for the clustering, scoring, selection and expansion of the indexer, in CuTe DSL for H200 and B300, with a Triton version of each, which runs on any GPU. For the attention, we reuse existing attention kernels, such as FlashAttention-3 [72], which read the index list as a page table.
Parallelism. We integrate LSA into SGLang with support for tensor, expert, and pipeline parallelism, and for context parallelism in both prefill and decode. Context parallelism splits the KV cache across ranks by token, but LSA replicates its bucket cache on every rank. Every rank thus builds the same index and selects the same index list. The ranks only exchange the index keys of the newest tokens. This design choice trades memory for communication. At 10M tokens on GLM-5.3-FLASH with context parallelism over 8 GPUs, each rank holds a 1.25 GiB share of the 10 GiB KV cache of a layer and the full 246 MiB bucket cache.
We provide further details on kernels in Appendix § 14.§ 1 and on parallelism in Appendix § 14.§ 2.
N-LSA
N-LSA (Figure 4) trains LSA natively with the model, using only the NTP loss. As the model learns with the selection from the start, we cluster the keys of the attention itself, without dedicated index projections. We use a single key-value head, such that one key per token serves every query head, and score each bucket by the best similarity any query head gives its centroid:
Figure 4. N-LSA design. The attention layer of N-LSA, which can be trained from scratch on the next-token loss alone.
where is the normalised query of head . We stop the gradient through the selection, and the NTP loss trains the attention projections only through the keys each query attends to, the selected keys plus the sink and the window. N-LSA therefore adds no parameters to the model and needs no teacher, no index projections and no auxiliary loss.
The attention layers of N-LSA use no positional encoding, as is becoming more common in hybrid models, and RoPE could be supported by clustering only the RoPE-free part of the keys, as D-LSA does on QWEN-3.8-27B.
Evaluation
In this section, we describe the models, the methods we compare, and their settings. Then, we evaluate the accuracy of D-LSA up to 10M tokens, and its performance per attention layer and end to end. Finally, we show that N-LSA can be trained natively at an accuracy close to full attention.
Setup
Models. D-LSA aims to keep the accuracy of a model while making its attention significantly cheaper. We compare each model with D-LSA against its baseline attention, in accuracy and in performance. We demonstrate D-LSA on both a model with full attention (QWEN-3.8-27B ([65 c])) and one with sparse attention (GLM-5.3-FLASH ([90])). On QWEN-3.8-27B, we also replace the attention with block sparse attention, inspired by QSA ([64]), as a second sparse baseline. We also compare flat to tree search where possible.
Methods. We compare the following methods:
▪ Full Attention: every query attends to every preceding key, the baseline attention of QWEN-3.8-27B.
▪ Block Sparse Attention (Flat): sparse attention whose indexer scores every block of four consecutive keys, as in QSA ([64]) and GLM-5.3-FLASH ([90]).
▪ Block Sparse Attention (Tree): sparse attention whose indexer searches a tree over blocks of the keys of the attention itself, as in HiP ([40]) and PISA ([74]). To represent these methods, we search a tree over blocks of keys directly on the KV cache, without training, which uses the full keys and favours accuracy over performance. We use blocks of eight keys, the most accurate of the sizes we compare (Appendix § 13.§ 7).
▪ D-LSA (Flat) and D-LSA (Tree): D-LSA with a flat search over every bucket centroid and with a tree search over the hierarchy of buckets. In decode, each request processes one token per step, and the tree search runs its levels one after another, which pays off only when many requests decode together. As we evaluate one request at a time, D-LSA (Tree) decodes with the flat search (Appendix § 14.§ 1).
▪ D-LSA (Training-Free): D-LSA (Flat) with the principal directions of the keys as both projections instead of trained ones.
Every trained indexer we compare is distilled in the same way to ensure fairness.
Design parameters. Table 2 lists the parameters of every method per model. For D-LSA, we found that a budget of long-range keys holds accuracy past 1M tokens better than smaller budgets, and these extra keys cost little next to the scan of the indexer they replace. On GLM-5.3-FLASH, D-LSA (Flat) selects long-range keys for queries before position 1M and beyond. We found that the smaller budget keeps accuracy below 1M tokens while making the attention faster. Tuning the budget further, for example per layer or per request, is left to future work. A small index space of dimensions keeps the scoring and the clustering cheap, and we found wider spaces to add little accuracy.
| QWEN-3.8-27B | GLM-5.3-FLASH | |||||||
| Unit | block | block | bucket | bucket | block | block | bucket | bucket |
| Search | ••• | ♣ | ••• | ♣ | ••• | ♣ | ••• | ♣ |
| 2,048 | 4,608 | 4,096 | 4,096 | 2,048 | 4,608 | 2,048/4,096 | 4,096 | |
| 512 | 576 | 1,024 | 1,024 | 512 | 576 | 512/1,024 | 1,024 | |
| -, - | 8, 16 | 4, 512 | 4, 512 | -, - | 8, 16 | 4, 512 | 4, 512 | |
| 128 | 256 | 64 | 64 | 128 | 512 | 64 | 64 | |
| 4 | 8 | 4 | 4 | 4 | 8 | 4 | 4 | |
| - | - | 512 | 512 | - | - | 512 | 512 | |
| - | - | 8, 16 | 8, 16 | - | - | 8, 16 | 8, 16 | |
| - | 2 | - | 4 | - | 2 | - | 4 | |
| - | - | - | 4,096 | - | - | - | 4,096 | |
| - | 576 | - | 2,048 | - | 576 | - | 2,048 |
Table 2. Parameters. Block Sparse Attention (Flat), Block Sparse Attention (Tree), D-LSA (Flat) and D-LSA (Tree) per model (symbols in Table 5). The search is flat (•••) or tree (♣), over blocks of consecutive keys or over buckets. Block Sparse Attention (Tree) always keeps the first, previous and current blocks among its blocks. On GLM-5.3-FLASH, D-LSA (Flat) uses the first and below 1M tokens and the second beyond.
Indexer distillation. We train on 352M tokens of documents of 16k to 1M tokens from seven domains, for 48k steps, each on 128 queries sampled from one document, with a learning rate of . This is a fraction of the 2.1B tokens of the dense warm-up of DSA. In Equation (2), the target keeps only the 2,048 keys with the most attention weight per query, renormalised over them, while still spans all of , and is calibrated per layer before the first step. Unlike DSA, we do not follow the distillation by training the whole model under sparse attention (sparse training), a step D-LSA also supports but we omit here due to its cost. We provide further details about the composition and construction of the dataset in Appendix § 11.
Benchmarks. We evaluate accuracy in three categories, each with a representative set of benchmarks. For knowledge, we use MMLU-Pro [80] (a stratified sample of 2,000 questions) and GPQA-Diamond [67]. For reasoning, we use AIME25 (four repeats) and MATH500 [46] (100 problems). For long context, we use LongBench v2 [3] (60 questions) and the 13 tasks of RULER [26] from 4k to 10M tokens (100 prompts per task up to 1M and 20 beyond). Evaluating each benchmark in full at ultra-long context is prohibitively expensive, and we therefore use the subsets given in parentheses.
PaperTrail. To better understand model capabilities in a less synthetic ultra-long context setting, we extend RULER with a set of new tasks called PaperTrail. The tasks are constructed by embedding a trail of fabricated facts in a large collection of scientific papers. All papers are recent, almost all published after the release date of the models, such that no model could have seen them in training. A question can only be answered by finding the facts of its trail and connecting them, as in a literature search. Samples are built in three domains: exoplanets, long-context machine learning and quantum error correction. The questions come in three tiers of increasing difficulty, where tier 1 recalls a fact from one paper, tier 2 links facts across two papers and tier 3 traces a trail across three or more. We provide further details about the construction of PaperTrail in Appendix § 12.
Model sampling. We configure each model’s sampling as recommended by its model card with generation up to 163,840 tokens (Appendix E). QWEN-3.8-27B reaches 10M tokens with YaRN [63] at factor 40 (42 for PaperTrail at 10M), while GLM-5.3-FLASH needs no extension as it has no explicit positional encoding.
Hardware and serving. We serve all models in SGLang [96] on H200 and B300 GPUs. To show that D-LSA can be integrated with the main parallelism strategies, we evaluate it with tensor parallelism (TP8) and with context parallelism in both prefill and decode (CP8). We show more details in Appendix § 14.§ 2. All accuracy measurements use context parallelism wherever the KV cache of a request outgrows the memory of a single GPU.
Accuracy
Figure 5 shows that on knowledge, reasoning and long-context tasks, D-LSA stays close to the accuracy of baseline Full Attention on QWEN-3.8-27B and of baseline sparse attention on GLM-5.3-FLASH. PaperTrail further shows how well the models find and connect facts across large collections of scientific papers.
Figure 5. D-LSA accuracy is on par with full attention and sparse attention baselines on downstream tasks. Shown on knowledge, reasoning and long-context tasks up to 10M tokens. Baseline scores are given with their 95% confidence interval.
Knowledge and reasoning. On knowledge, reasoning and LongBench v2, D-LSA stays within the 95% confidence interval of both baselines, even with reasoning growing to tens of thousands of tokens (Appendix E). We further show that D-LSA does not measurably affect the length of the reasoning.
Long context. On QWEN-3.8-27B, D-LSA (Flat) stays within 5 points of baseline Full Attention up to 1M tokens and exceeds it at 10M tokens, at 63.9 against 59.9 on RULER and 29.1 against 19.8 on PaperTrail. It also stays above Block Sparse Attention (Flat) on RULER from 128k tokens on, in a fair comparison, as both indexers are distilled the same way.
On GLM-5.3-FLASH, D-LSA (Flat) stays within 3 points of baseline sparse attention up to 1M tokens, trails it by up to 8 points at 2M to 5M tokens, and is within 4 points on RULER at 10M tokens. This comparison favours the baseline, which was trained together with
D-LSA (Flat) is distilled from attention that is already sparse (with no finetuning of the base model itself). On both models, MultiKey 2, MultiKey 3 and CWE fall the most for every method, the baselines included. We break down RULER by task and PaperTrail by tier in Appendix § 13.
Design comparison. D-LSA (Tree) follows D-LSA (Flat) on QWEN-3.8-27B, but loses accuracy past 1M tokens on GLM-5.3-FLASH, at 55.8 against 66.8 on RULER at 10M tokens. Block Sparse Attention (Tree) falls behind from 512k tokens, to about 63 on RULER at 1M tokens on both models, against 78 and 88 for D-LSA (Tree), and we therefore do not evaluate it beyond 1M tokens. Large blocks fall earlier (Appendix § 13.§ 7). D-LSA (Training-Free), without any training, stays close to D-LSA (Flat) on QWEN-3.8-27B, but trails it on GLM-5.3-FLASH.
D-LSA follows the accuracy of the baselines, showing that our semantic clustering can preserve accuracy up to 10M tokens. However, we do observe a steep drop in accuracy beyond 512k tokens for Block Sparse Attention (Tree).
Performance
Next, we evaluate the performance of D-LSA. We first measure a single attention layer and then the whole model in production serving.
Methodology. For one layer, we time the whole attention module, including its projections, the indexer and the attention, on a single GPU. In prefill, we time the last chunk of 32,768 queries before the given context. In decode, we time one step of one request with CUDA graph enabled (clustering is amortised across the steps). For the whole model, we serve each method in SGLang with prefill chunks of 32,768 tokens, and measure the time to first token (TTFT) and the time per output token (TPOT) of one request at a time, the median of three requests of 256 output tokens.
Figure 6 shows that the prefill speedup of D-LSA grows with the context. On GLM-5.3-FLASH, it rises from 3.5× at 1M to 22.7× at 10M tokens over baseline sparse attention, and on QWEN-3.8-27B to 72.4× at 10M tokens over baseline Full Attention. Below 1M tokens, the gain is smaller, at 2.1× at 512k tokens, as the scan of every block is still cheap. Extrapolated beyond 10M tokens, the prefill of D-LSA (Tree) grows subquadratically and overtakes the flat search, and at 100M tokens it is predicted to be more than 90× faster than baseline sparse attention.
Figure 6. D-LSA speedup of one global attention layer. Prefill and decode speedup over the baseline attention (log scale), Full Attention for QWEN-3.8-27B and Block Sparse Attention (Flat) for GLM-5.3-FLASH, required memory at 10M with one index key per block of four keys for Block Sparse Attention (Flat), and prefill scaling up to 100M (log-log), extrapolated beyond 10M (shaded).
Decode follows the same trend, with a speedup up to 4.6× over baseline sparse attention and 42.5× over baseline Full Attention at 10M tokens, with D-LSA (Tree) decoding with the flat search. On B300, D-LSA reaches 12.5× and 42.9× in prefill and 5.2× and 18.4× in decode at 10M tokens.
D-LSA also requires less memory for its indexer. Baseline sparse attention caches an index key for every block of four keys, while D-LSA caches one centroid in its smaller index space for every bucket of about eight keys, together with the id of each key. At 10M tokens, the index of D-LSA (Flat) takes 246 MiB per layer against 331 MiB in GLM-5.3-FLASH and 642 MiB in QSA. Larger buckets would shrink the index of D-LSA further.
End-to-end. Table 3 shows that the speedup of one layer carries over to the whole model at ultra-long context. On GLM-5.3-FLASH, D-LSA (Flat) is on par with baseline sparse attention up to 1M tokens with CP8 and already 1.5× faster to first token at 1M with TP8, and at 10M tokens reduces the TTFT by 2.2× and the TPOT by 2.1× with CP8, and by 5.8× and 1.9× with TP8. On QWEN-3.8-27B, it reduces them by 9.2× and 2.9× at 10M tokens against baseline Full Attention. On B300, D-LSA (Flat) reduces the TTFT and the TPOT at 10M tokens by 1.6× and 2.2× with CP8 and by 4.8× and 2.6× with TP8 on GLM-5.3-FLASH, and by 5.7× and 1.5× on QWEN-3.8-27B. The speedup of the whole model is expected to be smaller than that of one layer, as only the global attention layers become faster, while the other layers keep their cost.
| Setting | Method | TTFT (s) 512k | TTFT (s) 1M | TTFT (s) 10M | TPOT (ms) 512k | TPOT (ms) 1M | TPOT (ms) 10M |
| H200; QWEN-3.8-27B; TP8 DCP8 | Full Attention | 21.7 | 65.3 | 4582.2 | 7.8 | 9.1 | 32.1 |
| H200; QWEN-3.8-27B; TP8 DCP8 | Block Sparse Attention (Flat) | 1.61× | 2.30× | 8.02× | 0.89× | 0.91× | 0.89× |
| H200; QWEN-3.8-27B; TP8 DCP8 | D-LSA (Flat) | 1.03× | 1.51× | 9.21× | 0.87× | 1.00× | 2.87× |
| H200; QWEN-3.8-27B; TP8 DCP8 | D-LSA (Tree) | 0.96× | 1.41× | 8.83× | 0.87× | 1.00× | 2.87× |
| H200; GLM-5.3-FLASH; CP8 DCP8 | Full Attention | 8.5 | 18.1 | 451.5 | 7.6 | 8.2 | 17.9 |
| H200; GLM-5.3-FLASH; CP8 DCP8 | Block Sparse Attention (Flat) | 1.03× | 1.09× | 2.24× | 1.01× | 1.06× | 2.10× |
| H200; GLM-5.3-FLASH; CP8 DCP8 | D-LSA (Flat) | 0.93× | 0.98× | 2.14× | 0.97× | 1.03× | 2.10× |
| H200; GLM-5.3-FLASH; TP8 | Full Attention | 17.9 | 45.1 | 1300.9 | 5.8 | 6.3 | 18.9 |
| H200; GLM-5.3-FLASH; TP8 | Block Sparse Attention (Flat) | 1.22× | 1.51× | 5.83× | 0.99× | 1.06× | 1.91× |
| H200; GLM-5.3-FLASH; TP8 | D-LSA (Flat) | 0.98× | 1.21× | 5.62× | 0.95× | 1.03× | 1.89× |
| B300; QWEN-3.8-27B; TP8 | Full Attention | 10.6 | 31.6 | 2172.2 | 4.9 | 6.0 | 27.3 |
| B300; QWEN-3.8-27B; TP8 | Block Sparse Attention (Flat) | 0.79× | 0.82× | 0.84× | 0.82× | 0.90× | 1.33× |
| B300; QWEN-3.8-27B; TP8 | D-LSA (Flat) | 0.68× | 0.96× | 5.73× | 0.63× | 0.73× | 1.46× |
| B300; QWEN-3.8-27B; TP8 | D-LSA (Tree) | 0.63× | 0.91× | 5.40× | 0.63× | 0.73× | 1.46× |
| B300; GLM-5.3-FLASH; CP8 DCP8 | Full Attention | 6.5 | 13.5 | 273.1 | 6.9 | 7.4 | 17.3 |
| B300; GLM-5.3-FLASH; CP8 DCP8 | Block Sparse Attention (Flat) | 0.96× | 0.99× | 1.56× | 0.97× | 1.03× | 2.21× |
| B300; GLM-5.3-FLASH; CP8 DCP8 | D-LSA (Flat) | 0.78× | 0.81× | 1.48× | 0.96× | 1.03× | 2.22× |
| B300; GLM-5.3-FLASH; TP8 | Full Attention | 11.1 | 27.2 | 1186.4 | 5.2 | 5.7 | 15.6 |
| B300; GLM-5.3-FLASH; TP8 | Block Sparse Attention (Flat) | 1.15× | 1.39× | 4.76× | 0.96× | 1.03× | 2.56× |
| B300; GLM-5.3-FLASH; TP8 | D-LSA (Flat) | 0.91× | 1.09× | 4.28× | 0.95× | 1.03× | 2.56× |
Table 3. D-LSA speedup in production setting. Time to first token (TTFT) and time per output token (TPOT) of one request under SGLang. Full Attention, Block Sparse Attention (Flat), D-LSA (Flat) and D-LSA (Tree). Each block’s first row gives the baseline’s times and the others their speedup over it, with D-LSA at 10M in bold. QWEN-3.8-27B is served with tensor parallelism (TP8) and decode context parallelism (DCP8) on H200 and with TP8 on B300, GLM-5.3-FLASH with either context parallelism (CP8, with DCP8 and expert parallelism EP8) or TP8. Without DCP, Block Sparse Attention’s prefill indexer runs whole on every TP rank. All models run on a single node, except TP8 for GLM-5.3-FLASH at 10M on H200 (2 nodes with PP2); a B300 node holds it on one.
Native Training
Finally, we evaluate N-LSA, which is trained natively on the NTP loss alone.
Setup. We train a hybrid model of about 0.5B parameters with 24 layers, in which every fourth layer is an attention layer and the others are Gated DeltaNet layers (Yang et al., [86]), a form of linear attention (Schmidhuber, [70]; Schlag et al., [69]). Each attention layer has 8 query heads over a single key-value head of 128 dimensions, without positional encoding. We pre-train with a standard Warmup-Stable-Decay (WSD) learning rate schedule (Hu et al., 2024), on 100B tokens of FineWeb-Edu (Penedo et al., [62]) at a context of 4k tokens in the Warmup and Stable phases, followed by 12B tokens of LongDocs in the Decay phase, at 128k tokens (Appendix § 11). We compare N-LSA with Full Attention, Sliding Window Attention (SWA), and the native forms of block sparse attention, NSA (Yuan et al., [87]) for Block Sparse Attention (Flat) and PISA (Tang et al., [74]) for Block Sparse Attention (Tree), all in the same model architecture and on the same data. Every sparse method attends to about 2,560 keys per query. N-LSA selects long-range keys with flat search, next to a window of 512 keys and 4 sink keys. SWA attends to a window of 2,560 keys and 4 sink keys, NSA attends to 32 selected blocks of 64 keys and a window of 512 keys, and PISA attends to 40 blocks of 64 keys, which always include the first block and the two most recent blocks (window). We train N-LSA with flat search only, and leave native training with tree search to future work.
Figure 7 shows that N-LSA reaches the validation loss of Full Attention throughout the decay phase at 128k token context, at 1.085 against 1.085, while PISA, NSA and SWA trail at 1.100, 1.117 and 1.154. During pre-training at 4k tokens, all methods stay within 0.004 of each other, as their keys cover most of the context. Table 4 confirms this on downstream tasks. On commonsense reasoning ([10, 92]), all models are on par. On recall ([2]), N-LSA stays close to Full Attention and ahead of the other sparse methods. On RULER, N-LSA stays within 3.5 points of Full Attention up to 128k tokens, at 32.2 against 34.7 at 128k tokens, while NSA and PISA fall to 19.2 and 14.6, and SWA to 3.0 once the context outgrows its window.

Figure 7. N-LSA training speedup. Models trained natively on the NTP loss with Full Attention, SWA, NSA, PISA and N-LSA. Left: one attention layer, forward and backward, on one H200. Right: validation loss over the training time of the whole model in the decay phase (all on the same data and number of tokens).
| Commonsense | Recall | RULER | ||||||||||
| ARC-e | ARC-c | HellaSwag | SWDE | FDA | 4k | 8k | 16k | 32k | 64k | 128k | ||
| Full Attention | 50.0 | 30.0 | 45.6 | 67.8 | 79.3 | 53.9 | 46.0 | 42.5 | 38.1 | 35.6 | 34.7 | |
| SWA | 53.9 | 31.4 | 45.2 | 64.6 | 69.6 | 26.7 | 15.6 | 10.1 | 5.9 | 3.0 | 3.0 | |
| NSA | 53.8 | 30.1 | 45.8 | 62.2 | 72.2 | 39.4 | 34.6 | 31.6 | 25.2 | 18.9 | 19.2 | |
| PISA | 52.2 | 28.7 | 45.5 | 63.7 | 78.6 | 56.2 | 48.4 | 40.4 | 26.6 | 19.1 | 14.6 | |
| N-LSA | 54.6 | 30.8 | 45.5 | 66.4 | 79.2 | 52.2 | 42.5 | 41.5 | 38.4 | 32.9 | 32.2 |
Table 4. Native training accuracy. Accuracy (%) of models trained from scratch with Full Attention, SWA, NSA, PISA and N-LSA. Every sparse method attends to about 2,560 keys per query.
N-LSA also trains faster than Full Attention at long context. For one attention layer on a single GPU, the forward and backward pass of N-LSA is 1.1× faster than Full Attention with FlashAttention-3 ([72]) at 64k tokens and 1.6× faster at 128k tokens (Figure 7, left). End to end, the whole model completes its decay phase 1.24× faster at the same validation loss (Figure 7, right), as only every fourth layer is an attention layer. At shorter contexts, the selection of the keys costs more than it saves. Its peak memory is higher, at 6.8 GiB against 5.8 GiB per layer at 128k tokens,
partly because N-LSA keeps the indices of the selected keys of every query for the backward pass, which grow linearly with the context.
Overall, N-LSA trains natively at the accuracy of Full Attention and becomes faster as the context grows.
Discussion
LSA reduces the cost of finding the keys, but not of storing them, and the KV cache still grows linearly with the context. It is orthogonal to methods that compress the KV cache (DeepSeek-AI, [15]), share the index across layers (Bai et al., (2026); DeepSeek-AI [16]), or predict several tokens per step (Gloeckner et al., (2024); DeepSeek-AI [13]), and could be combined with each.
We observed tree search to lose accuracy faster than flat search past 1M tokens on GLM-5.3-FLASH, while it overtakes flat search in decode only from about 5M tokens with one request, and is predicted to do so in prefill only beyond 10M tokens. It remains open whether sparse training would close this gap beyond indexer distillation.
We train N-LSA at a small scale, with about 0.5B parameters, up to 128k tokens and with flat search only. Scaling native training to larger models, longer contexts and tree search remains open.
Conclusion
At ultra-long context, the cost of sparse attention lies in finding which keys to attend to. We introduce LSA, which clusters the keys in a learned latent space and replaces the scan of every key with a search over the clusters, flat or along a tree. D-LSA distils LSA into pre-trained models and stays close to the accuracy of their baseline attention up to 10M tokens. At 10M tokens, one attention layer of D-LSA prefills 22.7× and decodes 4.6× faster than sparse attention in GLM-5.3-FLASH, and 72.4× and 42.5× faster than Full Attention in QWEN-3.8-27B. End to end, this reduces the time to first token by 2.2× and 9.2×, respectively. N-LSA trains LSA natively on the next-token loss alone, at the accuracy of Full Attention and 1.6× faster per layer at 128k tokens. With tree search, the cost of finding the keys grows subquadratically, which makes LSA a step towards attention that scales well beyond 10M tokens. LSA thus provides a practical way forward for ultra-long context attention, both for distilling pre-trained models and for native training.
Acknowledgements
We thank Tri Dao for his valuable feedback and support. We also thank Daniel Woods, Roman Iurkov, Cyril Kondratenko, Manuel Gomez, and the rest of the Nebius team for excellent support with the compute platform.
Attention Sparsity
To test how many keys a query needs to keep accuracy, we let each query of QWEN-3.8-27B attend only to the sink, the window and its true top- keys (Figure 8). At 128k tokens, Top- Oracle Attention keeps 98% of the RULER accuracy of Full Attention down to , while each query attends to 0.8% of the keys. Single-needle retrieval holds down to , while MultiKey 2, MultiKey 3 and CWE fall first in accuracy. Therefore, sparse attention loses little if it finds the most relevant keys, which is what LSA is designed to do without scoring every key.
Figure 8. Top- Oracle Attention on RULER. QWEN-3.8-27B at 128k tokens. Each query attends to a 4-token sink, a 512-token window and the top- long-range keys, one list shared by all heads. Process: sum over all heads, pick top-, renormalise per head over the shared selection.
Method Details
Table 5 lists the symbols of LSA. Algorithm 1 clusters one cluster block into buckets, and Algorithm 2 is the beam search of D-LSA (Tree).
| Symbol | Description |
| context size | |
| , , | query position, key position, head |
| attention projections | |
| , | attention query (head ) and key |
| , | query heads, head dimension |
| , | index projections, index dimension |
| , | index query and key, unit norm |
| long-range keys of query (without sink and window) | |
| , | attention weight of the model / indexer for query and key |
| temperature of the indexer softmax | |
| , | keys summed per pool key, pool key |
| , | keys per cluster block, buckets per block |
| , | average and max number of keys per bucket |
| , | bucket key sum, keys in bucket |
| , | alignment, merge cost |
| , | bucket centroid (unit norm), bucket cache |
| , , | level, buckets per node, buckets per node block |
| , | node buckets kept per level, at most at the start level |
| , | bucket score, buckets selected per query |
| , , | long-range, sink and window keys |
| key positions query attends to |
Table 5. Notation. D-LSA symbols and their description.
Training Data
In this section, we report additional details on the training dataset we use to train D-LSA under indexer distillation, and also train N-LSA in the decay phase: LongDocs.
We use the following terminology:
Source: a collection of documents, such as Project Gutenberg.
Domain: a group of sources: books, academic, code, agentic, government, parliament and discussion.
Document: one item of a source, such as a book, a paper, a code repository or an agent trace.
Sequence: a contiguous part of one document, at a particular sequence length from 16k to 1M tokens.
Sources. LongDocs is a collection of long documents from publicly available sources and in-house agent traces. Eight sources are publicly available: Project Gutenberg, DOAB, arXiv, USGPO, UK Hansard and Wikimedia from the filtered releases of the Common Pile (Kandpal et al., [35]), The Stack (Kocetkov et al., [37]), of which we take the
Algorithm 1 Ward clustering of one cluster block
Require: index keys of a complete cluster block; sizes
Ensure: with centroids and their key positions
Pooling
1: for all pool keys , in parallel do
2: the sum of its consecutive
3: end for
4: start one bucket per pool key, ,
Merging
5: while more than buckets remain do
every pair of buckets, in parallel
6: ▷ Gram matrix
7: ,
8: ▷ merge cost
9: ,
the cheapest mutual pairs , at most as many as buckets above , in parallel
10: , , remove
11: end while
Bucket cache
12: for all buckets , in parallel do
13:
14: store and the positions of the keys of in
15: end for
D-LSA (Tree) only
16: merge blocks of buckets into node buckets
Algorithm 2 Beam search of one query in D-LSA (Tree)
Require: index query , levels of ; sizes
Ensure: the top- buckets for query
1: all node buckets of the lowest level with
Descent
2: while do
every candidate in , in parallel
3: ▷ node score
4: the best candidates of
5: the children of the open nodes of level
6:
7: end while
Selection
every bucket in , in parallel
8: ▷ bucket score
9: return the top- buckets of
Python and Java repositories, and the software engineering traces of Open-SWE-Traces (NVIDIA, 2026e). The in-house agent traces are long sessions of agents that play games or solve long-context tasks, and they make up 13.1% of the training tokens. We use every public source as released.
Sequences. Each sequence comes from one document and never joins two documents. A code document is one repository, its files joined in the order of their paths, and an agent trace is one session rendered as text. We hold out 2% of the documents for validation.
Distillation training. We distil every indexer on a mix of 352M tokens drawn from LongDocs (Figure 9). The mix is uniformly sampled across domain and sequence length, with a bias towards longer context windows at 512k and 1M. In particular, we give each sequence length 32M tokens, and twice and four times as many to 512k and 1M. Sequences of 1M tokens thus hold about a third of the training tokens. We experimentally determine this choice to be beneficial, with +0.036 accuracy on RULER at 1M (averaged over 3 seeds, notable lifts in MULTIKEY_2, MULTIKEY_3, +0.35 and +0.15 respectively), while accuracy from 4k to 512k stays roughly identical. Within each length, every domain receives an equal share of the tokens. A domain with too few documents of that length gives its share to the others (parliament stops at 256k tokens and discussion at 512k).
Figure 9. LongDocs. Tokens of the distillation mix by domain and source (left) and by sequence length (right), with the in-house agent traces hatched.
Native training. The decay training stage of N-LSA reads the whole collection rather than the mix, without the long-context agent sessions. The data is tokenised (18.9B tokens over 134,740 documents), and cut into sequence lengths the training requires.
PaperTrail
As part of this work, we introduce PaperTrail, an extension of the RULER (Hsieh et al., [26]) benchmark that tests models at ultra-long context in a more realistic setting, inspired by real-life scientific literature review (Figure 10). Instead of finding a needle in synthetic text, a model finds facts spread across real scientific papers and connects them, as in a literature search. We use the following terminology:
Figure 10. PaperTrail. ❶ A sample, its fabricated, host and background papers. ❷ One trail, introduced, revised and renamed, and still quoted in its old form after the revision. ❸ Questions by tier, and one whose answer is not in the papers. ❹ Context sizes, each holding every paper of the size below. ❺ Controls.
Sample: a collection of real arXiv preprints of 2026 from one domain.
Trail: a chain of papers that introduces a result, cites it, and later revises and renames it.
Fabricated paper: a paper we write for a trail.
Host paper: a real paper into which we insert passages that cite or use facts from a fabricated paper.
Insertion: a passage we add to a host paper, which only adds sentences and never changes the text of the authors.
Background paper: any other paper of a sample, real and unchanged.
❶ Context. PaperTrail has three samples, on exoplanets, long-context machine learning and quantum error correction, with 1,246 papers in total. Each sample hides two trails (Trail A and Trail B). A trail has two fabricated papers and 8 to 9 insertions in 6 to 7 host papers, and background papers fill the rest of the context. An insertion places its references where an author would place them, such that a host paper reads naturally as before. Papers in the context
are not presented in chronological order, and a host often comes before the paper it cites.
❷ Trails. The first fabricated paper of a trail introduces a result, for example AlloKV, a KV cache eviction method whose budget grows as the 0.43 power of the context length. Host papers cite and use it. Two to three weeks later, the second fabricated paper revises the value and renames the method, here to SuBA with an exponent of 0.61. Host papers posted after the revision either cite the new name or still quote the old value, as real literature often does.
❸ Questions. Each sample asks 30 scored questions, 10 per tier. A tier 1 question recalls a fact from one paper, a tier 2 question links two papers, and a tier 3 question links three or more papers, often the whole trail. Each sample also asks 6 questions whose answer is not in the papers, about an absent entity, a false premise or a near miss, which we report apart from the score (Table 8). The model ends its reply with the papers it used and its answer. A number counts within a relative tolerance of 2% by default, and a set of papers is scored by its F1 against the reference set.
| Exoplanets | Long-context ML | QEC | Average | |
| Not in corpus, QWEN-3.8-27B | ||||
| Full Attention | 100.0 | 100.0 | 100.0 | 100.0 |
| Block Sparse (Flat) | 100.0 | 100.0 | 83.3 | 94.4 |
| Block Sparse (Tree) | 100.0 | 100.0 | 88.9 | 96.3 |
| D-LSA (Flat) | 100.0 | 100.0 | 88.9 | 96.3 |
| D-LSA (Tree) | 100.0 | 94.4 | 94.4 | 96.3 |
| D-LSA (Training-Free) | 100.0 | 94.4 | 100.0 | 98.1 |
| Not in corpus, GLM-5.3-FLASH | ||||
| Block Sparse (Flat) | 100.0 | 88.9 | 100.0 | 96.3 |
| Block Sparse (Tree) | 100.0 | 88.9 | 94.4 | 94.4 |
| D-LSA (Flat) | 100.0 | 100.0 | 100.0 | 100.0 |
| D-LSA (Tree) | 100.0 | 88.9 | 100.0 | 96.3 |
| D-LSA (Training-Free) | 100.0 | 94.4 | 100.0 | 98.1 |
| Controls, GLM-5.3-FLASH | ||||
| Question alone | 0.0 | 0.0 | 0.0 | 0.0 |
| Hosts without insertions | 0.0 | 0.0 | 0.2 | 0.1 |
| Fabricated + insertions | 100.0 | 100.0 | 100.0 | 100.0 |
| Background alone | 0.0 | 0.0 | 0.0 | 0.0 |
Table 8. PaperTrail not-in-corpus questions and controls. Score (%) on the questions whose answer is not in the papers at 512k tokens, for Full Attention, Block Sparse Attention (Flat) and (Tree), and D-LSA (Flat), (Tree) and (Training-Free). The controls ask the scored questions over parts of a sample, with GLM-5.3-FLASH and D-LSA (Training-Free).
❹ Context sizes. We build each sample at 512k, 1M, 2M, 5M and 10M tokens, and every size holds every paper of the size below. The fabricated and host papers, at most 18 per sample, are the same at every size. The background papers grow the context from 25 to 30 papers at 512k to 386 to 465 papers at 10M, and spread the trails over the whole context. The trails are fabricated, and every paper up to 5M was posted after the release of both QWEN-3.8-27B and GLM-5.3-FLASH, guaranteeing that no model has seen these papers in training. The 10M contexts hold papers posted at most six weeks before its release.
❺ Controls. Four controls ask the same questions over parts of a sample: the question alone, the host papers without their insertions, the fabricated papers with the insertions alone, and the background papers alone. By construction, only the fabricated papers and the insertions hold the answers. We validate the controls with GLM-5.3-FLASH and D-LSA (Training-Free) in Table 8.
Accuracy
In this section, we show details of the evaluation setup, compare every method with its baseline on knowledge and reasoning, examine whether D-LSA changes the length of the reasoning, and break down the long-context results, RULER by task and PaperTrail by tier.
Evaluation Setup
Table 6 lists the setup of every benchmark. We sample QWEN-3.8-27B at temperature 1.0 with top-p 0.95, top-k 20, min-p 0 and no presence penalty, and GLM-5.3-FLASH at temperature 1.0 with top-p 0.95, as their model cards recommend. On RULER, both models decode greedily with reasoning disabled. On every benchmark with reasoning, at most 4.6% of the replies reach the output cap of 163,840 tokens for any model and method.
| Questions | Repeats | Reasoning | Max output | Metric | |
| MMLU-Pro | 2,000 | 1 | on | 163,840 | Exact match |
| GPQA-Diamond | 198 | 1 | on | 163,840 | Exact match |
| AIME25 | 30 | 4 | on | 163,840 | Math-Verify |
| MATH500 | 100 | 1 | on | 163,840 | Math-Verify |
| LongBench v2 | 60 | 1 | on | 163,840 | Accuracy |
| RULER | 100 (1M) / 20 (1M) | 1 | off | 4,096 | String match |
| PaperTrail | 90 | 3 | on | 163,840 | F1 |
Table 6. Evaluation setup. Questions, repeats, generation and metric of every benchmark, with RULER questions per task.
Knowledge and Reasoning
Table 7 compares every method with the baselines on knowledge, reasoning and LongBench v2. Every score lies within the 95% confidence interval of the respective baselines.
| MMLU-Pro | GPQA-D | AIME25 | MATH500 | LongBench v2 | |
| QWEN-3.8-27B | |||||
| Full Attention | 83.2 | 88.4 | 98.3 | 98.0 | 70.0 |
| Block Sparse (Flat) | +0.3 | +0.8 | +1.0 | 0.0 | |
| Block Sparse (Tree) | +0.2 | +2.0 | |||
| D-LSA (Flat) | +1.0 | ||||
| D-LSA (Tree) | +0.3 | +1.7 | +2.0 | ||
| D-LSA (Training-Free) | +1.0 | ||||
| 95% CI | |||||
| GLM-5.3-FLASH | |||||
| Block Sparse (Flat) | 85.4 | 85.9 | 97.5 | 98.0 | 70.0 |
| Block Sparse (Tree) | +0.2 | +2.0 | 0.0 | ||
| D-LSA (Flat) | +0.3 | +1.0 | 0.0 | ||
| D-LSA (Tree) | +0.3 | +0.5 | +1.7 | +1.0 | 0.0 |
| D-LSA (Training-Free) | +0.4 | +2.0 | +1.0 | +1.7 | |
| 95% CI |
Table 7. Accuracy compared to the baseline. Baseline scores are in bold. Other rows show the difference to the baseline in points. Differences smaller than the 95% confidence interval (CI) of the baseline are not significant.
Reasoning Length
Figure 12 compares the reasoning length of each method with the baseline of its model, question by question. For each question, we average the reasoning tokens over its repeats and divide them by those of the baseline on the same question, and each marker is the geometric mean of these ratios over the benchmark. The grey band of a benchmark is the median width of the 95% bootstrap intervals of its methods, and a marker inside it shows no measurable change. Every D-LSA design stays inside its band, except D-LSA (Tree) on GPQA-Diamond with QWEN-3.8-27B, which reasons 11% shorter.

Figure 12. Reasoning tokens against baselines. Reasoning tokens of each method relative to the baseline of its model, on the same questions. Grey bands mark the expected variation, and markers inside them show no measurable change in reasoning length.
RULER
Figure 11 shows the 13 tasks of RULER from 4k to 10M tokens, grouped into retrieval, tracing, aggregation and question answering. Single-needle retrieval stays at 95 or above for every method up to 5M tokens, except Block Sparse Attention (Tree), which falls to 88 and 71 at 1M tokens. MultiKey 2, MultiKey 3 and CWE fall to at most 5 at 10M tokens for every method, including the baselines, and bound the RULER average of every method. On QWEN-3.8-27B, D-LSA (Flat) exceeds baseline Full Attention at 10M tokens mostly on MultiQuery and HotpotQA, at 92 and 35 against 72 and 20. On GLM-5.3-FLASH, D-LSA (Tree) falls to 60, 56 and 55 on MultiKey 1, MultiValue and VT at 10M tokens, against 100 for baseline sparse attention, and these three tasks account for most of its gap to D-LSA (Flat).
Figure 11. D-LSA accuracy on RULER individual tasks. The 13 tasks of RULER from 4k to 10M tokens on QWEN-3.8-27B and GLM-5.3-FLASH, by task group, with their average.
PaperTrail
Figure 13 shows the three tiers of PaperTrail and their average from 512k to 10M tokens. The harder the tier, the earlier it falls. Tier 3 questions fall to at most 19.0 at 10M tokens for every method on both models, while tier 1 questions stay above 90 on GLM-5.3-FLASH. On QWEN-3.8-27B, D-LSA exceeds baseline Full Attention from 5M tokens on tier 1 and tier 2 questions, and D-LSA (Flat) reaches 53.3 and 32.2 against 32.2 and 22.2 at 10M tokens. On GLM-5.3-FLASH, D-LSA trails baseline sparse attention mostly
Figure 13. D-LSA accuracy on PaperTrail per tier. Tier 1, tier 2 and tier 3 questions, each averaged over the three domains and three repeats.
on tier 3 questions from 2M tokens, and D-LSA (Flat) also on tier 2 questions at 10M tokens, at 56.7 against 76.7.
Table 8 further reports the questions whose answer is not in the papers, at 512k tokens. Every D-LSA design recognises these questions within 3.7 points of the baseline of its model, at 96.3 to 100.0 against 100.0 for QWEN-3.8-27B and 96.3 for GLM-5.3-FLASH.
Block Size
Figure 14 compares Block Sparse Attention (Tree) with blocks of 8, 16 and 64 keys, all selecting the same number of keys. At short context, every block size matches the baseline. As the context grows, accuracy falls earlier and further the larger the block, on both benchmarks and both models, as a larger block summarises more keys in one score and can hide a single relevant key. Even with blocks of eight keys, the size we use in the main results, the tree baseline trails D-LSA (Flat) at long context.
Figure 14. Block size of Block Sparse Attention (Tree). RULER and PaperTrail up to 1M tokens with blocks of 8, 16 and 64 keys (), each selecting the same 4,608 keys, against the baseline of each model and D-LSA (Flat).
Performance
We first introduce the hardware, describe the kernels of the indexer (Appendix § 14.§ 1), and how D-LSA integrates with different parallelism strategies (Appendix § 14.§ 2): Tensor Parallelism (TP) and Context Parallelism (CP).
Hardware. We run on NVIDIA H200 (Hopper) and B300 (Blackwell Ultra) GPUs, eight per node (NVLink domain), with the details presented in Table 9.
| H200 | B300 | |
| Architecture | Hopper (sm_90) | Blackwell Ultra (sm_103) |
| SMs | 132 | 148 |
| Shared memory per SM | 228 KiB | 228 KiB |
| Matrix instructions | WGMMA | tcgen05 |
| Memory (HBM3e) | 141 GB | 288 GB |
| HBM bandwidth | 4.8 TB/s | 8 TB/s |
| NVLink per GPU | 900 GB/s | 1.8 TB/s |
| bf16 tensor | 989 Tflop/s | 2,250 Tflop/s |
| fp32 | 67 Tflop/s | 75 Tflop/s |
| Ridge point (bf16) | 206 flop/B | 281 flop/B |
Table 9. Hardware specs. Specifications of the GPUs we use, as given by NVIDIA [58 d;a;b].
Precision. The KV cache, the index keys and the bucket centroids are in bf16. For the bucket centroids, which every decode step of the flat search reads in full, bf16 halves the bytes of fp32 at little cost to the selection. In an offline study on GLM-5.3-FLASH, the bf16 indexer selects roughly 20 of
the 4,096 long-range keys of the flat search differently from an fp32 reference, and changes the attention mass that the selected keys cover by less than 0.0001. Bucket centroids in fp8 would halve their bytes again, but would select about six times as many keys differently, so we keep them in bf16. We do not observe similar sensitivity for the node centroids, which we set to fp8 in D-LSA (Tree).
Conventions. We run prefill over chunks of consecutive queries, and decode over steps of one new query per request. All sizes are per layer, and per chunk in prefill or per step in decode. We run D-LSA with index keys and queries of dimensions, cluster blocks of keys, buckets of keys on average and at least . Each query selects buckets, so that even buckets of keys should hold its long-range keys, and the expansion fills from about the best half of them on average. It also attends to sink keys and a window of keys. On GLM-5.3-FLASH, D-LSA (Flat) uses and for the queries in the first 1M positions of the context (Table 2), but for queries of a chunk at 10M tokens, the sizes above apply. Therefore, at a context of M tokens, a layer holds buckets, whose centroids take MiB.
Kernels
We refer to the steps of D-LSA by their badges in Figure 3, and Figure 15 shows how we implement each of them. We write kernels for three steps of the indexer: ❸ clustering, B scoring and selection of the buckets, and C expansion of the selected buckets into the index lists . These kernels are written in CuTe DSL, the Python-based domain-specific language of CUTLASS [76], to target H200 and B300 specifically. We also keep a Triton [77] version of each kernel, which is not tied to these two GPUs. We give the shapes and times of the kernels on H200, and note where B300 differs. For D attention, we rely on pre-existing attention kernels, that attend over the index lists unchanged. The only exception is CP (Appendix § 14.§ 2), for which we do add an attention kernel.

Figure 15. Implementation of the steps of D-LSA. Badges as in Figure 3.
Clustering
The clustering does relatively little work: one Gram matrix per cluster block, over its pooled keys, takes about 0.13 Gflop for a whole chunk, against 5.5 Tflop for the bucket scan at 10M tokens. Ward’s method, however, merges buckets in rounds that run one after the other. Each of these rounds scores every pair of buckets and merges the cheapest mutual pairs (Algorithm 1, lines 5-11), until buckets remain. The overall runtime of the clustering step is therefore set by the latency of these rounds. Our kernel cuts the time of each round by keeping its data on chip and avoiding round trips to the host between rounds. More precisely, the kernel keeps the state of a cluster block in shared memory for all its rounds, and spreads its search for partners over a thread-block cluster of up to eight streaming multiprocessors (SMs), which exchange their results through distributed shared memory. In D-LSA (Tree), each pairing that forms the node buckets (line 16) runs all its rounds in a single launch, whose thread blocks synchronise through counters in high-bandwidth memory (HBM), so the host neither launches nor reads anything between rounds.
B C Selection by threshold. D-LSA scores an index query against the centroids of the buckets of the cluster blocks before its own, and keeps its top- buckets. Over the index queries of a chunk, the scores form a matrix with one row per query, so we call the scores of query its row, in decode as well. The row holds scores, which grows with the position of the query, to about at the end of a context of tokens. In prefill and in decode alike, many thread blocks compute the scores of a row, each over its own span of the row, so no thread block sees the whole row, and the selection cannot take its top- directly. Instead, inspired by Floyd & Rivest [18] (who select from only the elements between two bounds estimated from a sample), before the scan, we set a threshold for each query, and a thread block compares each of its scores with . Then, it appends only the buckets whose score is at or above it to the list of the query: one comparison per score, without seeing the rest of the row. Only the buckets of , a small multiple of them (see below), are then
ranked.
We also evaluate an alternative inspired by the k-selection of FAISS (Johnson et al., [34]), which keeps the top- buckets of each span and merges them. With this design, every thread block has to rank its span, rather than compare each score once. Furthermore, the merge receives buckets from every span, rather than about in all. Overall, at 10M tokens in prefill, a row is split into 320 spans, which list 327,680 buckets, against about 1,536 in .
We aim for about buckets of the row to score at or above , with : enough to hold the top- with room for error, and few enough to keep short. A first pass, the threshold estimate, sets from a sample of buckets of the row (Figure 16):

Figure 16. Threshold estimate. For one query . Top: the sampled scores, with at or just below the -th best. Bottom: the row, of which about buckets score at or above and are listed, the top- among them.
The sample takes about one bucket in every of the row, so its -th best score ranks about -th in the row, and is placed at or just below it. This estimate is noisy: for independent samples, the count of buckets above would typically differ from by about . However, since neighbouring buckets are correlated, this is only a heuristic. Prefill fixes , so that its sample grows with the row, and decode fixes , and derives for each request.
The noise affects only the cost of the selection. The threshold sets how many buckets are listed, but not which are selected: when holds every bucket with , and at least of them, the top- buckets of the row all reach , and are thus all in . The estimate misses in two ways: comes out too high, and fewer than buckets score at or above it, or too low, and more score at or above it than has room for. leaves room for the noise on both sides, below and the rest of above. A query that the estimate misses anyway falls back to its whole row. In prefill, it scores all its buckets again and takes their top-, the work the threshold avoids, but for one query rather than a whole chunk. In decode, it takes the top- of the scores the scan wrote. The fallback thus costs little as long as few queries take it.
Each bucket of is packed with its score into one 64-bit integer : the bits of , in a form that keeps their order, in its upper 32 bits, and the bucket id, inverted, in its lower 32. A larger is thus a better score, or, at equal scores, a lower bucket id. The C expansion finds, by radix select ([1, 93]) over the , the cut at which the buckets of , taken best first, reach keys or buckets. It writes into the keys of the buckets above , in the order of , then the first keys of the bucket at until holds . If these buckets hold fewer than keys outside the sink and the window, holds fewer. These are the keys that an expansion of the buckets best first would select, in another order. The attention does not depend on the order, so is never sorted. Prefill and decode differ in where the scores go, how is set, and what a fallback costs.
B C Flat prefill. In prefill, the flat search selects the buckets of all the queries of a chunk at once, in one matrix product on tensor cores, whose kernel compares each score with as soon as the score is computed, without writing it to memory. We first show why the scores are not written, then how the product is spread over the GPU, and then the steps of Algorithm 3.
With the index queries of a chunk as the rows of , and the bucket centroids as the rows of , the rows of the chunk form one score matrix
At 10M tokens, computing takes floating-point operations (flop) on at least bytes of bf16 queries and centroids, and writing it takes bytes in fp32:
On an H200, at the peak of its tensor cores, computing would take at least . If every query and centroid were read from HBM only once, the products would have an arithmetic intensity of , two orders of magnitude above the ridge point of the H200. Under this ideal reuse, they can be bound by compute.
Writing , however, is bound by memory, and would take at least . Both are lower bounds from the peaks of the GPU, and the second is more than six times the first at any context, as both grow with . At 9M tokens, alone would already exceed the 141 GB of HBM of an H200. At long context, we therefore never write , and the selection writes only the lists .
As in a matrix multiplication (GEMM), the product is tiled (Figure 17). A thread block holds a tile of index queries in shared memory, and streams the centroids through it in tiles of (128 on B300). It computes one tile of at a time, on tensor cores, and keeps it on chip. Each thread block takes such a tile of queries and a span of up to 64 tiles of centroids. At 10M tokens, a row is thus split into 320 spans, and the chunk into thread blocks, which, on H200 runs 528 at a time, 4 on each of its 132 SMs, which is enough to keep every SM busy. At shorter contexts, the spans shrink, so that there are still enough thread blocks.
Figure 17. B Bucket scan on H200. Left: the thread blocks (colours) over tiles of index queries and spans of centroid tiles, and below, candidate scoring and C expansion for one query. Right: one thread block (Algorithm 3, lines 3–9).
The bucket scan differs from a GEMM only in its epilogue. Instead of writing each entry of , the epilogue compares it with the threshold (lines 6–7), and compacts the buckets at or above it in shared memory, so that each span appends them to in HBM with one atomic per query (line 9). With , a tile takes few products, and this
Algorithm 3 Flat prefill selection of one chunk
Require: index queries , , and centroids , in HBM buckets before the cluster block of each sizes , , , , , margin , tile sizes ,
Ensure: index lists , rows of KV-cache slots
Threshold estimate
every tile of queries part of the sample, in parallel
score a strided sample of the of , on tensor cores
a bin edge at or below the -th best sampled score
Bucket scan
every tile of queries span of centroid tiles, in parallel
load the tile’s from HBM into shared memory
for each tile of centroids of the span do
load its from HBM into shared memory
, on tensor cores ▷ score tile
compact its with in shared memory
end for
, for the tile’s , in HBM
Candidate scoring
every query , in parallel
in fp32, for
packed in one int64 ▷ ties by bucket
if overflowed, or fewer than of its then ; redo lines 10–11
Expansion
every query , in parallel
where , best first, has keys or buckets
keys in of , in listed order
first keys in of the bucket at
the sink and the window; pad with
The threshold estimate and bucket scan epilogue costs about as much as they do, so the two must overlap. On H200, the thread block splits its warps by role: one warp loads the tiles of centroids into shared memory with the Tensor Memory Accelerator (TMA), in a pipeline of two stages, while a warpgroup multiplies them on tensor cores (WGMMA) into two accumulators in registers, and compares one with the threshold while the tensor cores fill the other. On B300, the products accumulate in the tensor memory of the SM instead, from which the comparing threads load them. Triton emits the same instructions, but chooses this schedule itself and gives no control of shared memory, which is why we write the scan in CuTe DSL.
At short context, below 32,768 buckets (about 256k tokens), the selection costs more than writing , which then takes at most 4 GiB, and we instead write with one matrix product in PyTorch and take the top- buckets of each row.
In prefill, we fix , so that the count above is off by about , and , which sets the sample: every th bucket of the row, about 4% of its scores at any context. Algorithm 3 then selects in four steps. The threshold estimate scores the sample twice on tensor cores, once for the mean and the best of its scores and once to count them in 64 bins between the two, binning them straight from the accumulators so that the scores of the sample are never written, and sets to the lower edge of the highest bin with at least samples at or above it (lines 1-2). The bucket scan computes tile by tile, and lists the buckets whose score is at or above minus a margin (lines 3-9):
Candidate scoring then re-scores the buckets of , as vector search re-scores a shortlist [31]: it computes their score again in fp32, one dot product per bucket, and packs it into (lines 10-11).
The margin is there because the scan and candidate scoring, which multiply the same bf16 operands and both accumulate in fp32, accumulate in different orders, so and can differ: with unit-norm index queries and centroids and , by less than . The margin is ten times larger, so holds every bucket with . The list has room for buckets, twice the target . A query whose list overflows, or holds fewer than buckets at or above , falls back: it scores all its buckets again, and takes the top- (line 12). The expansion writes (lines 13-16).
CUDA graphs
A prefill chunk takes milliseconds, against which the launches of its kernels from the host cost little, and its selection reads the sizes of its lists back on the host, so we serve prefill without graphs. A decode step of the selection, however, takes only tens of microseconds, about as long as its launches, so SGLang captures each decode step, our kernels included, in a CUDA graph, and replays it with one launch from the host. A CUDA graph fixes the shapes of its kernels, so a graph sized for the longest context the server admits would launch the thread blocks of 10M tokens at every step, whatever the contexts of the batch. We therefore capture one graph per range of context lengths, up to tokens for (from 144k to 9M), and one graph for the longest context, and each step replays the narrowest graph that covers the longest request of the batch. The threshold estimate of the flat search samples across the width of the graph, so a request far shorter than its range would see too few samples and fall back to its full row. The factor 1.125 avoids this for the common case: a request of a power-of-two length, with its answer, sits at about 89% of its range. The clustering of a complete cluster block runs outside the graph, before the replay, once every decoded tokens, and the decode times of § 5 spread its cost over those tokens.
Flat decode
In decode, each request adds one new query per step, whose buckets the flat search selects. It does so in a single pass, which reads the centroids of the request Algorithm 4 Flat decode selection of one request
Require: index query and centroids of , in HBM the request’s pages of sizes , , , , ,
Ensure: index list , as in Algorithm 3
Threshold estimate
every part of the sample, in parallel
score a strided sample of the , on CUDA cores
the -th best sampled score, rounded down
Bucket scan
every span of the request’s buckets, in parallel
read each once from HBM, through the request’s pages
in fp32, into shared memory and HBM
wait for
, for in the span, in HBM
Listing
gather the spans of , in bucket order
if overflowed, or fewer than of its then top- of the written , as
Expansion
lines 13–16 of Algorithm 3 on
once, scores them, and lists as it goes. We first show why this one read sets its time, then how the pass is spread over the GPU, and then the steps of Algorithm 4.
In decode, is a single row: the one index query of a request scores each of its centroids once per step. With in Equation (6), a step at 10M tokens takes flop on at least bytes, and writing its row takes bytes in fp32:
Its arithmetic intensity flop/B is far below the ridge point: the selection is bound by memory, and reading the centroids once would take at least on H200. Prefill, bound by compute, can afford to read the centroids more than once: every tile of queries reads them all, and candidate scoring reads those of again. In decode, a second read would about double the bytes the scan reads, so Algorithm 4 selects within a single read of the centroids, besides the small sample of its threshold estimate. It can, however, afford to write every to HBM, which adds only 3% to the bytes it moves, so that its fallback takes the top- of the written scores, without reading the centroids again. The threshold is thus not strictly needed in decode, but it saves a selection over all written scores of each request: the expansion ranks only the buckets of .
With a single query, only the row is split: the scan takes spans of 256 buckets, one per thread block. The centroids of a request lie in pages of , as its keys lie in pages of the KV cache, and each thread block reads the of its span through the page table of the request. The threshold estimate and the scan score on the CUDA cores, with the same dot product: at 1 flop/B the scan is bound by HBM, not by arithmetic, and gains nothing from tensor cores. A request whose row fits in its list takes every bucket.
Each CUDA graph samples a fixed number of buckets, from 2,048 to 16,384, in runs of 8 neighbouring buckets of a page, spread evenly over the row. A graph that covers longer contexts thus samples a smaller share of the row, with a smaller and a noisier estimate, so the target grows with the graph, from to . We set for each graph by hand, so that still holds at least buckets, and fits in its buffer, when the count misses by four times the deviation above, . Algorithm 4 runs in four launches, followed by the expansion, and has no candidate scoring. The threshold estimate takes two launches: one scores the sample, and one finds its -th best score by radix select over the bits of the scores, and rounds it down (lines 1–2). The bucket scan writes the of each span, and appends the buckets with to , already packed as (lines 3–6). It compares the very scores that rank the buckets, so it needs no margin. The listing gathers into the spans that the thread blocks appended, in bucket order (line 7), and the expansion is that of Algorithm 3 (line 9). The list has room for 2,048 to 8,192 buckets, by graph, and a request whose list overflows, or holds fewer than buckets, falls back to the top- of the scores the scan wrote (line 8). Programmatic dependent launch ([61]), a feature of Hopper and Blackwell GPUs, lets a kernel start while the one before it still runs, and wait for its results only where it reads them (line 5). The scan thus reads and scores the centroids while is still being computed.
Tree prefill. In prefill, the beam search (Algorithm 2) starts at the lowest level with at most node buckets, a value we chose by a sweep of accuracy against time. It keeps a beam of node buckets on every level. The descent starts from every node bucket of this level (line 1), which one matrix product in cuBLAS scores in full. Each level of the descent (lines 2–7) then runs in one kernel (five or six launches in Triton), besides a matrix product in cuBLAS that scores the open node buckets of the level. In this kernel, a thread block holds the scores of one query in shared memory, one 32-bit word per candidate, up to the 227 KiB a thread block can take. It scores the children of the node buckets kept on the level above (line 3). It then finds the -th score by a radix select of four passes, each of which counts one byte of the words in a histogram of 256 bins (line 4). Finally, it writes the children of the node buckets it keeps as the candidates of the next level (line 5). Above about 24k node buckets, keeping the beam of the start level this way would leave a single thread block per SM, so it takes two launches instead: a listing of the node buckets above a sampled threshold, as in the flat search, and the keep of the beam. At level 0 (lines 8-9), the thread block writes the top- buckets as packed scores . The expansion of Algorithm 3 (lines 13-16) reads them as it reads .
Tree decode. In decode, the new queries of all the requests of a batch descend together, through the beam kernels of the prefill, but over a copy of the hierarchy at fixed addresses, and at shapes set by the longest context, so that a CUDA graph can replay the step. Each request adds one query per step, but the descent still runs its levels one after another, so its time barely changes with the number of requests, while the flat search scans every bucket once per request. Every query scores the levels from the top level of the longest context down, and holds no candidates above its own start level. A step takes 7 to 13 kernels from 128k to 10M tokens, as the number of levels grows with the context.
Other steps. The remaining steps of D-LSA are the projections and the L2 normalisation of the index keys and queries , the pooling, and the tables of . They consist of matrix products, reductions and a sort, which run as standard PyTorch operations, with the products through cuBLAS. These steps cost little next to the scoring. On GLM-5.3-FLASH, for example, each of the two projections of a chunk of 32,768 tokens from the 512 dimensions of the latent takes flop, 4.3 Gflop for both. This is under 1% of the scoring of the chunk at contexts of 1M tokens and beyond (Equation (6)).
Attention kernels. D-LSA attends through existing kernels, which only need to read the index lists in a layout of their own. The expansion writes the index list of every query as a row of int32 entries. Each entry is the slot in the KV cache of one key , and the row is padded with when the query has fewer keys to attend to. The index lists of a chunk thus form one matrix of rows. On GLM-5.3-FLASH, takes the place of the index list of the model’s own indexer, and the sparse attention of SGLang attends over it unchanged. On QWEN-3.8-27B, we pass every query to a paged attention kernel as a sequence of its own: its row is the page table, each page holds one slot, and the sequence length is . Table 10 lists the attention kernels we use, from FlashAttention-3 (FA3) (Shah et al., [72]) and FA4, FlashMLA (Li et al., [44]), the TRT-LLM kernels of FlashInfer (Ye et al., 2025) and Triton. Each is the default of SGLang (Zheng et al., [96]) for its model and GPU, except where that default cannot run. For instance, on B300, D-LSA on QWEN-3.8-27B attends through the Triton kernel of SGLang, as FlashAttention supports pages of a single slot only on Hopper. With CP8, GLM-5.3-FLASH decodes through a split-KV sparse kernel that we add to SGLang
| H200 | B300 | |||||
| Model | Phase | Full Attention | Block Sparse Attention (Flat) | D-LSA (Flat and Tree) | ||
| GLM-5.3-FLASH | Prefill | FA3 | FA3 | FA3 | TRT-LLM | TRT-LLM |
| GLM-5.3-FLASH | Decode | FlashMLA | FA3 | FA3 | Triton | TRT-LLM |
| QWEN-3.8-27B | Prefill | FA3 | Triton | FA3 | TRT-LLM | Triton |
| QWEN-3.8-27B | Decode | FA3 | Triton + FA4 | FA3 | TRT-LLM | Triton |
Table 10. Attention kernels. Kernel through which each method attends, by model, phase and GPU, at TP8 or on one GPU. Full Attention, Block Sparse Attention (Flat), and D-LSA (Flat and Tree). Kernels in italics are non-default SGLang options.
(Appendix F.2), and on B300 it prefills through a TileLang kernel (Wang et al., [79]). With decode context parallelism (DCP8, Appendix § 14.§ 2) on H200, Full Attention and Block Sparse Attention (Flat) on QWEN-3.8-27B decode through Triton kernels.
N-LSA training. The kernels above serve D-LSA at inference. N-LSA also needs kernels for training, for its forward and its backward pass, with which a layer is up to faster than with Full Attention at 128k tokens (§ 5.1).
A few changes stem from the architectural differences of N-LSA compared to D-LSA. A training step processes whole sequences, of up to 128k tokens in our runs, with no KV cache and no decode, so every query of a sequence selects at once, as in a prefill chunk as long as the sequence. The keys change at every step as the attention projections learn, so every step clusters all keys again, where D-LSA clusters each cluster block once and keeps its buckets in the cache. The selection clusters and scores the attention’s own normalised keys and queries, of dimension 128, and takes the best score of the 8 query heads (Equation (3)). At 128k tokens, a row holds at most 16,384 buckets, from which a query selects for keys, next to a window of 512 keys and 4 sink keys. Finally, every step runs a backward pass, which D-LSA never does at long context.
Two of these differences shape the kernels. As the rows are short, the selection places from the whole row rather than from a sample: a first pass over the row counts its scores in a histogram of 256 bins, and a second lists the buckets from the bin of the -th score up, both on tensor cores, and the top- of the list are then taken by their scores. The backward pass is the main addition. As N-LSA stops the gradient through the selection, the clustering and the scoring, the centroids need no backward pass, and the only new backward kernel is that of the sparse attention. For one query head, with logits , its forward pass, as in any attention kernel, runs over the queries, each query attending to the keys it selected and writing only its own output:
Its backward pass cannot run over the queries in the same way, as each key and its value receive a gradient of the NTP loss from every query that selected the key:
where , and the queries of lie anywhere in the sequence at or after . As the 8 query heads share one key-value head, and also sum over the heads. Run over the queries, the backward pass would have many thread blocks add into the gradient of the same key, with an atomic addition for each share. The backward kernels instead run over the keys, as the backward pass of FlashAttention Dao et al., [11] does for dense attention (Algorithm 5). With a sparse selection, this first requires inverting the selection, listing the queries of each group of keys (lines 1-2). A thread block then holds the gradients of a group of keys in registers, adds the shares of these queries into them (lines 4-9), and writes them with few atomic additions (line 11). The gradient of each query sums over its own , so a separate pass gathers it as in the forward pass, computing again (line 12). To invert it, the backward pass needs the selection of every query, the int32 long-range indices that the forward pass keeps, as the sink and the window follow from the positions: 1 GiB at 128k tokens, about the gap between the peak memory of N-LSA and that of Full Attention. Evaluating and implementing a port of these kernels to CuTe DSL, as for D-LSA, is left to future work.
With these kernels, § 5.1 times one attention layer of N-LSA on one H200, at batch 1 in bf16, against the same layer with causal attention in FA3. Each time covers the forward and the backward pass of the whole layer, from its input projections to its output projection, with the clustering and the selection.
Measurements. Table 11 times each step on one H200 with Nsight Compute, in a single profile with the clocks unlocked, with its CuTe DSL and its Triton kernels. The inputs are synthetic, at the shapes we serve: a prefill chunk of 32,768 queries at 10M tokens, a beam of over node buckets of , decode with one request at 10M tokens, the Ward clustering of the 64 cluster blocks of one prefill chunk, and the clustering of one node block of buckets into node buckets. The times are those of the kernels alone, without the PyTorch launches between them. The regime of a step is where its main CuTe DSL kernel sits: bound by neither as a radix select over packed integer keys (selection), or limited by its serial rounds (latency). The bucket scan, the largest step of the flat prefill, reaches 49% of the bf16 peak and uses 1% of the HBM bandwidth, so the tiles of queries reread the centroids mostly from the L2 cache. On real inputs, the index queries of 12 held-out documents of 1M tokens and 3 books of 10M tokens on GLM-5.3-FLASH, over all its sparse layers, holds 1,679 buckets on average at 1M tokens and 1,737 at 10M, never overflows, and falls back, with fewer than buckets above , for of the queries at 1M and at 10M.
| Step | CuTe DSL (ms) | Triton (ms) | Speedup | Regime |
| Flat prefill | ||||
| Threshold estimate | 2.02 | 8.85 | 4.38× | compute (28%) |
| Bucket scan | 11.39 | 24.82 | 2.18× | compute (49%) |
| Candidate scoring | 1.63 | 1.66 | 1.02× | memory (85%) |
| Expansion | 2.37 | 4.14 | 1.75× | selection |
| Total | 17.41 | 39.48 | 2.27× | |
| Tree prefill | ||||
| Start level | 5.28 | 5.80 | 1.10× | selection |
| One beam level | 6.25 | 7.85 | 1.26× | selection |
| Level 0 (top-) | 9.13 | 9.59 | 1.05× | memory (66%) |
| Clustering | ||||
| Ward (buckets) | 0.043 | 0.056 | 1.30× | latency |
| Node buckets | 0.405 | 1.387 | 3.42× | latency |
| Flat decode | ||||
| Decode select | 0.071 | 0.153 | 2.15× | memory (75%) |
Table 11. Indexer kernels. Time of each step of the indexer of D-LSA on one H200, with its CuTe DSL and its Triton kernels, and the regime of its main CuTe DSL kernel.
Algorithm 5 Sparse attention backward of N-LSA, for one query head
Require: , , , and , in HBM log-sum-exp and the long-range indices of , kept by the forward pass
Ensure: , , , writing for
Inversion
count, for each group of keys, the queries with
, laid out by a prefix sum over the counts
Key and value gradients
every query , in parallel
every group of keys, in parallel
hold , , , in registers; ,
for each query and do
end for
add , into zeroed HBM, with few atomic additions
Query gradients
every query , in parallel
▷ computed again
In decode, of the steps fall back at 1M, mostly as their lists overflow, and none at 10M. The decode scan reads at 3.58 TB/s, 75% of the HBM bandwidth. At 10M tokens, the CuTe DSL kernels are faster than the Triton ones in every step, from 1.02× for candidate scoring, which is bound by the HBM bandwidth in both languages, to 4.4× for the threshold estimate. On B300, they are faster by up to 4.3× in every step but level 0 of the tree search, where the two languages tie. Of the 22 ms that CuTe DSL saves on a prefill chunk at 10M tokens, 13.4 ms come from the bucket scan, mostly from overlapping its epilogue with its products, and 6.8 ms from the threshold estimate, whose Triton version writes the scores of its sample to HBM and reads them twice. The Triton versions of other steps take more launches or more passes: the node-bucket clustering takes three launches per round and reads two flags on the host every eight rounds (3.4×), and the expansion finds its cut one bit per pass rather than one byte (1.75×). Table 11 sums the kernel times of the four launches of the decode select (the sample, the threshold, the scan and the listing), 71 µs at 10M tokens. Replayed from a CUDA graph, where programmatic dependent launch overlaps them, they take 14.7 µs at 1M tokens and 60.7 µs at 10M, against 36.4 µs and 146.6 µs in Triton. With the expansion and the other kernels of the selection, a flat decode step selects in 35 µs at 1M tokens and 86 µs at 10M, and Table 12 compares it with the tree search. With one request, the flat search is faster up to about 5M tokens. With 4 and 16 requests, the tree search is faster from about 1M and 256k tokens, and with 16 requests it is 5.6× faster at 10M tokens. As we evaluate one request at a time, every result of D-LSA (Tree) decodes with the flat search, and only its prefill uses the tree. At 1M tokens, the four steps of the flat prefill take 5.2 ms in CuTe DSL against 9.4 ms in Triton. For a whole layer of GLM-5.3-FLASH on H200, CuTe DSL makes a prefill chunk at 10M tokens 1.35× faster than Triton.
| Requests | 128k Flat | 128k Tree | 1M Flat | 1M Tree | 10M Flat | 10M Tree |
| 1 | 30 | 38 | 35 | 49 | 86 | 68 |
| 4 | 32 | 40 | 56 | 54 | 211 | 81 |
| 16 | 42 | 47 | 109 | 82 | 700 | 125 |
Table 12. Tree search in decode. Time of the selection of one layer for one decode step, from the scoring of the centroids to the index lists, in µs, with the flat and the tree search, on one H200, replayed from a CUDA graph. The faster search is in bold.
Parallelism
We describe our integration of D-LSA with tensor (TP) and context parallelism (CP) in SGLang, for multi-head latent attention (MLA), as in GLM-5.3-FLASH, and for grouped-query attention (GQA), as in QWEN-3.8-27B. denotes the number of ranks of a parallel group, and one of them. We study setups with (size of our NVLink domain).
LSA is orthogonal to expert and pipeline parallelism: the former splits only the experts, and the latter keeps the index of a layer co-located with its attention on the same rank, so in this section we do not describe these two mesh configurations, but focus on TP and CP instead.
For both TP and CP, every rank holds the whole bucket cache , which trades memory for communication (Section 3). Table 13 summarises the communication that the indexer adds for the two mesh configurations. We discuss these in more detail below.
| Phase | Collective | Size | Frequency | |
| TP, MLA | both | ALLREDUCE of query head sums | 2 KiB / token | every chunk and step |
| TP, GQA | both | ALLREDUCE of partial index queries and keys | 512 B / token | every chunk and step |
| PCP, MLA | prefill | ALLREDUCE of index keys | — | — |
| DCP | decode | ALLREDUCE of index keys | 64 KiB / block | every steps |
Table 13. Communication added by the indexer. Collectives that the indexer of D-LSA adds to one layer for TP alone, PCP and DCP, with MLA (GLM-5.3-FLASH) and GQA (QWEN-3.8-27B), beyond those of the attention. Sizes are per rank, and decode frequencies per request.
Tensor parallelism. We first describe TP alone, without CP, as illustrated in Figure 18. For TP, work gets split by head: rank holds a set of query heads, and projects them into a partial index query :
Figure 18. D-LSA with tensor parallelism (TP8). Top: MLA (GLM-5.3-FLASH). Bottom: GQA (QWEN-3.8-27B). Indexer keys: 1 projection (from the replicated latents with MLA, a partial per KV head with GQA), 2 bucket cache (the same on every rank). Indexer queries: A sum over the rank’s query heads and projection (a partial, summed across ranks by one ALLREDUCE, before the projection with MLA), B selecting the index list (the same on every rank), C attention over the rank’s own query heads.
With MLA, the ranks sum the query heads on the latent instead, and apply to the sum after the ALLREDUCE below, which is equivalent by linearity. ❶ With MLA, the latent is shared by all heads, thus replicated on every rank. With GQA, the KV heads are split across ranks too, and an index key is projected from all of them, so rank projects the keys of the set of KV heads it contributes into a partial index key :
where are the rows of that project KV head . Likewise, query head uses only the rows of the KV head it attends, in place of in (11).
For TP8, the four KV heads of QWEN-3.8-27B are each replicated on two ranks: both contribute their own query heads, but only one counts the KV head in its , and the other contributes . One ALLREDUCE sums the partials, in fp32, into the projections of (1), before their normalisation:
B Every rank thus holds the same index query and selects the same from its copy of . C It attends over with its own query heads . 2 The clustering takes the index keys of a cluster block only once all its keys have arrived, over several chunks or decode steps. With MLA, a rank recomputes them from the latents in its KV cache, so they are never stored. With GQA, a rank holds only some KV heads, so it cannot recompute them alone. Instead, it stores once summed: the ALLREDUCE of (13) carries the partial key of a token together with the token’s own partial query, so storing it costs memory, per token, but no communication. Recomputing it would save that memory, at the cost of a second ALLREDUCE of the fp32 partial keys,
256 B per token, once each block is complete.
With MLA, the ALLREDUCE carries the 512 dimensions of the latent, 2 KiB. Since the KV cache is replicated, adds the same 24.5 B per token to the 1 KiB of KV cache as without parallelism.
With GQA, the ALLREDUCE carries the partial index query and key, 512 B. A rank of QWEN-3.8-27B holds one of its four KV heads, 1 KiB, to which the indexer adds 24.5 B of bucket cache and of stored index key.
Context parallelism
CP splits work by token in two ways (Liu et al., [49]; Yang et al. [85]), which we describe in the following, and illustrate in Figure 19.
Figure 19. D-LSA with context parallelism (CP8). Top: prefill of GLM-5.3-FLASH (PCP8 + DCP8), each rank holding a contiguous span of the query rows. Indexer keys: ❶ projection of the whole chunk (on every rank), ❷ bucket cache (the same on every rank). Indexer queries: A projection of the rank’s own rows, B selecting their index lists (other ranks’ rows lighter), C marking the selected prefix rows (a 1 B mask, merged across ranks), D gathering those rows from the ranks that hold them (padded), E attention of the rank’s own rows. Bottom: decode of GLM-5.3-FLASH (PCP8 + DCP8) and QWEN-3.8-27B (TP8 + DCP8). Indexer keys: ❶ projection of a complete cluster block (summed across ranks, once per block), ❷ bucket cache. Indexer queries: A projection, B selecting the index list (the same on every rank), C attention over the entries the rank holds, D merging the partial outputs of the ranks (by log-sum-exp).
Decode context parallelism (DCP), also called KV parallelism (Bhatia et al., [6]), splits the KV cache across the ranks, in both prefill and decode, so that each rank stores only of it, which lets a context of millions of tokens fit in HBM, and reads only its share in each decode step. SGLang deals the tokens out round-robin: token goes to rank , its slot in the KV cache modulo , and that rank holds key and thus computes index key .
Prefill context parallelism (PCP) also splits the query rows of a chunk , giving rank the rows , so that it selects the index lists of its own rows only. The index query is not split: with PCP, every rank holds in prefill all query heads of its rows, and in decode (and with GQA also in prefill), the ranks gather them, as the attention does anyway, so every rank computes itself.
We extend SGLang with PCP for GLM-5.3-FLASH, which leaves us free to choose its layout, for both the base model and LSA. We make each one contiguous span of , which lets the linear-attention layers pass their recurrent state along it. LSA does not depend on this split, but it is a natural fit: as every query attends a fixed budget of keys, a contiguous split already balances the sparse attention across ranks, whereas dense causal attention commonly uses a zigzag or striped split to balance its causal work [7, 85].
With MLA, we use both PCP and DCP (Figure 19, top), with no TP in the prefill attention, since SGLang was already developing PCP for GLM-5.3-FLASH, while it had none for QWEN-3.8-27B. This also suits LSA: PCP splits the prefill by query rows rather than by heads. A Every rank thus holds all query heads of its rows and computes their index queries , , alone, without the ALLREDUCE of Equation (13) that TP needs. For PCP, the attention already gathers the latents of the whole chunk. 1 Every rank thus projects the index keys of the whole chunk itself, and 2 clusters them into the same . The indexer thus adds no communication, as long as each chunk starts on a cluster block, as chunks of 32768 = cluster tokens do. B Each rank selects for its rows , but DCP splits the KV cache of the prefix across ranks, so most of the selected keys live on other ranks, . sparse attention, of the base model as of LSA, gathers only those rows. C Every rank marks the prefix rows that its lists , , select in a mask of one byte per prefix row, and one ALLREDUCE with the maximum gives every rank their union over the chunk, less the chunk itself, whose rows every rank already holds:
D Each rank then contributes the latents of the rows of it holds, padded to the largest count over the ranks, to one ALLGATHER, which gives every rank all of . Every rank lays them out with the rows of the chunk in one buffer, , and remaps , , onto it. E The attention kernel then runs unchanged on the remapped lists. Without the mask, every rank would gather the whole prefix, 1 KiB per row, or 10 GiB per layer at a prefix of tokens, while the mask costs 1 B per row. Empirically, in our runs of GLM-5.3-FLASH on a 10M-token prompt, the union of the index lists of all 32,768 queries of a late chunk covers only about 30% of the prefix, as nearby queries select largely the same keys, so the ALLGATHER moves, and every rank holds, about 30% of the prefix rather than all of it, up to the padding.
With GQA, we run attention with TP8 and add DCP8, without PCP: TP splits the prefill by head and computes the keys of the whole chunk on every rank, and DCP splits the KV cache, which is what lets a context of 10M tokens fit on one node. DCP8 replaces the split of the KV heads: every rank holds all four KV heads of the tokens it owns, , and gathers the query heads of the other ranks before the indexer and the attention, in prefill and decode. It thus needs neither the partial sums of Equation (13) nor a stored like MLA, it recomputes the index keys of its own tokens from its KV cache.
In decode (Figure 19, bottom), each new token is written to its rank , so the keys of a cluster block are spread across all ranks. 1 Once its keys are complete, every rank needs all their index keys , but can compute only those with . Each rank writes them into an buffer, one row per token of the block in order, with zeros in the rows of the other ranks, and one ALLREDUCE sums the buffers, 64 KiB per block in bf16, or 128 B per token. As every row is non-zero on one rank only, the sum gathers the keys of the block in order, where an ALLGATHER would group them by rank. 2 Every rank then clusters the same keys into the same buckets of . Until then, the keys of the block are covered by the window of the most recent keys, which is selected by position, so they need no index key. This ALLREDUCE runs before the CUDA graph of the decode step is replayed. In decode the attention splits the query heads across the ranks, by TP with GQA and, with MLA, by a split of the MLA heads that we add to SGLang, so every rank gathers all query heads of every request, as the attention does anyway. A Every rank thus projects the same index query , B selects the same , and C attends over the entries of it holds,
into a partial output , softmax-normalised over , with the log-sum-exp of its logits. D The ranks merge the partial outputs of each query head by their log-sum-exp (Dao et al., [12]):
As the partition , is the attention over , up to floating-point rounding.
Memory. Replicating trades memory for communication. With and the bytes per token of the KV cache and of the bucket cache, the persistent storage of the KV cache and the flat index, per layer and rank, is approximately
where B, and B for GLM-5.3-FLASH and 4,096 B for Qwen-3.8-27B. At M with DCP8, the bucket cache adds 246 MiB as measured, plus padding rows, to the GiB share of the KV cache of a layer of GLM-5.3-FLASH (+19%), and 5% on Qwen-3.8-27B. An index per rank over its own keys would avoid the ALLREDUCE of the keys, but its buckets and its would differ from those of a single GPU.
References
- [1]Alabi, T., Blanchard, J. D., Gordon, B., and Steinbach, R. Fast k-selection algorithms for graphics processing units. ACM Journal of Experimental Algorithmics, 17, 2012. doi: 10.1145/2133803.2345676.DOI
- [2]Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Zinsley, D., Zou, J., Rudra, A., and Ré, C. Simple linear attention language models balance the recall-throughput tradeoff. In Proceedings of the 41st International Conference on Machine Learning, 2024.arxiv.org/abs/2402.18668
- [3]Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., Tang, J., and Li, J. Long-Bench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025.arxiv.org/abs/2412.15204
- [5]Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.arxiv.org/abs/2004.05150
- [6]Bhatia, N., More, A., Borkar, R., Mitra, T., Matas, R., Zhao, R., Golub, M., Mudigere, D., Pharris, B., and Darvish Rouhani, B. Helix parallelism: Rethinking sharding strategies for interactive multi-million-token LLM decoding. arXiv preprint arXiv:2507.07120, 2025.arxiv.org/abs/2507.07120
- [7]Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J. Striped attention: Faster ring attention for causal transformers. arXiv preprint arXiv:2311.09431, 2023.arxiv.org/abs/2311.09431
- [8]Chen, Y., Zhang, J., Lu, B., Zhang, Q., Zhang, C., Liu, J., Luo, J., Liu, D., Jiang, H., Chen, Q., Ding, B., Yan, X., Jiang, J., Chen, C., Zhang, M., Li, C., Yang, Y., Yang, F., and Yang, M. RetroInfer: A vector storage engine for scalable long-context LLM inference. Proceedings of the VLDB Endowment, 19(5):1016–1031, 2026. doi: 10.14778/3796195.3796212.DOI
- [9]Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.arxiv.org/abs/1904.10509
- [10]Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
- [11]Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, volume 35, pp. 16344–16359. Curran Associates, Inc., 2022. doi: 10.52202/068431-1189.DOI
- [12]Dao, T., Haziza, D., Massa, F., and Sizov, G. Flash-Decoding for long-context inference, 2023. URL https://pytorch.org/blog/flash-decoding/.
- [13]DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
- [14]DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.arxiv.org/abs/2512.02556
- [15]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026a.arxiv.org/abs/2606.19348
- [16]DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv preprint arXiv:2609.19969, 2026b.
- [17]Desai, A., Yang, S., Cuadron, A., Zaharia, M., Gonzalez, J. E., and Stoica, I. HashAttention: Semantic sparsity for faster inference. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 13402–13418. PMLR, 2025.arxiv.org/abs/2412.14468
- [18]Floyd, R. W. and Rivest, R. L. Expected time bounds for selection. Communications of the ACM, 18(3):165–172, 1975. doi: 10.1145/360680.360691.DOI
- [19]Gao, Y., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H., Cao, T., Yang, F., and Yang, M. SeerAttention: Self-distilled attention gating for efficient long-context prefilling. In Advances in Neural Information Processing Systems, volume 38, pp. 62230–62253. Curran Associates, Inc., 2025. doi: 10.52202/085713-1869.DOI
- [20]Gao, Y., Guo, S., Cao, S., Xia, Y., Cheng, Y., Wang, L., Ma, L., Sun, Y., Ye, T., Dong, L., So, H. K-H., Hua, Y., Cao, T., Yang, F., and Yang, M. Sparse attention adaptation for long reasoning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=c5BOchM6J8.arxiv.org/abs/2506.08889
- [21]Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
- [23]Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., and Peste, A. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1–124, 2021. URL http://jmlr.org/papers/v22/21-0366.html.arxiv.org/abs/2102.00554
- [24]Hooper, C., Zhao, S., Manolache, L., Kim, S., Mahoney, M., Shao, S., Keutzer, K., and Gholami, A. Multipole attention for efficient long context reasoning. In Advances in Neural Information Processing Systems, volume 38, pp. 41954–41977. Curran Associates, Inc., 2025a. doi: 10.52202/085713-1255.DOI
- [25]Hooper, C. R. C., Kim, S., Mohammadzadeh, H., Mah-eswaran, M., Zhao, S., Paik, J., Mahoney, M. W., Keutzer, K., and Gholami, A. Squeezed attention: Accelerating long context length LLM inference. In Proceedings of the 63rd Annual Meeting of the Association of Computational Linguistics (Volume 1: Long Papers), pp. 32631–32652, Vienna, Austria, 2025b. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.1568.DOI
- [26]Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024.arxiv.org/abs/2404.06654
- [31]Jégou, H., Tavenard, R., Douze, M., and Amsaleg, L. Searching in one billion vectors: Re-rank with source coding. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 861–864. IEEE, 2011. doi: 10.1109/ICASSP.2011.5946540.DOI
- [32]Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems, volume 37, pp. 52481–52515. Curran Associates, Inc., 2024. doi: 10.52202/079017-1663.DOI
- [33]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VF8yNQM66.arxiv.org/abs/2310.06770
- [34]Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2021. doi: 10.1109/TBDATA.2019.2921572.
- [35]Kandpal, N., Lester, B., Raffel, C., Majstorovic, S., Biderman, S., Abbasi, B., Soldaini, L., Shippole, E., Cooper, A. F., Skowron, A., Kirchenbauer, J., Longpre, S., Swatiwaka, L., Albalak, A., Xu, Z., Penedo, G., Ben Al-lal, L., Bakouch, E., Pressman, J. D., Fan, H., Stander, D., Song, G., Gokaslan, A., Goldstein, T., Bartoldson, B. R., Kailkhura, B., and Murray, T. The Common Pile v0.1: An 8TB dataset of public domain and openly licensed text. arXiv preprint arXiv:2506.05209, 2025.arxiv.org/abs/2506.05209
- [36]Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. In The Eighth International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkgNKkHtvB.arxiv.org/abs/2001.04451
- [37]Kocetkov, D., Li, R., Ben Allal, L., Li, J., Mou, C., Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, M., Muñoz Ferrandis, C., Jernite, Y., Mitchell, M., Hughes, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Wolfe, T. The Stack: 3 TB of permissively licensed source code. Transactions on Machine Learning Research, 2023.arxiv.org/abs/2211.15533
- [38]Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., Bloom, R., Broadley, T., Du, H., Goodrich, B., Jurkovic, N., Miles, L., Nix, S., Lin, T., Parikh, N., Rein, D., Koba Sato, L. J., Wijk, H., Ziegler, D., Barnes, E., and Chan, L. Measuring AI ability to complete long software tasks. In Advances in Neural Information Processing Systems, volume 38, pp. 102382–102435. Curran Associates, Inc., 2025. doi: 10.52202/085713-3086.DOI
- [39]Lai, X., Xu, W., Yang, Y., Chen, Q., Xu, Y., Zeng, L., Li, X., Sun, H., Zhu, H., Zhang, V., Hu, J., Li, J., Gao, R., Li, Z., Zhu, S., Zhou, J., and Zhao, P. MiniMax sparse attention. arXiv preprint arXiv:2606.13392, 2026.arxiv.org/abs/2606.13392
- [40]Lee, H., Park, G., Lee, Y., Suh, J., Kim, J., Jeong, W., Kim, B., Lee, H., Jeon, M., and Hwang, S. J. A training-free sub-quadratic cost transformer model serving framework with hierarchically pruned attention. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id=PTcMzQqKmn.arxiv.org/abs/2406.09827
- [41]Lee, H., Park, G., Suh, J., and Hwang, S. J. InfiniteHiP: Extending language model context up to 3 million tokens on a single GPU. arXiv preprint arXiv:2502.08910, 2025b.arxiv.org/abs/2502.08910
- [42]Leng, J., Hu, X., Wang, J., Li, J., Wu, W., and Lu, Y. Understanding and improving length generalization in hierarchical sparse attention models. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=HqdSQk6q6qc.arxiv.org/abs/2510.17196
- [43]Li, D., Liu, Z., Lin, G., Hu, B., and Zhang, M. LycheeCluster: Efficient long-context inference with structure-aware chunking and hierarchical KV indexing. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 7607–7623, San Diego, California, United States, 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.376.DOI
- [44]Li, J., Liu, S., and Sun, Y. FlashMLA: Efficient multi-head latent attention kernels, 2025. URL https://github.com/deepseek-ai/FlashMLA.
- [46]Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2305.20050
- [47]Liu, D., Chen, M., Lu, B., Jiang, H., Han, Z., Zhang, Q., Chen, Q., Zhang, C., Ding, B., Zhang, K., Chen, C., Yang, F., Yang, Y., and Qiu, L. RetrievalAttention: Accelerating long-context LLM inference via vector retrieval. In Advances in Neural Information Processing Systems, volume 38, pp. 60539–60566. Curran Associates, Inc., 2025a. doi: 10.52202/085713-1816.DOI
- [48]Liu, G., Li, C., Zhao, J., Zhang, C., and Guo, M. ClusterKV: Manipulating LLM KV cache in semantic space for recallable compression. In 2025 62nd ACM/IEEE Design Automation Conference (DAC), pp. 1–7. IEEE, 2025b. doi: 10.1109/DAC63849.2025.11132479.DOI
- [49]Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.01889
- [52]Madaan, L., Bhojanapalli, S., Jain, H., and Jain, P. Treeformer: Dense gradient trees for efficient attention computation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=DWn1Eb2fK.arxiv.org/abs/2208.09015
- [53]Mao, Y., Li, M. Y., and Fox, E. B. Simplified sparse attention via gist tokens. arXiv preprint arXiv:2604.20920, 2026a.arxiv.org/abs/2604.20920
- [54]Mao, Y., Wang, Q., Ester, M., and Li, K. IceCache: Memory-efficient KV-cache management for long-sequence LLMs. In The Fourteenth International Conference on Learning Representations, 2026b. URL https://openreview.net/forum?id=yHxSKM9kdr.arxiv.org/abs/2604.10539
- [55]MiniMax. MiniMax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313, 2025.arxiv.org/abs/2501.08313
- [58]NVIDIA. NVIDIA H200 tensor core GPU. https://www.nvidia.com/en-us/data-center/h200/, 2026c. Accessed 2026-10-28.
- [61]NVIDIA. CUDA programming guide, section 4.5: Programmatic dependent launch and synchronization. https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/programmatic-dependent-launch.html, 2026f. Accessed 2026-10-30.
- [62]Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024.arxiv.org/abs/2406.17557
- [63]Peng, B., Quesnelle, J., Fan, H., and Shippole, E. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2309.00071
- [64]Qwen Team. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026a.
- [65]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026b. URL https://qwen.ai/blog?id=qwen3.8.
- [67]Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
- [68]Roy, A., Saffar, M., Vaswani, A., and Grangier, D. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53–68, 2021. doi: 10.1162/tacl_a_00353.DOI
- [69]Schlag, I., Irie, K., and Schmidhuber, J. Linear transformers are secretly fast weight programmers. In Proceedings of the 38th International Conference on Machine Learning, pp. 9355–9366, 2021.arxiv.org/abs/2102.11174
- [70]Schmidhuber, J. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992a. doi: 10.1162/neco.1992.4.1.131.DOI
- [71]Schmidhuber, J. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234–242, 1992b. doi: 10.1162/neco.1992.4.2.234.DOI
- [72]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems, volume 37, pp. 68658–68685. Curran Associates, Inc., 2024. doi: 10.52202/079017-2193.
- [74]Tang, B., Qin, Z., Pan, Y., Li, Z., and Liu, P. Block sparse attention with log-linear complexity. arXiv preprint arXiv:2609.31093, 2026.arxiv.org/abs/2609.31093
- [75]Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasicki, B., and Han, S. QUEST: Query-aware sparsity for efficient long-context LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47901–47911. PMLR, 2024.arxiv.org/abs/2406.10774
- [76]Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Atluri, A., Qiao, F., Majcher, P., Springer, P., Hohenbarch, M., Wang, J., and Gupta, M. CUTLASS, 2023. URL https://github.com/NVIDIA/cutlass.
- [77]Tillet, P., Kung, H. T., and Cox, D. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19. Association for Computing Machinery, 2019. doi: 10.1145/3315508.3329973. Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang, S., Chen, T., Kasicki, B., Grover, V., Krishnamurthy, A., and Ceze, L. FlashInfer: Efficient and customizable attention engine for LLM inference serving. In Proceedings of Machine Learning and Systems, volume 7, 2025.
- [78]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.arxiv.org/abs/1706.03762
- [79]Wang, L., Cheng, Y., Shi, Y., Tang, Z., Mo, Z., Xie, W., Ma, L., Xia, Y., Xue, J., Yang, F., and Yang, Z. TileLang: A composable tiled programming model for AI systems. arXiv preprint arXiv:2504.17577, 2025.arxiv.org/abs/2504.17577
- [80]Wang, L., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024.arxiv.org/abs/2406.01574
- [81]Ward, Jr., J. H. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301):236–244, 1963. doi: 10.1080/01621459.1963.10500845.DOI
- [82]Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., and Sun, M. InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory. In Advances in Neural Information Processing Systems, volume 37, pp. 119638–119661. Curran Associates, Inc., 2024a. doi: 10.52202/079017-3801.DOI
- [83]Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=NG7sS51zVF.arxiv.org/abs/2309.17453
- [85]Yang, A., Yang, J., Ibrahim, A., Xie, X., Tang, B., Sizov, G., Park, J., and Huang, J. Context parallelism for scalable million-token inference. In Proceedings of Machine Learning and Systems, volume 7. MLSys, 2025a.arxiv.org/abs/2411.01783
- [86]Yang, S., Kautz, J., and Hatamizadeh, A. Gated delta networks: Improving Mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025b.arxiv.org/abs/2412.06464
- [87]Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.1126.DOI
- [88]Yuan, J., Shinn, C., Xu, K., Cui, J., Klimiashvili, G., Xiao, G., Zheng, P., Li, B., Zhou, Y., Ye, Z., You, W., Zheng, T., Brown, D., Wang, P., Hoehenricher, M., Cai, R., Demouth, J., Owens, J. D., Hu, X., Han, S., Liu, T., and Mao, H. BLASST: Dynamic BLocked attention sparsity via softmax thresholding. In Proceedings of Machine Learning and Systems, volume 8, pp. 843–859. MLSys, 2026.arxiv.org/abs/2512.12087
- [89]Zaheer, M., Guruganes h, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pp. 17283–17297. Curran Associates, Inc., 2020.arxiv.org/abs/2007.14062
- [90]Z.ai. GLM-5.3-Flash, 2026. URL https://z.ai/blog/glm-5.3-flash.
- [91]Zan, W., Zhang, J., Tan, J., Liu, H., Wang, C., Li, X., Ma, D., Wu, G., Lu, Y., Li, F., Sun, Y., Pei, P., Xie, Y., and Cai, X. LongCat sparse attention: Taming the lightning via streaming-aware hierarchical cross-layer indexing. arXiv preprint arXiv:2608.01662, 2026.arxiv.org/abs/2608.01662
- [92]Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019.arxiv.org/abs/1905.07830
- [93]Zhang, J., Naruse, A., Li, X., and Wang, Y. Parallel top-K algorithms on GPU: A comprehensive study and new methods. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–13. ACM, 2023a. doi: 10.1145/3581784.3607062.DOI
- [94]Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, volume 36, pp. 34661–34710. Curran Associates, Inc., 2023b. doi: 10.52202/075280-1506.DOI
- [95]Zhao, W., Zhou, Z., Su, Z., Xiao, C., Li, Y., Li, Y., Zhang, Y., Zhao, W., Li, Z., Huang, Y., Sun, A., Han, X., and Liu, Z. InfLLM-V2: Dense-sparse switchable attention for seamless short-to-long adaptation. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=z2F9V0H6Vi.arxiv.org/abs/2509.24663
- [96]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozakyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2312.07104
- [97]Zhu, D., Zhou, X., Qin, S., Zhu, X., Ding, H., Zhong, S., Wen, Z., Xie, Z., Gou, C., Ren, L., Wang, Y., Zhong, J., Liu, R., Gao, T., Lin, Y., Zhang, J., Song, M., Qi, X., Wu, J., Zhang, C., Piao, Y., Niu, Z., Lin, H., Meng, L., Tang, P., Tang, C., Wu, S., Zheng, H., Liu, Y., Zhu, L., Wang, H., Ding, M., Wan, Z., Liu, H., Wang, S., Zhu, H., Zhang, X., Chai, N., Liu, Y., Lai, P., Yuan, S., Su, Z., Zhang, G., Zhou, W., Du, Y., Huang, W., and Shi, G. EdgeBench: Unveiling scaling laws of learning from real-world environments. arXiv preprint arXiv:2607.05155, 2026.arxiv.org/abs/2607.05155














