WorksComputer Science

Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention

Serdarcan Dilbaz, Arjun Krishna, Kiran Kamble, Daniel M. Bikel

Published Nov 02, 2026Fresh — added to the Hub within the last 30 daysv139 min read

AI contributions

Not recorded for this paper

View details

Ancillary data

1 code link

View

Abstract

Sparse attention cuts the cost of long-context inference by making the attention matrix sparse: each query attends only a subset of the sequence, chosen by a fixed pattern or, in recent production LLMs, learned per query. In DeepSeek Sparse Attention (DSA), the attention of DeepSeek-V3.2 and the GLM-5 family, every layer is preceded by a lightning indexer that scores the whole cache and keeps the k=2048 highest-scoring tokens for the layer to attend. At long context this indexer scan, not the sparse attention it feeds, is where the decode time goes. But adjacent layers select nearly the same tokens, so a layer can attend the selection of the previous indexer layer instead of its own. This removes the dependency: each indexer then runs concurrently with the attention of its layer. The resulting Multi-layer Indexer (MLI) needs no retraining and no new weights. Its default realization changes no kernel, running inside SGLang's production decode graph; an optional fused decode kernel accelerates it further on DeepSeek-V3.2. MLI composes with IndexCache (the cross-layer reuse GLM-5.2 ships as IndexShare): IndexCache drops the indexer from some layers, and MLI hides the scan on the rest. We serve DeepSeek-V3.2 and GLM-5 under IndexCache's protocol at retentions 1/2 to 1/32 (the fraction of layers that keep an indexer) and contexts 10K to 195K. At equal retention the composition decodes faster than IndexCache: up to 14.5% faster on DeepSeek-V3.2 and 9.8% on GLM-5 at 1/2, and 7.9% and 5.1% at 1/4, largest where IndexCache keeps the most indexers and growing with context. At retention 1/r it is faster still than IndexCache at 1/(2r) while keeping twice the fresh selections: at 128K it reaches 1.32x the unmodified model's decode throughput at 1/4, above IndexCache's 1.28x at 1/8. Its LongBench, RULER and LongBench v2 accuracy stays within a few percentage points of IndexCache's.

Introduction

Sparse attention cuts the cost of long-context decoding by making the attention matrix sparse: each query attends only a subset of the sequence, chosen by a fixed pattern or learned per query. Recent production LLMs take the learned route, with a lightweight selector that must run before attention; at long context it, not the sparse attention it feeds, is where the decode time goes. DeepSeek Sparse Attention (DSA) [12] is the attention mechanism of DeepSeek-V3.2 and of GLM-5, GLM-5.1 and GLM-5.2 [25, 23 b]. Consider decoding token $t$ with $L$ tokens in the cache. At each layer a lightning indexer with $H_I$ small heads scores every cached token $n \le L$ ,

I_{t,n} = \sum_{h=1}^{H_I} w_{t,h} \operatorname{ReLU}\left(q^{I}_{t,h} \cdot k^{I}_{n,h}\right), \tag*{(1)}

where $I_{t,n}$ is the indexer’s score for query $t$ and cached token $n$ , the superscript $I$ marks the indexer’s own projections (distinct from the layer’s attention), $h$ indexes its $H_I$ heads, and $\mathbf{q}^{I}_{t,h}$ , $\mathbf{k}^{I}_{n,h}$ are low-rank projections of the layer input with $w_{t,h}$ a per-head weight. The layer keeps the set $S_t$ of the $k=2048$ tokens with the highest score and runs attention over those tokens only. The selection is the sole coupling between the indexer and attention: attention cannot begin until the top- $k$ is known, and nothing else in the layer depends on it.

The two nevertheless run in sequence. Sparse attention gathers $k$ keys and its cost is flat in $L$ ; the indexer scans all $L$ keys. Beyond a few tens of thousands of tokens the scan is the largest term on the attention side of the layer, two thirds of it at 128K (Figure 1). Only the selection dependency forces the order, and adjacent DSA layers select strongly overlapping sets (§ 2), a cross-layer redundancy that reuse methods already exploit [3, 20, 6], so a layer’s attention need not wait for its own selection: an earlier layer’s serves.

Figure 1. Per-layer decode latency of the DSA attention path (DeepSeek-V3.2, H100, one request; kernels timed in isolation). (a) Standard DSA runs the indexer and attention in sequence, so their latencies add; the prefix scan grows with context and comes to dominate. (b) MLI overlaps them on a second stream, so the per-layer time is the bottleneck, not the sum; the full-width bar is the bottleneck and the striped bar the hidden component, the indexer split into scan and top- $k$ as in (a). Idealized ceiling; the realized gain is smaller and bandwidth-bound (§ 5.2).

The cost of the indexer per decode step is a product of two independent factors: how many indexers run, and how long each one is waited for. IndexCache [3] reduces the first, keeping the indexer on a fraction $1/r$ of the layers (§ 2). We reduce the second. The Multi-layer Indexer (MLI) lets layer $\ell$ attend the selection produced by the previous indexer layer. The indexer of layer $\ell$ then feeds a later layer, not its own attention, and the two run at the same time (in the idealized limit of Figure 1; the realized speedup is smaller, for reasons we trace in § 5.2). Every selection is still recomputed at every decode step; nothing is reused across steps. Because the two factors multiply and neither method touches the other’s, the two remedies compose: on the layers IndexCache keeps, MLI hides the scan that IndexCache still pays. Like IceFormer’s deployable acceleration [15], MLI needs no retraining, applies to every DSA model, and leaves the selections unchanged. This is the regime long context forces: KV memory caps how many requests decode together (six at 128K, two at 195K on our node), so very long context is served few-at-a-time, exactly where the indexer scan is exposed; the gain fades only when many short requests pack the batch and saturate memory bandwidth (§ 5.2).

Our contributions are the following.

A drop-in schedule that overlaps the indexer with attention. MLI (§ 3) makes a layer’s indexer and its attention independent, at any retention and lag depth, so they run concurrently; it needs no retraining and no new weights, runs in SGLang’s decode CUDA graph with the production kernels unchanged, and is faster still on DeepSeek-V3.2 with a fused decode kernel (§ 5.3).

Faster at equal retention, and one further halving for free. Serving DeepSeek-V3.2 and GLM-5 under IndexCache’s protocol (§ 5.1), IndexCache+MLI decodes faster than IndexCache at the same retention: up to 14.5% faster on DeepSeek-V3.2 and 9.8% on GLM-5 at $1/2$ , and 7.9% and 5.1% at $1/4$ ; and at retention $1/r$ it is faster than IndexCache at $1/2r$ while attending twice as many fresh selections, so a halving of the indexer budget comes at no speed cost. At 128K it reaches 1.32× the unmodified model’s throughput at $1/4$ , above IndexCache’s 1.28× at $1/8$ .

A shared-primitive fused kernel, and the case for co-design. On DeepSeek-V3.2 a fused decode kernel runs the sparse attention and the indexer’s prefix scan of a shared step in one launch, a per-request win (§ 5.3); it does not help GLM-5, which has fewer index heads per rank. Because the indexer scores the same group bilinear interaction as attention, this is evidence that DSA models should be co-designed around that shared primitive: a direction we argue for and partly realize, not a model we train.

The quality cost of staleness is small and recoverable. Across five long-context suites (§ 5.4), attending an older selection stays within a few points of IndexCache at every retention it reports; the arm with twice the fresh selections, now the faster of the two, matches IndexCache on LongBench and LongBench v2 and is within 3 points on RULER (§ 5.6), and distilling only the indexers, applied identically to both methods, recovers little at the mild retentions IndexCache reports but rescues the collapse at aggressive retention (§ 5.5).

Background

A DSA layer at decode time. Let $N$ be the number of layers and $h_\ell$ the input of layer $\ell$ for the current token. With $L$ tokens in the cache, layer $\ell$ performs three steps in order: (i) the indexer scores every cached token with (1), reading an index-key cache of $L$ entries; (ii) a top- $k$ over the $L scores produces the selection $S_{\ell}$ ; (iii) sparse Multi-head Latent Attention runs over the $k$ selected latent KV rows. Only step (i) grows with $L$ .

IndexCache. IndexCache [3] splits the layers into full layers, which run step (i), and shared layers, which copy the selection of the nearest preceding full layer. A uniform pattern with period $r$ keeps a fraction $1/r$ of the indexers; a greedy search against a calibration loss does better at equal retention, and the paper publishes searched patterns for GLM-5. GLM-5.2 adopts the uniform pattern with $r=4$ as IndexShare [24]. A full layer still waits for its own scan. IndexCache also describes a training-aware variant that distills each retained indexer toward the selections of the layers it serves.

Why a neighbouring layer’s selection is usable. The dependency between the indexer and attention is on the support of the softmax, not on any computation they share, and it disappears the moment attention takes its support from a selection that already exists when $h_{\ell}$ does. Such a selection is available if adjacent layers select similar tokens. On real activations (GovReport, $k=2048$ ) the Jaccard overlap between consecutive layers’ selections stays near 0.42 on DeepSeek-V3.2 and 0.26 on GLM-5 from 16K to 128K; whether that suffices is measured directly in § 5.4. IndexCache uses this by copying a selection forward; MLI attends the previous full layer’s selection while the current one is computed. Only the indexer’s placement moves, not its arithmetic.

Multi-layer Indexer

Schedule

We describe which selection each layer attends. Let $S_f$ denote the top-2048 tokens the indexer of full layer $f$ selects. Let $F \subseteq\{1,\ldots,N\}$ be the set of full layers, those that run the indexer; standard DSA has $F=\{1,\ldots,N\}$ , and IndexCache with retention $1/r$ has $\lvert F\rvert=N/r$ . For any layer $\ell$ let $p(\ell)$ be the nearest full layer at or before $\ell$ , so that $p(\ell)=\ell$ when $\ell\in F$ . Let $\pi_k$ step back $k$ full layers along $F$ (the lag depth), so $\pi_0$ is the identity. For a full layer $f$ and a depth $k\geq0$ let $\pi_k(f)$ be the full layer $k$ positions before $f$ in $F$ , with $\pi_0(f)=f$ . Every layer attends to the selection of one full layer:

\text{layer }\ell\text{ attends }S_{\sigma(\ell)}, \qquad\sigma(\ell)=\pi_k(p(\ell)). \tag*{(2)}

The retention $r$ and the depth $k$ are independent. With $k=0$ every layer attends the selection of its own full layer: this is standard DSA when $r=1$ and IndexCache when $r>1$ . With $k\geq1$ a full layer attends the selection made $k$ full layers earlier, and a shared layer attends what its full layer attended. This is MLI when $r=1$ and the composition of the two methods when $r>1$ : MLI acts on the full layers, IndexCache’s copies act on the shared layers. The first $k$ full layers of the forward pass have no predecessor in $F$ and use the standard schedule.

The point of (2) with $k\geq1$ is independence. The indexer of full layer $f$ reads $h_f$ and its output is first consumed by the layer that follows $f$ in $F$ ; nothing in layer $f$ waits for it, so the indexer can execute while the attention, the output projection and the feed-forward block of layer $f$ execute (Figure 3 in the appendix draws the layer schedule). Two properties separate this from IndexCache: under MLI a selection is at most $k$ full layers old and every full layer attends a selection recomputed for the current token, whereas under IndexCache a shared layer attends a selection up to $r-1$ layers old that was never recomputed for it. One serial scan per decode step remains, the first full layer of the forward pass; in a model whose first layers use dense attention it could be issued alongside the last dense layer, but DeepSeek-V3.2 and GLM-5 carry an indexer at every layer, so we keep it serial.

Layer schedule comparing the standard schedule and the Multi-layer Indexer — Figure 3. Layer schedule with depth $k=1$ , drawn for a dense layer followed by two DSA layers. Left: standard DSA. Attention at layer $\ell$ starts only after the indexer of layer $\ell$ has produced $S_\ell$ . Right: Multi-layer Indexer. The indexer of layer $\ell+1$ and the attention of layer $\ell$ read the same hidden state and run concurrently (green step); a selection is attended one step after it is produced.

Realization inside the decode graph

We implement Equation (2) in SGLang’s DSA decode path; its portable realization changes no kernel (a fused kernel, faster on DeepSeek-V3.2, is Section 5.3), and Figure 2 shows one decode step. At a full layer, as soon as the layer input exists, the indexer (its projections, the scan over the paged index-key cache, and the top- $k$ ) is enqueued on a second CUDA stream while attention on the main stream proceeds with the selection that Equation (2) assigns to the layer. The second stream is joined at the layer that first consumes the new selection, $k$ full layers later, and references to the indexer’s inputs and output are held until that join so the allocator cannot reuse their memory. The cross-stream edges are ordinary graph dependencies, so the whole schedule is captured once into the decode CUDA graph and replayed without host involvement; a per-layer registry carries selections between full layers, and shared layers receive the carried selection exactly as in IndexCache. Because the scan and the top- $k$ are the production kernels on the production buffers, each full layer produces the selection the standard schedule produces; only the layer that consumes it changes.

Two-stream realization across two full layers — Figure 2. Two-stream realization, depth 1, one decode step across two full layers. The indexer of layer $\ell$ forks onto the side stream once $h_\ell$ exists and runs under the attention, output projection and feed-forward block of layer $\ell$ ; its selection is joined at the prepare step of layer $\ell+1$ , which attends it. The whole pattern is captured once into the decode CUDA graph.

Recovering selection quality by distillation

MLI changes exactly one thing, the selection a layer attends, and the cost of that change can be paid back without touching the backbone. We train only the indexers of the full layers, about 0.5% of the parameters, so that the selection a full layer produces matches what the layers that will attend it would have selected for themselves: under MLI those consumers are the layers from the next full layer up to the one after it, under IndexCache the full layer itself and its shared span. The teacher is the unmodified model’s indexer at each consumer, frozen, read from the same forward pass. With $C(f)$ the consumers of full layer $f$ and $p^c$ , $\hat{p}^c$ the softmax teacher and student scores at consumer $c$ , the per-layer loss is

\mathcal{L}_f = \frac{1}{\lvert C(f)\rvert} \sum_{c \in C(f)} \left[ \lambda\operatorname{KL}(p^c \mathbin{\|} \hat{p}^f) + (1-\lambda)\mathcal{L}_{\mathrm{PL}}(\pi^c; \hat{\pi}^f) \right], \tag*{(3)}

where $\mathcal{L}_{\mathrm{PL}}$ is a listwise Plackett–Luce (ListMLE) term on the teacher’s top- $k$ ranking $\pi^c$ and $\lambda=0.5$ ; it fixes the ranking where the top- $k$ is decided (Section K). The same data, steps and loss are applied to IndexCache on its own consumer set, which is its training-aware variant, so distilled arms are always compared in pairs.

Experimental Setup

Models and serving. We serve DeepSeek-V3.2 (61 layers, 128 attention heads, 64 indexer heads) and GLM-5 (78 layers, 64 attention heads, 32 indexer heads) from their fp8 checkpoints with SGLang v0.5.16 on one node of eight H200 GPUs, with 8-way tensor parallelism, an fp8 KV cache, CUDA graphs enabled for every arm and $k=2048$ (a data-parallel-attention replication is in Section F).

Regimes and contexts. We report (i) per-request decode throughput at a single concurrent request; (ii) full-KV-cache decode throughput, where the request set is sized to the server’s KV capacity and every prompt is first warmed into the prefix cache so that the measured phase is saturated decode; and (iii) prefill latency. Contexts are 10K, 60K and 120K tokens on both models, plus 128K on DeepSeek-V3.2 and 195K (199,680 tokens) on GLM-5. Retention is uniform with $r \in\{2, 4, 8, 16, 32\}$ (IndexCache reports up to $1/8$ ). On GLM-5 we also run IndexCache’s published searched patterns (40 and 22 full layers of 78), with MLI applied to the same patterns.

Protocol and fairness. Every arm of a cell (one model, one regime, one context) runs in the same job on the same node, differing only in model-override flags, so a speed difference reflects the mechanism and not the setup. Three controls guard the comparison. (i) Drift: the unmodified model is booted first and again last, and the cell is reported only if the two agree within 2% (observed within 0.4%). (ii) Engagement: every MLI arm must log, on every tensor-parallel rank, the marker that the overlapped path was captured into the decode graph, with no fallback marker (Section A). (iii) IndexCache parity: IndexCache runs through its own released layer-skip knobs, and as a control its arms are also booted on the unmodified SGLang release in the same job, where our build matches the release within 0.2% on the base model and is 0.4 to 1.1% slower on the IndexCache arms (Section C), so the tables understate IndexCache by at most 1.1 percentage points. Distilled arms are always paired with IndexCache distilled by the same recipe on its own spans (§ 3.3). Measurement noise is in Section J, and paired bootstrap intervals for every quality contrast in § 17.

Quality. All evaluations are served through the same endpoints, with greedy decoding, and every delta is against the unmodified model on the same job. LongBench [1]: eleven English tasks, 100 examples each, inputs middle-truncated to 250,000 characters, task-native metrics, macro-averaged. LongBench v2 [2]: multiple-choice accuracy on the full 503-question set, inputs middle-truncated to 120K tokens, the official prompt, answer budget and answer extraction, each model’s non-thinking chat format (Section H). RULER [8]: needle retrieval, variable tracking and common-word extraction at 4K to 128K tokens, the benchmark’s own metrics. Needle retrieval: ten numeric needles in haystacks of 32K to 520K characters. Decode perplexity: perplexity of the model’s own greedy 512-token continuation after 4K to 64K tokens of GovReport text, five seeds; a deviation in either direction is drift.

Results

One free halving of retention

Our main result is per-request decode throughput, IndexCache and IndexCache+MLI side by side at every retention and context (Table 1), at a single concurrent request. Each model uses its best realization of MLI (§ 5.3): the fused decode kernel on DeepSeek-V3.2 and the deferred-stream schedule on GLM-5, where the fused kernel does not help; both attend exactly the selections the serial schedule would. We read the table two ways. Across a row, the composition is faster than IndexCache at equal retention. The gain is largest where IndexCache keeps the most indexers: at $1/2$ the composition decodes up to 14.5% faster than IndexCache on DeepSeek-V3.2 and 9.8% on GLM-5, at $1/4$ up to 7.9% and 5.1%, and it narrows toward parity at $1/32$ as fewer scans remain to hide. It grows with context and holds on IndexCache’s searched patterns. Down a column, IndexCache+MLI at $1/r$ is faster than IndexCache at $1/2r$ : at 128K and $1/4$ it is 3% faster than IndexCache at $1/8$ (1.32× the unmodified model against IndexCache’s 1.28×) despite half the retained indexers, and the same holds at $1/2$ and $1/8$ (on GLM-5 by 2 to 3%). Which mechanism buys the halving differs by model: deferred overlap on GLM-5, the fused kernel on DeepSeek-V3.2 (§ 5.3). Prefill latency, which MLI leaves unchanged, is in Section E.

DeepSeek-V3.2 retention	10K IndexCache	10K +MLI	60K IndexCache	60K +MLI	120K IndexCache	120K +MLI	128K IndexCache	128K +MLI
none (base, tok/s)	89.1		88.0		85.7		85.5
all (MLI alone)	+21.5		+19.8		+26.1		+25.1
$1/2$	+11.5	+25.3	+12.3	+25.2	+13.6	+30.1	+13.7	+29.8
$1/4$	+19.1	+27.3	+20.5	+28.3	+22.8	+32.5	+22.7	+32.3
$1/8$	+23.4	+28.5	+25.2	+29.8	+27.9	+33.7	+28.0	+33.7
$1/16$	+25.3	+26.3	+27.1	+28.2	+30.2	+31.6	+30.2	+31.7
$1/32$	+26.1	+26.9	+28.2	+28.5	+31.4	+32.2	+31.4	+32.2
GLM-5 retention	10K IndexCache	10K +MLI	60K IndexCache	60K +MLI	120K IndexCache	120K +MLI	195K IndexCache	195K +MLI
none (base, tok/s)	68.5		67.8		66.5		65.0
all (MLI alone)	+14.6		+14.4		+15.6		+16.8
$1/2$	+8.2	+17.6	+8.7	+17.9	+9.8	+19.8	+10.6	+21.4
$1/4$	+13.7	+18.8	+14.2	+19.2	+15.9	+21.6	+17.7	+23.7
$1/8$	+16.3	+19.4	+17.1	+20.4	+19.3	+22.8	+21.3	+25.0
$1/16$	+18.1	+20.0	+18.9	+20.9	+21.2	+23.3	+23.4	+25.7
$1/32$	+18.8	+20.2	+19.7	+21.1	+22.1	+23.5	+24.4	+25.8
$1/2$ searched	+8.4	+17.2	+8.7	+17.5	+9.8	+19.3	+10.9	+21.1
$1/4$ searched	+13.4	+18.5	+13.9	+19.1	+15.5	+21.4	+17.2	+23.4

Table 1. Per-request decode throughput at single concurrency. Base row: tokens/s of the unmodified model; all other cells: percentage change over base in the same cell, positive = faster, for IndexCache alone and for IndexCache+MLI at the same retention. All arms of a cell run in one job on one node. $1/16$ and $1/32$ extend IndexCache’s published sweep ( $1/2$ – $1/8$ ). DeepSeek-V3.2 +MLI cells use the fused-kernel realization (§ 5.3); GLM-5 uses the deferred-stream realization, which is faster there; both are selection-identical to the production top- $k$ .

Full KV cache. Table 5 (appendix) repeats the sweep with the KV cache full; the gain from MLI tracks the number of requests decoding together, largest where the context is long enough to hold few (Section D).

DeepSeek-V3.2 retention	10K IndexCache	10K +MLI	60K IndexCache	60K +MLI	120K IndexCache	120K +MLI	128K IndexCache	128K +MLI
none (base, tok/s)	1760.5		494.7		316.3		314.2
requests in flight	64		13		6		6
all (MLI alone)	-2.2		+4.3		+5.7		+9.8
1/2	+3.1	+1.9	+8.1	+10.5	+12.3	+16.3	+12.1	+18.4
1/4	+6.1	+2.7	+13.2	+12.6	+20.7	+20.1	+21.4	+23.6
1/8	+6.5	+2.6	+15.1	+12.4	+24.3	+24.2	+25.4	+24.7
1/16	+2.0	+0.1	+13.3	+12.8	+24.7	+25.5	+26.1	+26.9
1/32	+0.8	+0.4	+13.2	+12.3	+25.8	+26.6	+27.5	+27.5
GLM-5 retention	10K IndexCache	10K +MLI	60K IndexCache	60K +MLI	120K IndexCache	120K +MLI	195K IndexCache	195K +MLI
none (base, tok/s)	1129.6		322.2		158.2		103.0
requests in flight	43		7		3		2
all (MLI alone)	+4.4		+7.1		+5.6		+11.3
1/2	+5.7	+6.4	+9.1	+12.8	+10.6	+13.7	+11.4	+18.3
1/4	+8.2	+10.1	+14.7	+16.7	+15.9	+17.9	+18.6	+22.9
1/8	+9.2	+9.1	+17.6	+18.2	+19.8	+20.6	+22.6	+24.6
1/16	+7.3	+8.5	+23.2	+23.9	+23.3	+24.0	+24.9	+25.5
1/32	+8.5	+8.8	+27.8	+27.0	+25.3	+24.6	+25.9	+27.0
1/2 searched	+6.4	+6.2	+9.5	+11.1	+10.7	+14.1	+11.6	+17.2
1/4 searched	+9.4	+9.6	+14.6	+14.4	+16.6	+18.1	+17.9	+21.4

Table 5. Full-KV-cache decode throughput. Base row: aggregate tokens/s of the unmodified model with the request set sized to the KV cache (the number of requests decoding together is given per context); other cells: percentage change over base, positive = faster. Reference is the closing base run of the cell where the opening run was cold by more than 2%.

What limits the gain

With every indexer retained ( $r=1$ ) and one request, MLI raises decode throughput by 19.8 to 26.1% on DeepSeek-V3.2 (the fused kernel) and 14.4 to 16.8% on GLM-5 (the deferred stream) across 10K to 195K (Table 1, second row). The ceiling, obtained by removing every indexer that can be removed (IndexCache at $1/32$ retains two), is +31% on DeepSeek-V3.2 and +24% on GLM-5 at the longest context, so MLI hides about four-fifths of the exposed scan on DeepSeek-V3.2 and about two-thirds on GLM-5. The first full layer’s scan stays serial (no earlier layer to overlap), so even the ceiling is not fully reachable. Two ablations (Table 3) rule out scheduling as the further limit: a high-priority CUDA stream and a deeper $k=2$ or 3 window both change nothing. What remains is memory bandwidth: the scan hides only in the HBM bandwidth attention and the feed-forward block leave idle, which is why the gain vanishes once requests saturate it (Table 5, 64 requests; not profiled directly).

	DeepSeek-V3.2		GLM-5
	10K	128K	10K	195K
base (tok/s)	86.1	82.9	69.2	65.8
MLI depth 1	+22.0	+21.4	+13.6	+15.8
MLI depth 1, high-priority stream	+22.0	+21.0	+13.5	+15.6
MLI depth 2	+21.9	+21.8	+14.2	+16.4
MLI depth 3	+21.9	+21.8	+14.8	+16.8

Table 3.

A fused kernel where bandwidth allows

MLI has two realizations, both attending the production top- $k$ . The deferred stream changes no kernel and issues the next layer’s indexer on a second CUDA stream; the fused kernel runs the sparse attention and the prefix scan of the shared step in one launch, so the scan never competes for SMs with a separate launch. On DeepSeek-V3.2 the fused kernel is faster at every retention and context, by 4 to 6 percentage points (composition at $1/4$ , 128K: +32.3% against +27.5%), and Table 1 uses it there. On GLM-5 it loses 2 to 4 percentage points to the deferred stream, which Table 1 uses instead: GLM-5 has half as many index heads per rank and a smaller per-layer feed-forward window, so the single launch cannot hide the scan. Selections are bit-identical, so quality (§ 5.4) is unchanged. The fused kernel is per-request only: at full-KV-cache batch sizes it is issue-bound (about 20% slower than the stream at 64 requests), so Table 5 uses the deferred stream.

What MLI costs

The only behavioral change is which selection a layer attends. Table 2 pairs the two methods at the retentions IndexCache reports, under the serving protocol where every retained layer computes its selection at prefill and copies only at decode (Section H). With every indexer retained, MLI alone is within 0.4–0.8 points of base on LongBench and LongBench v2, and on RULER 2.1 below on DeepSeek-V3.2 / 2.3 above on GLM-5, inside the models’ own base spread. Retention is cheap at $1/2$ and $1/4$ : both arms stay within about 3 points of base on every suite. On top of IndexCache, MLI costs at most a point or two (Table 2); the paired 95% intervals resolve differences that small only on RULER, spanning zero on LongBench and LongBench v2 (§ 17).

DeepSeek-V3.2 retention	LongBench IndexCache	LongBench +MLI	RULER IndexCache	RULER +MLI	LongBench v2 IndexCache	LongBench v2 +MLI
none (base, score)	0.449		0.975		0.501
all (MLI alone)	-0.0		-2.1		-0.4
all (MLI alone), distilled	-0.4		-4.7		-4.4
$1/2$	-0.2	-0.3	-2.7	-4.0	+0.8	+0.4
$1/2$ distilled	-0.0	-0.8	-1.1	-3.7	-0.8	+0.0
$1/4$	-0.3	+0.5	-0.9	-0.4	-1.6	-0.4
$1/4$ distilled	+0.2	-0.4	-2.0	-4.0	-0.8	-0.8
$1/8$	-0.1	-1.1	-0.7	+0.3	-0.4	+0.4
$1/8$ distilled	-0.6	-0.5	-0.9	-2.7	+0.4	-0.4
GLM-5 retention	LongBench IndexCache	LongBench +MLI	RULER IndexCache	RULER +MLI	LongBench v2 IndexCache	LongBench v2 +MLI
none (base, score)	0.453		0.941		0.477
all (MLI alone)	-0.2		+2.3		-0.8
all (MLI alone), distilled	-1.3		+2.9		+0.8
$1/2$	-1.2	-1.1	+2.9	+2.1	+1.2	+0.4
$1/2$ distilled	-0.2	+0.4	+2.0	+2.2	+2.9	-2.1
$1/4$	-0.2	-0.4	+0.7	+1.6	+2.5	-0.4
$1/4$ distilled	-0.8	+0.9	+0.9	-1.1	-0.8	+2.1
$1/8$	-0.0	-1.4	+1.6	-1.3	+0.8	-1.7
$1/8$ distilled	-0.4	+0.4	+1.2	+1.2	+0.0	-2.1

Table 2. Quality at the retentions IndexCache reports, IndexCache and IndexCache+MLI side by side. Base row: absolute score of the unmodified model; other cells: percentage change over base, positive = better, with the better arm of each IndexCache/+MLI pair in bold. LongBench: macro-F1 over 11 tasks; RULER: mean over tasks and lengths 4K–128K; LongBench v2: accuracy on the full 503-question set (one question is 0.2 points, so differences under a point are within its resolution). Distilled rows: indexer-only distillation, same recipe and budget for both methods. Needle retrieval is 1.00 for every arm shown (it collapses only at $1/16$ – $1/32$ , outside this table; see Table 8) and decode perplexity is within ±6% (±11% on GLM-5) of base; Table 8 has every arm, metric and retention.

Recovering it

Distillation earns its place unevenly across the retention range, and where it does is exactly where the free halving matters most. At the mild retentions IndexCache reports the cost is already small, so distilling only the indexers (the recipe of Section 3.3, applied identically to both methods) has correspondingly little to recover and is neutral to slightly negative: distilled and undistilled arms are within 2.5 points of each other on LongBench and LongBench v2, and on RULER the distilled composition on DeepSeek-V3.2 sits 3 to 4 points below the unmodified model against 1 to 2 for distilled IndexCache (the intervals in Section I exclude zero for the former and not the latter). At the cliff it is decisive: at $1/16$ on GLM-5, distilling the indexers turns IndexCache’s collapse (LongBench −11.5, RULER −61, needle 0.00) into a recovery (LongBench −0.1, RULER −7.4, needle 1.00; Table 8), moving the usable range one halving further. Distillation swaps only indexer weights, so decode speed is unchanged. One mild-retention cell moves more (GLM-5 at $1/2$ on LongBench v2, −2.5; a single-seed swing).

DeepSeek-V3.2	LongBench	NIAH	PPL	RULER	LongBench v2
base	0.449	1.000	1.167	0.975	0.501
MLI	0.448 (-0.0%)	1.00	1.176 (+0.8%)	0.954 (-2.1%)	0.499 (-0.4%)
MLI (distilled)	0.447 (-0.4%)	1.00	1.203 (+3.1%)	0.929 (-4.7%)	0.479 (-4.4%)
IC $_{1/2}$	0.448 (-0.2%)	1.00	1.166 (-0.1%)	0.949 (-2.7%)	0.505 (+0.8%)
IC $_{1/2}$ (distilled)	0.448 (-0.0%)	1.00	1.160 (-0.6%)	0.965 (-1.1%)	0.497 (-0.8%)
IC $_{1/2}$ +MLI	0.447 (-0.3%)	1.00	1.132 (-3.0%)	0.936 (-4.0%)	0.503 (+0.4%)
IC $_{1/2}$ +MLI (distilled)	0.445 (-0.8%)	1.00	1.142 (-2.1%)	0.939 (-3.7%)	0.501 (+0.0%)
IC $_{1/4}$	0.447 (-0.3%)	1.00	1.193 (+2.3%)	0.966 (-0.9%)	0.493 (-1.6%)
IC $_{1/4}$ (distilled)	0.450 (+0.2%)	1.00	1.153 (-1.2%)	0.956 (-2.0%)	0.497 (-0.8%)
IC $_{1/4}$ +MLI	0.451 (+0.5%)	1.00	1.193 (+2.3%)	0.971 (-0.4%)	0.499 (-0.4%)
IC $_{1/4}$ +MLI (distilled)	0.447 (-0.4%)	1.00	1.162 (-0.4%)	0.936 (-4.0%)	0.497 (-0.8%)
IC $_{1/8}$	0.448 (-0.1%)	1.00	1.193 (+2.2%)	0.968 (-0.7%)	0.499 (-0.4%)
IC $_{1/8}$ (distilled)	0.446 (-0.6%)	1.00	1.168 (+0.1%)	0.967 (-0.9%)	0.503 (+0.4%)
IC $_{1/8}$ +MLI	0.444 (-1.1%)	1.00	1.149 (-1.5%)	0.978 (+0.3%)	0.503 (+0.4%)
IC $_{1/8}$ +MLI (distilled)	0.447 (-0.5%)	1.00	1.111 (-4.8%)	0.949 (-2.7%)	0.499 (-0.4%)
IC $_{1/16}$	0.445 (-0.7%)	1.00	1.175 (+0.7%)	0.961 (-1.4%)	0.473 (-5.6%)
IC $_{1/16}$ (distilled)	0.447 (-0.4%)	1.00	1.156 (-0.9%)	0.964 (-1.1%)	0.489 (-2.4%)
IC $_{1/16}$ +MLI	0.450 (+0.3%)	1.00	1.144 (-2.0%)	0.937 (-3.9%)	0.465 (-7.1%)
IC $_{1/16}$ +MLI (distilled)	0.447 (-0.3%)	1.00	1.142 (-2.1%)	0.958 (-1.7%)	0.493 (-1.6%)
IC $_{1/32}$	0.437 (-2.7%)	0.11	1.140 (-2.3%)	0.891 (-8.6%)	0.352 (-29.8%)
IC $_{1/32}$ (distilled)	0.444 (-1.1%)	0.89	1.166 (-0.1%)	0.912 (-6.4%)	0.475 (-5.2%)
IC $_{1/32}$ +MLI	0.426 (-5.2%)	0.11	1.155 (-1.1%)	0.817 (-16.2%)	0.314 (-37.3%)
IC $_{1/32}$ +MLI (distilled)	0.443 (-1.3%)	0.22	1.141 (-2.2%)	0.776 (-20.4%)	0.467 (-6.7%)
GLM-5	LongBench	NIAH	PPL	RULER	LongBench v2
base	0.453	1.000	1.163	0.941	0.477
MLI	0.452 (-0.2%)	1.00	1.061 (-8.8%)	0.963 (+2.3%)	0.473 (-0.8%)
MLI (distilled)	0.447 (-1.3%)	1.00	1.083 (-6.9%)	0.969 (+2.9%)	0.481 (+0.8%)
IC $_{1/2}$	0.448 (-1.2%)	1.00	1.061 (-8.7%)	0.969 (+2.9%)	0.483 (+1.2%)
IC $_{1/2}$ (distilled)	0.452 (-0.2%)	1.00	1.082 (-7.0%)	0.960 (+2.0%)	0.491 (+2.9%)
IC $_{1/2}$ +MLI	0.448 (-1.1%)	1.00	1.081 (-7.1%)	0.961 (+2.1%)	0.479 (+0.4%)
IC $_{1/2}$ +MLI (distilled)	0.455 (+0.4%)	1.00	1.108 (-4.7%)	0.962 (+2.2%)	0.467 (-2.1%)
IC $_{1/2}^{\mathrm{searched}}$	0.454 (+0.3%)	1.00	1.078 (-7.4%)	0.956 (+1.5%)	0.473 (-0.8%)
IC $_{1/2}^{\mathrm{searched}}$ +MLI	0.445 (-1.7%)	1.00	1.087 (-6.6%)	0.951 (+1.0%)	0.477 (+0.0%)
IC $_{1/4}$	0.452 (-0.2%)	1.00	1.074 (-7.7%)	0.948 (+0.7%)	0.489 (+2.5%)
IC $_{1/4}$ (distilled)	0.449 (-0.8%)	1.00	1.105 (-5.0%)	0.950 (+0.9%)	0.473 (-0.8%)
IC $_{1/4}$ +MLI	0.451 (-0.4%)	1.00	1.132 (-2.6%)	0.957 (+1.6%)	0.475 (-0.4%)
IC $_{1/4}$ +MLI (distilled)	0.457 (+0.9%)	1.00	1.090 (-6.2%)	0.931 (-1.1%)	0.487 (+2.1%)
IC $_{1/4}^{\mathrm{searched}}$	0.454 (+0.1%)	1.00	1.086 (-6.6%)	0.971 (+3.2%)	0.475 (-0.4%)
IC $_{1/4}^{\mathrm{searched}}$ +MLI	0.450 (-0.8%)	1.00	1.082 (-7.0%)	0.958 (+1.8%)	0.483 (+1.2%)
IC $_{1/8}$	0.453 (-0.0%)	1.00	1.076 (-7.4%)	0.957 (+1.6%)	0.481 (+0.8%)
IC $_{1/8}$ (distilled)	0.451 (-0.4%)	1.00	1.094 (-5.9%)	0.952 (+1.2%)	0.477 (+0.0%)
IC $_{1/8}$ +MLI	0.447 (-1.4%)	1.00	1.084 (-6.8%)	0.929 (-1.3%)	0.469 (-1.7%)
IC $_{1/8}$ +MLI (distilled)	0.455 (+0.4%)	1.00	1.110 (-4.6%)	0.953 (+1.2%)	0.467 (-2.1%)
IC $_{1/16}$	0.401 (-11.5%)	0.00	1.827 (+57.1%)	0.365 (-61.3%)	0.378 (-20.8%)
IC $_{1/16}$ (distilled)	0.453 (-0.1%)	1.00	1.105 (-5.0%)	0.871 (-7.4%)	0.485 (+1.7%)
IC $_{1/16}$ +MLI	0.405 (-10.6%)	0.00	1.894 (+62.8%)	0.541 (-42.6%)	0.388 (-18.8%)
IC $_{1/16}$ +MLI (distilled)	0.448 (-1.2%)	1.00	1.095 (-5.9%)	0.806 (-14.4%)	0.483 (+1.2%)
IC $_{1/32}$	0.375 (-17.3%)	0.00	2.499 (+114.9%)	0.894 (-47.5%)	0.336 (-29.6%)
IC $_{1/32}$ (distilled)	0.450 (-0.8%)	1.00	1.093 (-6.0%)	0.806 (-14.4%)	0.461 (-3.3%)
IC $_{1/32}$ +MLI	0.373 (-17.6%)	0.00	2.501 (+115.0%)	0.653 (-30.6%)	0.342 (-28.3%)
IC $_{1/32}$ +MLI (distilled)	0.445 (-1.8%)	0.78	1.124 (-3.4%)	0.839 (-10.9%)	0.427 (-10.4%)

Table 8. Quality vs the unmodified model (same job, same endpoints, greedy). LongBench: macro-F1 over 11 tasks; NIAH: worst score over haystacks up to 520K characters and both needle modes; PPL: perplexity of the model’s own greedy continuation, geometric mean over 4K–64K inputs and five seeds (a deviation in either direction is drift from the unmodified model); RULER: mean over tasks and lengths 4K–128K; LongBench v2: accuracy on the full 503-question set. Distilled rows use indexer-only span-aware distillation with the same recipe for both methods. A missing entry (–) is an evaluation whose server ran out of memory during prefill in two attempts.

At equal speed

The halving pair no longer runs at equal speed (the composition at $1/r$ is the faster arm; Table 1), and it gives up nothing in quality to buy that speed: both arms sit within a few points of base, and every 95% interval spans zero (§ 17, Table 9) except one RULER pair on DeepSeek-V3.2 ( $-3.1$ , [ $-5.9$ , $-0.6$ ]). So within the retentions IndexCache reports the extra freshness comes free; one halving beyond, at the cliff (§ 5.5), it also avoids collapse, as IndexCache degenerates at $1/16$ on GLM-5 and $1/32$ on DeepSeek-V3.2 while the composition one halving above does not (Table 8). These are single runs.

arm	decode	undistilled			distilled
		LongBench	RULER	LBv2	LongBench	RULER	LBv2
DeepSeek-V3.2, 128K
IndexCache $1/4$	+22.7	-0.3	-0.9	-1.6	+0.2	-2.0	-0.8
IndexCache+MLI $1/2$	+29.8	-0.3	-4.0	+0.4	-0.8	-3.7	+0.0
IndexCache $1/8$	+28.0	-0.1	-0.7	-0.4	-0.6	-0.9	+0.4
IndexCache+MLI $1/4$	+32.3	+0.5	-0.4	-0.4	-0.4	-4.0	-0.8
IndexCache $1/16$	+30.2	-0.7	-1.4	-5.6	-0.4	-1.1	-2.4
IndexCache+MLI $1/8$	+33.7	-1.1	+0.3	+0.4	-0.5	-2.7	-0.4
GLM-5, 195K
IndexCache $1/4$	+17.7	-0.2	+0.7	+2.5	-0.8	+0.9	-0.8
IndexCache+MLI $1/2$	+21.4	-1.1	+2.1	+0.4	+0.4	+2.2	-2.1
IndexCache $1/8$	+21.3	-0.0	+1.6	+0.8	-0.4	+1.2	+0.0
IndexCache+MLI $1/4$	+23.7	-0.4	+1.6	-0.4	+0.9	-1.1	+2.1
IndexCache $1/16$	+23.4	-11.5	-61.3	-20.8	-0.1	-7.4	+1.7
IndexCache+MLI $1/8$	+25.0	-1.4	-1.3	-1.7	+0.4	+1.2	-2.1

Table 9. The halving pair: IndexCache at $1/2r$ and IndexCache+MLI at $1/r$ , the latter with twice the fresh selections. Decode: per-request throughput change over base at the longest context of each model (IndexCache+MLI at $1/r$ is faster than IndexCache at $1/2r$ , not merely equal); quality: percentage change over base, positive = better.

Related Work

Sparse attention and selection. Fixed-pattern sparse attention [5, 4, 22] attends a static subset; the learned variants adapt it: NSA [21], MoBA [13], InfLLM-V2 [27] and SeerAttention [7] learn to select blocks or tokens, DSA [12] with a per-layer lightning indexer; Quest [18] estimates importance from cached key statistics and MInference [9] from offline-assigned per-head sparse patterns; eviction and offloading [26, 19, 11, 28, 14] shrink or relocate the cache rather than the read; and Sadhukhan et al. [2026] argue that at test time sparse attention sets the cost per token. All of these run selection before attention. Distillation into sparse attention has targeted whole attention layers [17]; we distill only the indexers of a model that already is sparse, toward its own selections. IndexCache [3] shares one DSA layer’s selection with neighbouring layers; TidalDecode [20] and Kascade [6] reuse attention-derived selections across layers in models without an indexer. These reduce how often selection runs; MLI reduces how long a layer waits for it, and composes with them.

Selection and attention as one primitive. Attention’s logit and the indexer’s score are both low-rank bilinear interactions of the layer input, so the indexer is a second consumer of what attention already computes. Selective Attention [10] makes a related point at training time: reusing attention’s own scores as the selection signal rather than a separate module. MLI exploits only the cross-layer overlap of the resulting selections, but our serving results argue for the tighter co-design: on DeepSeek-V3.2 a single fused kernel computes the attention and the indexer scan in one launch. Training a model whose indexer and attention share that primitive from the start is the natural next step.

Limitations

The gain of MLI falls with the number of requests decoding together. The fused kernel helps only DeepSeek-V3.2; on GLM-5 it loses to the deferred stream (§ 5.3). We evaluate GLM-5 as the family’s representative: GLM-5.1 shares its base model and DSA architecture (only the checkpoint differs), so our results carry over; GLM-5.2 ships IndexCache natively as IndexShare, so a comparison there would not be on equal footing and we leave it out of scope.

Conclusion

DSA places a lightning indexer in front of every attention layer, and the indexer scan is a serial dependency of the step it feeds. It need not be. Adjacent DSA layers select almost the same tokens, so a layer can attend to the selection an earlier indexer already produced and run its own scan alongside attention rather than before it. The change is drop-in: no retraining, no new weights, and it composes with IndexCache. Serving DeepSeek-V3.2 and GLM-5, it makes IndexCache faster at every retention, by up to 14.5% in decode throughput where the indexers are densest, and buys a further halving of the indexer budget for free, since at half the retention the composition is already faster than IndexCache while attending twice as many fresh selections. The indexer only recomputes the bilinear interaction attention already forms, so the scan is not a separate cost to hide but a second reader of one primitive; the step this invites is to co-design DSA around that primitive rather than place a selector ahead of it.

Serving Protocol and Gates

Server. SGLang v0.5.16 with our overlay, --tp-size 8, --mem-fraction-static 0.83, decode CUDA graphs captured for batch sizes 1, 2, 4 (per-request cells) or 1 to 64 (full-KV-cache cells), context limit 163,840 (DeepSeek-V3.2) or 202,752 (GLM-5), one server boot per arm. Method arms differ from base only in JSON model-override arguments (index_topk_freq, index_topk_pattern, index_reuse_lag_refresh, index_reuse_lag_refresh_deferred, index_reuse_lag_depth, index_distilled_weights).

Regimes. Per-request: sglang.benchmark.one_batch_server at batch size 1, input length $L$ , 128 output tokens; we report the last decode throughput and the last time-to-first-token. Full KV cache: the request set is sized to the server’s KV capacity, $n=\lfloor\operatorname{max\_total\_tokens}/(L+1024)\rfloor$ capped at 64; a first bench_serving pass with the same prompts warms them into the radix cache, and a second pass decodes 1024 tokens per request at concurrency $n$ ; we report the aggregate decode throughput of the second pass. (bench_serving’s raw output throughput is prefill-dominated at 128K and is not reported.)

Gates. Every cell (one model, one regime, one context) runs all of its arms in one Slurm job on one node, in the order base, method arms, base again. A number is reportable only if the cell passes all of: (S) the result file exists and reports a positive steady throughput; (A) the closing base repeat is within 2% of the opening one; (E) engagement: the server log contains the marker proving the intended path ran (first lag_refresh_deferred for MLI, first lag_depth_k for depth $k$ ) and none of the fallback strings that the overlay emits when a path is bypassed; (I) plausibility: an arm that only removes or hides a scan cannot exceed the indexer-free ceiling and cannot be slower than base by more than a tolerance; (C) the overlay build’s base matches the unmodified release on the same node (Section C). Verdicts are written next to the cell; the table generator reads only PASS arms from cells whose two base runs pass (S), (E) and (A).

Selection identity. The stream realization runs the production scan and top- $k$ kernels on the production buffers, so its selections are the serial schedule’s by construction.

Ablations: Stream Priority and Lag Depth

Depth 2 and 3 track depth 1 within about a percentage point at $r=1$ (above). We do not characterize quality for $k>1$ : since deeper windows give no speed benefit, there is no reason to trade selection freshness for a larger lag. The indexer-free ceiling (+31% DeepSeek-V3.2, +24% GLM-5; Table 1, $1/32$ ) sits well above every row here, so what the scheduling ablations leave on the table is not scheduling.

IndexCache Parity Control

Our build is within 0.2% of the release on base and 0.4 to 1.1% slower on the IndexCache arms. The IndexCache numbers in the main tables are from our build, so they understate stock IndexCache by Table 3: MLI scheduling ablation with every indexer retained ( $r=1$ ), the deferred-stream realization on the same overlay and recipe as Table 1 (HH_LAG_NO_ALT). Per-request decode throughput relative to the unmodified model on the same node, drift-gated (base within 2% of a closing base), at the two ends of the context range. Deepening the lag window to $k=2$ or 3 full layers and giving the indexer a high-priority CUDA stream both leave the gain unchanged, so scheduling is not the limit. The DeepSeek cells here ran on nodes with a lower absolute base than Table 1 (82.9–86.1 tok/s vs $\sim85.5$ tok/s), so their percentages run a few percentage points high; the deferred arm reaches the same absolute throughput ( $\sim100.7$ tok/s at 128K, matching § 5.3).

Table 4: Unmodified SGLang release (stock) vs our build (ours), same job and node, per-request decode tok/s; IndexCache arms use the release’s own index_topk_freq. Closing base sentinels: 0.0 to 0.2%.

	DeepSeek-V3.2				GLM-5
	10K		128K		10K		128K
	stock	ours	stock	ours	stock	ours	stock	ours
base	89.4	89.4	85.8	85.6	68.4	68.2	66.3	66.1
IndexCache $1/2$	100.6	99.6	98.2	97.4	74.8	74.0	73.5	72.7
IndexCache $1/4$	107.3	106.2	106.1	105.0	78.0	77.7	77.2	76.9

Table 4.

up to 1.1 percentage points; the equal-retention margins in Table 1 exceed that comfortably at $1/2$ to $1/8$ and shrink to within it at $1/16$ and $1/32$ . The down-column result survives the correction: even against stock IndexCache, IndexCache+MLI at $1/r$ is faster than stock IndexCache at $1/2r$ , by 2 to 6 percentage points on DeepSeek-V3.2 and by up to 3 on GLM-5.

Full-KV-Cache Throughput

Table 5 (appendix) repeats the sweep with the KV cache full. The number of requests decoding together is set by the context (64 at 10K down to 6 at 128K on DeepSeek-V3.2, 43 down to 2 on GLM-5), and the MLI gain follows it. At 64 requests MLI alone is 2.2% slower than the unmodified model and the composition trails IndexCache by up to 4 points; at every lower concurrency (43 and below) MLI alone is positive (4.3 to 11.3%), the composition is within about 3 points of IndexCache either way, and at the longest context it runs up to 6 points above IndexCache at $1/2$ . The single-request gain therefore carries to the saturated server wherever the context holds few requests, the regime where the scan dominates the step; only when many short requests pack together does the feed-forward block saturate bandwidth by itself and the gain vanish (§ 5.2).

Prefill Latency

MLI acts on decode only; prefill runs the standard schedule, so the composition inherits IndexCache’s prefill gain (from skipping the indexer on shared layers during prefill) and adds nothing to it; time-to-first-token matches IndexCache within measurement noise.

Attention Data-Parallel Replication

IndexCache reports its numbers under attention data-parallelism. We replicate our cells under --enable-dp-attention --dp-size 8 (attention and indexer of a request on one GPU, MoE tensor-parallel across eight) with the same gates. Table 6 gives the per-request result; at single concurrency this layout leaves seven attention ranks idle, so base is slower than under tensor-parallel attention

DeepSeek-V3.2 retention	128K IndexCache	128K +MLI	GLM-5 retention	195K IndexCache	195K +MLI
none (base, tok/s)	56.6		none (base, tok/s)	47.5
all (MLI alone)	+3.9		all (MLI alone)	+5.8
1/2	+8.9	+10.5	1/2	+6.6	+10.3
1/4	+13.3	+14.3	1/4	+10.7	+11.8

Table 6. Replication under attention data-parallelism (–enable-dp-attention –dp-size 8, IndexCache’s parallelism): per-request decode throughput (tokens/s, single concurrency) relative to the unmodified model in the same cell; positive = faster. Decode throughput is the client-side output tokens / (latency − TTFT), which agrees with the server-side metric within 0.5% on the TP cells.

(56.6 vs 85.5 tok/s at 128K) and every indexer remedy hides or removes a smaller share of the step: MLI +3.9%, IndexCache 1/4 +13.3%, the composition +14.3%. The composition remains additive over IndexCache at every retention measured. tbl:7 gives full-KV-cache throughput under this layout.

Searched Patterns

We use IndexCache’s published GLM-5 searched patterns [3] (40 and 22 full layers of 78 at 1/2 and 1/4), applying MLI to the same patterns.

Full Quality Table and Frontier

LongBench v2 protocol. We follow the official evaluation script’s non-CoT setting: its prompt, a 128-token answer budget, and its extraction of “The correct answer is (X)”, with greedy decoding where the script samples at temperature 0.1. An answer without an extractable letter is scored as wrong, as in the script. Under this budget 6% of the unmodified DeepSeek-V3.2 model’s answers on

DeepSeek-V3.2	128K			GLM-5	195K
retention		IndexCache	+MLI	retention		IndexCache	+MLI
none (base, tok/s)		65.0		none (base, tok/s)		12.3
requests in flight		4		requests in flight		1
all (MLI alone)		+2.3		all (MLI alone)		+1.6
$\frac{1}{2}$		+13.6	+12.6	$\frac{1}{2}$		+11.8	+10.7
$\frac{1}{4}$		+21.2	+21.4	$\frac{1}{4}$		+14.8	+15.6
$\frac{1}{8}$		+25.6	+25.2	$\frac{1}{8}$		+18.3	+18.3
$\frac{1}{16}$		+26.1	+27.5	$\frac{1}{16}$		+20.8	+19.9
$\frac{1}{32}$		+28.9	+29.5	$\frac{1}{32}$		+21.5	+20.6

caption: tbl:7: Replication under attention data-parallelism: full-KV-cache aggregate decode throughput relative to the unmodified model; positive = faster.

:::

the full 503-question set are unparsed because the model begins a step-by-step explanation and is cut off; this applies to every arm alike and is part of the reported accuracy.

Serving protocol for reuse arms. Every arm that retains a fraction of the indexers runs with each retained layer computing its own selection at prefill and copying it forward only at decode (index_prefill_full_indexer), the configuration appropriate to models that ship indexer weights at every layer. Copying a stale selection at prefill as well corrupts the context encoding at aggressive retention and long context; that configuration is not reported.

Thinking mode. LongBench v2 is evaluated with each model’s non-thinking format. A control on the first 50 questions (4096-token budget) showed no lift for DeepSeek-V3.2 (0.52 thinking vs 0.62 non-thinking, 9 traces truncated) and, for GLM-5, reasoning that overran the budget on half the questions; the sweep was therefore run without thinking.

Frontier. Figure 4 plots per-request decode speedup against LongBench for IndexCache and IndexCache+MLI at every retention, undistilled and distilled, at the longest context of each model: the composition sits up and to the left of IndexCache at matched retention, and one halving further in it avoids the quality collapse IndexCache falls into.

Decode speedup over base and LongBench change versus base for DeepSeek-V3.2 and GLM-5 — Figure 4. What adding MLI to IndexCache does at each retention, at the longest measured context of each model. Left: per-request decode speedup over the unmodified model; right: LongBench macro-F1 change (symmetric-log axis so the 1/32 collapse fits). Light: IndexCache alone; dark: IndexCache+MLI at the same retention; hollow: distilled arms.

Confidence Intervals for the Quality Contrasts

Tables 10 and 11 give paired bootstrap 95% intervals for every quality contrast the text makes: each arm against the unmodified model, IndexCache+MLI against IndexCache at equal retention, and IndexCache+MLI at $1/r$ against IndexCache at $1/2r$ (equal speed). The resample is over examples and is applied to both arms of a contrast, so the interval is for the paired difference; intervals marked * exclude zero.

Measurement Noise and Compute

Three sources of variation bound every speed number. Within a boot: repeating the per-request measurement five times on one warm server gives a standard error of 0.01 to 0.03 tok/s per arm, so the equal-retention gaps carry 95% intervals of about $\pm0.1$ percentage point (DeepSeek-V3.2 at 128K: $1/4$ +4.70 $\pm$ 0.11, $1/8$ +2.36 $\pm$ 0.14; GLM-5 at 195K: $1/4$ +4.12 $\pm$ 0.06, $1/8$ +2.61 $\pm$ 0.06). Between boots in a cell: the closing run of the unmodified model is within 0.4% of the opening one in every certified cell. Between cells: each replicate cell and its sweep cell, run on different days and nodes, differ by 0.8% (DeepSeek-V3.2) and 0.1% (GLM-5) on the unmodified model and by up to 0.5 and 0.2 percentage point on the equal-retention gaps. The between-cell figure is the one to read against the tables; the equal-retention margins at $1/2$ to $1/8$ exceed it several times over.

All experiments ran on nodes of eight H200 GPUs (141 GB each) on an internal Slurm cluster. The reported speed sweep (20 cells plus 4 data-parallel and 2 replicate-CI cells, at 2 to 4 node-hours each) is about 90 node-hours; the quality matrix (48 arms) about 75; and distillation (18 runs) about 70. Including the fused-kernel realization and cells voided by the gates, the project used roughly three times the reported compute.

Distillation Recipe

Let $C(f)$ be the consumers of full layer $f$ (§ 3.3). For a query row $t$ let $I^c_{t,:}$ be the unmodified model’s indexer scores at consumer $c$ and $\hat{I}^f_{t,:}$ , the scores of the indexer being trained at $f$ . The loss is

\mathcal{L}_f = \frac{1}{\lvert C(f)\rvert}\sum_{c\in C(f)}\left[\lambda\,\operatorname{KL}\left(\operatorname{softmax}(I^c_{t,:})\mathbin{\|}\operatorname{softmax}(\hat{I}^f_{t,:})\right)+(1-\lambda)\mathcal{L}_{\mathrm{PL}}\left(\pi^c_t;\hat{I}^f_{t,:}\right)\right], \tag*{(4)}

where $\mathcal{L}_{\mathrm{PL}}$ is the negative Plackett–Luce log-likelihood of the teacher’s ranking $\pi^c_t$ of its top- $k$ tokens under the student’s scores, with $\lambda=0.5$ and $k=2048$ . The KL term alone leaves the ranking near the selection boundary free; the listwise term fixes it there.

Only the lightning indexers of the retained (full) layers are trained; the backbone, attention and the shared layers’ behavior are frozen. Teacher signal: the unmodified model’s indexer logits at the consumer layers, captured by forward hooks in a frozen forward over the same input. For full layer $\ell$ with next full layer $\ell'$ , the consumers are layers $\ell+1,\ldots,\ell'-1$ under MLI (each attends $\mathcal{S}^{(\ell)}$ one full layer late) and $\ell,\ldots,\ell'-1$ under IndexCache (the retained layer also serves itself); the student at $\ell$ is trained against each consumer in its span, with the same per-consumer weight. Objective: 0.5 KL between the consumer’s softmaxed teacher scores and the student’s, plus 0.5 listwise Plackett–Luce likelihood of the teacher’s top-2048 ordering under the student (the hybrid objective), 512 query rows sampled per step, causal masking. Data: GovReport documents packed into 8 streams of 32,768 tokens (DeepSeek-V3.2) or 16,384 (GLM-5). Optimization: 300 steps, AdamW, learning rate $10^{-3}$ , no weight decay, gradient clipping at 1.0, seed 0, two H200 nodes per run. One run per (model, retention, span rule); the weights load at serving time through index_distilled_weights and change nothing else about the server. The identical recipe, data and budget are used for the IndexCache distilled arms, differing only in the span rule.

DeepSeek-V3.2	LongBench	RULER	LongBench v2
$IC_{1/16}$ (distilled) − base	-0.4 [-2.2, +1.3]	-1.1 [-3.3, +0.3]	-2.4 [-6.3, +1.6]
$IC_{1/2}$ (distilled) − base	-0.0 [-1.8, +1.6]	-1.1 [-3.9, +1.6]	-0.8 [-4.4, +2.8]
$IC_{1/32}$ (distilled) − base	-1.1 [-3.0, +0.7]	-6.4 [-9.9, -3.1]*	-5.2 [-9.5, -0.8]*
$IC_{1/4}$ (distilled) − base	+0.2 [-1.7, +2.1]	-2.0 [-4.5, +0.4]	-0.8 [-4.0, +2.4]
$IC_{1/8}$ (distilled) − base	-0.6 [-2.4, +1.2]	-0.9 [-2.5, +0.4]	+0.4 [-3.6, +4.4]
$IC_{1/16}$ − base	-0.7 [-2.4, +0.9]	-1.4 [-3.6, +0.2]	-5.6 [-10.7, -0.4]*
$IC_{1/2}$ − base	-0.2 [-2.0, +1.7]	-2.7 [-4.9, -0.5]*	+0.8 [-2.8, +4.4]
$IC_{1/32}$ − base	-2.7 [-4.7, -0.8]*	-8.6 [-11.9, -5.5]*	-29.8 [-38.1, -21.4]*
$IC_{1/4}$ − base	-0.3 [-1.9, +1.3]	-0.9 [-2.4, +0.1]	-1.6 [-4.8, +1.6]
$IC_{1/8}$ − base	-0.1 [-2.0, +1.8]	-0.7 [-2.2, +0.4]	-0.4 [-3.6, +2.8]
$IC_{1/16}$ +MLI (distilled) − base	-0.3 [-2.1, +1.4]	-1.7 [-3.8, -0.2]*	-1.6 [-5.6, +2.0]
$IC_{1/2}$ +MLI (distilled) − base	-0.8 [-2.6, +1.0]	-3.7 [-6.4, -1.2]*	+0.0 [-4.0, +4.0]
$IC_{1/32}$ +MLI (distilled) − base	-1.3 [-3.2, +0.5]	-20.4 [-23.2, -17.5]*	-6.7 [-11.5, -2.4]*
$IC_{1/4}$ +MLI (distilled) − base	-0.4 [-2.1, +1.2]	-4.0 [-7.0, -1.0]*	-0.8 [-4.0, +2.4]
$IC_{1/8}$ +MLI (distilled) − base	-0.5 [-2.2, +1.2]	-2.7 [-4.4, -1.1]*	-0.4 [-4.0, +3.2]
$IC_{1/16}$ +MLI − base	+0.3 [-1.6, +2.1]	-3.9 [-6.3, -1.7]*	-7.1 [-12.3, -2.0]*
$IC_{1/2}$ +MLI − base	-0.3 [-2.1, +1.4]	-4.0 [-7.1, -1.2]*	+0.4 [-2.0, +2.8]
$IC_{1/32}$ +MLI − base	-5.2 [-7.5, -2.9]*	-16.2 [-20.1, -12.3]*	-37.3 [-46.0, -28.6]*
$IC_{1/4}$ +MLI − base	+0.5 [-1.2, +2.2]	-0.4 [-3.2, +2.1]	-0.4 [-3.2, +2.4]
$IC_{1/8}$ +MLI − base	-1.1 [-2.9, +0.6]	+0.3 [-0.1, +0.7]	+0.4 [-2.8, +3.6]
MLI − base	-0.0 [-1.1, +1.0]	-2.1 [-4.1, -0.2]*	-0.4 [-3.2, +2.8]
MLI (distilled) − base	-0.4 [-2.0, +1.3]	-4.7 [-7.3, -2.3]*	-4.4 [-9.5, +0.8]
equal retention 1/2: +MLI − IndexCache	-0.2 [-1.2, +0.9]	-1.4 [-4.1, +1.1]	-0.4 [-3.6, +2.8]
equal retention 1/2: +MLI − IndexCache (distilled)	-0.7 [-2.2, +0.7]	-2.6 [-5.1, -0.3]*	+0.8 [-3.2, +4.8]
equal retention 1/4: +MLI − IndexCache	+0.8 [-0.1, +1.8]	+0.5 [-2.4, +3.2]	+1.2 [-2.4, +4.8]
equal retention 1/4: +MLI − IndexCache (distilled)	-0.7 [-1.9, +0.6]	-2.0 [-4.3, +0.0]	+0.0 [-4.0, +4.0]
equal retention 1/8: +MLI − IndexCache	-1.0 [-2.2, +0.0]	+1.0 [+0.0, +2.5]	+0.8 [-2.0, +3.6]
equal retention 1/8: +MLI − IndexCache (distilled)	+0.1 [-1.1, +1.3]	-1.8 [-4.2, +0.7]	-0.8 [-4.4, +2.8]
equal retention 1/16: +MLI − IndexCache	+1.0 [-0.3, +2.3]	-2.5 [-4.6, -0.6]*	-1.6 [-6.7, +3.6]
equal retention 1/16: +MLI − IndexCache (distilled)	+0.1 [-1.0, +1.2]	-0.6 [-1.2, +0.0]	+0.8 [-2.8, +4.4]
equal retention 1/32: +MLI − IndexCache	-2.5 [-4.3, -0.8]*	-7.6 [-11.4, -3.4]*	-7.5 [-15.5, +0.0]
equal retention 1/32: +MLI − IndexCache (distilled)	-0.2 [-1.6, +1.2]	-14.0 [-16.7, -11.6]*	-1.6 [-5.6, +2.4]
equal speed: IndexCache+MLI 1/2 − IndexCache 1/4	-0.0 [-1.1, +1.1]	-3.1 [-6.0, -0.7]*	+2.0 [-1.2, +5.2]
equal speed: IndexCache+MLI 1/2 − IndexCache 1/4 (distilled)	-1.0 [-2.5, +0.5]	-1.7 [-3.2, -0.4]*	+0.8 [-3.6, +5.2]
equal speed: IndexCache+MLI 1/4 − IndexCache 1/8	+0.6 [-0.6, +1.9]	+0.4 [-2.5, +3.1]	+0.0 [-3.2, +3.2]
equal speed: IndexCache+MLI 1/4 − IndexCache 1/8 (distilled)	+0.2 [-1.0, +1.3]	-3.2 [-6.6, +0.2]	-1.2 [-4.8, +2.0]
equal speed: IndexCache+MLI 1/8 − IndexCache 1/16	-0.4 [-1.5, +0.7]	+1.7 [+0.2, +3.8]*	+6.0 [+0.8, +11.1]*
equal speed: IndexCache+MLI 1/8 − IndexCache 1/16 (distilled)	-0.0 [-1.1, +1.0]	-1.5 [-3.4, +0.2]	+2.0 [-1.6, +5.6]
equal speed: IndexCache+MLI 1/16 − IndexCache 1/32	+3.0 [+1.3, +4.7]*	+4.7 [+1.5, +7.9]*	+22.6 [+14.3, +31.3]*
equal speed: IndexCache+MLI 1/16 − IndexCache 1/32 (distilled)	+0.8 [-0.4, +2.1]	+4.7 [+1.7, +7.4]*	+3.6 [-1.2, +8.3]

Table 10. DeepSeek-V3.2. Paired bootstrap 95% intervals for the contrasts in the text (percentage change over the unmodified model’s score; 10,000 resamples of examples, the same resample applied to both arms; LongBench resampled within each of its 11 tasks and RULER within each of its 30 task $\times$ length cells, both macro-averaged as in Table 8). *: interval excludes zero. Examples per arm: LongBench 1,100, RULER 234, LongBench v2 as stated in Table 8.

GLM-5	LongBench	RULER	LongBench v2
$IC_{1/16}$ (distilled) − base	-0.1 [-2.2, +2.0]	-7.4 [-10.2, -4.7]*	+1.7 [-2.9, +6.2]
$IC_{1/2}$ (distilled) − base	-0.2 [-2.2, +1.8]	+2.0 [+0.4, +3.6]*	+2.9 [-1.7, +7.9]
$IC_{1/32}$ (distilled) − base	-0.8 [-3.0, +1.5]	-14.4 [-18.6, -10.2]*	-3.3 [-9.6, +2.5]
$IC_{1/4}$ (distilled) − base	-0.8 [-2.9, +1.1]	+0.9 [-2.4, +4.1]	-0.8 [-5.0, +3.3]
$IC_{1/8}$ (distilled) − base	-0.4 [-2.4, +1.5]	+1.2 [-1.4, +3.7]	+0.0 [-4.6, +4.6]
$IC_{1/16}$ − base	-11.5 [-14.4, -8.5]*	-61.3 [-64.9, -57.7]*	-20.8 [-27.9, -13.8]*
$IC_{1/2}$ − base	-1.2 [-3.2, +0.9]	+2.9 [+0.9, +4.9] *	+1.3 [-2.1, +4.6]
$IC_{1/32}$ − base	-17.3 [-20.7, -13.9]*	-47.5 [-53.0, -42.2]*	-29.6 [-36.7, -22.1]*
$IC_{1/4}$ − base	-0.2 [-2.4, +1.9]	+0.7 [-1.3, +2.6]	+2.5 [-1.7, +6.7]
$IC_{1/8}$ − base	-0.0 [-2.1, +2.1]	+1.6 [+0.2, +3.1]*	+0.8 [-2.9, +4.6]
$IC_{1/16}$ +MLI (distilled) − base	-1.2 [-3.6, +1.1]	-14.4 [-17.7, -11.2]*	+1.3 [-3.8, +6.2]
$IC_{1/2}$ +MLI (distilled) − base	+0.4 [-1.5, +2.4]	+2.2 [-1.8, +6.1]	-2.1 [-6.7, +2.5]
$IC_{1/32}$ +MLI (distilled) − base	-1.8 [-4.1, +0.4]	-10.9 [-15.1, -6.9]*	-10.4 [-17.5, -3.8]*
$IC_{1/4}$ +MLI (distilled) − base	+0.9 [-1.2, +2.9]	-1.1 [-3.5, +1.4]	+2.1 [-2.9, +7.1]
$IC_{1/8}$ +MLI (distilled) − base	+0.4 [-1.8, +2.6]	+1.2 [-0.3, +2.7]	-2.1 [-7.1, +2.9]
$IC_{1/16}$ +MLI − base	-10.6 [-13.5, -7.7]*	-42.6 [-47.1, -38.0]*	-18.7 [-25.4, -12.1]*
$IC_{1/2}$ +MLI − base	-1.1 [-3.1, +0.9]	+2.1 [+0.7, +3.4]*	+0.4 [-3.8, +4.6]
$IC_{1/32}$ +MLI − base	-17.6 [-21.0, -14.2]*	-30.6 [-35.5, -25.8]*	-28.3 [-35.4, -21.3]*
$IC_{1/4}$ +MLI − base	-0.4 [-2.3, +1.3]	+1.6 [-0.6, +3.7]	-0.4 [-4.6, +4.2]
$IC_{1/8}$ +MLI − base	-1.4 [-3.7, +0.9]	-1.3 [-4.1, +1.3]	-1.7 [-5.8, +2.5]
$IC_{\mathrm{searched}}^{1/2}$ +MLI − base	-1.7 [-3.7, +0.2]	+1.0 [-1.4, +3.2]	+0.0 [-3.8, +3.8]
$IC_{\mathrm{searched}}^{1/4}$ +MLI − base	-0.8 [-2.9, +1.3]	+1.8 [-0.6, +4.0]	+1.3 [-2.5, +5.4]
$IC_{\mathrm{searched}}^{1/2}$ − base	+0.3 [-1.9, +2.5]	+1.5 [-1.4, +4.7]	-0.8 [-5.0, +3.3]
$IC_{\mathrm{searched}}^{1/4}$ − base	+0.1 [-1.9, +2.3]	+3.2 [+1.6, +4.6]*	-0.4 [-4.6, +3.8]
MLI − base	-0.2 [-2.2, +1.7]	+2.3 [+0.4, +4.3]*	-0.8 [-5.0, +3.3]
MLI (distilled) − base	-1.3 [-3.5, +0.9]	+2.9 [+0.6, +5.2]*	+0.8 [-4.2, +5.8]
equal retention $1/2$ : +MLI − IndexCache	+0.1 [-2.1, +2.2]	-0.8 [-2.6, +0.8]	-0.8 [-5.0, +3.3]
equal retention $1/2$ : +MLI − IndexCache (distilled)	+0.6 [-1.3, +2.6]	+0.2 [-3.3, +4.0]	-5.0 [-9.2, -1.2]*
equal retention $1/4$ : +MLI − IndexCache	-0.2 [-2.3, +1.8]	+0.9 [-1.8, +3.3]	-2.9 [-7.1, +1.3]
equal retention $1/4$ : +MLI − IndexCache (distilled)	+1.7 [-0.6, +4.0]	-2.0 [-4.7, +0.8]	+2.9 [-1.7, +7.5]
equal retention $1/8$ : +MLI − IndexCache	-1.4 [-3.6, +0.7]	-3.0 [-5.2, -0.7]*	-2.5 [-6.2, +1.2]
equal retention $1/8$ : +MLI − IndexCache (distilled)	+0.8 [-1.5, +3.1]	+0.0 [-2.5, +2.6]	-2.1 [-6.7, +2.5]
equal retention $1/16$ : +MLI − IndexCache	+0.8 [-1.6, +3.3]	+18.7 [+14.4, +23.0]*	+2.1 [-3.8, +7.9]
equal retention $1/16$ : +MLI − IndexCache (distilled)	-1.1 [-3.5, +1.2]	-7.0 [-10.4, -3.5]*	-0.4 [-5.0, +4.2]
equal retention $1/32$ : +MLI − IndexCache	-0.3 [-2.6, +2.0]	+16.9 [+12.0, +21.8]*	+1.2 [-3.3, +5.4]
equal retention $1/32$ : +MLI − IndexCache (distilled)	-1.1 [-3.5, +1.4]	+3.5 [+0.0, +6.9]*	-7.1 [-13.8, -0.8]*
equal speed: IndexCache+MLI $1/2$ − IndexCache $1/4$	-0.8 [-3.1, +1.4]	+1.4 [-0.7, +3.5]	-2.1 [-6.7, +2.1]
equal speed: IndexCache+MLI $1/2$ − IndexCache $1/4$ (distilled)	+1.2 [-0.9, +3.4]	+1.3 [-0.4, +3.1]	-1.2 [-5.8, +3.3]
equal speed: IndexCache+MLI $1/4$ − IndexCache $1/8$	-0.4 [-2.8, +2.0]	+0.0 [-2.4, +2.0]	-1.2 [-5.4, +2.5]
equal speed: IndexCache+MLI $1/4$ − IndexCache $1/8$ (distilled)	+1.3 [-0.8, +3.4]	-2.3 [-5.3, +0.7]	+2.1 [-2.9, +7.1]
equal speed: IndexCache+MLI $1/8$ − IndexCache $1/16$	+10.1 [+6.8, +13.3]*	+59.9 [+55.8, +63.8]*	+19.2 [+12.5, +26.2]*
equal speed: IndexCache+MLI $1/8$ − IndexCache $1/16$ (distilled)	+0.5 [-1.7, +2.7]	+8.7 [+6.0, +11.3]*	-3.7 [-8.3, +1.2]
equal speed: IndexCache+MLI $1/16$ − IndexCache $1/32$	+6.7 [+4.2, +9.4]*	+5.0 [-0.4, +10.4]	+10.8 [+5.0, +16.7]*
equal speed: IndexCache+MLI $1/16$ − IndexCache $1/32$ (distilled)	-0.5 [-2.7, +1.7]	-0.0 [-3.7, +3.7]	+4.6 [-2.1, +11.2]

Table 11. GLM-5. Paired bootstrap 95% intervals for the contrasts in the text (percentage change over the unmodified model’s score; 10,000 resamples of examples, the same resample applied to both arms; LongBench resampled within each of its 11 tasks and RULER within each of its 30 task×length cells, both macro-averaged as in Table 8). *: interval excludes zero. Examples per arm: LongBench 1,100, RULER 234, LongBench v2 as stated in Table 8.

References

[1]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 3119–3137, 2024.arxiv.org/abs/2308.14508
[2]Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3639–3664, 2025.arxiv.org/abs/2412.15204
[3]Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li. Index-cache: Accelerating sparse attention via cross-layer index reuse. arXiv preprint arXiv:2603.12201, 2026.arxiv.org/abs/2603.12201
[4]Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.arxiv.org/abs/2004.05150
[5]Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.arxiv.org/abs/1904.10509
[6]Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra, and Ramachandran Ramjee. Kascade: A practical sparse attention method for long-context llm inference. arXiv preprint arXiv:2512.16391, 2025.arxiv.org/abs/2512.16391
[7]Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, et al. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410.13276, 2024.arxiv.org/abs/2410.13276
[8]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024.arxiv.org/abs/2404.06654
[9]Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems, 37:52481–52515, 2024.arxiv.org/abs/2407.02490
[10]Yaniv Leviathan, Matan Kalman, and Yossi Matias. Selective attention improves transformer. In International Conference on Learning Representations, volume 2025, pages 4389–4414, 2025.arxiv.org/abs/2410.02703
[11]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947–22970, 2024.arxiv.org/abs/2404.14469
[12]Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.arxiv.org/abs/2512.02556
[13]Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. Advances in Neural Information Processing Systems, 38:17790–17815, 2026.arxiv.org/abs/2502.13189
[14]Cheng Luo, Zefan Cai, Hanshi Sun, Jinqi Xiao, Bo Yuan, Wen Xiao, Junjie Hu, Jiawei Zhao, Beidi Chen, and Anima Anandkumar. Headinfer: Memory-efficient llm inference by head-wise offloading. arXiv preprint arXiv:2502.12574, 2025.arxiv.org/abs/2502.12574
[15]Yuzhen Mao, Martin Ester, and Ke Li. Iceformer: Accelerated inference with long-sequence transformers on cpus. In International Conference on Learning Representations, volume 2024, pages 14858–14875, 2024.arxiv.org/abs/2405.02842
[17]Vijaykaarti Sundarapandiyan, Tom Goldstein, and Ashwinee Panda. Foreign sparse attention: Effective distillation into sparse attention. In ICML 2025 Workshop on Long-Context Foundation Models, 2025.
[18]Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024.arxiv.org/abs/2406.10774
[19]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, volume 2024, pages 21875–21895, 2024.arxiv.org/abs/2309.17453
[20]Lijie Yang, Zihao Zhang, Zhuofu Chen, Zikun Li, and Zihao Jia. Titldecode: Fast and accurate llm decoding with position persistent sparse attention. In International Conference on Learning Representations, volume 2025, pages 5823–5844, 2025.arxiv.org/abs/2410.05076
[21]Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Jean Wang, Zhiping Xia, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23078–23097, 2025.arxiv.org/abs/2502.11089
[22]Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297, 2020.arxiv.org/abs/2007.14062
[23]Z.ai. GLM-5.1 model card. https://huggingface.co/zai-org/GLM-5.1, 2026a.
[24]Z.ai. GLM-5.2 model card. https://huggingface.co/zai-org/GLM-5.2, 2026b. IndexShare: one indexer shared across every four sparse-attention layers.
[25]Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.arxiv.org/abs/2602.15763
[26]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in neural information processing systems, 36:34661–34710, 2023.arxiv.org/abs/2306.14048
[27]Weilin Zhao, Zihan Zhou, Chaojun Xiao, Yuxuan Li, Yanghao Li, Yudi Zhang, Weilun Zhao, Zhen Li, Yuxiang Huang, Ao Sun, et al. Infllm-v2: Dense-sparse switchable attention for seamless short-to-long adaptation. In International Conference on Learning Representations, volume 2026, pages 69374–69390, 2026.arxiv.org/abs/2509.24663
[28]Yuanbing Zhu, Zhenheng Tang, Xiang Liu, Ang Li, Bo Li, Xiaowen Chu, and Bo Han. Oraclekv: Oracle guidance for question-independent kv cache eviction. In ICML 2025 Workshop on Long-Context Foundation Models, 2025.