WorksComputer Science
Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining
Abstract
The full-attention layers of large language models (LLMs) make every generated token more expensive than the previous one, slowing models down on long-context tasks. Hybrid models reduce this cost by replacing most full-attention layers with linear-attention layers, but the first still dominate decoding at long contexts. We accelerate these remaining layers without any training or weight changes using combined sparse attention: each layer attends to a sliding window of recent tokens, a pinned prefix, and a fixed budget of most relevant tokens in the context. The window size and budget are found on a per-layer basis using a multi-objective search algorithm. Across six hybrid models from three families, from 4B to 397B parameters, the selected configurations preserve their baseline model's accuracy on 21 long-context datasets. Decoding 64k/256k tokens becomes up to 1.26x/2.22x faster on Qwen3.5-4B, 1.20x/2.23x on Qwen3.5-122B-A10B, and 1.16x/2.06x on Ornith-1.0-397B.
Introduction
Current implementations of self-attention (flash-attention (Dao et al., [4])) cost time and memory in the sequence length . For autoregressive inference, the memory capacity is usually the binding constraint: the KV cache grows linearly with every generated token and with every concurrent request. At long context, the full-attention layers consume the majority of the decode-step budget and determine how many concurrent requests a deployment can serve (Figure 1).
Figure 1. Time spent decoding one token by Qwen3.5-4B in each kernel (averaging 256 tokens), at different context lengths.
Two lines of work have emerged to defuse this bottleneck. The first is architectural: replace softmax attention with a recurrent, convolutional, or windowed mixer whose per-step cost is constant in . Mamba (Gu & Dao, 2024}), RWKV (Peng et al., [21]), Hyena (Poli et al., [22]), Gated DeltaNet (Yang et al., [30]), and the sliding-window layers of Gemma 4 (Gemma Team et al., [6]) belong here, together with the hybrid models (Qwen3.5 (Qwen Team, [23]), Gemma 4, gpt-oss (OpenAI et al., [20]), MiniMax-Text-01 (MiniMax et al., [18]), Kimi-K3 (Kimi Team, [11])) that interleave a minority of full-attention layers with cheaper attention layers. The accuracy of hybrids is competitive, but decisions such as how many full-attention layers to use, where to place them, and their window size are frozen into the released weights at pretraining time.
The second line is post-training linearization: take a pre-trained Transformer, swap its attention for a linear variant, and recover the accuracy with continued training. Most works focus on smaller models, up to 8B parameters; Lol-Cats (Zhang et al., [33]) is one exception, applied to Llama 3.1 405B. Besides focusing on smaller models, these works often require large amounts of training data, with SUPRA (Mercat et al., [17]) reporting 5% of pretraining, and LawCat (Liu et al., [16]) reporting 0.1% of pretraining. LolCats use a model with size comparable to our work, and report spending 454 H100 GPU-hours in attention transfer and LoRA fine-tuning. Although Jet-Nemotron-2B obtained a 12.0x speedup over its base model at a 64k-token context, it was trained on 600B tokens, using 18,328 H100 GPU-hours (Gu et al., 2025}). It is not practical to expend this compute and data cost for every new released model, motivating the search for cheaper alternatives. This work builds upon hybrid models whose remaining full-attention layers still carry the long-context cost, and reduces their computational cost without further training by limiting their attention. Qwen3.5-27B keeps 16 full-attention layers out of 64; gpt-oss-120b, 18 out of 36; Ornith-397B, 15 out of 60. These layers read keys per step, and they are the ones that would benefit from optimizations. The question that motivates this work is therefore:
Given a pre-trained hybrid model, how much can we restrict the attention of its full-attention layers at no accuracy degradation, without retraining and without editing weights?
Attention maps are known to be highly concentrated, making training-free KV eviction (H2O [34], DuoAttention [28]) work, but this is an empirical claim, and recent large-scale evaluation [19] shows it is strongly task- and scale-dependent. We hypothesize that, since attention is sparse at long context, each query only needs the keys of a small, query-dependent subset of positions. By reducing attention in models, we obtain speedups by skipping the bottleneck of decoding in long-context tasks, full attention (Figure 1). This motivates searching for configurations that minimize the cost of attention through linearization techniques, while aiming to preserve the accuracy of the baseline models.
Our challenge is therefore twofold: first, identifying which positions carry information depending on the query and the layer, not only on the position; second, minimizing the attention budget on a layer-by-layer basis. We combine a plain sliding window ([2, 31, 10]) with a dynamic query-aware top- selection on paged KV (Quest [26]) on a layer-by-layer basis, and let a multi-objective search decide the window and budget of every layer.
Contributions:
A per-layer, training-free sparse-attention configuration search algorithm that combines a sliding window, a pinned prefix, and query-aware paged selection, installed into vLLM [13].
A multi-objective NSGA-II [5] search heuristic that returns per-layer (window, budget) pairs that optimize both long-context accuracy and estimated decoding cost. The search costs 8.6–34.2 GPU-hours on A100 GPUs for models ranging from 4 to 120B parameters, and 31.3–58.4 GPU-hours on a x4 GB200 GPU node for models with 122B and 397B parameters.
An evaluation on six hybrid models from three families (Qwen3.5 at 4B / 27B / 122B, Ornith 397B, Gemma-4-31B, gpt-oss-120b) across 21 long-context benchmarks. To our knowledge, this is the first training-free sparse-attention method evaluated at the 100B+ scale.
Headline results. On every model, at least one configuration preserves the baseline accuracy on every long-context dataset, with averages within 0.5 percentage points (pp) of the baseline. At 64k/256k tokens and batch size 4, these configurations decode faster on Qwen3.5-4B, on Qwen3.5-122B-A10B, and on Ornith-1.0-397B. The gains are smaller in the other models: on the dense Qwen3.5-27B ( at 64k tokens) and Gemma-4-31B ( at 128k tokens) models, where a larger chunk of the decoding time is used to read weights (Figure 7, in appendix), and on gpt-oss-120b ( at 128k tokens), where our method preserved most of the full-attention layers. At short context, every configuration is slower than the unmodified model.
Figure 7. Average time spent in each kernel category when decoding 256 tokens, starting at a context length of 64k tokens.
Related Work
Reducing the cost of self-attention in deployed language models is studied extensively. We group prior work by the computational requirements: full pretraining, a post-training phase of continued pretraining, or no training at all. Table 1 summarizes the landscape and locates this work in it.
| Method | Approach | Training required | Long context | Largest model reported |
| Pretraining – architecture decided before weights are released | ||||
| Mamba, Mamba-3 | SSM | full pretraining | synthetic only (1M), 2k on text | 1.5–2.8B |
| Hyena | long convolution | full pretraining | synthetic recall (131k), 2k on text | 1.3B |
| RWKV | linear recurrence | full pretraining | 16k | 14B |
| Qwen3.5, Gemma 4, gpt-oss, kimi-k3 | hybrid full + linear/SWA | full pretraining | yes (1M) | 2.8T |
| Post-training linearization – continued training on top of a released model | ||||
| Hedgehog | learned softmax-like feature map | LoRA conversion | not evaluated | 7B |
| LoLCATs | linear attention swap | 40M tokens, LoRA | 8k, single dataset | 405B |
| Liger | gated recurrence | 20M tokens, LoRA | not evaluated | 8B |
| LAWCAT | distillation + convolution | 94M tokens, LoRA | up to 22k | 7B |
| SUPRA | linear uptraining | 20B tokens, uptraining | 16k | 7B |
| HALO | hybrid distillation | 2.3B tokens, distillation | yes (256k) | 8B |
| JetNemotron | PostNAS + retraining | 600B tokens | yes (64k) | 4B |
| Training-free sparse / windowed attention on frozen weights | ||||
| StreamingLLM | window + sinks | none | no retrieval beyond window | 70B |
| H2O | KV eviction | none | 2k | 30B |
| SnapKV | prefill-time top- | none | yes (380k) | 47B |
| Quest | query-aware paged top- | none | yes (100k) | 7B |
| DuoAttention | retrieval-head split | gates only, 2k steps | yes (1M) | 70B |
| Ours | per-layer SWA + prefix + Quest + NSGA-II | none2 | yes (21 datasets, up to 64k tokens) | 397B |
Table 1. Methods for reducing attention costs in LLMs. The first block covers pretraining methods, the second covers post-training, and the third is training-free. Our pipeline is training-free and tested in hybrid models across three model families (Qwen3.5, Gemma 4, gpt-oss). Green: no training, long-context capability, or more than 300B parameters; orange: post-training, limited long-context capability, or 10–300B parameters; red: full pretraining or over 1B token training, no long-context evaluation, or fewer than 10B parameters.
Pretrained Linear and Hybrid Architectures
Mamba ((Gu & Dao, 2024)), Hyena ([22]), and RWKV ([21]) replace softmax attention with a recurrence or a long convolution whose cost grows linearly or quasi-linearly with . They require a full pretraining run, are relatively small (under 15B parameters), and their long-context ability is either shown on synthetic tasks only (Mamba and Hyena, whose language models are trained on contexts of at most 2k tokens) or limited ([21]). The hybrid models studied here (Qwen3.5, Gemma 4, gpt-oss) as well as the recent Kimi-K3 ([11]) keep a minority of full-attention layers in an otherwise linear or windowed stack, while retaining long-context capabilities. In every case, the design decisions (ratio of full to linear layers, placement, window size of the sliding ones) are fixed by the pretraining run and baked into the released weights. These weights are the starting point of our work, not an alternative to it: we take the hybrid as given and crop only the full-attention layers that remain.
Post-Training Linearization
A second line keeps a released Transformer and converts its attention layers into cheaper alternatives with additional training. Hedgehog [32] learns a feature map that mimics softmax attention and converts a 7B model for a single task with LoRA; LoLCATs [33] transfers attention to a feature map and recovers the accuracy with LoRA on 40M tokens for an 8B model, and scales the recipe to 70B and 405B models; Liger [14] reuses the key projections as gates and fine-tunes with LoRA
| Model | Thro. | NSGA-II | Accuracy |
| Qwen3.5-4B | 10.5 | 8.6 | 97.2 |
| Qwen3.5-27B | 22.9 | 34.2 | 465.4 |
| Gemma-4-31B | 38.4 | 27.5 | 693.4 |
| gpt-oss-120b | 13.5 | 16.2 | 220.2 |
| Total (A100) | 85.3 | 86.5 | 1476.2 |
| Qwen3.5-122B-A10B | 52.3 | 31.3 | 116.2 |
| Ornith-1.0-397B | 93.1 | 58.4 | 163.6 |
| Total (GB200) | 145.3 | 89.7 | 279.8 |
Table 2. Total number of GPU-hours spent on the throughput (Thro.), NSGA-II, and accuracy runs.
on 20M tokens; LAWCAT ([16]) adds a convolution to the linear attention and distills it on 94M tokens for a 7B model; SUPRA ([17]) uptrains every weight on 20B tokens; and Halo ([3]) distills a hybrid student from a Transformer teacher on 2.3B tokens. Jet-Nemotron ((Gu et al., 2025)) uses 600B tokens and 18k GPU-hours, obtaining a 12.0x speedup on Qwen2.5-1.5B. The cheapest of these conversions cost a few GPU-hours, but all of them change the weights, and only LAWCAT and Halo report long-context retrieval beyond 8k tokens (up to 22k and 256k tokens, respectively), on models of at most 8B parameters. At the frontier scales (120B, 400B) that this work targets, only LoLCATs has been applied, and its only long-context test, passkey retrieval at 8k tokens on the 8B model, fails unless the model is linearized on that same task.
Training-Free Sparse Attention
Like our work, these methods leave the weights untouched and change what the attention attends to. StreamingLLM ([27]) pins the first few tokens as attention sinks, which lets a plain sliding window keep producing coherent text past the window, but cannot retrieve information that has left it; H2O ([34]) evicts keys by accumulated attention mass; SnapKV ((Li et al., 2024)) selects the keys to keep at the end of the prefill.
Since these selections are fixed, or only shrink, once the generation starts, they fail if the information later needed was evicted. Quest ([26]) addresses this failure mode by re-selecting pages of the paged KV cache at every decoding step, at the cost of scoring every page at every step. RazorAttention ([25]) and DuoAttention ([28]) split the attention heads into retrieval heads, which keep the full KV cache, and streaming heads, which keep a window; RazorAttention identifies the retrieval heads without training, whereas DuoAttention learns one gate per head on synthetic data. Except for SnapKV (up to 47B), all of these methods were evaluated on long-context tasks with models of at most 8B parameters; DuoAttention reaches 70B, but only on short-context benchmarks.
We build directly on this third line. Our contribution over it is twofold: (i) we treat the hyperparameters that these methods expose (window , Quest budget , refresh interval , pinned prefix size) as a per-layer vector and search it with NSGA-II ([5]), rather than fixing one value globally; and (ii) we demonstrate the method on hybrids of up to 397B parameters, more than five times larger than any prior training-free report in Table 1, on long-context tasks of up to 64k tokens, and across three distinct model families.
Methodology
Following the illustration in Figure 2, our pipeline receives as input a model and evaluation samples used to ensure accuracy is preserved. This method discovers new configurations using the following steps, explained in detail in this section: (1) it calculates a surrogate cost table with different window/budget pairs; (2) it generates an initial set of candidate solutions; (3) those solutions are evaluated based on their accuracy and estimated cost, and a Pareto front is calculated; (4) the best solutions are selected to be combined and mutated, generating a new set of solutions, and (5) when the run is over, we extract up to 3 solutions from the Pareto front (best-accuracy, best-speed, and balanced).
Figure 2. Overview of the pipeline used to obtain configurations using NSGA-II. (1) We fill an table with measured times with different window/budget values to be used by the surrogate cost function; (2) we generate the initial solutions; (3) the solutions are evaluated in the evaluation samples and using the cost function; (4) the best solutions are combined and mutated, and (5) when the search is over, we extract 3 configurations from the Pareto front.
Large Language Models
As previously seen in Figure 1, for long-context tasks, two kernel categories dominate the decoding time: loading weights and full attention. Since this pipeline focuses on reducing the computational cost of attention, without modifying the other aspects of decoding, users should take into consideration the loading times, as they will limit the maximum speedup obtainable by reconfiguring the LLM.
As an example, we profile the decoding of 3 models at a 64k-token context (Figure 7, in the appendix). At opposite extremes, we have Qwen3.5-4B (29.4ms in decoding, out of which 6.8 ms are loading weights) and Qwen3.5-27B (47.1ms and 19.4ms, respectively). Because this operation is unavoidable in our pipeline, the best speedup we obtained at this context length was 3.03x and 1.78x, respectively.
Given this constraint, we recommend using mixture-of-experts (MoE) models, due to their lower number of active parameters, reducing their loading time; as well as using hybrid models, giving priority to models with higher ratios of linear- to full-attention layers, reducing our search space.
Query-aware Sparse Attention
During decoding, a full-attention layer reads the keys and values of every previous token, so the cost of each new token grows with the sequence length . In the layers selected by a configuration, we replace this attention with a sparse attention that reads a bounded number of keys per step, independent of , while every other layer, and every weight of the model, is left unchanged. The keys that each modified layer reads, how often they are re-selected, and what the method does and does not change are described below.
From windows to query-aware pages: A plain sliding window forgets every token that leaves the window, and preserving the first tokens as attention sinks (Xiao et al., [27]) only partially recovers the accuracy lost. Information that is relevant to the current query can be anywhere in the sequence. For this reason, every modified layer attends to three sets of keys during decoding: (1) the most recent tokens (the window), (2) the first 256 tokens of the sequence, preserving part of the input (Xiao et al., [27]), and (3) the pages selected by Quest (Tang et al., [26]), which are the most relevant pages to the current query. The KV cache is divided into 32-token pages, which are summarised by the per-channel minimum and maximum of their keys. At each decoding step, the pages are scored, and the top pages are attended to until a budget of tokens is used. The page selection is computed per KV head, so different heads can retrieve different parts of the sequence. By combining the three sparse-attention methods, we obtain a scheme with (1) a sliding window, (2) one static sink, and (3) dynamic sinks, updated every tokens, illustrated in Figure 3.

Figure 3. Combined sparse attention: one static sink (pinned prefix, 256 tokens, as 4 × 🟨), sliding window ( = 6 × 🟦), and dynamic (Quest) sinks ( = 4 × 🟩), refreshed every tokens (=3).
Refresh interval: Scoring every page at every step is the dominant cost of Quest. We therefore recompute the selection every decoding steps, and keep it fixed in between, widening the window so that no token falls between the window and the stale selection. We use by default and refer to (selection at every step) as per-step.
Prefill and memory: Only decoding is modified: the prefill runs vLLM’s exact attention and fills the page summaries. The KV cache is kept at its full size, since any page may be selected later. As such, the method reduces the number of keys read per decoding step, and therefore the decoding time, but it does not reduce the KV cache memory footprint.
Search for New Configurations
Our methodology combines three sparse-attention methods and then searches for a set of parameters to minimize the model’s cost while preserving its accuracy. Currently, our search algorithm is focused on minimizing two parameters: the sliding window size and the number of tokens retrieved by Quest. Since we hypothesize that each layer has different needs, we use NSGA-II as a parameter search algorithm to automatically search for heterogeneous configurations.
Encoding of the Search Space
Each full-attention layer has two components we are interested in limiting: its sliding window size and a Quest budget . For a model with full-attention layers, a solution in our search space is defined as a sequence, such that, and , where is the possible values for Quest budget values chosen to be , and is the set of possible sliding window sizes (). For example, a layer using will attend to the last 512 tokens, the 256 pinned tokens, and the 256 tokens selected by Quest.
Obtaining configurations
Our pipeline (Figure 2) is build around using NSGA-II (Deb et al. [5]) as a multi-objective search algorithm, using two objectives: maximizing accuracy and minimizing estimated cost. NSGA-II follows these steps: during initialization, NSGA-II generates a set of uniform solutions, as well as randomly generated solutions within our search space. These solutions are evaluated using the objective functions, and
then solutions are selected via tournament for crossover and mutation, generating a new set of solutions, continuing the cycle. Tournament: Two solutions are randomly selected and the dominating one (better in both objective) wins. If there is no dominating solution, the one furthest away from the other solutions in the objective space is selected (crowding distance). Crossover: Two solutions are split in the same index, swapping the later halves, generating 2 offspring. Mutation: Each parameter has a chance of being multiplied or divided by 2.
When the search ends, we pick 3 candidate from the Pareto front (list of non-dominated solutions) that are within 3 percentage points (3pp) of the baseline model, ensuring the quality of the configurations chosen. We name these configurations Best Accuracy (the most accurate point), Best Speed (the point with the lowest estimated cost), and Balanced (the point closest to the utopia point, with both objectives rescaled to over the front).
Surrogate Cost Function: Table 2 shows the GPU-hours spent in this work, with the first columns being the time spent measuring the token/s throughput of the final solutions (up to 6 solutions per model). From these values, we conclude that measuring throughput in long-context tasks would be a very time-consuming task. We reduce the costs of measuring throughput of candidate solutions using the following expression as a low-cost surrogate cost function:
where is the measured time of the base model, is the time spent in one full-attention layer, and is the time spent in a layer with window and budget ( for an unmodified layer). The values of are measured for every pair by measuring the time it takes to go through a layer with that configuration 64k times using batch size 8.
Evaluation
While our pipeline uses a search algorithm to automatically find configurations, we acknowledge that it is possible to search for configurations manually, despite that placing a cost in deployment that is measurable in human-hours. As such, we compare two ways of obtaining a configuration: a uniform configuration with parameters decided through manual trial and error, and the configurations returned by our search algorithm.
Uniform configuration (manual): Every full-attention layer uses the same window and budget , with the 256 pinned tokens and a refresh interval of (R32), or (R1, per-step). These values are the result of a long sequence of manual experiments, in which we first tried plain sliding windows, then windows with sinks, SnapKV-placed sinks, and Quest with different page sizes, budgets, and refresh intervals. Its main limitation is that it applies the same configuration to every layer and every model, despite our assumption that different models and layers have different needs.
NSGA-II (automatic): We use NSGA-II [5] as a multi-objective search algorithm, using two objectives: maximizing accuracy and minimizing the estimated cost. The accuracy objective is the mean score on two long-context datasets: BABILong-qa1 at 64k tokens [12] and LongBench Qasper [1]. Each candidate is evaluated on the first 100 examples of each dataset. The initial population is seeded with three uniform configurations (; ), and the rest is sampled at random. The runs were targeted to fit into a 15-hour wall time block, resulting in the following number of iterations (it.) and population size (ps) per setup: 1xA100 (4 it., ps 10), 2xA100 (3 it., ps 8), 4xB200 (3 it., ps 10).
Hybrid models: We apply our methodology to six representative hybrid models from three families (Table 3). In every model, only a minority of the decoder layers compute a full softmax-attention matrix; the remaining layers either linear attention (Gated DeltaNet (GDN)) [30] in the Qwen3.5 family [23, 29] or sliding-window attention (Gemma 4 [6] and gpt-oss [20]). Since those layers are already or bounded by their own window, they are not modified in any way in this study: the search space is exactly the set of full-attention layers.
| Model | Type | Total | Active | Layers | Full attn. | Other layers | Context | Hardware |
| Qwen3.5-4B | dense | 4B | 4B | 32 | 8 | GDN | 262k | 1×A100 |
| Qwen3.5-27B | dense | 27B | 27B | 64 | 16 | GDN | 262k | 1×A100 |
| Qwen3.5-122B-A10B-FP8 | MoE | 122B | 10B | 48 | 12 | GDN | 262k | 4×GB200 |
| Ornith-1.0-397B-FP8 | MoE | 397B | 17B | 60 | 15 | GDN | 262k | 4×GB200 |
| Gemma-4-31B | dense | 30.7B | 30.7B | 60 | 10 | SWA-1024 | 262k | 2×A100 |
| gpt-oss-120b | MoE | 117B | 5.1B | 36 | 18 | SWA-128 | 131k | 2×A100 |
Table 3. Models studied in this work and hardware used. For multi-GPU runs, we use tensor parallelism of 2 and 4.
In the remainder of this section, we discuss the configurations discovered through NSGA-II, followed by the speedups obtained by applying both the uniform and NSGA-II configurations, and finally the accuracy on the long-context benchmark, showing that this methodology is applied without affecting the models’ accuracy.
Configurations Discovered Through NSGA-II
The NSGA-II algorithm is used to search for configurations that maximize accuracy while minimizing their estimated decoding cost. The configurations obtained have been placed in the appendix (Section A.5). After running the search algorithm, we obtain up to three configurations, all within 3pp of the baseline accuracy: Best-Accuracy, Best-Speed, and Balanced, which are described below.
Qwen3.5-4B: Best-Accuracy keeps only one layer at full attention, and crops the other 7 layers with windows between 256 and 4096 and budgets between 256 and 2048. Balanced crops all 8 layers, with a window of 256 on 6 of them, and Best Speed crops all 8 layers to , .
Qwen3.5-27B: Every pick crops all 16 layers. Best Speed uses , on every layer, and Balanced differs from it only in a budget of 2048 on layer 7. Best Accuracy keeps a window of 256 on 11 layers, widens it to 2048–4096 on the other 5 (layers 3, 31, 51, 55 and 59), and uses budgets between 256 and 4096.
Gemma-4-31B: Best-Accuracy crops only 2 of the 10 layers (layer 11 to and layer 59 to ), and Best Speed crops 6, keeping layers 35, 51 and 47 at full attention. The Balanced point of the front coincides with Best-Accuracy, so it is not reported separately.
gpt-oss-120b: The front contained a single configuration, which crops 4 of the 18 layers (layers 3, 5, 25 and 33), and is therefore reported as Best-Accuracy only.
Qwen3.5-122B: The front contained 4 configurations, and the three picks are close to each other: on the 21 datasets, their average accuracy is within 0.2 pp of each other, and their speed is within 1%.
Ornith-397B: Best Speed is a uniform configuration (, on all 15 layers), while Best Accuracy and Balanced use different windows and budgets per layer. The front contained 3 configurations, and the uniform configuration had a budget of 256.
Considering that out top-3 models have 8 to 15 full-attention layers, there are – possible configurations, and we only try 24–40 configurations per model. We leave a large portion of the search space unexplored, which LLMs could explore more efficiently as decision makers [9]. Despite the limited number of explored solutions, it is interesting to say that the majority of the configurations cropped all or most full-attention layers, indicating the current models can still be vastly improved.
Controlled throughput benchmark
The wall-clock time from the accuracy runs cannot tell us how much faster the model has become. In those runs, the output length is a free variable, making the measured time dependent on the length of the answer, and not on the speed of the model. As such, we make a controlled throughput benchmark, in which we measure how long each configuration (baseline, best-accuracy, best-speed, balanced, R32 and R1) performs the same work at different batch sizes.
For each batch size , we generated synthetic 128-token prompts, and every request generates exactly tokens (end-of-sequence ignored). The engine is stepped manually, and the wall-clock time at which every request in the batch has produced tokens is recorded for . The throughput at length is defined as . A batch size stops at the largest checkpoint whose KV cache fits in memory ( within 90% of the capacity vLLM reports. The results are presented as one figure per model with the top-3 best speedups in Figure 4, and all models in the appendix (Figure 8). Table 4 shows the values for batch size 4.
Figure 4. Token/s throughput using different configurations and batch sizes. The baseline models are faster for short sequences due to the computation overhead from Quest. For longer sequences starting from 64k tokens, we see reasonable speedups in Qwen-4B, Qwen-122B, and Ornith. These speedups range 1.16–1.26x in context lengths of 64k tokens and 2.06–2.23x for 256k tokens
| Model / Configuration | Generated tokens | 16k | 64k | 256k |
| Qwen3.5-4B | Baseline | 472.3 | 382.4 | 217.6 |
| Best Accuracy | 0.99× (466.8) | 1.19× (453.4) | 1.87× (407.3) | |
| Best Speed | 1.02× (483.5) | 1.26× (483.5) | 2.22× (483.0) | |
| R32 | 1.01× (478.3) | 1.25× (478.2) | 2.20× (477.7) | |
| Qwen3.5-27B | Baseline | 106.8 | 96.7 | — |
| Best Accuracy | 0.97× (103.4) | 1.07× (103.4) | — | |
| Balanced | 1.00× (107.0) | 1.11× (107.0) | — | |
| R32 | 0.97× (104.0) | 1.08× (104.0) | — | |
| Gemma-4-31B | Baseline | 141.0 | 122.3 | — |
| Best Accuracy | 0.99× (139.5) | 1.01× (123.3) | — | |
| Best Speed | 0.99× (139.4) | 1.05× (127.9) | — | |
| R32 | 0.99× (139.5) | 1.09× (133.1) | — | |
| gpt-oss-120b | Baseline | 529.8 | 382.3 | — |
| Best Acc. / Speed | 0.87× (461.3) | 0.97× (371.6) | — | |
| R32 | 0.61× (322.9) | 0.84× (322.0) | — | |
| Qwen3.5-122B-A10B-FP8 | Baseline | 690.2 | 540.8 | 290.8 |
| Best Accuracy | 0.93× (640.0) | 1.19× (642.4) | 2.21× (642.9) | |
| Balanced | 0.94× (647.1) | 1.20× (648.2) | 2.23× (648.3) | |
| R32 | 0.92× (637.9) | 1.18× (639.0) | 2.20× (639.3) | |
| Ornith-1.0-397B-FP8 | Baseline | 484.0 | 389.8 | 219.4 |
| Best Accuracy | 0.91× (439.3) | 1.10× (426.9) | 1.75× (383.3) | |
| Best Speed / R32 | 0.93× (450.5) | 1.16× (451.0) | 2.06× (450.9) |
Table 4. Decode speedup over the baseline at the same context length and batch size 4, followed by the throughput in brackets (tokens/s, all requests). For each model, we show the results of the baseline, Best Accuracy, the fastest between Balance and Best Speed configurations, and R32 (applying a uniform 1024-token budget to all layers, updating quest every 32 steps). The GPT-OSS run returned a single configuration. Ornith’s Best Speed and R32 configurations are the same. ‘—’ marks a point that was not measured either because the context is beyond the model’s limit (GPT-OSS) or the KV cache would not fit into memory at batch size 4 (Qwen-27B and Gemma runs).
Figure 8. Tokens-per-second throughput using different configurations and batch sizes. We see that the baseline models are faster for short sequences due to the computation overhead from Quest. For longer sequences starting from 64k tokens, we see reasonable speedups in Qwen-4B, Qwen-122B, and Ornith. These speedups range 1.16–1.26x in context lengths of 64k tokens and 2.06–2.23x for 256k tokens
Short contexts are slower. While Quest allows us to save on computation by only considering the top- tokens, this also introduces an additional computational overhead that makes the configurations slower than the baseline for context lengths up to 16k–32k tokens (excluding gpt-oss-120b). This motivated us to only rerank the Quest pages once every 32 tokens, vastly reducing the computational overhead that results in the R1 configuration being the slowest.
Long contexts are faster. In full-attention blocks, the cost of
generating each token grows linearly with the context length, making each token more computationally expensive than the previous one. Since our approach has a computational cost independent of the context length (despite KV cache still growing normally), Quest’s computation overhead is overshadowed by the full-attention costs in long contexts. At the longest context, the speedups reach 1.26/2.22× on Qwen3.5-4B, 1.20/2.23× on Qwen3.5-122B-A10B, and 1.16/2.06× on Ornith-1.0-397B, for batch size 4.
Computational overhead of Quest. Quests need to rank all attention pages every decode step and every layer, or every 32 decode steps if you follow our methodology. This adds an overhead that is highly dependent on the number of full-attention layers within the model. When we profiled our configuration during decode (Figure 7, in appendix), we noticed that this overhead is minimal, especially when done only once every 32 decode steps.
Baseline model bottleneck: This method was not very effective in the Qwen3.5-27B and Gemma-4-31B models. Both of these methods are dense, resulting in a reasonable amount of time being the time spent reading weights, making the savings on compute less noticeable in the throughput. Similarly, Qwen3.5-4B is also a dense model, but due to its small size, the time spent reading parameters is small enough to obtain a good speedup from optimizing the computational side of decode.
Across the 3 models with the best speedups, we obtained speedups in the range of 1.16–1.26x in context lengths of 64k tokens and 2.06–2.23x when pushing the context length to 256k tokens. In addition to this, it should be noted that the configuration in which Quest is applied to all layers became constant in terms of token/s produced. As will be shown next, these configurations did not degrade the accuracy of the baseline models, making these speedups more impressive.
Accuracy in Long-context tasks
We evaluate the configurations on 21 long-context datasets: (1) BABILong qa1–qa5 (Kuratov et al., [12]) at 64k tokens; and (2) the 16 English subset of LongBench (Bai et al., [1]) (single- and multi-document QA, summarisation, few-shot learning, synthetic retrieval and code completion). We use the full test set of every dataset, excluding the 100 examples of BABILong-qa1 and Qasper that were used during the search, so the reported accuracy is measured on held-out data.
Protocol: Each configuration is compared with the baseline model. Decoding is greedy, with thinking disabled and the maximum number of new tokens defined per dataset (256 for BABILong, and LongBench’s own limits). LongBench is scored with its official metrics (F1, ROUGE-L, accuracy, and edit similarity for code), and BABILong by answer match. Before scoring, typographic characters (narrow and thin spaces or commas between digit groups, non-breaking hyphens and curly quotes) are normalized to ASCII, as some models emit them in place of the characters in the references. A configuration is considered to preserve accuracy on a dataset if its score is within 3pp of the baseline.
Prompts: Each model receives its own chat template. For gpt-oss, which uses the harmony format, the assistant turn is opened directly in its final channel, so the answer budget is not spent on its analysis channel, and LongBench’s few-shot and code-completion subsets, which LongBench runs as raw completions, are also wrapped in the chat template.
Table 5 shows the accuracy values for each configuration on the 21 long-context datasets. In this table, we highlight in green and red values that are 1 percentage point above or below the baseline, and consider a configuration successful if they do not degrade the results of the baseline model on any of the datasets by more than 3pp. The later comments will be focused on Qwen-4B, Qwen-122B, and Ornith, since they obtained the biggest speedup.
| Dataset | Qwen-4B Base | Qwen-4B Acc | Qwen-4B Bal | Qwen-4B Spd | Qwen-4B R32 | Qwen-4B R1 | Qwen-27B Base | Qwen-27B Acc | Qwen-27B Bal | Qwen-27B Spd | Qwen-27B R32 | Qwen-27B R1 | gpt-oss Base | gpt-oss Acc | gpt-oss R32 | gpt-oss R1 | Gemma-31B Base | Gemma-31B Acc | Gemma-31B Spd | Gemma-31B R32 | Gemma-31B R1 | Qwen-122B Base | Qwen-122B Acc | Qwen-122B Bal | Qwen-122B Spd | Qwen-122B R32 | Qwen-122B R1 | Ornith-397B Base | Ornith-397B Acc | Ornith-397B Bal | Ornith-397B Spd/R32 |
| qa1 | 61.0 | 60.9 | 60.8 | 60.7 | 60.9 | 60.7 | 92.2 | 91.9 | 92.1 | 92.1 | 92.0 | 92.1 | 47.6 | 47.3 | 47.3 | 47.2 | 75.0 | 74.8 | 74.8 | 74.8 | 74.8 | 87.2 | 87.2 | 87.2 | 87.1 | 87.1 | 87.2 | 85.6 | 85.6 | 85.6 | 85.6 |
| qa2 | 38.4 | 38.3 | 38.2 | 38.1 | 38.2 | 38.4 | 50.7 | 51.2 | 50.4 | 50.5 | 49.8 | 50.7 | 29.0 | 29.2 | 30.2 | 29.8 | 41.4 | 41.7 | 41.8 | 41.5 | 41.5 | 44.0 | 43.9 | 43.9 | 43.9 | 43.9 | 43.9 | 32.9 | 31.9 | 31.3 | 31.7 |
| qa3 | 26.3 | 26.1 | 26.2 | 26.2 | 25.9 | 26.1 | 30.3 | 30.5 | 30.4 | 30.7 | 30.5 | 30.7 | 26.0 | 26.0 | 25.8 | 25.6 | 28.2 | 27.4 | 27.4 | 27.4 | 27.4 | 32.3 | 32.3 | 32.3 | 32.3 | 32.3 | 32.3 | 8.4 | 7.1 | 7.8 | 5.5 |
| qa4 | 59.7 | 59.5 | 59.6 | 59.5 | 59.5 | 59.5 | 57.1 | 57.0 | 56.7 | 56.9 | 57.1 | 56.9 | 45.4 | 45.0 | 44.4 | 45.5 | 85.9 | 86.4 | 86.4 | 86.4 | 86.4 | 67.5 | 67.5 | 67.5 | 67.5 | 67.5 | 67.5 | 61.3 | 61.3 | 61.3 | 61.3 |
| qa5 | 72.1 | 72.0 | 72.0 | 72.0 | 72.0 | 72.0 | 80.4 | 80.5 | 80.6 | 80.6 | 80.3 | 80.4 | 61.7 | 62.0 | 62.0 | 61.8 | 81.5 | 81.6 | 81.6 | 81.6 | 81.6 | 83.9 | 83.9 | 83.9 | 83.9 | 83.9 | 83.9 | 84.7 | 84.7 | 84.7 | 84.7 |
| Mus | 44.9 | 45.3 | 44.8 | 44.8 | 45.4 | 45.4 | 55.3 | 54.6 | 54.8 | 55.7 | 55.5 | 55.5 | 33.6 | 33.3 | 32.5 | 33.2 | 52.7 | 52.5 | 52.5 | 52.5 | 52.5 | 57.2 | 57.2 | 57.2 | 57.2 | 57.2 | 57.2 | 57.1 | 57.1 | 57.1 | 57.2 |
| HQA | 64.7 | 64.6 | 64.8 | 64.4 | 64.6 | 65.1 | 67.4 | 67.7 | 67.6 | 67.5 | 67.4 | 68.1 | 49.0 | 51.5 | 51.7 | 50.5 | 67.3 | 68.0 | 68.0 | 68.0 | 68.0 | 70.6 | 70.6 | 70.5 | 70.5 | 70.5 | 70.5 | 69.3 | 69.3 | 69.4 | 69.4 |
| 2Wiki | 57.4 | 57.4 | 57.3 | 57.8 | 57.4 | 57.6 | 67.9 | 68.0 | 66.7 | 67.8 | 67.5 | 67.9 | 38.4 | 38.6 | 39.0 | 38.9 | 71.4 | 71.9 | 71.9 | 71.9 | 71.9 | 70.5 | 70.5 | 70.5 | 70.5 | 70.5 | 70.5 | 73.2 | 73.2 | 73.2 | 73.0 |
| NQA | 30.1 | 30.0 | 30.0 | 30.1 | 29.9 | 29.3 | 33.3 | 33.2 | 33.3 | 33.2 | 33.2 | 33.8 | 23.2 | 24.8 | 24.6 | 23.2 | 32.5 | 32.5 | 32.5 | 32.5 | 32.6 | 32.6 | 32.6 | 32.6 | 32.6 | 32.6 | 32.6 | 33.2 | 33.2 | 33.2 | 33.2 |
| Qas | 46.3 | 45.7 | 44.0 | 43.9 | 45.8 | 45.9 | 50.7 | 49.9 | 48.1 | 49.2 | 50.5 | 51.2 | 37.5 | 37.6 | 36.0 | 37.4 | 51.4 | 51.7 | 51.6 | 48.4 | 49.4 | 47.2 | 48.6 | 48.6 | 47.0 | 48.8 | 47.9 | 49.7 | 49.9 | 47.8 | 49.9 |
| GovR | 31.9 | 31.0 | 29.4 | 27.9 | 30.4 | 31.8 | 32.1 | 30.9 | 29.2 | 29.2 | 30.7 | 31.7 | 22.2 | 22.1 | 21.4 | 21.9 | 28.3 | 28.4 | 28.1 | 27.4 | 27.8 | 30.8 | 30.2 | 30.2 | 30.6 | 30.2 | 31.0 | 29.9 | 30.0 | 29.3 | 29.1 |
| LCC | 52.7 | 52.0 | 51.4 | 51.4 | 52.3 | 51.8 | 67.5 | 68.6 | 68.6 | 67.2 | 67.4 | 67.2 | 10.4 | 10.5 | 10.2 | 10.1 | 6.0 | 6.0 | 5.5 | 6.1 | 6.2 | 68.1 | 68.8 | 67.7 | 68.5 | 68.9 | 68.6 | 76.1 | 76.0 | 75.9 | 76.2 |
Table 5. Accuracy (%) in each dataset and their average. Base = baseline, Acc / Bal / SpdB = accuracy-best, balanced, and speed-best configurations, R32 = uniform window 256 with Quest budget 1024 re-selected every 32 decode steps, R1 the same at every step. Red / green mark a drop and a gain of more than 1pp against the baseline.
No visible loss in accuracy: These values show that our experiments were successful. The Best Accuracy and R32 configurations showed no accuracy degradation in any of the models and datasets. The Best Speed configuration obtained the best throughput while only reducing accuracy in three datasets when applied to the Qwen-4B model.
Qwen3.5-4B: The Best Accuracy, Balanced, and R32 configurations showed no degradation in accuracy in any individual dataset, and their average accuracy stayed within 0.5 percentage points of the baseline model. This indicates that our 1.19x–1.26x (64k tokens) and 1.87x–2.22x (256k tokens) speedups do not bring down the quality of the model.
Qwen3.5-122B: All of the final configurations tested using this model passed all the 21 long-context datasets, and their average accuracy stayed within 0.2 percentage points of the baseline. Once again, this validates that this method can reliably speed up models. Similarly to the previous model, we obtained 1.18–1.20x speedups in contexts with 64k tokens and 2.20–2.23x for 256k tokens.
Ornith-397B: Similarly to the previous model, there is no accuracy degradation in individual benchmarks, and the average accuracy of the configurations stays within 0.4pp of the baseline. In addition to the configurations tested in the previous models, we try pushing the uniform configuration to a lower budget, damaging the accuracy in 4 datasets.
The results show that this combination of pinned prefix + Quest + sliding window attention is an effective method to accelerate models in long-context tasks with little to no damage to the models’ capabilities.
Testing Qwen3.5-4B for Code Optimization
This method shows good results at preserving accuracy in long-context benchmarks, whether the configurations were obtained manually or through NSGA-II. We decided to extend the evaluation to kernel optimization using the PolyBench repository [24]. In this task, we attempt to optimize 4 kernels (gemm, correlation, heat-3d and lu) in parallel using batch size 4, and a sequence of requests until the context length reaches 256k tokens. The use case is detailed in the appendix (Section A.1).
For this optimization task, we compare the Qwen3.5-4B model with 3 configurations: Best-Accuracy and two uniform configurations using and 4096. We
do a series of prompts to the model, evaluating the proposed optimization at the end of each prompt. Prompt 1 described the problem; prompts 2-4 recommended focusing on memory access order; prompts 5-8 loop ordering and tiling; prompts 9-12 OpenMP; prompts 13-16 SIMD and vectorization; prompts 17-20 combining previous implementations and retuning parameters, and all prompts until reaching a 256k-token context alternated between asking the model for a different loop structure and tuning parameters.
For these kernels, the baseline model sped up 3 kernels (gemm, correlation, and heat-3d) and could not provide a valid solution for lu. The uniform configuration with produced a similar speedup in 2 kernels (gemm and correlation), and a small speedup in lu, failing to speed up correlation. Despite the task requiring a longer budget than the one used for the accuracy benchmarks, this uniform configuration preserved its 410 token/s throughput until the end of the run, while the baseline model reached the 256k token limit at 140 token/s.
Implementation and Computational Environment
Implementation in vLLM
We implement the method inside vLLM ([13]), a widely used inference engine. vLLM supports all six models in this study; however, none of them allow changing the attention of a full-attention layer through model configuration. We replace the attention class of each model at construction time, so that only the layers selected by a configuration are modified, without editing the weights or the configuration files. The replacement is installed as a vLLM plugin, so the method runs on an unmodified vLLM installation. Appendix § 7.§ 2 details the overrides, the Triton kernels, and how the configuration is verified on every GPU. Our implementation is publicly available at GitHub1
Computational environment
Our experiments were run on two systems. The first is a node with 2x AMD EPYC 7453 28-Core, 503 GiB Host memory, 7x NVIDIA A100 80GB PCIe (1935 GB/s (HBM2e) and peak BF16 of 312 TFLOP/s), with vLLM (0.24.0), PyTorch (2.11.0+cu129), transformers (5.13.0), datasets (5.0.1), Python (3.12), attention backends flash_attn (Qwen3.5) and triton_attn (Gemma 4 and gpt-oss, whose head sizes and attention sinks require it), JIT-compiled by Triton 3.6.0, compilation: vllm compile (inductor), CUDA graph mode: full and piecewise. The second is a node with 4x NVIDIA GB200 (189 GB each, aarch64), used for Qwen3.5-122B-A10B-FP8 and Ornith-1.0-397B-FP8 with tensor parallelism of 4, with vLLM (0.27.1), PyTorch (2.13.0+cu130) and the flashinfer attention backend.
Limitations
Speedups only at long context: Our configurations are slower than the baseline at short context, and only become faster after 32k to 64k tokens, depending on the model and the batch size. When the prompt is short, the first part of the output is therefore generated more slowly than by the unmodified model. The method is not a general solution to the slowdown of LLMs: it is a trade-off that pays off when long contexts dominate the workload.
Memory: Quest keeps the KV cache at full size (any page may be selected), so the method reduces compute but not memory. Plain SWA frees KV blocks on vLLM’s sliding path and so also reduces memory, but at the cost of lowering the model’s accuracy. Similarly, SnapKV also saves on memory by retaining only the top tokens of the input, but the static sinks did not perform well in preliminary work.
Conclusions
Hybrid models reduce the cost of long-context inference by replacing most of their attention layers with linear or sliding-window layers, but the full-attention layers that remain dominate the decoding time at long context. We linearize the computational cost of the full attention layers of 6 models by limiting the amount of tokens they attend to the first 256 tokens, the last tokens (sliding window attention), and the most relevant tokens using Quest. and are two parameters that had to be discovered for each layer, resulting in a heterogeneous LLM configuration.
For each model, we compare 6 different configurations: the baseline model; two uniform configurations with the same parameters in all layers (=1024, =256) that update the top tokens every step and every 32 steps; and three configurations obtained using NSGA-II as a multi-objective search algorithm (Best Accuracy, Best Speed, and Balanced).
Our results show that we can obtain 1.16–1.26x and 2.06–2.23x speedups (batch size 4) at context lengths of 64k and 256k tokens, respectively, when using hybrid MoE models (Qwen3.5-122B and Ornith-397B), or dense models with a low parameter count (Qwen3.5-4B). For other dense models (Qwen3.5-27B and Gemma-4-31B), the speedups obtained were limited to 1.11x and 1.09x, respectively. For gpt-oss-120b, we only obtained minor speedups near the model’s context length limit. The experiments showed no visible loss in the accuracy of the proposed configurations.
We motivate this work by asking how much we can restrain the attention of full-attention layers without accuracy degradation. These results obtained by combining three sparse-attention methods tell us that we can place heavy restraints in the models, as long as the sparse-attention method is dynamic.
Appendix
Use case: Polybench
Despite the configurations successfully passing the long-context benchmarks, we try applying them to a simple use case: optimizing 4 polybench benchmarks. To do so, we select 4 kernels () and optimize all of them in parallel using batch size 4. For each kernel we make several prompts asking the Qwen3.5-4B model for additional optimization:
Prompt 1: See Listing 1, applied to gemm.
Prompts 2–4: “focus on memory access order: loop interchange so the innermost loop walks contiguous memory.”
Prompts 5–8: “focus on cache blocking (tiling); experiment with the tile sizes.”
Prompts 9–12: “parallelize with OpenMP over the 4 threads (work split, scheduling, no data races).”
Prompts 13–16: “focus on SIMD: make the inner loops vectorize (restrict, #pragma omp simd, unrolling, AVX2/FMA intrinsics).”
Prompts 17–20: “combine the best ideas so far and tune their parameters.”
Prompts 21+: Until a context length of 256k tokens is reached, we alternate every 4 attempts between “Try a substantially different loop structure or algorithmic formulation.” and “Tune your best version further: tile sizes, unroll factors, scheduling, alignment.”
In Figure 5 we see that the decode speed does not slow down even when moving long-context. Results wise, as seen in Figure 6, despite this being a small non-coding model, it obtained a speedup in 3 out of 4 kernels, providing no correct solutions in the last kernel. By comparison, the uniform configuration also showed similar results, obtaining a speedup in 3 ou of 4 kernels. The best-accuracy configuration, despite passing all the 21 long-context benchmarks, was unable to provide optimizations in this task.
Figure 5. Throughput of the baseline Qwen3.5-4B model, best-accuracy configuration, and two uniform configurations with SWA and Quest budgets or 2048 and 4096 tokens.
Figure 6. Speedup obtained in 4 polybench kernels (gemm, correlation, heat-3d and lu) as the conversation with each configuration moved towards 256k tokens. The lack of lines implies there was no correct code in that interval.
Modifications to vLLM
Per-layer sliding windows in vLLM
As previously mentioned, our pipeline is based on vLLM [13]. vLLM supports per-layer windows, allowing us to easily replace full-attention layers with SWA, which changes both the KV-cache allocation and the arguments passed to the FlashAttention kernel. Since the window values are defined per layer, it allows us to easily explore using different window sizes over the model, rather than experimenting with a homogeneous SWA implementation.
Qwen3.5: vLLM’s implementation reuses
Listing 1. Prompt 1, applied to gemm.
You are optimizing a C kernel from the
PolyBench/C 4.2.1 suite for speed. The
full benchmark file is below; only the
function ‘kernel_gemm‘ is yours to
change.
Machine and build:
- AMD EPYC 7453 (Zen 3): AVX2 + FMA, no AVX
-512; 4 cores for your program
OMP_NUM_THREADS=4).
- gcc 13: `gcc -O3 -march=native -fopenmp -
DEXTRALARGE_DATASET`. Problem sizes
come from gemm.h, EXTRALARGE dataset.
- DATA_TYPE is double.
Rules:
- Keep the exact signature of `kernel_gemm
`. You may add helper functions, #
include's (e.g. <omp.h>, <immintrin.h>)
and pragmas above it. No external
libraries (no BLAS).
- The output arrays must match the original
to a relative tolerance of 1e-06.
Each attempt is compiled, checked and timed
(median of 3 runs) and I report the
result. We will iterate for many
attempts: keep improving on your best
version.
Reply with the complete `kernel_gemm`
function (and any helpers) in a single
```c code block.
gemm.h:
```c <full header>```
gemm.c:
```c <full source>```
Qwen3NextAttention, never passing a window, and the window is not reachable through any config override. Instead of editing the config, we use the prototype in Listing 2 as a wrapper, allowing us to define a window size.
This window is applied to prompt and generation jointly: a query at position attends to , with no attention sink or global tokens retained. This is the plain banded mask of Longformer’s local component, not StreamingLLM’s sink-preserving variant [27].
Query-aware sparse attention in vLLM
Our pipeline is based on vLLM [13]. In none of the six models is the attention of a full-attention layer reachable through a config override: Qwen3.5 reuses Qwen3NextAttention, and Gemma 4 and gpt-oss build their layers from the module-level Attention class. Instead of editing the configs or the weights, we replace that class in each model module with the factory in Listing 3. It reads the layer index from the module prefix vLLM builds while constructing the model and rewires only the layers selected by the configuration.
Learned sinks: For gpt-oss, the per-head sink logit is added to the softmax denominator of the sparse attention, , which reproduces the model’s attention exactly when every page is selected.
Multi-GPU deployment: With tensor parallelism, vLLM builds the model in spawned worker processes, which do not inherit the override in the parent process. A vLLM plugin reinstalls it in every worker from an environment variable. After initialisation, every rank reports which layers it modified, and the run is aborted if any rank differs from the intended configuration.
Listing 2. Sliding-window attention override
def attention_with_sliding_window(*args,
prefix: str = "", **kwargs):
window = windows.get(
extract_layer_index(prefix))
if window is not None:
kwargs["per_layer_sliding_window"]
= window
return original_attention(*args, prefix
=prefix, **kwargs)
qwen3_next.Attention =
attention_with_sliding_window
Listing 3. Per-layer attention override
def attention_factory(*args, prefix: str =
"", **kwargs):
layer = original_attention(*args,
prefix=prefix, **kwargs)
idx = extract_layer_index(prefix)
if idx in config:
window, budget = config[idx]
apply_quest_attention(layer, window
, budget, pinned=num_sinks)
return layer
for module in(qwen3_next, gemma4, gpt_oss):
module.Attention = attention_factory
Profiling vLLM
Full Attention vs Uniform Configurations
This section gives illustrates the time spent in each kernel category when decoding long-context sequences. Figure 7 illustrates the average time spent decoding each token when the context length is already at 64k tokens. We see that despite replacing full attention with quest saves us a lot in compute, the next bottleneck of the system is the time spent in loading weights. We also see that updating Quest at every decode step results in the throughput being slower when compared to decoding every 8, 32 or 64 steps. Despite we updating Quest every 32 tokens, this figure shows that there would be little loss in throughput if we updated every 8 tokens instead.
Throughput
Figure 8 shows the tokens per second throughput of all configurations using difference batch sizes.
Configurations
Table 6 shows the configurations obtained for each model.
| Qwen3.5-4B | 3 | 7 | 11 | 15 | 19 | 23 | 27 | 31 | ||||||||||
| Best Accuracy | 256 | 4096 | 1024 | 4096 | full | 2048 | 256 | 1024 | ||||||||||
| 512 | 256 | 512 | 2048 | – | 2048 | 2048 | 2048 | |||||||||||
| Balanced | 256 | 512 | 256 | 4096 | 256 | 256 | 256 | 256 | ||||||||||
| 1024 | 256 | 512 | 2048 | 1024 | 512 | 512 | 2048 | |||||||||||
| Best Speed | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | ||||||||||
| 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | |||||||||||
| Uniform 1024 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | ||||||||||
| 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | |||||||||||
| Qwen3.5-27B | 3 | 7 | 11 | 15 | 19 | 23 | 27 | 31 | 35 | 39 | 43 | 47 | 51 | 55 | 59 | 63 | ||
| Best Accuracy | 4096 | 256 | 256 | 256 | 256 | 256 | 256 | 2048 | 256 | 256 | 256 | 256 | 4096 | 2048 | 4096 | 256 | ||
| 1024 | 512 | 512 | 256 | 1024 | 512 | 512 | 512 | 512 | 4096 | 1024 | 256 | 512 | 512 | 512 | 512 | |||
| Balanced | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | ||
| 512 | 2048 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | |||
| Best Speed | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | ||
| 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | 512 | |||
| Uniform 1024 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | ||
| 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | |||
| Gemma-4-31B | 5 | 11 | 17 | 23 | 29 | 35 | 41 | 47 | 53 | 59 | ||||||||
| Best Accuracy | full | 1024 | full | full | full | full | full | full | full | 4096 | ||||||||
| – | 1024 | – | – | – | – | – | – | – | 1024 | |||||||||
| Best Speed | full | 4096 | 512 | 1024 | 256 | full | full | full | 1024 | 2048 | ||||||||
| – | 256 | 256 | 1024 | 1024 | – | – | – | 2048 | 2048 | |||||||||
| Uniform 1024 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | ||||||||
| 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | |||||||||
| gpt-oss-120b | 1 | 3 | 5 | 7 | 9 | 11 | 13 | 15 | 17 | 19 | 21 | 23 | 25 | 27 | 29 | 31 | 33 | 35 |
| Best Acc. / Speed | full | 4096 | 1024 | full | full | full | full | full | full | full | full | full | 256 | full | full | full | 512 | full |
| – | 256 | 1024 | – | – | – | – | – | – | – | – | – | 512 | – | – | – | 1024 | – | |
| Uniform 1024 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 |
| 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | |
| Qwen3.5-122B-A10B-FP8 | 3 | 7 | 11 | 15 | 19 | 23 | 27 | 31 | 35 | 39 | 43 | 47 | ||||||
| Best Accuracy | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 512 | ||||||
| 1024 | 1024 | 1024 | 2048 | 1024 | 1024 | 1024 | 1024 | 1024 | 4096 | 1024 | 1024 | |||||||
| Balanced | 256 | 256 | 256 | 256 | 256 | 256 | 512 | 256 | 256 | 256 | 256 | 512 | ||||||
| 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 512 | |||||||
| Best Speed | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 512 | ||||||
| 1024 | 1024 | 1024 | 1024 | 1024 | 512 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | |||||||
| Uniform 1024 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | ||||||
| 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | |||||||
| Ornith-1.0-397B-FP8 | 3 | 7 | 11 | 15 | 19 | 23 | 27 | 31 | 35 | 39 | 43 | 47 | 51 | 55 | 59 | |||
| Best Accuracy | 256 | 4096 | 256 | 2048 | 2048 | 256 | 256 | full | 512 | 512 | 256 | 512 | full | 4096 | 256 | |||
| 1024 | 4096 | 256 | 4096 | 512 | 1024 | 4096 | – | 4096 | 1024 | 1024 | 1024 | – | 1024 | 1024 | ||||
| Balanced | 512 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 1024 | |||
| 1024 | 256 | 1024 | 4096 | 512 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | ||||
| Best Speed / Uniform 1024 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | |||
| 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | 1024 | ||||
| Uniform 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | |||
| 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 | 256 |
Table 6. Configurations evaluated for each model: the window and Quest budget (tokens) of every full-attention layer, indexed by its position in the model. ‘full’ marks a layer left at full attention, whose budget is irrelevant (‘–’). On gpt-oss-120b, the search front contained a single configuration; on Ornith, Best Speed is the Uniform 1024 configuration.
References
- [1]Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. LongBench: A bilingual, multilingual benchmark for long context understanding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119–3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172/.DOI
- [2]Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer, 2020. URL https://arxiv.org/abs/2004.05150.
- [3]Chen, Y., Thai, Z. L., Zhou, Z., Zhang, Z., Shen, X., Wang, S., Xiao, C., Han, X., and Liu, Z. Hybrid linear attention done right: Efficient distillation and effective architectures for extremely long contexts, 2026. ICML 2026.arxiv.org/abs/2601.22156
- [4]Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2205.14135
- [5]Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6 (2):182–197, April 2002. ISSN 1089-778X. doi: 10.1109/4235.996017. URL http://dx.doi.org/10.1109/4235.996017.
- [6]Gemma Team, Abd, S. E., Aggarwal, V., AlGayres, R., et al. Gemma 4 technical report, 2026.arxiv.org/abs/2607.02770
- [9]Iba, H., Batista, J. E., and Xu, J. LLMs for Metaheuristics, pp. 157–195. Springer Nature Singapore, Singapore, 2026. ISBN 978-981-95-8597-7. doi: 10.1007/978-981-95-8597-7_4. URL https://doi.org/10.1007/978-981-95-8597-7_4.
- [10]Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825.
- [11]Kimi Team. Kimi k3: Open frontier intelligence, 2026.arxiv.org/abs/2607.24653
- [12]Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., and Burtsev, M. Babiling: Testing the limits of llms with long context reasoning-in-a-haystack. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 106519–106554. Curran Associates, Inc., 2024. doi: 10.52202/079017-3381.DOI
- [13]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, 2023. URL https://doi.org/10.1145/3600006.3613165.
- [14]Lan, D., Sun, W., Hu, J., Du, J., and Cheng, Y. Linearizing large language models to gated recurrent structures. In Singh, A., Fazel, M., Hsu, D., Lacoste-Julien, S., Berkekamp, F., Maharaj, T., Wagstaff, K., and Zhu, J. (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 32452–32466. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/lan25b.html.arxiv.org/abs/2503.01496
- [16]Liu, Z., Kundu, S., Jiang, L., Li, A., Ronanki, S., Bodapati, S. B., Datta, G., and Beerel, P. A. LAW-CAT: Efficient distillation from quadratic to linear attention with convolution across tokens for long context modeling. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 20865–20881, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.1138. URL https://aclanthology.org/2025.findings-emnlp.1138/.
- [17]Mercat, J., Vasiljevic, I., Keh, S., Arora, K., Dave, A., Gaidon, A., and Kollar, T. Linearizing large language models. In First Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2405.06640
- [18]MiniMax, Li, A., Gong, B., Yang, B., et al. MiniMax-01: Scaling foundation models with lightning attention, 2025.arxiv.org/abs/2501.08313
- [19]Nawrot, P., Li, R., Huang, R., Ruder, S., Marchisio, K., and Ponti, E. M. The sparse frontier: Sparse attention trade-offs in transformer llms, 2026. URL https://arxiv.org/abs/2504.17768.
- [20]OpenAI, Agarwal, S., Ahmad, L., Ai, J., et al. gpt-oss-120b & gpt-oss-20b model card, 2025.
- [21]Peng, B., Albalak, A., Anthony, Q., Albalak, A., Arcandino, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., Lau, H., Lin, J., Mantri, K. S. I., Mom, F., Saito, A., Song, G., Tang, X., Wind, J., Woźniak, S., Zhang, Z., Zhou, Q., Zhu, J., and Zhu, R.-J. RWKV: Reinventing RNNs for the transformer era. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048–14077, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.936. URL https://aclanthology.org/2023.findings-emnlp.936/.
- [22]Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Re, C. Hyena hierarchy: Towards larger convolutional language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28043–28078. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/poli23a.html.arxiv.org/abs/2302.10866
- [23]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
- [24]Reisinger, M. MatthiasJReisinger/PolyBenchC-4.2.1, August 2024. URL https://github.com/MatthiasJReisinger/PolyBenchC-4.2.1.
- [25]Tang, H., Lin, Y., Lin, J., Han, Q., Hong, S., Yao, Y., and Wang, G. RazorAttention: Efficient KV cache compression through retrieval heads. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2407.15891.
- [26]Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasicki, B., and Han, S. QUEST: Query-aware sparsity for efficient long-context LLM inference. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47901–47911. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/tang241.html.arxiv.org/abs/2406.10774
- [27]Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=NG7sS51zVF.arxiv.org/abs/2309.17453
- [28]Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. Efficient long-context llm inference with retrieval and streaming heads. In Yue, Y., Garg, A., Peng, N., Sha, F., and Yu, R. (eds.), International Conference on Learning Representations, volume 2025, pp. 37228–37253, 2025.arxiv.org/abs/2410.10819
- [29]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 technical report, 2025a. URL https://arxiv.org/abs/2505.09388.
- [30]Yang, S., Kautz, J., and Hatamizadeh, A. Gated delta networks: Improving Mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025b. URL https://openreview.net/forum?id=r8H7x hYPwz.arxiv.org/abs/2412.06464
- [31]Zaheer, M., Gurug anesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/c8512d142a2d849725f31a9a7a361ab9-Abstract.html.arxiv.org/abs/2007.14062
- [32]Zhang, M., Bhatia, K., Kumbong, H., and Re, C. The hedge-hog & the porcupine: Expressive linear attentions with softmax mimicry. In Kim, B., Yue, Y., Chaudhuri, S., Fragkiadaki, K., Khan, M., and Sun, Y. (eds.), International Conference on Learning Representations, volume 2024, pp. 53551–53580, 2024.
- [33]Zhang, M., Arora, S., Chalamala, R., Spector, B., Wu, A., Ramesh, K., Singhal, A., and Re, C. Lolcats: On low-rank linearizing of large language models. In Yue, Y., Garg, A., Peng, N., Sha, F., and Yu, R. (eds.). International Conference on Learning Representations, volume 2025, pp. 46200–46253, 2025.
- [34]Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z. A., and Chen, B. H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 34661–34710. Curran Associates, Inc., 2023. doi: 10.52202/075280-1506.DOI






