WorksComputer Science
Jet-QSparse: Accelerating Agentic LLM Inference with 4-Bit Sparse Attention
Abstract
Long-horizon reasoning and agentic workloads strain KV-cache capacity and decode attention throughput. We present Jet-QSparse, a training-free method that combines compact NVFP4 cache storage with native FP4 query-key computation on Blackwell tensor cores. A dense scan scores the full key history and selects 128-token blocks for sparse FP16 value aggregation with FP32 accumulation, preserving higher precision in the output-forming step while enabling contiguous tile loads. Calibrated key centering and at most two BF16 pages per sequence and layer address quantization error; FP8 prefill uses a temporary arena while the persistent cache is packed directly from BF16 activations. Across four hybrid LLMs, a budget of 128 blocks per KV head shows no detected accuracy loss on the evaluated mathematical reasoning, coding, and agentic benchmarks, although some smaller budgets incur losses. With 64 blocks, Qwen3.8-27B achieves a end-to-end decode speedup over dense BF16 at 128K context and batch size 16 on one GB300, and the measured Qwen KV pools provide - BF16's token capacity.
Introduction
Reasoning and agentic language models process long histories of intermediate reasoning, tool outputs, and interaction. The key–value (KV) cache grows with these histories and limits the number of requests that fit on a GPU. Dense decode attention reads and processes the history for every generated token, increasing latency as the sequence grows. Efficient inference requires a smaller persistent cache and cheaper attention while preserving task accuracy.
KV quantization reduces the bytes stored per token ([24, 14]), while sparse decode reduces the data fetched and processed for each query ([43, 40] (Yuan et al. 2026)). Blackwell supports native NVFP4 computation: packed 4-bit operands reduce memory traffic and feed block-scaled tensor-core operations [29] (Du et al. 2026). An efficient attention method must coordinate the precision used to score keys, the blocks selected for value aggregation, and the layout used to fetch them. Selection adds work, irregular accesses can limit hardware utilization, and errors from quantization and sparsification must be evaluated together.
Jet-QSparse combines compact NVFP4 storage and dense native FP4 scoring with sparse higher-precision aggregation in a simple, training-free design. Its NVFP4 cache retains the complete token history. At each decode step, native FP4 query–key computation scores every key and estimates block attention masses. Query-dependent selection restricts the output-forming product to selected blocks, using FP16 operands and FP32 accumulation. Calibrated key centering and BF16 copies of the first and current blocks address quantization error within two pages per sequence and layer.
We group 128 consecutive tokens per cache page to align selection with contiguous memory loads and tensor-core tiles. Query heads sharing a KV head reuse selected blocks, and selection executes on the GPU within CUDA graphs. Prepacked scales feed native QK computation, and pure decode steps fuse query packing with cache writes. The implementation integrates into a production serving engine. Prefill uses FP8 attention while the persistent NVFP4 cache is packed directly from BF16 activations.
We study sparsification under quantization through attention-output analysis, dense NVFP4 controls, and block-budget sweeps. The analysis relates selection error to omitted mass and the gap between selected and omitted value averages. Long-context diagnostics show that aggressive sparsity can impair aggregation over distributed evidence while largely preserving needle retrieval; agentic tests reveal model-dependent failure patterns. These findings motivate evaluating precision and sparsity together at the task level.
We evaluate Jet-QSparse on Qwen3.6-35B-A3B, Qwen3.8-27B, Nemotron 3 Nano, and Nemotron 3.5 Lightning. These hybrid models span different generations, dense or mixture-of-experts (MoE) feedforward layers, and different proportions of full-attention layers. We also deploy Qwen3.8-2.4T-A95B across eight nodes. On the two Qwens, the measured cache pools provide 2.58–2.76× the KV-token capacity of dense BF16 under the same allocation policy. At 128K context and batch size eight on one GB300, 64 selected blocks per KV head accelerate Qwen3.8-27B and Qwen3.6-35B-A3B decode by 1.53× and 1.32× over dense BF16, respectively. Every compared method fits this workload. Both configurations show no detected accuracy loss on five reasoning, coding, and agentic benchmarks at the evaluation’s statistical resolution; a 128-block budget meets this criterion across all four models. Figure 1 compares speed, accuracy, and occupied KV memory at this shared budget.
Figure 1. SWE-bench Verified accuracy (mean ± SD, three seeds) versus separate GB300 decode speedups (128K prompt/4K output, batch 8); Jet-QSparse uses . Dashed lines mark BF16 accuracy. Leaders connect memory callouts to measured points. Semicircle areas are proportional to occupied KV (% of BF16): left, 4K/16K (one run); right, 128K/4K (two runs), both at batch 8. Panels share one area scale; x-axis ranges differ. Bytes include pages, metadata, and active BF16 copies, excluding unused reservations, recurrent state, and prefill workspace.
Smaller budgets incur losses on some model–benchmark pairs, which we report alongside the performance gains.
Our contributions are:
We introduce Jet-QSparse, a simple, training-free attention method that combines compact NVFP4 storage and native low-precision scoring with sparse higher-precision aggregation. Its block-based cache aligns selection with contiguous memory access and tensor-core execution, enabling faster decoding and lower KV memory use within a production serving engine.
We systematically study the additional error from sparsifying quantized attention and its effects on long-context and agentic tasks. We relate selection error to omitted mass and value differences, and use Jet-QSparse’s dense NVFP4 controls and budget sweeps to examine task sensitivity and model-dependent failure patterns.
We evaluate Jet-QSparse across Qwen and Nemotron model families on mathematical reasoning, coding, and agentic benchmarks, demonstrating decode speedups and increased KV capacity. A 128-block budget shows no detected accuracy loss on the five application benchmarks across all four models, and we extend the evaluation to a 2.4T-parameter model with 95B active parameters.
Related Work
Hybrid Attention Models. Linear attention uses fixed-size states for recurrent decoding [20]. Mamba uses selective state-space recurrences; Mamba-2 relates them to structured attention through state-space duality (Gu & Dao, 2024; Dao & Gu, 2024). Gated DeltaNet (GDN) combines state-decay gates with delta-rule updates to key–value associations (Yang et al., 2025). Hybrids interleave these recurrent mechanisms with softmax attention. Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, and Qwen3.8-27B each interleave three GDN blocks with one gated full-attention block [34, 35 b]. Nemotron 3 Nano and Nemotron 3.5 Lightning combine Mamba-2, MoE, and six full-attention GQA blocks in 52-block backbones [28, 30]. Jet-QSparse reduces the KV-cache costs of these attention layers without changing recurrent or feedforward blocks.
Attention kernels and serving systems. FlashAttention-2/3/4 improve dense attention through tiling, work partitioning, and hardware-specific pipelines [8, 41] (Zadouri et al. 2026). PagedAttention and FlashInfer provide cache management and paged execution [21] (Ye et al. 2025). TensorRT-LLM integrates paged execution, quantized caches, and multiple attention backends (NVIDIA, 2026b;d). Its trtllm-gen backend pairs NVFP4 KV with FP8 queries (NVIDIA, 2026c); Jet-QSparse uses NVFP4 Q and K in its dense scan.
Low-bit arithmetic and KV storage. SageAttention uses key centering and INT8 QK; SageAttention2 combines INT4 QK with FP8 PV; SageAttention3 extends both products to NVFP4 (Zhang et al., 2025c;a;b). Attn-QAT studies quantization-aware training and provides kernels with NVFP4 QK and higher-precision PV; Hardware-Aware FP4 FlashAttention-4 quantizes probabilities for noncausal forward attention [16] (Zhang et al. 2026). KIVI quantizes keys per channel and values per token with a full-precision residual buffer; KVQuant combines pre-RoPE key quantization with nonuniform codebooks and separately stored outliers; QuaRot uses invariant rotations [24, 14, 3]. BitDecoding maps dense low-bit-cache attention to tensor cores, including native MXFP4/NVFP4 computation on Blackwell (Du et al., 2026).
Query-dependent sparse decode. Quest bounds page scores using channel-wise key minima and maxima [43]. SparQ estimates token importance from query coordinates with large magnitudes, fetches selected full K/V vectors, and compensates for estimated omitted mass using a running value mean [40]. Loki uses an offline per-head PCA basis to rank tokens in reduced dimensions before attending to selected full-dimensional vectors [42]. Jet-QSparse scans all NVFP4 keys and selects blocks by globally normalized attention mass.
KV eviction and retention. H2O keeps recent tokens and attention heavy hitters; StreamingLLM keeps sinks and a rolling window; SnapKV compresses the prompt cache using a trailing observation window [46] (Zhang et al. 2023; Li et al. 2024). DuoAttention separates full-history retrieval and streaming heads through optimized gates [47], while RocketKV combines prompt eviction with per-step selection [4]. Tri-Attention periodically evicts generation KV using calibrated pre-RoPE query statistics and trigonometric analysis [26]. ThinKV combines thought-dependent precision with progressive eviction [38]. Jet-QSparse retains every token in NVFP4 and selects aggregation blocks at each step.
Sparse prefill and trained selectors. MInference, FlexPrefill, and XAttention use training-free sparse prefill patterns to reduce QK and PV work [18, 22] (Xu et al. 2025); PAROAttention combines block sparsity and low-bit arithmetic for visual generation (Zhao et al., 2025). SeerAttention trains block gates, and SeerAttention-R extends learned, GQA-shared selection to reasoning decode [11, 12]. MoBA uses parameter-free routing in sparsely trained models; Native Sparse Attention and DeepSeek Sparse Attention introduce learned compression or indexing [1] (Lu et al. 2025; Yuan et al. 2025). Jet-QSparse calibrates existing full-attention layers without weight updates.
Sparse aggregation after scoring. SpargeAttention combines pooled Q/K selection with online-softmax PV skipping (Zhang et al., 2025d). BLASST compares tile and running maxima to skip softmax, V loads, and PV without a separate selection pass (Yuan et al., 2026). Jet-QSparse uses a separate dense NVFP4 scan to select shared 128-token blocks under a fixed budget or capped tail-mass policy, then performs FP16 PV with FP32 accumulation. Caps may override the mass target.
Jet-QSparse: Hardware-Optimized 4-Bit Sparse Attention
Jet-QSparse uses NVFP4 as both a compact storage format and a native tensor-core operand. Storing the KV cache in NVFP4 reduces its footprint and read traffic; computing directly from packed queries and keys uses Blackwell’s FP4 tensor cores to score the full context. We retain FP16 operands and FP32 accumulation for PV, the weighted sum that forms the attention output. Sparsity reduces the cost of this higher-precision step by restricting aggregation to selected 128-token blocks.
Figure 2 summarizes the pipeline: dense NVFP4 scoring followed by sparse higher-precision aggregation. Blocks align selection with contiguous memory loads and tensor-core tiles. Calibrated key centering and BF16 copies of the first and current blocks address quantization error within a two-page allowance. The method targets single-token causal decode in full-attention layers and requires no model-weight updates. Prefill uses a separate FP8 path (§ 6.4).
Figure 2. Jet-QSparse decode. Logical blocks pair key and value pages for 128 consecutive tokens; internal divisions are schematic. The dense scan emits FP32 statistics from every key block. First/current-block statistics are recomputed from BF16 keys before normalization. Each query-head group (Ainslie et al., [2]) shares selected K/V blocks, always including the current block. Selected scores are recomputed for online softmax. FP16 unnormalized weights multiply reconstructed FP16 values with FP32 accumulation; denominator weights remain FP32. Merged partials are normalized and cast to BF16. BF16 paths use the same quantized query and FP16 operands. Reconstruction retains NVFP4 storage error. An unselected first page may still be read and masked.
NVFP4 Cache and Block Layout
Aligning selection with memory access. Token-level selection can scatter memory accesses and create irregular workloads. We group consecutive tokens into a block, storing its keys and values in matching cache pages (Figure 3). We select whole token blocks. Each KV head occupies contiguous regions for packed codes, micro-block scales, and row scales within a page. A page table locates these regions so kernels can load contiguous tiles even when selected pages are not adjacent in physical memory. Using whole blocks trades finer selection granularity for regular memory access and block-sized matrix products.
Figure 3. Blocked cache layout and lifetime. (a) K/V pages cover matching 128-token ranges; logical neighbors need not be physically adjacent. (b) Bytes cover one head’s contiguous K or V region; compression excludes fixed buffers. (c) Token 384 opens block 3, reusing the current BF16 slot. Block 0’s BF16 copy (Xiao et al., [46]; Hooper et al., [14]) and all NVFP4 copies persist; the live row is enlarged. (d) BF16 inputs feed both paths independently. FP8 pages become reusable; the arena remains reserved.
Packing keys and values
A row contains elements for one token and one KV head. Its NVFP4 reconstruction is , where is an E2M1 code, is an E4M3 scale shared by a micro-block of 16 elements, and is an FP32 row scale. For zero-based element index , . Scaling the row’s maximum magnitude by gives , where 6 and 448 are the E2M1 and E4M3 maxima. Each micro-block’s maximum magnitude then sets its E4M3 scale, and its elements are rounded to E2M1. Zero scales are replaced by one, so an all-zero row reconstructs as zero. Keys and values are packed independently; the query uses the same format. Including scale metadata, a row occupies
This is 4.625 bits per element at and 4.75 at and 3.37 packed-row compression relative to BF16, excluding fixed buffers.
Centering keys
A component shared across keys can consume quantization range without affecting their relative attention scores. We subtract a calibrated mean for each layer and KV head after applying RoPE, where used. In exact arithmetic, subtracting this common vector adds the same constant to every logit of a query and leaves softmax unchanged (Zhang et al., 2025c). Quantizing the centered keys can still change attention. An offline attention-output error screen determines which heads use a nonzero mean; the remaining heads use .
Retaining two BF16 pages
We retain BF16 copies of block 0 and the current block alongside their NVFP4 copies. The side buffer holds centered keys rounded to BF16 and the original BF16 values, covering at most two distinct blocks per sequence and layer. We pack every row into NVFP4 on its first write. After the current page advances, we read a completed noninitial block from its existing NVFP4 copy; block 0’s BF16 copy persists throughout the sequence. We mask unused positions in the current page. Selection determines whether block 0 contributes to the sparse output.
Dense Scoring for Block Selection
Selecting blocks by their share of attention requires a normalizer over the full context. We obtain it with a dense NVFP4 scan that reads keys and scales without fetching values or materializing the attention matrix.
Consider one KV head and its group of query heads. Let be the cached sequence length, including the current token, the number of live blocks, and the live token positions in block . Denote the reconstructed NVFP4 query by . The effective operands , are the BF16 copies in the first and current blocks and the reconstructed NVFP4 operands elsewhere. In exact arithmetic,
Thus, measures a block’s importance under the method’s dense attention distribution. Dense here refers to coverage of all keys.
The scan records a reference maximum and an exponential sum for each block and query head. A correction kernel replaces the first and current blocks’ statistics using their BF16 keys and the same quantized query. After this correction, we rescale the statistics to a common maximum and normalize across blocks. This gives selection and aggregation the same effective operands. We write for the kernel’s FP32 estimates of the ideal masses in Equation (2).
Selecting Blocks for Grouped-Query Attention
Each KV head uses one selected block set for all query heads in its group, sharing the fetched K/V tiles. A fixed budget controls aggregation work, while a tail-mass policy adapts the selected count to the attention distribution. Both policies include the current block . Figure 4 illustrates their selections for one query-head group.
Figure 4. Illustrative block attention for two query heads. Green outlines mark selected blocks. Fixed includes current block 7; counts are exact, ranks approximate. Tail selection unions the head sets and current block, yielding six blocks with nonbinding four-block caps. Binding caps can override the mass target. Omitted percentages follow head order.
Fixed block budget. We rank each block by the largest mass assigned to it within the group, . A block can therefore rank highly when any head assigns it substantial mass. Setting ensures that the current block occupies one of the selected slots. A 16-step threshold bisection finds the ranking boundary; any remaining slots are filled from the final boundary band in ascending block order. The selected count is exact, while the top- ranking is approximate.
We evaluate , selecting at most 4,096, 8,192, or 16,384 token positions per KV head. All blocks are selected when the context fits within the budget. A fixed count does not bound an individual head’s omitted mass: blocks preferred by other heads can fill the shared budget.
Tail-mass budget. Each query head has an omitted-mass budget and a calibrated cap fraction . We set
to the 99th percentile of the context fraction needed on held-out prompts at omitted mass 0.125. Calibration uses 16-token pages; decode converts the fraction to a cap on 128-token blocks, , ignoring FP32 rounding in this expression.
The tail threshold keeps the sum of masses below it within ; the cap threshold retains at most blocks. We find both by 16-step bisection on , taking the feasible lower endpoint for the tail constraint and the feasible upper endpoint for the cap:
The cap takes precedence over the mass budget and applies to , not the final group union. Thresholding can underfill a head’s cap, even leaving an empty set. In the evaluated configurations, for every head: and 0.60 correspond to omitted-mass budgets of 0.60 and 0.40. Only varies across heads. sets a mass target rather than a fixed block fraction.
Sparse Value Aggregation
We fetch the selected pages and recompute their QK products, avoiding a write and reread of token-level scores from the dense scan. With , the ideal output is
This renormalizes softmax over selected keys.
Effect of omitted mass. Let be dense attention on the same effective operands and the omitted mass. For , let be the normalized output of the omitted keys. Then
The diameter is the largest pairwise distance between effective values; the error is zero at . With exact masses and arithmetic, the tail policy ensures when its cap does not bind, i.e., . The group union can only reduce omitted mass. A binding cap removes this guarantee, and fixed provides no such bound. FP32 mass estimation perturbs the ideal guarantee; quantization error relative to BF16 attention is a separate source of error.
Higher precision for the weighted sum. The gather kernel multiplies FP16 unnormalized softmax weights by FP16 reconstructed values, accumulating in FP32. This preserves higher precision in the output sum while sparsity limits the number of values it processes. Reconstruction retains NVFP4 storage error; it cannot restore BF16 values.
The precision choice also matches the cache layout: NVFP4 scales run along the head dimension, the reduction dimension of QK, whereas PV reduces over tokens. The gather kernel reconstructs selected values as it consumes them. A separate kernel processes selected BF16 pages with FP16 operands. The kernels emit partial outputs and softmax statistics for a stable merge, followed by a BF16 output cast. The denominator uses FP32 weights; rounding the numerator weights to FP16 and the remaining kernel arithmetic introduce error beyond Equation (4).
Execution and Prefill
Realizing the decode savings. The implementation uses TensorRT-LLM’s paged cache manager, CuTe ([7]) decode kernels, and Triton ([45]) selection and BF16-page kernels. We store K scales once in the layout consumed by the block-scaled MMA. Pure decode steps fuse query packing with the new token’s K/V store. Block lists remain on the device, and both selection policies execute within CUDA graphs.
A sparse step reads every key in the scan, then rereads selected keys and fetches their values. In a streaming model without cache reuse, the K/V-read volume relative to dense NVFP4 attention approaches , where is the selected block fraction. This approximation excludes block-statistic traffic, fixed BF16-page overheads, and page padding. It captures read savings when a small fraction is selected at long context and extra reads when most blocks are selected. The dense NVFP4 control directly attends all blocks with the same cache and precision choices, omitting the scan and selector to isolate the cost and benefit of selection.
Separating prefill from decode storage. Prefill uses FlashInfer’s FP8 context attention with calibrated per-layer power-of-two scales. Before attention, the store packs the persistent cache directly from BF16 K/V, updates the two BF16 pages, and writes a temporary FP8 arena. Decode therefore incurs no FP8-to-NVFP4 conversion. We release a request’s arena pages when prefill finishes; decode reads only the NVFP4 pool and BF16 side pages. Requests without arena space fall back to context attention over expanded cache operands. Cache-capacity measurements must account for the arena’s device-memory reservation and the BF16 buffers allocated across request slots.
Evaluation
Experimental Setup
Models and tasks. We evaluate Qwen3.8-27B, Qwen3.6-35B-A3B, Nemotron 3 Nano, and Nemotron 3.5 Lightning, changing only full-attention layers. We measure accuracy on GPQA-Diamond ([39]), AIME25 ([27]), LiveCodeBench v6 ([17]), Terminal-Bench 2.1 ([44]), and SWE-bench Verified ([19, 33]), with three seeds per method.
Implementation and baselines. The four-model evaluation uses frozen TensorRT-LLM snapshots on GB300, with tensor parallelism one and a 256K context window. All methods share the harness and seeds. We compare dense BF16 attention with Quest ([43]), SparQ ([40]), and TriAttention ([26]). Their aggregation budgets are 2,048 tokens, the top 1/16 of positions, and the prompt plus 4,096 generated tokens, respectively. On the Nemotrons, whose attention layers use no positional encoding, our TriAttention integration uses identity phase factors, leaving a calibrated mean-query dot product and a bandwise norm correction. Each baseline uses one untuned operating point. For Jet-QSparse, we evaluate dense NVFP4, two tail-mass settings, and . We selected for the main comparison after evaluating these settings. These baseline re-implementations use Triton; Jet-QSparse uses CuTe decode kernels alongside Triton selection and BF16-page kernels, so timings also reflect implementation choices. The systems sweep also evaluates TensorRT-LLM’s FP8 KV cache for speed and capacity ([32]).
Measurements. Accuracy and speed use separate workloads. Speedups are ratios of BF16 to method decode time per output token (TPOT), measured on one GB300 per arm with real text and CUDA graphs. The Qwen3 sweep uses two replicates, 32K–252K prompts, batches 1–64, and 4,096 generated tokens; we omit cells that exceed an arm’s measured pool. Output throughput includes prefill; time to first token (TTFT) is separate. KV capacity counts allocated token slots; bytes per token describe packed storage.
We compare accuracy using paired tests clustered by problem or task. No detected loss means that no task in the stated benchmark set has a negative difference from BF16 at ; it does not establish equivalence. We also report Benjamini–Hochberg (BH) adjusted values Benjamini & Hochberg, [5], calling a negative result at but a nominal loss. Approximate minimum detectable differences at 80% power (MDD) describe resolution in percentage points. Appendix A gives the tests, correction families, run settings, and calibration checks. We exclude a separate SWE-bench comparison after a shared writable environment exposed solution artifacts across runs (Appendix A).
Accuracy–Efficiency Trade-off
At , Jet-QSparse combines faster decode and compact storage with no detected accuracy loss on the two Qwens and Lightning (Table 1). Across these models, MDD ranges from 1.17–3.76 points on reasoning and coding, 2.44–2.80 on SWE-bench, and 6.24–8.58 on Terminal-Bench. Nano has one nominal loss: AIME25 decreases by 1.98 points (95% interval [−3.77, −0.37], , ). Increasing the budget to gives no detected loss on the five application benchmarks across all four models. Section 4.4 analyzes precision and budgets.
At 128K context and batch 16, generating 1,024 tokens, speeds up Qwen decode by 1.94× and 1.57×, and Nemotron decode by 1.11× and 1.13×. Its packed cache uses 28.9–29.7% of BF16’s bytes per token. Median contexts in the accuracy runs are 6.9K–39.2K on reasoning tasks and 30K–65K per agentic call, below 128K.
The baselines make different trade-offs (Table 1). SparQ has no detected loss on the Qwens, while its layout uses 1.5× the BF16 bytes. Quest incurs large agentic losses on the Qwens; TriAttention loses long-reasoning accuracy. Jet-QSparse outpaces every fitting baseline in Table 1 while using less KV storage; SparQ’s Qwen3.8 pool cannot admit this batch.
Decode Performance and Serving Capacity
Gains grow with batch and context. At , decode speedup rises from 0.95–1.14× at batch one to 2.00× on Qwen3.8 and 2.01× on Qwen3.6 at 252K, with batches 8 and 16, respectively (Appendix D). At BF16’s largest supported batches, outpaces every fitting baseline in all eight tests (Table 7). Figure 5 compares scaling across block budgets and dense precision baselines; at fixed batch eight, the 252K gains are 2.00×/1.66×.

Figure 5. Jet-QSparse scaling with batch size and context length, compared with dense NVFP4, FP8 KV, and dense BF16 on one GB300. Speedups use 4,096 output tokens and two replicates. (a,b) Batch sweeps from 2 to 16 at 128K context; (c,d) context sweeps at batch eight. Jet-QSparse uses . Axes are logarithmic; we plot batches that fit BF16. Each method uses its own BF16 timing reference; Appendix § 14 describes the comparison across measurements.
Native low-precision baselines. FP8 KV reaches 1.47×/1.48× decode speedup on the Qwens and often matches or exceeds at batch one. At these two 252K workloads, dense NVFP4 achieves 1.51×/1.56×; selection adds approximately 1.33×/1.30×. The dense control captures the cache format, arithmetic, and kernels together. The read model approaches of dense NVFP4 traffic, so the scan limits gains from smaller budgets.
Kernel-level speedup. Separate GB300 per-layer benchmarks at head dimension 256 give 1.98–4.01× the served BF16 kernel’s speed, including selection and BF16-page work (Figure 6). Appendix § 12 gives the timing protocol, detailed measurements, and FA4 baseline checks.
Figure 6. GB300 single-layer decode speedup over TensorRT-LLM BF16 at head dimension 256. Adapted FA4 uses GQA packing with or without split-KV, whichever is faster. Jet-QSparse is timed separately, including BF16-page work and selection for .
Long generation. With 1K prompts, 16K outputs, and batch 64, runs at 0.97–1.08× BF16 speed across the four models: dense scanning and selection offset gather savings at short contexts. TriAttention performs no eviction in the 4K-output sweep; with 16K outputs it reaches 1.16×/1.06× on the Qwens with eviction kernels compiled before timing. Appendix § 14 gives the full comparison.
Appendix D gives the full comparison.
Capacity and serving throughput. Measured NVFP4 pools grow 2.34M→6.05M tokens (2.58×) on Qwen3.8 and 6.82M→18.80M (2.76×) on Qwen3.6; FP8 KV provides 1.90×/1.85× BF16 capacity. Recurrent-state and FP8-arena reservations keep NVFP4 below its 3.46× packed-row ideal. Qwen workloads exceeding BF16 capacity with 48 or 64 requests (128K prompts, 8K outputs) gain 1.99–2.16× BF16 throughput from faster computation and larger admitted batches; both Qwen3.8 pools overflow. The separate 4K-output sweep’s two 252K workloads yield 1.57×/1.71× throughput including prefill. Across all four models, FP8 prefill improves single-request TTFT 1.17–1.32× at 128K and 1.25–1.46× at 250K.
Design Ablations and Error Analysis
Table 2. Paired AIME25 changes from BF16 (points). Red ↓: . Final adds centering, two BF16 pages, FP8 prefill, and CuTe. Lower rows use dense NVFP4 with the original prefill. No listed gain survives BH correction.
Precision and calibration. We screen precision choices offline and measure task accuracy (Table 2). Across all 12 Nemotron budget pairs, the recipe raises AIME25 by 1.46–2.50 points over matched controls, though no individual gain survives BH correction. These comparisons change centering, BF16 retention, prefill, and the Nemotron kernel implementation together. Dense controls separate BF16 retention from centering; centering adds no detected reasoning or coding gain over two pages. At , all 20 comparisons with dense NVFP4 show no detected loss.
| Configuration | Nano | Lightning |
| , unfixed | −4.13↓ | −3.16↓ |
| , final | −1.98↓ | −1.01 |
| Dense, no BF16 pages | −2.74↓ | −2.33↓ |
| Dense, two BF16 pages | −2.12↓ | +0.66 |
| + key centering | −1.63↓ | −0.38 |
Table 2.
Quantization changes trajectories. Diagnostic runs use simulated NVFP4 on H100 or packed NVFP4 on GB200, without BF16 pages or FP8 prefill and with different budget and agentic settings (Appendix § 11). On 16 Qwen3.6 SWE contexts, NVFP4 changes 3.45% of teacher-forced top-1 tokens; greedy continuations diverge after a median 2.5 steps. This cohort is separate from the agentic outcome comparisons in Appendix § 13. Table 1 provides the matched final-engine evaluation of the complete Jet-QSparse precision recipe, rather than quantization alone.
| Model | Method | GPQA-D | AIME25 | LCB v6 | TB 2.1 | SWE-V | Decode× | KV× |
| Qwen3.8 27B | BF16 | 1.00 | 1.000 | |||||
| Jet-QSparse | 1.94 | 0.289 | ||||||
| Quest | 1.68 | 1.055 | ||||||
| SparQ | n/f | 1.501 | ||||||
| TriAttention | ||||||||
| Qwen3.6 35B-A3B | BF16 | 1.00 | 1.000 | |||||
| Jet-QSparse | 1.57 | 0.289 | ||||||
| Quest | 1.32 | 1.050 | ||||||
| SparQ | 1.09 | 1.502 | ||||||
| TriAttention | ||||||||
| Nemotron 3 Nano | BF16 | 1.00 | 1.000 | |||||
| Jet-QSparse | 1.11 | 0.297 | ||||||
| Quest | 1.07 | 1.042 | ||||||
| SparQ | 0.99 | 1.504 | ||||||
| TriAttention | ||||||||
| Nemotron 3.5 Lightning | BF16 | 1.00 | 1.000 | |||||
| Jet-QSparse | 1.13 | 0.297 | ||||||
| Quest | 1.05 | 1.042 | ||||||
| SparQ | 0.97 | 1.504 | ||||||
| TriAttention |
Table 1. Four-model comparison. Accuracy is mean SD over runs (%). BF16 has six runs on GPQA-D, AIME25, and LCB, and three on TB and SWE; other rows have three runs. Red entries with mark negative paired differences from BF16 at ; marks positive differences at , not claimed as gains. Decode is TPOT speedup at 128K context, batch 16, and 1,024 output tokens, measured separately from accuracy; bold marks the fastest fitting method. n/f: the batch exceeds the method’s KV-pool capacity. KV is packed bytes per token relative to BF16, including format metadata but excluding serving reservations. §TriAttention does not evict before 4,224 generated tokens; ‡per retained token.
Sparsity affects distributed evidence. Figure 7(a) compares the configurations on RULER, indexed by their SWE-bench block fractions. Count caps balance attention coverage against gather work: when a head needs more blocks to meet its mass target, the cap takes precedence. The union across query heads can still vary in size; fixed instead budgets the shared KV-group list directly. Common-word losses reach 19.8 points for Qwen3.6 and 18.5 for Nano, while their 13-task averages fall by 2.43 and 1.94 points. Needle retrieval remains largely preserved across the tested budgets: all eight Qwen3.6 needle families score 100%, and Nano’s needle scores stay within one point of dense. Figure 8 shows aggregation losses with known fixed-budget fractions. The selection-error relation (Equation (5)) connects this sensitivity to omitted mass and the gap between selected and omitted value averages. Our final-engine RULER diagnostic ([15]) uses 13 tasks, three lengths (32K/128K/256K), and three 100-document datasets per task and length. Table 3 reports its budget effects. We report RULER as a separate diagnostic.

Figure 7. Precursor sparsity study. All block fractions were measured during SWE-bench agentic runs. Hollow markers denote count caps: mass-threshold selection with an additional per-head block limit. (a) RULER common-word and 13-task score changes from each model’s dense reference. The x-axis identifies configurations by their SWE-bench fractions, not their RULER traffic. (b) SWE-bench changes from BF16 (Qwen3.6) or same-stack dense attention (Nano); bars show half the two-seed spread. Numbers count head/layer pairs retaining almost no history. (c) Unfinished rates use available Qwen seeds and Nano seed one.
Figure 8. Fixed-budget RULER study on Nemotron 3 Nano in the precursor engine. Scores are changes from dense attention at the same context length (128K, 160K, or 192K). Brick: common-word extraction; gray: 13-task average. Filled markers use blocks per KV head; hollow markers use . The gathered fraction is . Dashed: fit to all six common-word points.
| Budget | Qwen3.8 | Qwen3.6 | Nano | Lightning |
| +0.12 | −0.85 | −0.78 | −1.55 | |
| +0.28 | −0.16 | −0.45 | −0.83 | |
| +0.28 | −0.02 | −0.29 | −0.45 |
Table 3. Final-engine RULER changes from BF16, averaged over 13 tasks and three context lengths (points). Bold marks negative differences at and . Output caps affect Qwen3.8’s positive aggregates, concealing variable-tracking losses.
Fixed budgets and context length. Figure 8 shows Nano’s common-word loss increasing with smaller gathered fractions, while the task average changes much less. The six-cell
fit gives 16.5 points per halving (SE 0.5), over 0.52–1.56%. For a binding budget, the per-step fraction is approximately . A practical starting rule is therefore to choose for the configured maximum context and keep it fixed during decoding: 128 blocks at 256K and 512 at 1M both gather 6.25% at maximum length. The 1M setting remains untested; equal fractions need not retain equal attention mass.
Agentic failures differ by model. Figure 7(b) and Figure 7(c) compare accuracy and completion. Qwen3.6 stays near BF16 through ; tighter caps coincide with losses up to 11 points and 17.3% unfinished episodes, versus 3.6% across preceding settings. Unfinished episodes yield no gradable result after time/context limits or abnormal exits. Nano loses 5.6 points on average over comparable fractions; lower accuracy among finished episodes accounts for at least 87% of seed-one losses of 2.3 points or more. In the main evaluation, no Qwen3.6 SWE setting falls below BF16 at . Nano’s two lowest-fraction losses remain nominal, with no detected deficit against dense NVFP4.
Budget choice and prefill precision. In the four-model study, retains at least 94% of speed. At , all four models meet the five-benchmark accuracy criterion; this budget covers 44–83% of reference reasoning/coding contexts per step. Small average RULER losses remain (Table 3). A separate FP8-prefill effect costs Qwen3.8 10.1–14.2 points on variable tracking at 256K by echoing the answer prefix. Dense NVFP4 controls measure the additional effect of sparsification; measuring task-level interaction with quantization requires a matching BF16 sparse arm, which we did not evaluate.
Scaling to a 2.4T-Parameter Model
We deploy Qwen3.8-2.4T-A95B [37] across eight four-GPU GB300 nodes, with 32-way tensor and expert parallelism and attention data parallelism. All arms use the same FP8 weights; BF16 refers to the reference KV cache. We evaluate GPQA-Diamond, AIME25, LiveCodeBench, and Terminal-Bench with two seeds per Jet-QSparse arm. The reference has four runs on the first three tasks and two on Terminal-Bench.
All four configurations show no detected loss on the four evaluated tasks (Table 4). At , MDD80 is 2.11 points on GPQA, 2.25 on LiveCodeBench, and 8.99 on Terminal-Bench. AIME25 is near its ceiling: the paired change is −0.36 points with a 95% interval of [−1.04, +0.29], and only five problems contribute nonzero paired differences. We did not evaluate SWE-bench at this scale.
| Method | GPQA-D | AIME25 | LCB | TB |
| BF16 | ||||
| Dense NVFP4 | ||||
Table 4. 2.4T accuracy (mean SD, %). No arm falls below BF16 at on any of the four tasks.
At , decode speedups range from 1.07–1.22× across 32K–250K workloads (Table 5). The measured pool holds 2,360,832 tokens per rank versus 835,584 for BF16 (2.825×). Twelve requests per rank, each with a 128K prompt and 8K output, exceed BF16 capacity twofold but fit Jet-QSparse. Whole-deployment throughput, including prefill, improves 1.487× versus 1.423× for dense NVFP4.
| Method | Decode (×) 32K×2 | Decode (×) 128K×6 | Decode (×) 250K×3 | Output rate (×) |
| Dense BF16 | 1.000 | 1.000 | 1.000 | 1.000 |
| Dense NVFP4 | 1.058 | 1.161 | 1.094 | 1.423 |
| 1.076 | 1.297 | 1.177 | 1.480 | |
| 1.074 | 1.216 | 1.149 | 1.487 | |
| 1.046 | 1.234 | 1.154 | 1.472 |
Table 5. 2.4T speedups over dense BF16 on 32 GB300 GPUs. Decode columns give prompt length × requests per rank. Output rate covers 12 requests per rank, each with a 128K prompt and 8K output, and includes prefill. Ratios use two-run means, except , 64 use three-run medians; their 128K replicate ranges overlap.
Attention share and memory. In the profiling study, we infer from traced kernel times that attention accounts for 15–21% of the BF16 step. Reducing the NVFP4 pool to approximately BF16 capacity frees 41.5 GiB per GPU. With large preallocated pools, the benefit instead appears as capacity; in a replayed Terminal-Bench load, the reserved FP8 arena offsets live-KV savings. Appendix § 14 separates allocated memory from live demand and describes these measurements.
Conclusion
Jet-QSparse combines compact NVFP4 storage and native QK computation with sparse, higher-precision value aggregation in a simple, training-free design. Its block layout aligns selection with contiguous memory access and tensor-core execution. With 64 blocks per KV head, both Qwen models show no detected accuracy loss on five application benchmarks and reach up to 2.01× end-to-end decode speedup over dense BF16 on separate 252K-context workloads. Their measured KV pools hold 2.58–2.76× as many tokens. At 128 blocks, all four Qwen and Nemotron models show no detected accuracy loss across mathematical reasoning, coding, and agentic benchmarks. At 2.4T parameters, compact KV storage expands capacity while expert computation and communication limit decode gains.
Quantization can change the attention distribution used for block selection. The additional selection error depends on omitted mass and the difference between selected and omitted value averages. Across all four models, the 64-block configuration shows no detected additional loss relative to the final dense NVFP4 control on the five application benchmarks. Our diagnostics show larger losses on aggregation over distributed evidence under aggressive sparsity, while needle retrieval remains largely preserved. These findings guide joint precision and budget calibration through attention-output probes and end-to-end task validation.
AI USAGE DISCLOSURE
Claude (under Claude Code) is used for kernel debugging (mainly tracing runtime errors and portability issues), experiment monitoring, and related work lookup. ChatGPT (website) is used for wording polishing of the paper.
Evaluation Protocol and Statistical Resolution
Runs and comparisons. The four-model accuracy study uses pinned checkpoints and frozen TensorRT-LLM snapshots: fsnap-10020414 for the final Jet-QSparse arms and final1001 for the BF16 reference and baseline re-runs. Within each model and benchmark, arms share the harness configuration, sampling settings, 262,144-token context limit, and strict tool-call parsers. Each arm runs seeds 42–44. On GPQA, AIME25, and LiveCodeBench, the reference averages two independent BF16 runs per seed; Terminal-Bench and SWE-bench have one BF16 run per seed. Tables report means and sample standard deviations across runs, whereas the paired tests operate on item outcomes. We measure speed on separate workloads; the timing cells do not measure latency within these accuracy runs.
Full-attention layers account for 16 of 64 layers on Qwen3.8-27B, 10 of 40 on Qwen3.6-35B-A3B, and six of 52 on each Nemotron. The Qwens use head dimension 256 and the Nemotrons 128. The 252K sweep prompts use 258,048 of 262,144 tokens, leaving 4,096 for output.
Each seed evaluates 198 GPQA questions, 30 AIME25 problems with 16 samples each, 454 LiveCodeBench v6 problems from the August 2024 to May 2025 evaluation window, 89 Terminal-Bench tasks, and 500 SWE-bench Verified instances. Each agent call generates at most 16,384 tokens.
We chose the headline after evaluating all six configurations. We selected the accuracy fixes on the same benchmark problems later used for evaluation; repeat runs provide new samples, not new problems. Each sparse baseline uses one untuned operating point, and the comparisons are not sparsity-matched. These choices limit conclusions about an optimal budget or the relative accuracy of independently tuned methods.
Timing and capacity eligibility. The common timing workload in Table 1 uses 131,072 prompt tokens, batch size 16, and 1,024 output tokens on all four models. Figure 1 uses the same prompt length, batch size eight, and 4,096 output tokens from the separate Qwen sweep. Speed runs use real text and CUDA graphs on one GB300 per arm. TPOT is the per-token decode time of the request prefilled last; speedup divides the same run’s dense BF16 TPOT by the arm’s. SparQ’s Qwen3.8 cell in Table 1 is marked n/f: its 2,113,536-token demand exceeds the measured 1,576,928-token pool.
The batch/context sweeps use 4,096 output tokens and two replicates. Baselines and Jet-QSparse run in separate campaigns, each normalized to its own dense BF16 anchor. A same-node comparison reproduces the Jet-QSparse grid with median absolute deviations of 0.08% on Qwen3.8 and 0.85% on Qwen3.6 (maxima 0.83% and 2.73%). This check supports the comparison but does not eliminate cross-campaign uncertainty. Reported fixed-batch cells require all requests to fit without suspension. We use measured pool capacities where available and the validated cache-manager allocation rule otherwise. We mark larger batches as not fitting and exclude the Qwen3.8 FP8-KV 64K64 cell after a request suspension.
SWE evaluation integrity. We exclude a thirteen-arm Qwen3.8 SWE comparison over all 500 instances and seeds 42–44 because shared writable runtime files compromised task isolation. The sandboxes shared a Python installation containing a task solution and notes. Twenty retained episodes across ten arms and 19 tasks read these files; one matching-task episode subsequently made the identical edit. This establishes solution propagation, not its effect on the success rate. Incomplete tool histories prevent bounding total exposure, so we exclude the whole comparison rather than filter only known readers. The separate FP8-KV/BLASST evaluation’s BF16 control differs from Table 1’s reference by +4.83 points (95% CI [+2.95, +6.71], ). We exclude the associated FP8-KV and BLASST SWE measurements from the accuracy comparison. This audit cannot certify environments outside the excluded comparison.
Paired, clustered inference. An item is a (seed, problem) pair on the reasoning benchmarks, with AIME25’s 16 samples averaged within that item, or a (seed, task or instance) pair on the agentic benchmarks. We exclude SWE-bench pairs with zero-token outcomes. Let be the arm-minus-reference difference in percentage points. Items cluster by problem, task, or instance across seeds. For cluster , define , , and . With seeded sign draws, independently uniform over for each cluster, the estimate and two-sided Monte Carlo test are
The 95% interval inverts this test: it contains shifts for which testing gives . This inversion assumes a constant per-item shift. The sign-flip test is exact for independent cluster sums symmetric about zero under the null, but approximate with a two-run reference.
Calibration and shared-reference uncertainty. Design-based grid simulations reject 3.3–4.7% of nulls at nominal 5%, with 94.2–96.2% interval coverage in the simulated shifted cases. Real-data null checks reject 0.0–3.3% on three models but 4.4–13.3% on Nano, indicating that Nano’s grid intervals are about 15% too narrow. Agentic tests have neither a null family nor a coverage simulation. All contrasts for a model share its BF16 runs, so a high reference run can produce deficits across several configurations. Clustering by task handles repeated tasks across seeds but does not by itself remove whole-run shifts. Post hoc pooling and sign analyses are descriptive; they do not determine configuration verdicts.
Multiplicity and verdicts
For ordered -values, Benjamini–Hochberg correction [5] gives
The final-method family has : four models, five benchmarks, and six configurations comprising dense NVFP4, , and . Baseline -values use the full family of comparisons: the 60 displayed contrasts cover three baselines, four models, and five benchmarks; 20 additional comparisons remain in the correction family but are not displayed. We apply multiplicity correction to RULER in separate diagnostic families and exclude it from the five-benchmark criterion; we adopted this separation after an initial combined analysis. A negative estimate with is a detected loss if , and nominal otherwise. A configuration shows no detected loss only when none of its five cells is negative at , irrespective of the correction family. Positive differences are not claimed as gains. The positive regression dependence condition for BH [6] is unproven for these two-sided tests; sharing a reference does not establish it. We did not apply the Benjamini–Yekutieli correction [6].
Detection resolution. Using a normal approximation with interval half-width , we report
This approximates the loss detectable at with 80% power, not an equivalence margin or a confidence bound on loss. Multiplicity requires stronger evidence: a lone rejection among 120 needs and has only about 29% power at one . Discrete outcomes further limit resolution. With nonzero cluster sums, the exact two-sided sign-flip is at least ; fewer than six informative clusters cannot yield . Near-ceiling ties can also make the normal MDD approximation optimistic. Smaller losses remain possible after a non-rejection.
The 2.4T extension. This campaign uses two seeds per Jet-QSparse arm, four BF16 grid runs, and two BF16 Terminal-Bench runs, with one BH family of 16 contrasts across four benchmarks. We deferred SWE-bench, so this study cannot meet the five-benchmark criterion. Test and verdict rules preceded the runs. We chose two seeds after reading two LiveCodeBench pilots, and chose the four-run reference with BF16 grid scores known. We decided the final run set before Jet-QSparse Terminal-Bench started, but registered it after reading the grid and one BF16 Terminal-Bench run, 20 hours after the first Jet-QSparse Terminal-Bench launch and before reading any of its rows. Two runs had started and one had finished by registration. Under equal independent sampling variance, two seeds increase by approximately relative to three; each cell nevertheless uses its estimated resolution.
Exploratory error studies. The precursor studies in § 8 use a separate engine, precision recipe, and agentic protocol. Appendix § 13 details the heterogeneous quantization comparisons and their descriptive pool. The token disagreement study uses 16 contexts from one model and is not paired with those task–outcome comparisons. The context-scaling fit uses six single-run cells with shared controls within each length and different documents across lengths; its standard error does not represent fresh-document uncertainty. We selected the agentic budget range from Qwen’s outcomes; Nano’s finished-episode decomposition uses seed one. These observations suggest patterns to investigate; the final-engine controls provide separate checks in the main text.
Kernel measurements
| Method | Qwen3.8 32K×32 | Qwen3.8 128K×16 | Qwen3.6 32K×32 | Qwen3.6 128K×16 |
| TRT-LLM BF16 | 580.6 | 1,135 | 291.5 | 571.7 |
| FA4, as shipped | 3,515 | 7,499 | 2,195 | 4,934 |
| FA4, adapted | 741.9 | 1,465 | 374.1 | 747.0 |
| JQS, dense | 301.0 | 545.8 | 170.2 | 293.4 |
| JQS, | 237.2 | 283.2 | 147.5 | 175.4 |
Table 6. Single-layer decode times in s on GB300, head dimension 256. Qwen3.8 has 24 query/four KV heads; Qwen3.6 has 16/two. JQS denotes Jet-QSparse, timed separately with BF16-page work and selection for .
The TensorRT-LLM/FlashAttention-4 study uses FA4 4.0.0b19, CUDA-event timings with L2 flushes, and two replicate medians with at least 20 timing repetitions. Decode runs under CUDA graphs. The tested FA4 path at head dimension 256 lacks native GQA packing and split-KV. The adapted result is the faster of GQA packing alone and packing with an external split-KV/combine step. We time Jet-QSparse’s final decode kernels in a separate benchmark with seven shuffled rounds and include scan, selection, sparse attention, BF16-page operations, and merging where applicable. The descriptive ratios in Figure 6 compare matching head geometries, batches, and context lengths. Differences in harnesses, clocks, and timing aggregation prevent an exact decomposition of end-to-end speed.
Figure 6 and Figure 9 use TRT-LLM time divided by method time. The decode reference is the BF16 attention kernel itself. The prefill reference is the corresponding served context operator, using either one chunk or 8K chunks on both sides. FA4 prefill takes 0.899–0.945 of the one-chunk baseline time and 0.936–0.968 with 8K chunks. These are attention measurements, not a separate FA4 end-to-end serving arm.

Figure 9. FA4 speedup over TRT-LLM BF16 (log scale). (a) Decode at 128K: as shipped (hollow, dashed) or adapted (filled, solid). (b) Prefill at batch one: one chunk (filled, solid) or 8K chunks (hollow, dashed). Circles and squares denote the Qwen3.8 and Qwen3.6 head geometries.
Precursor Quantization Diagnostics
Figure 10 summarizes four earlier agentic comparisons of NVFP4 and BF16. These studies differ in hardware, arithmetic, benchmark, and inclusion of incomplete episodes. They provide exploratory evidence separate from the matched final-engine comparisons in Table 1.
Figure 10. Exploratory agentic accuracy changes from BF16 to NVFP4, with 95% intervals. Study protocols differ; rows are not matched repetitions across models. SWE denotes SWE-bench Verified; TB denotes Terminal-Bench 2.1. The descriptive pool uses studies 1, 3, and 4; study 2 shares study 1’s BF16 reference.
Study protocols. Studies 1 and 2 use simulated NVFP4 with BF16 arithmetic on H100 for Nemotron 3 Nano SWE-bench Verified. They share one BF16 reference and include only episodes completed by both arms: 468 and 477 of 500 problems, respectively. Study 3 evaluates Qwen3.6 on all 500 SWE-bench Verified problems using packed NVFP4 on GB200. Its records conflict on sampling settings and whether decode uses native FP4 arithmetic or BF16 over a dequantized cache, limiting attribution to quantization alone. Study 4 evaluates Qwen3.6 on 89 Terminal-Bench 2.1 tasks with four seeds per arm on GB200 and native FP4 scoring at prefill and decode.
Descriptive pooling. The inverse-variance pool of studies 1, 3, and 4 gives −0.04 points [95% CI −2.2, +2.1]. We exclude study 2 because it shares study 1’s BF16 reference. The pool assumes a common percentage-point effect across these different models and benchmarks; it does not reconcile their protocols or completion criteria. It establishes neither equivalence nor the accuracy of the final Jet-QSparse recipe. The 16-context probe in Section 4.4 uses a separate cohort and cannot link token disagreement to task outcomes.
Additional System Measurements
Batch, Context, and Block Budget
Table 7 compares baselines at the largest batch dense BF16 accommodates at each context. It also reports each method’s best output rate, including prefill, over its tested batches. This output rate reflects both faster execution and additional capacity. Ratios use each campaign’s own BF16 reference; Appendix § 11 describes the timing comparison and its consistency check.
| Method | Pool ratio | TPOT speedup 32K | TPOT speedup 64K | TPOT speedup 128K | TPOT speedup 252K | Best output-rate ratio 32K | Best output-rate ratio 64K | Best output-rate ratio 128K | Best output-rate ratio 252K |
| Qwen3.8-27B | |||||||||
| Quest | 0.95 | 1.10 | 1.48 | 1.70 | 1.83 | 1.07 | 1.24 | 1.29 | 1.27 |
| SparQ | 0.67 | 1.15 | n/f | n/f | n/f | 1.10 | 0.91 | 0.91 | 0.92 |
| TriAttention | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.01 | 1.01 |
| FP8 KV | 1.90 | 1.30 | 1.46 | 1.47 | 1.47 | 1.50 | 1.28 | 1.50 | 1.55 |
| Dense NVFP4 | 2.58 | 1.30 | 1.49 | 1.51 | 1.51 | 1.52 | 1.49 | 1.54 | 1.60 |
| Jet-QSparse | 2.58 | 1.41 | 1.84 | 1.95 | 2.00 | 1.63 | 1.69 | 1.76 | 1.81 |
| Qwen3.6-35B-A3B | |||||||||
| Quest | 0.96 | 1.04 | 1.32 | 1.53 | 1.74 | 1.03 | 1.19 | 1.26 | 1.25 |
| SparQ | 0.69 | 1.09 | 1.17 | 1.18 | 1.20 | 1.07 | 1.11 | 1.09 | 1.08 |
| TriAttention | 1.00 | 1.01 | 1.00 | 1.01 | 1.00 | 1.01 | 1.00 | 1.01 | 1.01 |
| FP8 KV | 1.85 | 1.25 | 1.38 | 1.43 | 1.48 | 1.21 | 1.31 | 1.57 | 1.58 |
| Dense NVFP4 | 2.76 | 1.25 | 1.43 | 1.48 | 1.56 | 1.21 | 1.34 | 1.63 | 1.81 |
| Jet-QSparse | 2.76 | 1.31 | 1.68 | 1.87 | 2.01 | 1.26 | 1.48 | 1.88 | 2.08 |
Table 7.
We compute capacity ratios from measured attention-pool sizes. The cache manager divides its quota between attention pages and recurrent states using an average request length of 16,384 tokens. Reducing bytes per attention token therefore increases the recurrent-state allocation as well as token capacity. This explains why FP8 KV provides capacity and NVFP4 , below their storage-only ratios. SparQ’s additional key copy reduces its pool to of BF16. The allocation rule matches measured pool counts within 0.1% and predicts approximately NVFP4 capacity on the Nemotrons. We exclude fixed-batch timings that require multiple admission waves.
The sparse curves in Figure 11 vary the gather budget while retaining the dense scan. At BF16-fitting batches, is at most 12% slower than on the Qwens; the difference reaches 19% at Qwen3.8’s 32K×64 cell beyond
Figure 11. Block-budget sweep on one GB300, with 4,096 output tokens and two replicates. TPOT speedup is relative to dense BF16 at its largest fitting batch: Qwen3.8 uses 32K×32 and 252K×8; Qwen3.6 uses 32K×64 and 252K×16. Circles and solid lines denote Qwen3.8; squares and dashed lines denote Qwen3.6. Jade marks sparse budgets; brick marks dense NVFP4 at “all.” BF16 capacity. The scan and non-attention layers limit the benefit of reducing the gather alone.
Table 7. Measured KV-pool capacity and performance on the Qwens. Pool is the token capacity relative to dense BF16. TPOT columns use BF16’s largest batch at each context: 32/32/16/8 on Qwen3.8 and 64/64/32/16 on Qwen3.6. Output-rate columns divide each method’s best throughput, including prefill, by BF16’s best over the tested batches. All cells generate 4,096 tokens; n/f means the batch does not fit the measured pool. The Qwen3.8 FP8-KV 64K×64 cell is excluded after a request suspension; its cells beyond BF16 capacity run with idle neighboring GPUs. Jet-QSparse and baseline ratios come from separate campaigns.
Table 8. Long-generation speedups: 1,024-token prompts, 16,384 outputs, and batch 64. Top: TPOT relative to dense BF16; two replicates, with baselines and Jet-QSparse in separate campaigns. *Eviction kernels compiled before timing; two same-node GPU pairs with iteration statistics enabled. †Rebased on other runs’ BF16 reference, with approximately 5% uncertainty. Bottom: one attention layer at 8.5K×64 on random data, relative to FlashInfer BF16; Jet-QSparse kernels omit the accuracy fixes.
Long Generation and Kernel Scope
TriAttention first evicts after 4,224 generated tokens. With a 1K prompt, its cache cycles between 5,120 and 5,248 tokens while dense attention reaches 17,408. All 64 requests fit the BF16 pool in Table 8, so this experiment measures shorter reads, not additional admitted requests. At 4,096 output tokens in the batch sweep, no eviction occurs. Compiling the eviction kernels before timing avoids charging a one-time compilation to decode. The Qwen measurements use this warm-up; the Nemotron cells retain the standard timing protocol. Same-method cross-campaign ratios differ by up to 0.05× in the Qwen3.8 long-generation cell, so small gaps across campaigns are descriptive.
| Q3.8 | Q3.6 | Nano | Lightning | |
| End to end | ||||
| Quest | 0.69 | 0.81 | 0.86 | 0.91 |
| SparQ | 1.02 | 0.97 | 0.97 | 0.93 |
| TriAttention | 1.16* | 1.06* | 1.01 | 0.96 |
| Dense NVFP4 | 1.14 | 1.07 | 1.00† | 0.99 |
| Jet-QSparse | 1.08 | 1.01 | 0.98† | 0.97 |
| Conditional limit | 1.43 | 1.19 | 1.08 | 1.06 |
| One attention layer | ||||
| Quest | 2.04 | 1.89 | 1.31 | |
| SparQ | 1.16 | 0.88 | 0.74 | |
| TriAttention | 1.54 | 1.48 | 1.40 | |
| Dense NVFP4 | 1.66 | 1.49 | 0.94 | |
| Jet-QSparse | 1.18 | 1.00 | 0.55 |
Table 8.
Short-prompt KV memory. Figure 1 also measures memory with eight requests, 4K prompts, and 16K outputs on one GB300 per method. All requests finish without pauses. This single-run check sets maximum batch size to 8, sequence length to 20,480, and prefill token budget to 8,192. We convert measured peak used pages and page sizes to occupied bytes, including layout metadata, and add the two BF16 pages for each active Jet-QSparse request. Ratios use the matching engine’s BF16 control; both dense controls have identical attention-memory values. Occupied bytes exclude recurrent states, unused reservations, and the FP8 prefill arena. The 128K/4K comparison instead uses the two-run sweep with configured maxima of 64 requests and 262,144 tokens for sequence length and the prefill token budget.
The conditional limits in Table 8 use , assuming non-attention time is unchanged, with estimated attention shares of 30%, 16%, 7%, and 6%. These estimates scale the 8.5K single-layer times by the number of attention layers; they approximate a generation whose context changes throughout decoding. At these short contexts, gathers a traffic-weighted block fraction of 0.72, so the read model in § 6.4 predicts times the dense NVFP4 K/V traffic.
Single-layer speed does not fully predict serving speed. Quest’s layer times, with its first two attention layers kept dense, imply 1.01–1.15× end-to-end speedup under an unchanged non-attention remainder, but measured speed is 0.69–0.91×. We did not decompose this gap, which limits algorithm-level conclusions from its serving ratios.
The 2.4T Decode Step and Memory Use
Table 9 attributes 15–21% of the BF16 step to attention and shows how reduces that component. This profiling study uses separate runs from the deployment measurements in Table 5. At 32K23, sampled traced kernels additionally attribute 20.0 ms to routed experts, 18.7 ms to MoE communication including waits, and 14.9 ms to BF16 GEMMs. These sampled-rank timings do not give an untraced whole-system decomposition. The non-attention remainder also varies between arms, so is a conditional limit rather than an unconditional bound on speedup.
| Cell | BF16 step | BF16 | Attention | Limit | speedup | |
| 32K23 | 69.6 | 10.2 | 4.0 | 14.7% | 1.17 | 1.10 |
| 128K6 | 58.3 | 10.5 | 2.8 | 18.0% | 1.22 | 1.20 |
| 250K3 | 47.0 | 9.8 | 2.7 | 20.9% | 1.26 | 1.13 |
Table 9. 2.4T systems study on 32 GPUs, using . Step times and speedups use TPOT estimates with traced windows replaced by untraced means (two allocations). Attention times are rank-0 kernel attributions; the BF16 attention share is inferred because profiling overhead exceeds its tolerance. The limit assumes unchanged non-attention time. Times are milliseconds.
Allocated memory and live demand. With the pool using 85% of free memory, measured NVML plateaus are 265.03 GiB per GPU for dense BF16 and 262.74 GiB for ; reservation keeps these footprints nearly constant across loads. Reducing the NVFP4 pool fraction to 0.45 to approximate BF16’s token capacity lowers its plateau to 225.40 GiB. Against the corresponding BF16 reference mean of 266.85 GiB, this frees 41.45 GiB per GPU, or approximately 1,327 GiB per 32-GPU copy. Propagating the bimodal BF16 reference’s spread gives an SD of 3.38 GiB per GPU, despite agreement between the two NVFP4 replicates. Normalizing to exactly 835,584 tokens gives a calculated 42.76 GiB saving. Associated TPOT changes are unresolved at MDD values of 1.29–5.37 ms.
A replay of 1,113 Terminal-Bench calls measures demand under an agentic arrival pattern. It reproduces token counts and timing, not task content, and uses two paired measurements. Subtracting unused reserved KV space from measured allocation gives peak memory needs of 200.97 GiB for BF16 and 205.31 GiB for , a derived difference of GiB. These are accounting estimates, not NVML footprints. Peak live KV averages about 105K/94K tokens across the two pairs, well below either pool. Hypothetically shrinking the FP8 arena to its observed peak would change the difference to GiB in the two pairs; we did not measure that allocation policy. We amended the accounting filters after a diagnostic measurement. In these replayed calls, decode accounts for 97.8% of client model time, with 68.7–69.3% of decode time in steps that also prefill another request.
Other execution choices. FP8 non-expert weights reduce the 32K23 step by 3.3–3.5 ms. The reduction is resolved for (MDD 1.3 ms), but not for dense BF16 (4.2 ms); the sparse-to-dense speedup change remains within MDD 0.07–0.14. Attention TP16 on four nodes achieves 8.9–9.7 ms for one request and 250/319 output tokens per second per node for dense BF16/NVFP4 at 128K30. We did not evaluate fixed-budget speculation on this layout because each KV group has four query heads instead of sixteen. A low- GEMM dispatcher saves 1.5–2.4 ms in isolated step measurements. We evaluated these weight and execution variants for performance only; they carry no accuracy claim.
References
- [1]DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models. Technical report, DeepSeek-AI, 2025. URL https://arxiv.org/abs/2512.02556.
- [2]Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghavi, S. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.298. URL https://aclanthology.org/2023.emnlp-main.298/.DOI
- [3]Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems, volume 37, pp. 100213–100240. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/b5b939436789f76108b90da5e81af7c–Abstract-Conference.html.arxiv.org/abs/2404.00456
- [4]Behnam, P., Fu, Y., Zhao, R., Tsai, P.-A., Yu, Z., and Tumanov, A. RocketKV: Accelerating long-context LLM inference via two-stage KV cache compression. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 3358–3392. PMLR, 2025. URL https://proceedings.mlr.press/v267/behnam25a.html.arxiv.org/abs/2502.14051
- [5]Benjamini, Y. and Hochberg, Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995. doi: 10.1111/j.2517-6161.1995.tb02031.x.DOI
- [6]Benjamini, Y. and Yekutieli, D. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4):1165–1188, 2001. doi: 10.1214/aos/1013699998.DOI
- [7]Cecka, C. CuTe layout representation and algebra. arXiv preprint arXiv:2603.02298, 2026. URL https://arxiv.org/abs/2603.02298.
- [8]Dao, T. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2307.08691.
- [11]Gao, Y., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H. K.-H., Cao, T., Yang, F., and Yang, M. SeerAttention: Self-distilled attention gating for efficient long-context prefilling. In Advances in Neural Information Processing Systems, volume 38, pp. 55846–55869. Curran Associates, Inc., 2025. doi: 10.52202/085713-1869.DOI
- [12]Gao, Y., Guo, S., Cao, S., Xia, Y., Cheng, Y., Wang, L., Ma, L., Sun, Y., Ye, T., Dong, L., So, H. K.-H., Hua, Y., Cao, T., Yang, F., and Yang, M. Sparse attention adaptation for long reasoning. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=c5BOCHM6J8.arxiv.org/abs/2506.08889
- [14]Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, volume 37, pp. 1270–1303. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/028fccf85435d39a40c4d61b42c99a4–Abstract-Conference.html.arxiv.org/abs/2401.18079
- [15]Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https://arxiv.org/abs/2404.06654.
- [16]Hu, R. Hardware-aware FP4 FlashAttention-4. arXiv preprint arXiv:2609.04105, 2026. URL https://arxiv.org/abs/2609.04105.
- [17]Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S. I., Solar-Lezama, A., Sen, K., and Stoica, I. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/94074d5a072d28f f75a76dabed43767-Abstract-Conference.html.
- [18]Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems, volume 37, pp. 52481–52515. Curran Associates, Inc., 2024.arxiv.org/abs/2407.02490
- [19]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narsimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e30c629acfe6cbe9ca88fb84-Abstract-Conference.html.arxiv.org/abs/2310.06770
- [20]Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165. PMLR, 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html.arxiv.org/abs/2006.16236
- [21]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626. ACM, 2023. doi: 10.1145/3600006.3613165.DOI
- [22]Lai, X., Lu, J., Luo, Y., Ma, Y., and Zhou, X. FlexPrefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OfjllbeIrT.arxiv.org/abs/2502.20766
- [24]Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.DOI
- [26]Mao, W., Lin, X., Huang, W., Xie, Y., Fu, T., Zhuang, B., Han, S., and Chen, Y. TriAttention: Efficient long reasoning with trigonometric KV compression. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pp. 86458–86477. PMLR, 2026. URL https://proceedings.mlr.press/v306/mao26e.html.arxiv.org/abs/2604.04921
- [27]Mathematical Association of America. American Invitational Mathematics Examination (AIME) 2025. Competition problems, 2025. URL https://web.archive.org/web/20250316185218/https://maa.org/maa-invitational-competitions/. AIME I and AIME II.
- [28]NVIDIA. Nemotron 3 Nano: Open, efficient mixture-of-experts hybrid Mamba-Transformer model for agentic reasoning. Technical report, NVIDIA, December 2025a. URL https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Nano-Technical-Report.pdf.arxiv.org/abs/2512.20848
- [29]NVIDIA. Pretraining large language models with NVFP4. arXiv preprint arXiv:2509.25149, 2025b. URL https://arxiv.org/abs/2509.25149.
- [30]NVIDIA. Nemotron 3.5 Super and Lightning: Open, efficient mixture-of-experts hybrid Mamba-Transformer models for agentic AI. Technical report, NVIDIA, October 2026a. URL https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf.arxiv.org/abs/2610.38095
- [32]NVIDIA. Quantization in TensorRT-LLM. TensorRT-LLM documentation, 2026d. URL https://nvidia.github.io/TensorRT-LLM/latest/features/quantization.html. Accessed 2026-10-13.
- [33]OpenAI. Introducing SWE-bench Verified. OpenAI blog, August 2024. URL https://openai.com/index/introducing-swe-bench-verified/.
- [34]Qwen Team. Qwen3-Next-80B-A3B-Instruct. Hugging Face model card, September 2025. URL https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct. Accessed 2026-10-16.
- [35]Qwen Team. Qwen3.6-35B-A3B. Hugging Face model card, April 2026a. URL https://huggingface.co/Qwen/Qwen3.6-35B-A3B. Accessed 2026-10-16.
- [37]Qwen Team. Qwen3.8-2.4T-A95B-FP8. Hugging Face model card, August 2026c. URL https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8. Accessed 2026-10-24.
- [38]Ramachandran, A., Neseem, M., Sakr, C., Venkatesan, R., Khailany, B., and Krishna, T. ThinKV: Thought-adaptive KV cache compression for efficient reasoning models. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=M3CeHnZKNC.arxiv.org/abs/2510.01290
- [39]Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98.arxiv.org/abs/2311.12022
- [40]Ribar, L., Chelombiev, I., Hudlass-Galley, L., Blake, C., Luschi, C., and Orr, D. SparQ attention: Bandwidth-efficient LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42558–42583. PMLR, 2024. URL https://proceedings.mlr.press/v235/ribar24a.html.arxiv.org/abs/2312.04985
- [41]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems, volume 37, pp. 68658–68685. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/7ede97c3e082c6df10a8d6103a2eebd2-Abstract-Conference.html.arxiv.org/abs/2407.08608
- [42]Singhania, P., Singh, S., He, S., Feizi, S., and Bhatele, A. Loki: Low-rank keys for efficient sparse attention. In Advances in Neural Information Processing Systems, volume 37, pp. 16692–16723. Curran Associates, Inc., 2024. doi: 10.52202/079017-0532. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/1e027da6bec9c9eb2ec37951ceecae93-Abstract-Conference.html.DOI
- [43]Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasicki, B., and Han, S. Quest: Query-aware sparsity for efficient long-context LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47901–47911. PMLR, 2024. URL https://proceedings.mlr.press/v235/tang241.html.arxiv.org/abs/2406.10774
- [44]The Terminal-Bench Team. Terminal-Bench 2.1. Benchmark release, May 2026. URL https://www.tbench.ai/news/terminal-bench-2-1.
- [45]Tillet, P., Kung, H. T., and Cox, D. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19. ACM, 2019. doi: 10.1145/3315508.3329973.DOI
- [46]Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2309.17453.
- [47]Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=cFu7ze7xUm.arxiv.org/abs/2410.10819







