betaFeedback

WorksComputer Science

Jet-QSparse: Accelerating Agentic LLM Inference with 4-Bit Sparse Attention

Abstract

Long-horizon reasoning and agentic workloads strain KV-cache capacity and decode attention throughput. We present Jet-QSparse, a training-free method that combines compact NVFP4 cache storage with native FP4 query-key computation on Blackwell tensor cores. A dense scan scores the full key history and selects 128-token blocks for sparse FP16 value aggregation with FP32 accumulation, preserving higher precision in the output-forming step while enabling contiguous tile loads. Calibrated key centering and at most two BF16 pages per sequence and layer address quantization error; FP8 prefill uses a temporary arena while the persistent cache is packed directly from BF16 activations. Across four hybrid LLMs, a budget of 128 blocks per KV head shows no detected accuracy loss on the evaluated mathematical reasoning, coding, and agentic benchmarks, although some smaller budgets incur losses. With 64 blocks, Qwen3.8-27B achieves a 1.94×1.94\times end-to-end decode speedup over dense BF16 at 128K context and batch size 16 on one GB300, and the measured Qwen KV pools provide 2.582.58-2.76×2.76\times BF16's token capacity.

Introduction

Reasoning and agentic language models process long histories of intermediate reasoning, tool outputs, and interaction. The key–value (KV) cache grows with these histories and limits the number of requests that fit on a GPU. Dense decode attention reads and processes the history for every generated token, increasing latency as the sequence grows. Efficient inference requires a smaller persistent cache and cheaper attention while preserving task accuracy.

KV quantization reduces the bytes stored per token ([24, 14]), while sparse decode reduces the data fetched and processed for each query ([43, 40] (Yuan et al. 2026)). Blackwell supports native NVFP4 computation: packed 4-bit operands reduce memory traffic and feed block-scaled tensor-core operations [29] (Du et al. 2026). An efficient attention method must coordinate the precision used to score keys, the blocks selected for value aggregation, and the layout used to fetch them. Selection adds work, irregular accesses can limit hardware utilization, and errors from quantization and sparsification must be evaluated together.

Jet-QSparse combines compact NVFP4 storage and dense native FP4 scoring with sparse higher-precision aggregation in a simple, training-free design. Its NVFP4 cache retains the complete token history. At each decode step, native FP4 query–key computation scores every key and estimates block attention masses. Query-dependent selection restricts the output-forming PVPV product to selected blocks, using FP16 operands and FP32 accumulation. Calibrated key centering and BF16 copies of the first and current blocks address quantization error within two pages per sequence and layer.

We group 128 consecutive tokens per cache page to align selection with contiguous memory loads and tensor-core tiles. Query heads sharing a KV head reuse selected blocks, and selection executes on the GPU within CUDA graphs. Prepacked scales feed native QK computation, and pure decode steps fuse query packing with cache writes. The implementation integrates into a production serving engine. Prefill uses FP8 attention while the persistent NVFP4 cache is packed directly from BF16 activations.

We study sparsification under quantization through attention-output analysis, dense NVFP4 controls, and block-budget sweeps. The analysis relates selection error to omitted mass and the gap between selected and omitted value averages. Long-context diagnostics show that aggressive sparsity can impair aggregation over distributed evidence while largely preserving needle retrieval; agentic tests reveal model-dependent failure patterns. These findings motivate evaluating precision and sparsity together at the task level.

We evaluate Jet-QSparse on Qwen3.6-35B-A3B, Qwen3.8-27B, Nemotron 3 Nano, and Nemotron 3.5 Lightning. These hybrid models span different generations, dense or mixture-of-experts (MoE) feedforward layers, and different proportions of full-attention layers. We also deploy Qwen3.8-2.4T-A95B across eight nodes. On the two Qwens, the measured cache pools provide 2.58–2.76× the KV-token capacity of dense BF16 under the same allocation policy. At 128K context and batch size eight on one GB300, 64 selected blocks per KV head accelerate Qwen3.8-27B and Qwen3.6-35B-A3B decode by 1.53× and 1.32× over dense BF16, respectively. Every compared method fits this workload. Both configurations show no detected accuracy loss on five reasoning, coding, and agentic benchmarks at the evaluation’s statistical resolution; a 128-block budget meets this criterion across all four models. Figure 1 compares speed, accuracy, and occupied KV memory at this shared budget.

Figure 1. SWE-bench Verified accuracy (mean ± SD, three seeds) versus separate GB300 decode speedups (128K prompt/4K output, batch 8); Jet-QSparse uses N=64N = 64. Dashed lines mark BF16 accuracy. Leaders connect memory callouts to measured points. Semicircle areas are proportional to occupied KV (% of BF16): left, 4K/16K (one run); right, 128K/4K (two runs), both at batch 8. Panels share one area scale; x-axis ranges differ. Bytes include pages, metadata, and active BF16 copies, excluding unused reservations, recurrent state, and prefill workspace.

Smaller budgets incur losses on some model–benchmark pairs, which we report alongside the performance gains.

Our contributions are:

  1. We introduce Jet-QSparse, a simple, training-free attention method that combines compact NVFP4 storage and native low-precision scoring with sparse higher-precision aggregation. Its block-based cache aligns selection with contiguous memory access and tensor-core execution, enabling faster decoding and lower KV memory use within a production serving engine.

  1. We systematically study the additional error from sparsifying quantized attention and its effects on long-context and agentic tasks. We relate selection error to omitted mass and value differences, and use Jet-QSparse’s dense NVFP4 controls and budget sweeps to examine task sensitivity and model-dependent failure patterns.

  1. We evaluate Jet-QSparse across Qwen and Nemotron model families on mathematical reasoning, coding, and agentic benchmarks, demonstrating decode speedups and increased KV capacity. A 128-block budget shows no detected accuracy loss on the five application benchmarks across all four models, and we extend the evaluation to a 2.4T-parameter model with 95B active parameters.

Related Work

Hybrid Attention Models. Linear attention uses fixed-size states for recurrent decoding [20]. Mamba uses selective state-space recurrences; Mamba-2 relates them to structured attention through state-space duality (Gu & Dao, 2024; Dao & Gu, 2024). Gated DeltaNet (GDN) combines state-decay gates with delta-rule updates to key–value associations (Yang et al., 2025). Hybrids interleave these recurrent mechanisms with softmax attention. Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, and Qwen3.8-27B each interleave three GDN blocks with one gated full-attention block [34, 35 b]. Nemotron 3 Nano and Nemotron 3.5 Lightning combine Mamba-2, MoE, and six full-attention GQA blocks in 52-block backbones [28, 30]. Jet-QSparse reduces the KV-cache costs of these attention layers without changing recurrent or feedforward blocks.

Attention kernels and serving systems. FlashAttention-2/3/4 improve dense attention through tiling, work partitioning, and hardware-specific pipelines [8, 41] (Zadouri et al. 2026). PagedAttention and FlashInfer provide cache management and paged execution [21] (Ye et al. 2025). TensorRT-LLM integrates paged execution, quantized caches, and multiple attention backends (NVIDIA, 2026b;d). Its trtllm-gen backend pairs NVFP4 KV with FP8 queries (NVIDIA, 2026c); Jet-QSparse uses NVFP4 Q and K in its dense scan.

Low-bit arithmetic and KV storage. SageAttention uses key centering and INT8 QK; SageAttention2 combines INT4 QK with FP8 PV; SageAttention3 extends both products to NVFP4 (Zhang et al., 2025c;a;b). Attn-QAT studies quantization-aware training and provides kernels with NVFP4 QK and higher-precision PV; Hardware-Aware FP4 FlashAttention-4 quantizes probabilities for noncausal forward attention [16] (Zhang et al. 2026). KIVI quantizes keys per channel and values per token with a full-precision residual buffer; KVQuant combines pre-RoPE key quantization with nonuniform codebooks and separately stored outliers; QuaRot uses invariant rotations [24, 14, 3]. BitDecoding maps dense low-bit-cache attention to tensor cores, including native MXFP4/NVFP4 computation on Blackwell (Du et al., 2026).

Query-dependent sparse decode. Quest bounds page scores using channel-wise key minima and maxima [43]. SparQ estimates token importance from query coordinates with large magnitudes, fetches selected full K/V vectors, and compensates for estimated omitted mass using a running value mean [40]. Loki uses an offline per-head PCA basis to rank tokens in reduced dimensions before attending to selected full-dimensional vectors [42]. Jet-QSparse scans all NVFP4 keys and selects blocks by globally normalized attention mass.

KV eviction and retention. H2O keeps recent tokens and attention heavy hitters; StreamingLLM keeps sinks and a rolling window; SnapKV compresses the prompt cache using a trailing observation window [46] (Zhang et al. 2023; Li et al. 2024). DuoAttention separates full-history retrieval and streaming heads through optimized gates [47], while RocketKV combines prompt eviction with per-step selection [4]. Tri-Attention periodically evicts generation KV using calibrated pre-RoPE query statistics and trigonometric analysis [26]. ThinKV combines thought-dependent precision with progressive eviction [38]. Jet-QSparse retains every token in NVFP4 and selects aggregation blocks at each step.

Sparse prefill and trained selectors. MInference, FlexPrefill, and XAttention use training-free sparse prefill patterns to reduce QK and PV work [18, 22] (Xu et al. 2025); PAROAttention combines block sparsity and low-bit arithmetic for visual generation (Zhao et al., 2025). SeerAttention trains block gates, and SeerAttention-R extends learned, GQA-shared selection to reasoning decode [11, 12]. MoBA uses parameter-free routing in sparsely trained models; Native Sparse Attention and DeepSeek Sparse Attention introduce learned compression or indexing [1] (Lu et al. 2025; Yuan et al. 2025). Jet-QSparse calibrates existing full-attention layers without weight updates.

Sparse aggregation after scoring. SpargeAttention combines pooled Q/K selection with online-softmax PV skipping (Zhang et al., 2025d). BLASST compares tile and running maxima to skip softmax, V loads, and PV without a separate selection pass (Yuan et al., 2026). Jet-QSparse uses a separate dense NVFP4 scan to select shared 128-token blocks under a fixed budget or capped tail-mass policy, then performs FP16 PV with FP32 accumulation. Caps may override the mass target.

Jet-QSparse: Hardware-Optimized 4-Bit Sparse Attention

Jet-QSparse uses NVFP4 as both a compact storage format and a native tensor-core operand. Storing the KV cache in NVFP4 reduces its footprint and read traffic; computing QKTQK^{T} directly from packed queries and keys uses Blackwell’s FP4 tensor cores to score the full context. We retain FP16 operands and FP32 accumulation for PV, the weighted sum that forms the attention output. Sparsity reduces the cost of this higher-precision step by restricting aggregation to selected 128-token blocks.

Figure 2 summarizes the pipeline: dense NVFP4 scoring followed by sparse higher-precision aggregation. Blocks align selection with contiguous memory loads and tensor-core tiles. Calibrated key centering and BF16 copies of the first and current blocks address quantization error within a two-page allowance. The method targets single-token causal decode in full-attention layers and requires no model-weight updates. Prefill uses a separate FP8 path (§ 6.4).

Figure 2. Jet-QSparse decode. Logical blocks pair key and value pages for 128 consecutive tokens; internal divisions are schematic. The dense scan emits FP32 statistics from every key block. First/current-block statistics are recomputed from BF16 keys before normalization. Each query-head group (Ainslie et al., [2]) shares selected K/V blocks, always including the current block. Selected scores are recomputed for online softmax. FP16 unnormalized weights multiply reconstructed FP16 values with FP32 accumulation; denominator weights remain FP32. Merged partials are normalized and cast to BF16. BF16 paths use the same quantized query and FP16 operands. Reconstruction retains NVFP4 storage error. An unselected first page may still be read and masked.

NVFP4 Cache and Block Layout

Aligning selection with memory access. Token-level selection can scatter memory accesses and create irregular workloads. We group B=128B = 128 consecutive tokens into a block, storing its keys and values in matching cache pages (Figure 3). We select whole token blocks. Each KV head occupies contiguous regions for packed codes, micro-block scales, and row scales within a page. A page table locates these regions so kernels can load contiguous tiles even when selected pages are not adjacent in physical memory. Using whole blocks trades finer selection granularity for regular memory access and block-sized matrix products.

Figure 3. Blocked cache layout and lifetime. (a) K/V pages cover matching 128-token ranges; logical neighbors need not be physically adjacent. (b) Bytes cover one head’s contiguous K or V region; compression excludes fixed buffers. (c) Token 384 opens block 3, reusing the current BF16 slot. Block 0’s BF16 copy (Xiao et al., [46]; Hooper et al., [14]) and all NVFP4 copies persist; the live row is enlarged. (d) BF16 inputs feed both paths independently. FP8 pages become reusable; the arena remains reserved.

Packing keys and values

A row contains dd elements for one token and one KV head. Its NVFP4 reconstruction is x^i=axsx,β(i)cx,i\hat{x}_i = a_xs_{x,\beta(i)}c_{x,i}, where cx,ic_{x,i} is an E2M1 code, sx,β(i)s_{x,\beta(i)} is an E4M3 scale shared by a micro-block of 16 elements, and axa_x is an FP32 row scale. For zero-based element index ii, β(i)=⌊i/16⌋\beta(i) = \lfloor i/16 \rfloor. Scaling the row’s maximum magnitude by 1/(6⋅448)1/(6 \cdot448) gives axa_x, where 6 and 448 are the E2M1 and E4M3 maxima. Each micro-block’s maximum magnitude then sets its E4M3 scale, and its elements are rounded to E2M1. Zero scales are replaced by one, so an all-zero row reconstructs as zero. Keys and values are packed independently; the query uses the same format. Including scale metadata, a row occupies

wd=d2+d16+4 bytes.(1)w_d = \frac{d}{2} + \frac{d}{16} + 4 \text{ bytes}. \tag*{(1)}

This is 4.625 bits per element at d=256d = 256 and 4.75 at d=128:3.46×d = 128: 3.46\times and 3.37×\times packed-row compression relative to BF16, excluding fixed buffers.

Centering keys

A component shared across keys can consume quantization range without affecting their relative attention scores. We subtract a calibrated mean μ\mu for each layer and KV head after applying RoPE, where used. In exact arithmetic, subtracting this common vector adds the same constant to every logit of a query and leaves softmax unchanged (Zhang et al., 2025c). Quantizing the centered keys can still change attention. An offline attention-output error screen determines which heads use a nonzero mean; the remaining heads use μ=0\mu= 0.

Retaining two BF16 pages

We retain BF16 copies of block 0 and the current block alongside their NVFP4 copies. The side buffer holds centered keys rounded to BF16 and the original BF16 values, covering at most two distinct blocks per sequence and layer. We pack every row into NVFP4 on its first write. After the current page advances, we read a completed noninitial block from its existing NVFP4 copy; block 0’s BF16 copy persists throughout the sequence. We mask unused positions in the current page. Selection determines whether block 0 contributes to the sparse output.

Dense Scoring for Block Selection

Selecting blocks by their share of attention requires a normalizer over the full context. We obtain it with a dense NVFP4 scan that reads keys and scales without fetching values or materializing the attention matrix.

Consider one KV head and its group G\mathcal{G} of query heads. Let LL be the cached sequence length, including the current token, nb=⌈L/B⌉n_b = \lceil L/B \rceil the number of live blocks, and Ib\mathcal{I}_b the live token positions in block bb. Denote the reconstructed NVFP4 query by q^h\hat{q}_h. The effective operands kjek_j^e, vjev_j^e are the BF16 copies in the first and current blocks and the reconstructed NVFP4 operands elsewhere. In exact arithmetic,

zhj=q^hTkjed,phj=exp⁡(zhj)∑t<Lexp⁡(zht),z_{hj} = \frac{\hat{q}_h^{\mathsf{T}} k_j^e}{\sqrt{d}}, \qquad p_{hj} = \frac{\exp(z_{hj})}{\sum_{t<L} \exp(z_{ht})},
Phb=∑j∈Ibphj.(2)P_{hb} = \sum_{j \in\mathcal{I}_b} p_{hj}. \tag*{(2)}

Thus, PhbP_{hb} measures a block’s importance under the method’s dense attention distribution. Dense here refers to coverage of all keys.

The scan records a reference maximum and an exponential sum for each block and query head. A correction kernel replaces the first and current blocks’ statistics using their BF16 keys and the same quantized query. After this correction, we rescale the statistics to a common maximum and normalize across blocks. This gives selection and aggregation the same effective operands. We write Pˉhb\bar{P}_{hb} for the kernel’s FP32 estimates of the ideal masses in Equation (2).

Selecting Blocks for Grouped-Query Attention

Each KV head uses one selected block set UU for all query heads in its group, sharing the fetched K/V tiles. A fixed budget controls aggregation work, while a tail-mass policy adapts the selected count to the attention distribution. Both policies include the current block b⋆=nb−1b_\star= n_b - 1. Figure 4 illustrates their selections for one query-head group.

Figure 4. Illustrative block attention for two query heads. Green outlines mark selected blocks. Fixed N=3N = 3 includes current block 7; counts are exact, ranks approximate. Tail selection unions the head sets and current block, yielding six blocks with nonbinding four-block caps. Binding caps can override the mass target. Omitted percentages follow head order.

Fixed block budget. We rank each block by the largest mass assigned to it within the group, πb=max⁡h∈GPˉhb\pi_b = \max_{h \in\mathcal{G}} \bar{P}_{hb}. A block can therefore rank highly when any head assigns it substantial mass. Setting πb⋆=2\pi_{b_\star} = 2 ensures that the current block occupies one of the n=min⁡(N,nb)n = \min(N, n_b) selected slots. A 16-step threshold bisection finds the ranking boundary; any remaining slots are filled from the final boundary band in ascending block order. The selected count is exact, while the top-NN ranking is approximate.

We evaluate N=32,64,128N = 32, 64, 128, selecting at most 4,096, 8,192, or 16,384 token positions per KV head. All blocks are selected when the context fits within the budget. A fixed count does not bound an individual head’s omitted mass: blocks preferred by other heads can fill the shared budget.

Tail-mass budget. Each query head hh has an omitted-mass budget τh\tau_h and a calibrated cap fraction fhf_h. We set

fhf_h to the 99th percentile of the context fraction needed on held-out prompts at omitted mass 0.125. Calibration uses 16-token pages; decode converts the fraction to a cap on 128-token blocks, ch=min⁡(nb,max⁡(1,⌊fhL/B⌋))c_h = \min(n_b, \max(1, \lfloor f_h L/B \rfloor)), ignoring FP32 rounding in this expression.

The tail threshold θhT\theta_h^T keeps the sum of masses below it within τh\tau_h; the cap threshold θhC\theta_h^C retains at most chc_h blocks. We find both by 16-step bisection on [0,1][0, 1], taking the feasible lower endpoint for the tail constraint and the feasible upper endpoint for the cap:

θh=max⁡(θhT,θhC),Kh={b:Pˉhb≥θh},\theta_h = \max(\theta_h^T, \theta_h^C), \qquad K_h = \{b : \bar{P}_{hb} \ge\theta_h\},
U={b⋆}∪⋃h∈GKh.(3)U = \{b_\star\} \cup\bigcup_{h \in\mathcal{G}} K_h. \tag*{(3)}

The cap takes precedence over the mass budget and applies to KhK_h, not the final group union. Thresholding can underfill a head’s cap, even leaving an empty set. In the evaluated configurations, τh=1−ρ\tau_h = 1 - \rho for every head: ρ=0.40\rho= 0.40 and 0.60 correspond to omitted-mass budgets of 0.60 and 0.40. Only fhf_h varies across heads. ρ\rho sets a mass target rather than a fixed block fraction.

Sparse Value Aggregation

We fetch the selected pages and recompute their QK products, avoiding a write and reread of token-level scores from the dense scan. With K(U)=⋃b∈UIbK(\mathcal{U}) = \bigcup_{b \in U} \mathcal{I}_b, the ideal output is

ohU=∑j∈K(U)phjvje∑j∈K(U)phj.(4)o_h^U = \frac{\sum_{j \in K(\mathcal{U})} p_{hj} v_j^e}{\sum_{j \in K(\mathcal{U})} p_{hj}}. \tag*{(4)}

This renormalizes softmax over selected keys.

Effect of omitted mass. Let ohDo_h^{\mathrm{D}} be dense attention on the same effective operands and εh=∑j∉K(U)phj\varepsilon_h = \sum_{j\notin K(U)} p_{hj} the omitted mass. For 0<εh<10 < \varepsilon_h < 1, let ohOo_h^{\mathrm{O}} be the normalized output of the omitted keys. Then

ohD−ohU=εh(ohO−ohU),o_h^{\mathrm{D}} - o_h^{\mathrm{U}} = \varepsilon_h(o_h^{\mathrm{O}} - o_h^{\mathrm{U}}),
∥ohD−ohU∥≤εhdiam⁡{vjD:j<L}.(5)\left\lVert o_h^{\mathrm{D}} - o_h^{\mathrm{U}} \right\rVert\leq\varepsilon_h \operatorname{diam}\{v_j^{\mathrm{D}}: j < L\}. \tag*{(5)}

The diameter is the largest pairwise distance between effective values; the error is zero at εh=0\varepsilon_h = 0. With exact masses and arithmetic, the tail policy ensures εh≤τh\varepsilon_h \leq\tau_h when its cap does not bind, i.e., θhT≥θhC\theta_h^{\mathrm{T}} \geq\theta_h^{\mathrm{C}}. The group union can only reduce omitted mass. A binding cap removes this guarantee, and fixed NN provides no such bound. FP32 mass estimation perturbs the ideal guarantee; quantization error relative to BF16 attention is a separate source of error.

Higher precision for the weighted sum. The gather kernel multiplies FP16 unnormalized softmax weights by FP16 reconstructed values, accumulating in FP32. This preserves higher precision in the output sum while sparsity limits the number of values it processes. Reconstruction retains NVFP4 storage error; it cannot restore BF16 values.

The precision choice also matches the cache layout: NVFP4 scales run along the head dimension, the reduction dimension of QK, whereas PV reduces over tokens. The gather kernel reconstructs selected values as it consumes them. A separate kernel processes selected BF16 pages with FP16 operands. The kernels emit partial outputs and softmax statistics for a stable merge, followed by a BF16 output cast. The denominator uses FP32 weights; rounding the numerator weights to FP16 and the remaining kernel arithmetic introduce error beyond Equation (4).

Execution and Prefill

Realizing the decode savings. The implementation uses TensorRT-LLM’s paged cache manager, CuTe ([7]) decode kernels, and Triton ([45]) selection and BF16-page kernels. We store K scales once in the layout consumed by the block-scaled MMA. Pure decode steps fuse query packing with the new token’s K/V store. Block lists remain on the device, and both selection policies execute within CUDA graphs.

A sparse step reads every key in the scan, then rereads selected keys and fetches their values. In a streaming model without cache reuse, the K/V-read volume relative to dense NVFP4 attention approaches 12+κ\frac{1}{2} + \kappa, where κ\kappa is the selected block fraction. This approximation excludes block-statistic traffic, fixed BF16-page overheads, and page padding. It captures read savings when a small fraction is selected at long context and extra reads when most blocks are selected. The dense NVFP4 control directly attends all blocks with the same cache and precision choices, omitting the scan and selector to isolate the cost and benefit of selection.

Separating prefill from decode storage. Prefill uses FlashInfer’s FP8 context attention with calibrated per-layer power-of-two scales. Before attention, the store packs the persistent cache directly from BF16 K/V, updates the two BF16 pages, and writes a temporary FP8 arena. Decode therefore incurs no FP8-to-NVFP4 conversion. We release a request’s arena pages when prefill finishes; decode reads only the NVFP4 pool and BF16 side pages. Requests without arena space fall back to context attention over expanded cache operands. Cache-capacity measurements must account for the arena’s device-memory reservation and the BF16 buffers allocated across request slots.

Evaluation

Experimental Setup

Models and tasks. We evaluate Qwen3.8-27B, Qwen3.6-35B-A3B, Nemotron 3 Nano, and Nemotron 3.5 Lightning, changing only full-attention layers. We measure accuracy on GPQA-Diamond ([39]), AIME25 ([27]), LiveCodeBench v6 ([17]), Terminal-Bench 2.1 ([44]), and SWE-bench Verified ([19, 33]), with three seeds per method.

Implementation and baselines. The four-model evaluation uses frozen TensorRT-LLM snapshots on GB300, with tensor parallelism one and a 256K context window. All methods share the harness and seeds. We compare dense BF16 attention with Quest ([43]), SparQ ([40]), and TriAttention ([26]). Their aggregation budgets are 2,048 tokens, the top 1/16 of positions, and the prompt plus 4,096 generated tokens, respectively. On the Nemotrons, whose attention layers use no positional encoding, our TriAttention integration uses identity phase factors, leaving a calibrated mean-query dot product and a bandwise norm correction. Each baseline uses one untuned operating point. For Jet-QSparse, we evaluate dense NVFP4, two tail-mass settings, and N=32,64,128N = 32, 64, 128. We selected N=64N = 64 for the main comparison after evaluating these settings. These baseline re-implementations use Triton; Jet-QSparse uses CuTe decode kernels alongside Triton selection and BF16-page kernels, so timings also reflect implementation choices. The systems sweep also evaluates TensorRT-LLM’s FP8 KV cache for speed and capacity ([32]).

Measurements. Accuracy and speed use separate workloads. Speedups are ratios of BF16 to method decode time per output token (TPOT), measured on one GB300 per arm with real text and CUDA graphs. The Qwen3 sweep uses two replicates, 32K–252K prompts, batches 1–64, and 4,096 generated tokens; we omit cells that exceed an arm’s measured pool. Output throughput includes prefill; time to first token (TTFT) is separate. KV capacity counts allocated token slots; bytes per token describe packed storage.

We compare accuracy using paired tests clustered by problem or task. No detected loss means that no task in the stated benchmark set has a negative difference from BF16 at p<0.05p < 0.05; it does not establish equivalence. We also report Benjamini–Hochberg (BH) adjusted qq values Benjamini & Hochberg, [5], calling a negative result at p<0.05p < 0.05 but q≥0.10q \ge0.10 a nominal loss. Approximate minimum detectable differences at 80% power (MDD80_{80}) describe resolution in percentage points. Appendix A gives the tests, correction families, run settings, and calibration checks. We exclude a separate SWE-bench comparison after a shared writable environment exposed solution artifacts across runs (Appendix A).

Accuracy–Efficiency Trade-off

At N=64N = 64, Jet-QSparse combines faster decode and compact storage with no detected accuracy loss on the two Qwens and Lightning (Table 1). Across these models, MDD80_{80} ranges from 1.17–3.76 points on reasoning and coding, 2.44–2.80 on SWE-bench, and 6.24–8.58 on Terminal-Bench. Nano has one nominal loss: AIME25 decreases by 1.98 points (95% interval [−3.77, −0.37], p=0.0088p = 0.0088, q=0.165q = 0.165). Increasing the budget to N=128N = 128 gives no detected loss on the five application benchmarks across all four models. Section 4.4 analyzes precision and budgets.

At 128K context and batch 16, generating 1,024 tokens, N=64N = 64 speeds up Qwen decode by 1.94× and 1.57×, and Nemotron decode by 1.11× and 1.13×. Its packed cache uses 28.9–29.7% of BF16’s bytes per token. Median contexts in the accuracy runs are 6.9K–39.2K on reasoning tasks and 30K–65K per agentic call, below 128K.

The baselines make different trade-offs (Table 1). SparQ has no detected loss on the Qwens, while its layout uses 1.5× the BF16 bytes. Quest incurs large agentic losses on the Qwens; TriAttention loses long-reasoning accuracy. Jet-QSparse outpaces every fitting baseline in Table 1 while using less KV storage; SparQ’s Qwen3.8 pool cannot admit this batch.

Decode Performance and Serving Capacity

Gains grow with batch and context. At N=64N = 64, decode speedup rises from 0.95–1.14× at batch one to 2.00× on Qwen3.8 and 2.01× on Qwen3.6 at 252K, with batches 8 and 16, respectively (Appendix D). At BF16’s largest supported batches, N=64N = 64 outpaces every fitting baseline in all eight tests (Table 7). Figure 5 compares scaling across block budgets and dense precision baselines; at fixed batch eight, the 252K gains are 2.00×/1.66×.

Four line plots showing Jet-QSparse scaling with batch size and context length for Qwen3.8 and Qwen3.6

Figure 5. Jet-QSparse scaling with batch size and context length, compared with dense NVFP4, FP8 KV, and dense BF16 on one GB300. Speedups use 4,096 output tokens and two replicates. (a,b) Batch sweeps from 2 to 16 at 128K context; (c,d) context sweeps at batch eight. Jet-QSparse uses N=32,64,128N = 32, 64, 128. Axes are logarithmic; we plot batches that fit BF16. Each method uses its own BF16 timing reference; Appendix § 14 describes the comparison across measurements.

Native low-precision baselines. FP8 KV reaches 1.47×/1.48× decode speedup on the Qwens and often matches or exceeds N=64N = 64 at batch one. At these two 252K workloads, dense NVFP4 achieves 1.51×/1.56×; selection adds approximately 1.33×/1.30×. The dense control captures the cache format, arithmetic, and kernels together. The read model approaches 12+κ\frac{1}{2} + \kappa of dense NVFP4 traffic, so the scan limits gains from smaller budgets.

Kernel-level speedup. Separate GB300 per-layer benchmarks at head dimension 256 give N=64N = 64 1.98–4.01× the served BF16 kernel’s speed, including selection and BF16-page work (Figure 6). Appendix § 12 gives the timing protocol, detailed measurements, and FA4 baseline checks.

Figure 6. GB300 single-layer decode speedup over TensorRT-LLM BF16 at head dimension 256. Adapted FA4 uses GQA packing with or without split-KV, whichever is faster. Jet-QSparse is timed separately, including BF16-page work and selection for N=64N = 64.

Long generation. With 1K prompts, 16K outputs, and batch 64, N=64N = 64 runs at 0.97–1.08× BF16 speed across the four models: dense scanning and selection offset gather savings at short contexts. TriAttention performs no eviction in the 4K-output sweep; with 16K outputs it reaches 1.16×/1.06× on the Qwens with eviction kernels compiled before timing. Appendix § 14 gives the full comparison.

Appendix D gives the full comparison.

Capacity and serving throughput. Measured NVFP4 pools grow 2.34M→6.05M tokens (2.58×) on Qwen3.8 and 6.82M→18.80M (2.76×) on Qwen3.6; FP8 KV provides 1.90×/1.85× BF16 capacity. Recurrent-state and FP8-arena reservations keep NVFP4 below its 3.46× packed-row ideal. Qwen workloads exceeding BF16 capacity with 48 or 64 requests (128K prompts, 8K outputs) gain 1.99–2.16× BF16 throughput from faster computation and larger admitted batches; both Qwen3.8 pools overflow. The separate 4K-output sweep’s two 252K workloads yield 1.57×/1.71× throughput including prefill. Across all four models, FP8 prefill improves single-request TTFT 1.17–1.32× at 128K and 1.25–1.46× at 250K.

Design Ablations and Error Analysis

Table 2. Paired AIME25 changes from BF16 (points). Red ↓: p<0.05p < 0.05. Final adds centering, two BF16 pages, FP8 prefill, and CuTe. Lower rows use dense NVFP4 with the original prefill. No listed gain survives BH correction.

Precision and calibration. We screen precision choices offline and measure task accuracy (Table 2). Across all 12 Nemotron budget pairs, the recipe raises AIME25 by 1.46–2.50 points over matched controls, though no individual gain survives BH correction. These comparisons change centering, BF16 retention, prefill, and the Nemotron kernel implementation together. Dense controls separate BF16 retention from centering; centering adds no detected reasoning or coding gain over two pages. At N=64N = 64, all 20 comparisons with dense NVFP4 show no detected loss.

ConfigurationNanoLightning
N=64N = 64, unfixed−4.13↓−3.16↓
N=64N = 64, final−1.98↓−1.01
Dense, no BF16 pages−2.74↓−2.33↓
Dense, two BF16 pages−2.12↓+0.66
+ key centering−1.63↓−0.38

Table 2.

Quantization changes trajectories. Diagnostic runs use simulated NVFP4 on H100 or packed NVFP4 on GB200, without BF16 pages or FP8 prefill and with different budget and agentic settings (Appendix § 11). On 16 Qwen3.6 SWE contexts, NVFP4 changes 3.45% of teacher-forced top-1 tokens; greedy continuations diverge after a median 2.5 steps. This cohort is separate from the agentic outcome comparisons in Appendix § 13. Table 1 provides the matched final-engine evaluation of the complete Jet-QSparse precision recipe, rather than quantization alone.

ModelMethodGPQA-DAIME25LCB v6TB 2.1SWE-VDecode×KV×
Qwen3.8 27BBF1690.1±0.990.1 \pm0.999.0±0.499.0 \pm0.486.7±0.886.7 \pm0.870.0±2.670.0 \pm2.682.7±3.782.7 \pm3.71.001.000
Jet-QSparse N=64N=6490.9±1.790.9 \pm1.798.3±0.398.3 \pm0.388.5±1.788.5 \pm1.771.9±1.171.9 \pm1.183.5±4.383.5 \pm4.31.940.289
Quest89.1±1.189.1 \pm1.198.1±0.498.1 \pm0.481.5±0.481.5 \pm0.427.7±3.627.7 \pm3.64.7±0.24.7 \pm0.21.681.055
SparQ89.2±1.189.2 \pm1.198.7±0.498.7 \pm0.487.6±0.687.6 \pm0.672.3±1.772.3 \pm1.783.7±2.983.7 \pm2.9n/f1.501
TriAttention88.4±0.588.4 \pm0.579.0±1.879.0 \pm1.866.4±0.866.4 \pm0.873.0±5.673.0 \pm5.682.7±2.382.7 \pm2.31.00§1.00^{\S}1.000‡1.000^{\ddagger}
Qwen3.6 35B-A3BBF1684.8±1.484.8 \pm1.491.8±0.691.8 \pm0.679.5±1.579.5 \pm1.541.6±4.141.6 \pm4.168.5±1.468.5 \pm1.41.001.000
Jet-QSparse N=64N=6483.3±2.283.3 \pm2.291.2±0.691.2 \pm0.678.5±0.378.5 \pm0.341.2±1.741.2 \pm1.770.6±1.670.6 \pm1.61.570.289
Quest82.0±2.082.0 \pm2.076.1±2.176.1 \pm2.159.6±1.059.6 \pm1.022.1±0.622.1 \pm0.619.4±0.819.4 \pm0.81.321.050
SparQ83.0±0.683.0 \pm0.691.1±0.391.1 \pm0.377.7±0.877.7 \pm0.840.1±1.740.1 \pm1.769.3±0.569.3 \pm0.51.091.502
TriAttention81.1±1.181.1 \pm1.163.7±1.363.7 \pm1.372.8±0.672.8 \pm0.641.9±2.341.9 \pm2.369.3±1.469.3 \pm1.41.00§1.00^{\S}1.000‡1.000^{\ddagger}
Nemotron 3 NanoBF1672.5±2.272.5 \pm2.288.9±1.088.9 \pm1.069.6±0.669.6 \pm0.64.1±1.74.1 \pm1.721.9±1.821.9 \pm1.81.001.000
Jet-QSparse N=64N=6472.7±1.872.7 \pm1.886.9±0.586.9 \pm0.568.3±1.468.3 \pm1.45.2±1.75.2 \pm1.720.7±1.720.7 \pm1.71.110.297
Quest72.4±1.972.4 \pm1.985.5±0.885.5 \pm0.867.4±0.667.4 \pm0.64.9±1.74.9 \pm1.719.3±1.719.3 \pm1.71.071.042
SparQ71.2±1.771.2 \pm1.787.2±0.487.2 \pm0.468.6±0.368.6 \pm0.32.6±1.72.6 \pm1.717.7±1.317.7 \pm1.30.991.504
TriAttention70.7±0.970.7 \pm0.971.8±1.071.8 \pm1.059.1±1.359.1 \pm1.37.1±1.37.1 \pm1.321.6±2.121.6 \pm2.11.00§1.00^{\S}1.000‡1.000^{\ddagger}
Nemotron 3.5 LightningBF1676.3±2.076.3 \pm2.087.6±0.687.6 \pm0.679.8±0.979.8 \pm0.924.0±0.624.0 \pm0.648.4±0.248.4 \pm0.21.001.000
Jet-QSparse N=64N=6477.6±0.877.6 \pm0.886.6±1.786.6 \pm1.779.1±0.479.1 \pm0.424.3±1.724.3 \pm1.748.8±0.948.8 \pm0.91.130.297
Quest74.4±0.874.4 \pm0.885.8±1.985.8 \pm1.977.5±0.177.5 \pm0.120.6±1.720.6 \pm1.743.1±1.043.1 \pm1.01.051.042
SparQ75.1±1.175.1 \pm1.188.2±0.388.2 \pm0.379.0±0.379.0 \pm0.321.3±4.121.3 \pm4.138.2±1.038.2 \pm1.00.971.504
TriAttention72.9±1.872.9 \pm1.872.7±1.272.7 \pm1.269.4±0.769.4 \pm0.723.2±3.923.2 \pm3.947.8±2.147.8 \pm2.11.00§1.00^{\S}1.000‡1.000^{\ddagger}

Table 1. Four-model comparison. Accuracy is mean ±\pm SD over runs (%). BF16 has six runs on GPQA-D, AIME25, and LCB, and three on TB and SWE; other rows have three runs. Red entries with ↓\downarrow mark negative paired differences from BF16 at p<0.05p < 0.05; ↑\uparrow marks positive differences at p<0.05p < 0.05, not claimed as gains. Decode is TPOT speedup at 128K context, batch 16, and 1,024 output tokens, measured separately from accuracy; bold marks the fastest fitting method. n/f: the batch exceeds the method’s KV-pool capacity. KV is packed bytes per token relative to BF16, including format metadata but excluding serving reservations. §TriAttention does not evict before 4,224 generated tokens; ‡per retained token.

Sparsity affects distributed evidence. Figure 7(a) compares the configurations on RULER, indexed by their SWE-bench block fractions. Count caps balance attention coverage against gather work: when a head needs more blocks to meet its mass target, the cap takes precedence. The union across query heads can still vary in size; fixed NN instead budgets the shared KV-group list directly. Common-word losses reach 19.8 points for Qwen3.6 and 18.5 for Nano, while their 13-task averages fall by 2.43 and 1.94 points. Needle retrieval remains largely preserved across the tested budgets: all eight Qwen3.6 needle families score 100%, and Nano’s needle scores stay within one point of dense. Figure 8 shows aggregation losses with known fixed-budget fractions. The selection-error relation (Equation (5)) connects this sensitivity to omitted mass and the gap between selected and omitted value averages. Our final-engine RULER diagnostic ([15]) uses 13 tasks, three lengths (32K/128K/256K), and three 100-document datasets per task and length. Table 3 reports its budget effects. We report RULER as a separate diagnostic.

Precursor sparsity study plots for RULER accuracy, SWE-bench accuracy, and SWE-bench unfinished

Figure 7. Precursor sparsity study. All block fractions were measured during SWE-bench agentic runs. Hollow markers denote count caps: mass-threshold selection with an additional per-head block limit. (a) RULER common-word and 13-task score changes from each model’s dense reference. The x-axis identifies configurations by their SWE-bench fractions, not their RULER traffic. (b) SWE-bench changes from BF16 (Qwen3.6) or same-stack dense attention (Nano); bars show half the two-seed spread. Numbers count head/layer pairs retaining almost no history. (c) Unfinished rates use available Qwen seeds and Nano seed one.

Figure 8. Fixed-budget RULER study on Nemotron 3 Nano in the precursor engine. Scores are changes from dense attention at the same context length (128K, 160K, or 192K). Brick: common-word extraction; gray: 13-task average. Filled markers use N=16N = 16 blocks per KV head; hollow markers use N=8N = 8. The gathered fraction is 128N/L128N/L. Dashed: fit to all six common-word points.

BudgetQwen3.8Qwen3.6NanoLightning
N=32N = 32+0.12−0.85−0.78−1.55
N=64N = 64+0.28−0.16−0.45−0.83
N=128N = 128+0.28−0.02−0.29−0.45

Table 3. Final-engine RULER changes from BF16, averaged over 13 tasks and three context lengths (points). Bold marks negative differences at p<0.05p < 0.05 and q<0.10q < 0.10. Output caps affect Qwen3.8’s positive aggregates, concealing variable-tracking losses.

Fixed budgets and context length. Figure 8 shows Nano’s common-word loss increasing with smaller gathered fractions, while the task average changes much less. The six-cell

fit gives 16.5 points per halving (SE 0.5), over 0.52–1.56%. For a binding budget, the per-step fraction is approximately κ=128N/L\kappa= 128N/L. A practical starting rule is therefore to choose NN for the configured maximum context and keep it fixed during decoding: 128 blocks at 256K and 512 at 1M both gather 6.25% at maximum length. The 1M setting remains untested; equal fractions need not retain equal attention mass.

Agentic failures differ by model. Figure 7(b) and Figure 7(c) compare accuracy and completion. Qwen3.6 stays near BF16 through κ=0.048\kappa= 0.048; tighter caps coincide with losses up to 11 points and 17.3% unfinished episodes, versus 3.6% across preceding settings. Unfinished episodes yield no gradable result after time/context limits or abnormal exits. Nano loses 5.6 points on average over comparable fractions; lower accuracy among finished episodes accounts for at least 87% of seed-one losses of 2.3 points or more. In the main evaluation, no Qwen3.6 SWE setting falls below BF16 at p<0.05p < 0.05. Nano’s two lowest-fraction losses remain nominal, with no detected deficit against dense NVFP4.

Budget choice and prefill precision. In the four-model study, N=64N = 64 retains at least 94% of N=32N = 32 speed. At N=128N = 128, all four models meet the five-benchmark accuracy criterion; this budget covers 44–83% of reference reasoning/coding contexts per step. Small average RULER losses remain (Table 3). A separate FP8-prefill effect costs Qwen3.8 10.1–14.2 points on variable tracking at 256K by echoing the answer prefix. Dense NVFP4 controls measure the additional effect of sparsification; measuring task-level interaction with quantization requires a matching BF16 sparse arm, which we did not evaluate.

Scaling to a 2.4T-Parameter Model

We deploy Qwen3.8-2.4T-A95B [37] across eight four-GPU GB300 nodes, with 32-way tensor and expert parallelism and attention data parallelism. All arms use the same FP8 weights; BF16 refers to the reference KV cache. We evaluate GPQA-Diamond, AIME25, LiveCodeBench, and Terminal-Bench with two seeds per Jet-QSparse arm. The reference has four runs on the first three tasks and two on Terminal-Bench.

All four configurations show no detected loss on the four evaluated tasks (Table 4). At N=64N = 64, MDD80 is 2.11 points on GPQA, 2.25 on LiveCodeBench, and 8.99 on Terminal-Bench. AIME25 is near its ceiling: the paired change is −0.36 points with a 95% interval of [−1.04, +0.29], and only five problems contribute nonzero paired differences. We did not evaluate SWE-bench at this scale.

MethodGPQA-DAIME25LCBTB
BF1692.2±0.792.2 \pm0.799.3±0.399.3 \pm0.389.2±0.489.2 \pm0.482.6±5.682.6 \pm5.6
Dense NVFP491.7±0.491.7 \pm0.499.6±0.399.6 \pm0.389.2±0.989.2 \pm0.981.5±2.481.5 \pm2.4
N=32N = 3291.7±0.491.7 \pm0.499.0±0.399.0 \pm0.388.7±0.588.7 \pm0.578.7±1.678.7 \pm1.6
N=64N = 6492.4±0.792.4 \pm0.799.0±0.099.0 \pm0.089.6±0.989.6 \pm0.979.8±3.279.8 \pm3.2
N=128N = 12892.7±1.192.7 \pm1.199.2±0.099.2 \pm0.090.5±0.090.5 \pm0.080.3±0.880.3 \pm0.8

Table 4. 2.4T accuracy (mean ±\pm SD, %). No arm falls below BF16 at p<0.05p < 0.05 on any of the four tasks.

At N=64N = 64, decode speedups range from 1.07–1.22× across 32K–250K workloads (Table 5). The measured pool holds 2,360,832 tokens per rank versus 835,584 for BF16 (2.825×). Twelve requests per rank, each with a 128K prompt and 8K output, exceed BF16 capacity twofold but fit Jet-QSparse. Whole-deployment throughput, including prefill, improves 1.487× versus 1.423× for dense NVFP4.

MethodDecode (×) 32K×2Decode (×) 128K×6Decode (×) 250K×3Output rate (×)
Dense BF161.0001.0001.0001.000
Dense NVFP41.0581.1611.0941.423
N=32N = 321.0761.2971.1771.480
N=64N = 641.0741.2161.1491.487
N=128N = 1281.0461.2341.1541.472

Table 5. 2.4T speedups over dense BF16 on 32 GB300 GPUs. Decode columns give prompt length × requests per rank. Output rate covers 12 requests per rank, each with a 128K prompt and 8K output, and includes prefill. Ratios use two-run means, except N=32N = 32, 64 use three-run medians; their 128K replicate ranges overlap.

Attention share and memory. In the N=32N = 32 profiling study, we infer from traced kernel times that attention accounts for 15–21% of the BF16 step. Reducing the NVFP4 pool to approximately BF16 capacity frees 41.5 GiB per GPU. With large preallocated pools, the benefit instead appears as capacity; in a replayed Terminal-Bench load, the reserved FP8 arena offsets live-KV savings. Appendix § 14 separates allocated memory from live demand and describes these measurements.

Conclusion

Jet-QSparse combines compact NVFP4 storage and native QK computation with sparse, higher-precision value aggregation in a simple, training-free design. Its block layout aligns selection with contiguous memory access and tensor-core execution. With 64 blocks per KV head, both Qwen models show no detected accuracy loss on five application benchmarks and reach up to 2.01× end-to-end decode speedup over dense BF16 on separate 252K-context workloads. Their measured KV pools hold 2.58–2.76× as many tokens. At 128 blocks, all four Qwen and Nemotron models show no detected accuracy loss across mathematical reasoning, coding, and agentic benchmarks. At 2.4T parameters, compact KV storage expands capacity while expert computation and communication limit decode gains.

Quantization can change the attention distribution used for block selection. The additional selection error depends on omitted mass and the difference between selected and omitted value averages. Across all four models, the 64-block configuration shows no detected additional loss relative to the final dense NVFP4 control on the five application benchmarks. Our diagnostics show larger losses on aggregation over distributed evidence under aggressive sparsity, while needle retrieval remains largely preserved. These findings guide joint precision and budget calibration through attention-output probes and end-to-end task validation.

AI USAGE DISCLOSURE

Claude (under Claude Code) is used for kernel debugging (mainly tracing runtime errors and portability issues), experiment monitoring, and related work lookup. ChatGPT (website) is used for wording polishing of the paper.

Evaluation Protocol and Statistical Resolution

Runs and comparisons. The four-model accuracy study uses pinned checkpoints and frozen TensorRT-LLM snapshots: fsnap-10020414 for the final Jet-QSparse arms and final1001 for the BF16 reference and baseline re-runs. Within each model and benchmark, arms share the harness configuration, sampling settings, 262,144-token context limit, and strict tool-call parsers. Each arm runs seeds 42–44. On GPQA, AIME25, and LiveCodeBench, the reference averages two independent BF16 runs per seed; Terminal-Bench and SWE-bench have one BF16 run per seed. Tables report means and sample standard deviations across runs, whereas the paired tests operate on item outcomes. We measure speed on separate workloads; the timing cells do not measure latency within these accuracy runs.

Full-attention layers account for 16 of 64 layers on Qwen3.8-27B, 10 of 40 on Qwen3.6-35B-A3B, and six of 52 on each Nemotron. The Qwens use head dimension 256 and the Nemotrons 128. The 252K sweep prompts use 258,048 of 262,144 tokens, leaving 4,096 for output.

Each seed evaluates 198 GPQA questions, 30 AIME25 problems with 16 samples each, 454 LiveCodeBench v6 problems from the August 2024 to May 2025 evaluation window, 89 Terminal-Bench tasks, and 500 SWE-bench Verified instances. Each agent call generates at most 16,384 tokens.

We chose the headline N=64N = 64 after evaluating all six configurations. We selected the accuracy fixes on the same benchmark problems later used for evaluation; repeat runs provide new samples, not new problems. Each sparse baseline uses one untuned operating point, and the comparisons are not sparsity-matched. These choices limit conclusions about an optimal budget or the relative accuracy of independently tuned methods.

Timing and capacity eligibility. The common timing workload in Table 1 uses 131,072 prompt tokens, batch size 16, and 1,024 output tokens on all four models. Figure 1 uses the same prompt length, batch size eight, and 4,096 output tokens from the separate Qwen sweep. Speed runs use real text and CUDA graphs on one GB300 per arm. TPOT is the per-token decode time of the request prefilled last; speedup divides the same run’s dense BF16 TPOT by the arm’s. SparQ’s Qwen3.8 cell in Table 1 is marked n/f: its 2,113,536-token demand exceeds the measured 1,576,928-token pool.

The batch/context sweeps use 4,096 output tokens and two replicates. Baselines and Jet-QSparse run in separate campaigns, each normalized to its own dense BF16 anchor. A same-node N=32N = 32 comparison reproduces the Jet-QSparse grid with median absolute deviations of 0.08% on Qwen3.8 and 0.85% on Qwen3.6 (maxima 0.83% and 2.73%). This check supports the comparison but does not eliminate cross-campaign uncertainty. Reported fixed-batch cells require all requests to fit without suspension. We use measured pool capacities where available and the validated cache-manager allocation rule otherwise. We mark larger batches as not fitting and exclude the Qwen3.8 FP8-KV 64K×\times64 cell after a request suspension.

SWE evaluation integrity. We exclude a thirteen-arm Qwen3.8 SWE comparison over all 500 instances and seeds 42–44 because shared writable runtime files compromised task isolation. The sandboxes shared a Python installation containing a task solution and notes. Twenty retained episodes across ten arms and 19 tasks read these files; one matching-task episode subsequently made the identical edit. This establishes solution propagation, not its effect on the success rate. Incomplete tool histories prevent bounding total exposure, so we exclude the whole comparison rather than filter only known readers. The separate FP8-KV/BLASST evaluation’s BF16 control differs from Table 1’s reference by +4.83 points (95% CI [+2.95, +6.71], p=0.0001p = 0.0001). We exclude the associated FP8-KV and BLASST SWE measurements from the accuracy comparison. This audit cannot certify environments outside the excluded comparison.

Paired, clustered inference. An item is a (seed, problem) pair on the reasoning benchmarks, with AIME25’s 16 samples averaged within that item, or a (seed, task or instance) pair on the agentic benchmarks. We exclude SWE-bench pairs with zero-token outcomes. Let did_i be the arm-minus-reference difference in percentage points. Items cluster by problem, task, or instance across seeds. For cluster cc, define Dc=∑i∈cdiD_c = \sum_{i\in c} d_i, nc=∣c∣n_c = |c|, and T=∑cDcT = \sum_c D_c. With F=10,000F = 10{,}000 seeded sign draws, independently uniform over {−1,+1}\{-1,+1\} for each cluster, the estimate and two-sided Monte Carlo test are

Δ^=∑cDc∑cnc,Tf=∑csfcDc\widehat{\Delta} = \frac{\sum_c D_c}{\sum_c n_c}, \qquad T_f = \sum_c s_{fc}D_c
p=1+#{f:∣Tf∣≥∣T∣}F+1.(6)p = \frac{1 + \#\{f : |T_f| \ge|T|\}}{F + 1}. \tag*{(6)}

The 95% interval inverts this test: it contains shifts δ0\delta_0 for which testing Dc−δ0ncD_c - \delta_0 n_c gives p≥0.05p \ge0.05. This inversion assumes a constant per-item shift. The sign-flip test is exact for independent cluster sums symmetric about zero under the null, but approximate with a two-run reference.

Calibration and shared-reference uncertainty. Design-based grid simulations reject 3.3–4.7% of nulls at nominal 5%, with 94.2–96.2% interval coverage in the simulated shifted cases. Real-data null checks reject 0.0–3.3% on three models but 4.4–13.3% on Nano, indicating that Nano’s grid intervals are about 15% too narrow. Agentic tests have neither a null family nor a coverage simulation. All contrasts for a model share its BF16 runs, so a high reference run can produce deficits across several configurations. Clustering by task handles repeated tasks across seeds but does not by itself remove whole-run shifts. Post hoc pooling and sign analyses are descriptive; they do not determine configuration verdicts.

Multiplicity and verdicts

For ordered pp-values, Benjamini–Hochberg correction [5] gives

q(i)=min⁡j≥imin⁡(1,mp(j)j).(7)q_{(i)}=\min_{j\ge i}\min\left(1,\frac{mp_{(j)}}{j}\right). \tag*{(7)}

The final-method family has m=120m = 120: four models, five benchmarks, and six configurations comprising dense NVFP4, N=32,64,128N = 32, 64, 128, and ρ=0.40,0.60\rho= 0.40, 0.60. Baseline qq-values use the full family of m=80m = 80 comparisons: the 60 displayed contrasts cover three baselines, four models, and five benchmarks; 20 additional comparisons remain in the correction family but are not displayed. We apply multiplicity correction to RULER in separate diagnostic families and exclude it from the five-benchmark criterion; we adopted this separation after an initial combined analysis. A negative estimate with p<0.05p < 0.05 is a detected loss if q<0.10q < 0.10, and nominal otherwise. A configuration shows no detected loss only when none of its five cells is negative at p<0.05p < 0.05, irrespective of the correction family. Positive differences are not claimed as gains. The positive regression dependence condition for BH [6] is unproven for these two-sided tests; sharing a reference does not establish it. We did not apply the Benjamini–Yekutieli correction [6].

Detection resolution. Using a normal approximation with interval half-width η≈z0.975σ\eta\approx z_{0.975}\sigma, we report

MDD80=(z0.975+z0.80)σ≈1.429η.(8)\mathrm{MDD}_{80}=(z_{0.975}+z_{0.80})\sigma\approx1.429\eta. \tag*{(8)}

This approximates the loss detectable at p<0.05p < 0.05 with 80% power, not an equivalence margin or a confidence bound on loss. Multiplicity requires stronger evidence: a lone rejection among 120 needs p≤0.00083p \le0.00083 and has only about 29% power at one MDD80\mathrm{MDD}_{80}. Discrete outcomes further limit resolution. With kk nonzero cluster sums, the exact two-sided sign-flip pp is at least 2/2k2/2^k; fewer than six informative clusters cannot yield p<0.05p < 0.05. Near-ceiling ties can also make the normal MDD approximation optimistic. Smaller losses remain possible after a non-rejection.

The 2.4T extension. This campaign uses two seeds per Jet-QSparse arm, four BF16 grid runs, and two BF16 Terminal-Bench runs, with one BH family of 16 contrasts across four benchmarks. We deferred SWE-bench, so this study cannot meet the five-benchmark criterion. Test and verdict rules preceded the runs. We chose two seeds after reading two LiveCodeBench pilots, and chose the four-run reference with BF16 grid scores known. We decided the final run set before Jet-QSparse Terminal-Bench started, but registered it after reading the grid and one BF16 Terminal-Bench run, 20 hours after the first Jet-QSparse Terminal-Bench launch and before reading any of its rows. Two runs had started and one had finished by registration. Under equal independent sampling variance, two seeds increase MDD80\mathrm{MDD}_{80} by approximately 3/2=1.22\sqrt{3/2}=1.22 relative to three; each cell nevertheless uses its estimated resolution.

Exploratory error studies. The precursor studies in § 8 use a separate engine, precision recipe, and agentic protocol. Appendix § 13 details the heterogeneous quantization comparisons and their descriptive pool. The token disagreement study uses 16 contexts from one model and is not paired with those task–outcome comparisons. The context-scaling fit uses six single-run cells with shared controls within each length and different documents across lengths; its standard error does not represent fresh-document uncertainty. We selected the agentic budget range from Qwen’s outcomes; Nano’s finished-episode decomposition uses seed one. These observations suggest patterns to investigate; the final-engine controls provide separate checks in the main text.

Kernel measurements

MethodQwen3.8 32K×32Qwen3.8 128K×16Qwen3.6 32K×32Qwen3.6 128K×16
TRT-LLM BF16580.61,135291.5571.7
FA4, as shipped3,5157,4992,1954,934
FA4, adapted741.91,465374.1747.0
JQS, dense301.0545.8170.2293.4
JQS, N=64N = 64237.2283.2147.5175.4

Table 6. Single-layer decode times in μ\mus on GB300, head dimension 256. Qwen3.8 has 24 query/four KV heads; Qwen3.6 has 16/two. JQS denotes Jet-QSparse, timed separately with BF16-page work and selection for N=64N = 64.

The TensorRT-LLM/FlashAttention-4 study uses FA4 4.0.0b19, CUDA-event timings with L2 flushes, and two replicate medians with at least 20 timing repetitions. Decode runs under CUDA graphs. The tested FA4 path at head dimension 256 lacks native GQA packing and split-KV. The adapted result is the faster of GQA packing alone and packing with an external split-KV/combine step. We time Jet-QSparse’s final decode kernels in a separate benchmark with seven shuffled rounds and include scan, selection, sparse attention, BF16-page operations, and merging where applicable. The descriptive ratios in Figure 6 compare matching head geometries, batches, and context lengths. Differences in harnesses, clocks, and timing aggregation prevent an exact decomposition of end-to-end speed.

Figure 6 and Figure 9 use TRT-LLM time divided by method time. The decode reference is the BF16 attention kernel itself. The prefill reference is the corresponding served context operator, using either one chunk or 8K chunks on both sides. FA4 prefill takes 0.899–0.945 of the one-chunk baseline time and 0.936–0.968 with 8K chunks. These are attention measurements, not a separate FA4 end-to-end serving arm.

FA4 decode speedup by batch and FA4 prefill speedup

Figure 9. FA4 speedup over TRT-LLM BF16 (log scale). (a) Decode at 128K: as shipped (hollow, dashed) or adapted (filled, solid). (b) Prefill at batch one: one chunk (filled, solid) or 8K chunks (hollow, dashed). Circles and squares denote the Qwen3.8 and Qwen3.6 head geometries.

Precursor Quantization Diagnostics

Figure 10 summarizes four earlier agentic comparisons of NVFP4 and BF16. These studies differ in hardware, arithmetic, benchmark, and inclusion of incomplete episodes. They provide exploratory evidence separate from the matched final-engine comparisons in Table 1.

Figure 10. Exploratory agentic accuracy changes from BF16 to NVFP4, with 95% intervals. Study protocols differ; rows are not matched repetitions across models. SWE denotes SWE-bench Verified; TB denotes Terminal-Bench 2.1. The descriptive pool uses studies 1, 3, and 4; study 2 shares study 1’s BF16 reference.

Study protocols. Studies 1 and 2 use simulated NVFP4 with BF16 arithmetic on H100 for Nemotron 3 Nano SWE-bench Verified. They share one BF16 reference and include only episodes completed by both arms: 468 and 477 of 500 problems, respectively. Study 3 evaluates Qwen3.6 on all 500 SWE-bench Verified problems using packed NVFP4 on GB200. Its records conflict on sampling settings and whether decode uses native FP4 arithmetic or BF16 over a dequantized cache, limiting attribution to quantization alone. Study 4 evaluates Qwen3.6 on 89 Terminal-Bench 2.1 tasks with four seeds per arm on GB200 and native FP4 scoring at prefill and decode.

Descriptive pooling. The inverse-variance pool of studies 1, 3, and 4 gives −0.04 points [95% CI −2.2, +2.1]. We exclude study 2 because it shares study 1’s BF16 reference. The pool assumes a common percentage-point effect across these different models and benchmarks; it does not reconcile their protocols or completion criteria. It establishes neither equivalence nor the accuracy of the final Jet-QSparse recipe. The 16-context probe in Section 4.4 uses a separate cohort and cannot link token disagreement to task outcomes.

Additional System Measurements

Batch, Context, and Block Budget

Table 7 compares baselines at the largest batch dense BF16 accommodates at each context. It also reports each method’s best output rate, including prefill, over its tested batches. This output rate reflects both faster execution and additional capacity. Ratios use each campaign’s own BF16 reference; Appendix § 11 describes the timing comparison and its N=32N = 32 consistency check.

MethodPool ratioTPOT speedup 32KTPOT speedup 64KTPOT speedup 128KTPOT speedup 252KBest output-rate ratio 32KBest output-rate ratio 64KBest output-rate ratio 128KBest output-rate ratio 252K
Qwen3.8-27B
Quest0.951.101.481.701.831.071.241.291.27
SparQ0.671.15n/fn/fn/f1.100.910.910.92
TriAttention1.001.001.001.001.001.001.001.011.01
FP8 KV1.901.301.461.471.471.501.281.501.55
Dense NVFP42.581.301.491.511.511.521.491.541.60
Jet-QSparse N=64N = 642.581.411.841.952.001.631.691.761.81
Qwen3.6-35B-A3B
Quest0.961.041.321.531.741.031.191.261.25
SparQ0.691.091.171.181.201.071.111.091.08
TriAttention1.001.011.001.011.001.011.001.011.01
FP8 KV1.851.251.381.431.481.211.311.571.58
Dense NVFP42.761.251.431.481.561.211.341.631.81
Jet-QSparse N=64N = 642.761.311.681.872.011.261.481.882.08

Table 7.

We compute capacity ratios from measured attention-pool sizes. The cache manager divides its quota between attention pages and recurrent states using an average request length of 16,384 tokens. Reducing bytes per attention token therefore increases the recurrent-state allocation as well as token capacity. This explains why FP8 KV provides 1.90/1.85×1.90/1.85\times capacity and NVFP4 2.58/2.76×2.58/2.76\times, below their storage-only ratios. SparQ’s additional key copy reduces its pool to 0.67/0.690.67/0.69 of BF16. The allocation rule matches measured pool counts within 0.1% and predicts approximately 1.88×1.88\times NVFP4 capacity on the Nemotrons. We exclude fixed-batch timings that require multiple admission waves.

The sparse curves in Figure 11 vary the gather budget while retaining the dense scan. At BF16-fitting batches, N=128N = 128 is at most 12% slower than N=32N = 32 on the Qwens; the difference reaches 19% at Qwen3.8’s 32K×64 cell beyond

Figure 11. Block-budget sweep on one GB300, with 4,096 output tokens and two replicates. TPOT speedup is relative to dense BF16 at its largest fitting batch: Qwen3.8 uses 32K×32 and 252K×8; Qwen3.6 uses 32K×64 and 252K×16. Circles and solid lines denote Qwen3.8; squares and dashed lines denote Qwen3.6. Jade marks sparse budgets; brick marks dense NVFP4 at “all.” BF16 capacity. The scan and non-attention layers limit the benefit of reducing the gather alone.

Table 7. Measured KV-pool capacity and performance on the Qwens. Pool is the token capacity relative to dense BF16. TPOT columns use BF16’s largest batch at each context: 32/32/16/8 on Qwen3.8 and 64/64/32/16 on Qwen3.6. Output-rate columns divide each method’s best throughput, including prefill, by BF16’s best over the tested batches. All cells generate 4,096 tokens; n/f means the batch does not fit the measured pool. The Qwen3.8 FP8-KV 64K×64 cell is excluded after a request suspension; its cells beyond BF16 capacity run with idle neighboring GPUs. Jet-QSparse and baseline ratios come from separate campaigns.

Table 8. Long-generation speedups: 1,024-token prompts, 16,384 outputs, and batch 64. Top: TPOT relative to dense BF16; two replicates, with baselines and Jet-QSparse in separate campaigns. *Eviction kernels compiled before timing; two same-node GPU pairs with iteration statistics enabled. †Rebased on other runs’ BF16 reference, with approximately 5% uncertainty. Bottom: one attention layer at 8.5K×64 on random data, relative to FlashInfer BF16; Jet-QSparse kernels omit the accuracy fixes.

Long Generation and Kernel Scope

TriAttention first evicts after 4,224 generated tokens. With a 1K prompt, its cache cycles between 5,120 and 5,248 tokens while dense attention reaches 17,408. All 64 requests fit the BF16 pool in Table 8, so this experiment measures shorter reads, not additional admitted requests. At 4,096 output tokens in the batch sweep, no eviction occurs. Compiling the eviction kernels before timing avoids charging a one-time compilation to decode. The Qwen measurements use this warm-up; the Nemotron cells retain the standard timing protocol. Same-method cross-campaign ratios differ by up to 0.05× in the Qwen3.8 long-generation cell, so small gaps across campaigns are descriptive.

Q3.8Q3.6NanoLightning
End to end
Quest0.690.810.860.91
SparQ1.020.970.970.93
TriAttention1.16*1.06*1.010.96
Dense NVFP41.141.071.00†0.99
Jet-QSparse N=64N = 641.081.010.98†0.97
Conditional limit1.431.191.081.06
One attention layer
Quest2.041.891.31
SparQ1.160.880.74
TriAttention1.541.481.40
Dense NVFP41.661.490.94
Jet-QSparse N=64N = 641.181.000.55

Table 8.

Short-prompt KV memory. Figure 1 also measures memory with eight requests, 4K prompts, and 16K outputs on one GB300 per method. All requests finish without pauses. This single-run check sets maximum batch size to 8, sequence length to 20,480, and prefill token budget to 8,192. We convert measured peak used pages and page sizes to occupied bytes, including layout metadata, and add the two BF16 pages for each active Jet-QSparse request. Ratios use the matching engine’s BF16 control; both dense controls have identical attention-memory values. Occupied bytes exclude recurrent states, unused reservations, and the FP8 prefill arena. The 128K/4K comparison instead uses the two-run sweep with configured maxima of 64 requests and 262,144 tokens for sequence length and the prefill token budget.

The conditional limits in Table 8 use 1/(1−a)1/(1-a), assuming non-attention time is unchanged, with estimated attention shares of 30%, 16%, 7%, and 6%. These estimates scale the 8.5K single-layer times by the number of attention layers; they approximate a generation whose context changes throughout decoding. At these short contexts, N=64N = 64 gathers a traffic-weighted block fraction of 0.72, so the read model in § 6.4 predicts 12+0.72=1.22\frac{1}{2} + 0.72 = 1.22 times the dense NVFP4 K/V traffic.

Single-layer speed does not fully predict serving speed. Quest’s layer times, with its first two attention layers kept dense, imply 1.01–1.15× end-to-end speedup under an unchanged non-attention remainder, but measured speed is 0.69–0.91×. We did not decompose this gap, which limits algorithm-level conclusions from its serving ratios.

The 2.4T Decode Step and Memory Use

Table 9 attributes 15–21% of the BF16 step to attention and shows how N=32N = 32 reduces that component. This profiling study uses separate runs from the deployment measurements in Table 5. At 32K×\times23, sampled traced kernels additionally attribute 20.0 ms to routed experts, 18.7 ms to MoE communication including waits, and 14.9 ms to BF16 GEMMs. These sampled-rank timings do not give an untraced whole-system decomposition. The non-attention remainder also varies between arms, so 1/(1−a)1/(1-a) is a conditional limit rather than an unconditional bound on speedup.

CellBF16 stepBF16Attention N=32N = 32aaLimitspeedup
32K×\times2369.610.24.014.7%1.171.10
128K×\times658.310.52.818.0%1.221.20
250K×\times347.09.82.720.9%1.261.13

Table 9. 2.4T systems study on 32 GPUs, using N=32N = 32. Step times and speedups use TPOT estimates with traced windows replaced by untraced means (two allocations). Attention times are rank-0 kernel attributions; the BF16 attention share aa is inferred because profiling overhead exceeds its tolerance. The limit 1/(1−a)1/(1-a) assumes unchanged non-attention time. Times are milliseconds.

Allocated memory and live demand. With the pool using 85% of free memory, measured NVML plateaus are 265.03 GiB per GPU for dense BF16 and 262.74 GiB for N=32N = 32; reservation keeps these footprints nearly constant across loads. Reducing the NVFP4 pool fraction to 0.45 to approximate BF16’s token capacity lowers its plateau to 225.40 GiB. Against the corresponding BF16 reference mean of 266.85 GiB, this frees 41.45 GiB per GPU, or approximately 1,327 GiB per 32-GPU copy. Propagating the bimodal BF16 reference’s spread gives an SD of 3.38 GiB per GPU, despite agreement between the two NVFP4 replicates. Normalizing to exactly 835,584 tokens gives a calculated 42.76 GiB saving. Associated TPOT changes are unresolved at MDD80_{80} values of 1.29–5.37 ms.

A replay of 1,113 Terminal-Bench calls measures demand under an agentic arrival pattern. It reproduces token counts and timing, not task content, and uses two paired measurements. Subtracting unused reserved KV space from measured allocation gives peak memory needs of 200.97 GiB for BF16 and 205.31 GiB for N=32N = 32, a derived difference of +4.34±0.18+4.34 \pm0.18 GiB. These are accounting estimates, not NVML footprints. Peak live KV averages about 105K/94K tokens across the two pairs, well below either pool. Hypothetically shrinking the FP8 arena to its observed peak would change the difference to −4.75/−4.49-4.75/-4.49 GiB in the two pairs; we did not measure that allocation policy. We amended the accounting filters after a diagnostic measurement. In these replayed calls, decode accounts for 97.8% of client model time, with 68.7–69.3% of decode time in steps that also prefill another request.

Other execution choices. FP8 non-expert weights reduce the 32K×\times23 step by 3.3–3.5 ms. The reduction is resolved for N=32N = 32 (MDD80_{80} 1.3 ms), but not for dense BF16 (4.2 ms); the sparse-to-dense speedup change remains within MDD80_{80} 0.07–0.14. Attention TP16 on four nodes achieves 8.9–9.7 ms for one request and 250/319 output tokens per second per node for dense BF16/NVFP4 at 128K×\times30. We did not evaluate fixed-budget speculation on this layout because each KV group has four query heads instead of sixteen. A low-MM GEMM dispatcher saves 1.5–2.4 ms in isolated step measurements. We evaluated these weight and execution variants for performance only; they carry no accuracy claim.

References

  1. [1]DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models. Technical report, DeepSeek-AI, 2025. URL https://arxiv.org/abs/2512.02556.
  2. [2]Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghavi, S. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.298. URL https://aclanthology.org/2023.emnlp-main.298/.DOI
  3. [3]Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems, volume 37, pp. 100213–100240. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/b5b939436789f76108b90da5e81af7c–Abstract-Conference.html.arxiv.org/abs/2404.00456
  4. [4]Behnam, P., Fu, Y., Zhao, R., Tsai, P.-A., Yu, Z., and Tumanov, A. RocketKV: Accelerating long-context LLM inference via two-stage KV cache compression. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 3358–3392. PMLR, 2025. URL https://proceedings.mlr.press/v267/behnam25a.html.arxiv.org/abs/2502.14051
  5. [5]Benjamini, Y. and Hochberg, Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995. doi: 10.1111/j.2517-6161.1995.tb02031.x.DOI
  6. [6]Benjamini, Y. and Yekutieli, D. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4):1165–1188, 2001. doi: 10.1214/aos/1013699998.DOI
  7. [7]Cecka, C. CuTe layout representation and algebra. arXiv preprint arXiv:2603.02298, 2026. URL https://arxiv.org/abs/2603.02298.
  8. [8]Dao, T. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2307.08691.
  9. [11]Gao, Y., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H. K.-H., Cao, T., Yang, F., and Yang, M. SeerAttention: Self-distilled attention gating for efficient long-context prefilling. In Advances in Neural Information Processing Systems, volume 38, pp. 55846–55869. Curran Associates, Inc., 2025. doi: 10.52202/085713-1869.DOI
  10. [12]Gao, Y., Guo, S., Cao, S., Xia, Y., Cheng, Y., Wang, L., Ma, L., Sun, Y., Ye, T., Dong, L., So, H. K.-H., Hua, Y., Cao, T., Yang, F., and Yang, M. Sparse attention adaptation for long reasoning. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=c5BOCHM6J8.arxiv.org/abs/2506.08889
  11. [14]Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, volume 37, pp. 1270–1303. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/028fccf85435d39a40c4d61b42c99a4–Abstract-Conference.html.arxiv.org/abs/2401.18079
  12. [15]Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https://arxiv.org/abs/2404.06654.
  13. [16]Hu, R. Hardware-aware FP4 FlashAttention-4. arXiv preprint arXiv:2609.04105, 2026. URL https://arxiv.org/abs/2609.04105.
  14. [17]Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S. I., Solar-Lezama, A., Sen, K., and Stoica, I. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/94074d5a072d28f f75a76dabed43767-Abstract-Conference.html.
  15. [18]Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems, volume 37, pp. 52481–52515. Curran Associates, Inc., 2024.arxiv.org/abs/2407.02490
  16. [19]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narsimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e30c629acfe6cbe9ca88fb84-Abstract-Conference.html.arxiv.org/abs/2310.06770
  17. [20]Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165. PMLR, 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html.arxiv.org/abs/2006.16236
  18. [21]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626. ACM, 2023. doi: 10.1145/3600006.3613165.DOI
  19. [22]Lai, X., Lu, J., Luo, Y., Ma, Y., and Zhou, X. FlexPrefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OfjllbeIrT.arxiv.org/abs/2502.20766
  20. [24]Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.DOI
  21. [26]Mao, W., Lin, X., Huang, W., Xie, Y., Fu, T., Zhuang, B., Han, S., and Chen, Y. TriAttention: Efficient long reasoning with trigonometric KV compression. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pp. 86458–86477. PMLR, 2026. URL https://proceedings.mlr.press/v306/mao26e.html.arxiv.org/abs/2604.04921
  22. [27]Mathematical Association of America. American Invitational Mathematics Examination (AIME) 2025. Competition problems, 2025. URL https://web.archive.org/web/20250316185218/https://maa.org/maa-invitational-competitions/. AIME I and AIME II.
  23. [28]NVIDIA. Nemotron 3 Nano: Open, efficient mixture-of-experts hybrid Mamba-Transformer model for agentic reasoning. Technical report, NVIDIA, December 2025a. URL https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Nano-Technical-Report.pdf.arxiv.org/abs/2512.20848
  24. [29]NVIDIA. Pretraining large language models with NVFP4. arXiv preprint arXiv:2509.25149, 2025b. URL https://arxiv.org/abs/2509.25149.
  25. [30]NVIDIA. Nemotron 3.5 Super and Lightning: Open, efficient mixture-of-experts hybrid Mamba-Transformer models for agentic AI. Technical report, NVIDIA, October 2026a. URL https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf.arxiv.org/abs/2610.38095
  26. [32]NVIDIA. Quantization in TensorRT-LLM. TensorRT-LLM documentation, 2026d. URL https://nvidia.github.io/TensorRT-LLM/latest/features/quantization.html. Accessed 2026-10-13.
  27. [33]OpenAI. Introducing SWE-bench Verified. OpenAI blog, August 2024. URL https://openai.com/index/introducing-swe-bench-verified/.
  28. [34]Qwen Team. Qwen3-Next-80B-A3B-Instruct. Hugging Face model card, September 2025. URL https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct. Accessed 2026-10-16.
  29. [35]Qwen Team. Qwen3.6-35B-A3B. Hugging Face model card, April 2026a. URL https://huggingface.co/Qwen/Qwen3.6-35B-A3B. Accessed 2026-10-16.
  30. [37]Qwen Team. Qwen3.8-2.4T-A95B-FP8. Hugging Face model card, August 2026c. URL https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8. Accessed 2026-10-24.
  31. [38]Ramachandran, A., Neseem, M., Sakr, C., Venkatesan, R., Khailany, B., and Krishna, T. ThinKV: Thought-adaptive KV cache compression for efficient reasoning models. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=M3CeHnZKNC.arxiv.org/abs/2510.01290
  32. [39]Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98.arxiv.org/abs/2311.12022
  33. [40]Ribar, L., Chelombiev, I., Hudlass-Galley, L., Blake, C., Luschi, C., and Orr, D. SparQ attention: Bandwidth-efficient LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42558–42583. PMLR, 2024. URL https://proceedings.mlr.press/v235/ribar24a.html.arxiv.org/abs/2312.04985
  34. [41]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems, volume 37, pp. 68658–68685. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/7ede97c3e082c6df10a8d6103a2eebd2-Abstract-Conference.html.arxiv.org/abs/2407.08608
  35. [42]Singhania, P., Singh, S., He, S., Feizi, S., and Bhatele, A. Loki: Low-rank keys for efficient sparse attention. In Advances in Neural Information Processing Systems, volume 37, pp. 16692–16723. Curran Associates, Inc., 2024. doi: 10.52202/079017-0532. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/1e027da6bec9c9eb2ec37951ceecae93-Abstract-Conference.html.DOI
  36. [43]Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasicki, B., and Han, S. Quest: Query-aware sparsity for efficient long-context LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47901–47911. PMLR, 2024. URL https://proceedings.mlr.press/v235/tang241.html.arxiv.org/abs/2406.10774
  37. [44]The Terminal-Bench Team. Terminal-Bench 2.1. Benchmark release, May 2026. URL https://www.tbench.ai/news/terminal-bench-2-1.
  38. [45]Tillet, P., Kung, H. T., and Cox, D. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19. ACM, 2019. doi: 10.1145/3315508.3329973.DOI
  39. [46]Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2309.17453.
  40. [47]Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=cFu7ze7xUm.arxiv.org/abs/2410.10819

Paper details

vlnc:W_pta77t4g
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code, ChatGPT (website)

View statement
Looking for related papers…
Contents