WorksComputer Science
NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent Attention
Abstract
Long-context reasoning and agentic workloads make key-value (KV) cache capacity and attention computation the dominant costs of serving frontier large language models. We present NVFP4-MLA, the first design to push both KV-cache storage and attention computation of multi-head latent attention (MLA) to FP4. NVFP4-MLA co-designs the quantization algorithm with the attention kernel and the cache system. Selective residual compensation preserves model accuracy by adding NVFP4 residuals only to the quantization-sensitive channels. Two-dimensional quantization blocks make the shared latent cache compatible with the different reduction directions of the two attention batched matrix multiplications (BMMs), removing requantization between them. On DeepSeek-R1 and Kimi-K3, NVFP4-MLA matches BF16 MLA average accuracy across nine benchmarks while compressing the MLA KV cache by 3.20x and 3.05x relative to BF16 MLA. On NVIDIA Rubin GPUs, NVFP4-MLA speeds up the attention kernel over FP8 MLA by up to 2.82x on DeepSeek-R1 and 2.71x on Kimi-K3, and lowers p90 time to first token in agentic multi-turn serving by up to 3.5x and 2.1x, respectively. In disaggregated DeepSeek-R1 serving, it also raises per-GPU throughput by up to 2.30x.
Introduction
Multi-head latent attention (MLA) represents keys and values in a shared low-rank latent space and is the common attention pattern of several popular open-source large language models, including DeepSeek-V3 ([3]), DeepSeek-R1 ([5]), Kimi-K2.6, Kimi-K3 ((Kimi Team, 2026)), and LongCat-Flash ((Meituan LongCat Team, 2025)), which use MLA in all or part of their attention layers. The low-rank latent shrinks the key-value (KV) cache, but the cache still grows with sequence length and serving concurrency, and every decoding step reads all of it, which puts pressure on both memory capacity and bandwidth. Agentic workloads sustain this pressure across turns and tool calls because they preserve and reuse growing contexts, which keeps the KV cache resident for longer. When GPU memory cannot hold these caches, the server evicts them and recomputes the prefix in the next turn, which increases time to first token. A smaller cache allows longer contexts, keeps more tokens and requests in GPU memory, reduces cache eviction and recomputation, and increases serving concurrency.
Beyond memory, computation can also be under pressure. In grouped-query attention (GQA), each KV head serves only the few query heads in its group, whereas all MLA heads share the latent cache during decoding, so MLA performs more computation per cached byte (Appendix § 8). Model-native multi-token prediction (MTP) verifies several queries against the same latent cache. Reusing the cache across these queries raises arithmetic intensity further and makes MLA decoding compute-bound (§ 4.4.2), so math throughput matters as much as cache bytes. Together, these pressures motivate us to apply NVFP4, a 4-bit format that recent NVIDIA Tensor Cores support natively, to both KV-cache storage and attention computation in MLA, so that MLA benefits from smaller storage and higher math throughput.
We introduce NVFP4-MLA, which brings unified NVFP4 storage and attention computation to MLA. Its design addresses the challenges of applying FP4 quantization directly to MLA. First, direct FP4 quantization degrades accuracy. Outliers in the MLA cache concentrate in the last few channels, so selective residual compensation adds an NVFP4 residual only to these channels and keeps all elements in FP4 (§ 4.2.2). Second, MLA shares one latent representation between keys and values. Default block-wise quantization groups this latent along channels, which matches the channel reduction of the first batched matrix multiplication (BMM1) but not the token reduction of BMM2. NVFP4-MLA introduces two-dimensional quantization blocks that are compatible with both reduction dimensions and therefore remove the need to requantize the shared latent cache for the second BMM (§ 3.3). Third, the higher arithmetic intensity makes math throughput a bottleneck. NVFP4-MLA therefore also quantizes queries and softmax scores online, so that both BMMs run on native FP4 Tensor Cores and exploit the higher FP4 math throughput. A corresponding cache system and kernel pipeline implement this design, and Figure 1 gives an overview.
Figure 1. Overview of NVFP4-MLA. (a) NVFP4 MLA cache: two-dimensional NoPE blocks shared by both BMMs, and RoPE with an NVFP4 residual. (b) NVFP4 MLA kernel: BMM1, online softmax fused with score quantization, and BMM2, with score construction and accumulator rescaling split across warp groups (WG). (c) Dataflow: score-operand reuse and split KV. (d) End-to-end integration in disaggregated serving: prefill, KV cache transfer, decode, and speculative decoding.
Our main contributions are:
We design selective residual compensation, which keeps model accuracy by adding NVFP4 residuals to quantization-sensitive channels without mixed precision.
We design two-dimensional quantization blocks and a corresponding cache system that let the shared quantized representation serve both BMMs without requantization.
We build an efficient kernel pipeline. The pipeline quantizes queries and softmax scores online, so both BMMs benefit from the higher throughput of FP4 Tensor Cores.
We evaluate NVFP4-MLA on two frontier open MLA models, DeepSeek-R1 and Kimi-K3, across nine benchmarks that span reasoning, coding, instruction following, long context, and agentic tool use. NVFP4-MLA matches BF16 MLA average accuracy on both models while reducing the MLA KV cache by and relative to BF16 MLA. On NVIDIA Rubin GPUs, NVFP4-MLA speeds up the attention kernel over FP8 MLA by up to on DeepSeek-R1 and on Kimi-K3. In the AgentX benchmark, the smaller cache lowers p90 time to first token by up to and , respectively. In disaggregated DeepSeek-R1 serving on the MLPerf and 8K/16K Reasoning workloads, per-GPU throughput reaches up to that of FP8 MLA, and is and higher on average, respectively.
§ 2 reviews MLA and the NVFP4 format. § 3 presents selective residual compensation, two-dimensional quantization blocks, and the corresponding cache system and kernel pipeline. § 4 evaluates the accuracy, memory footprint, and performance of NVFP4-MLA on DeepSeek-R1 and Kimi-K3. § 5 compares NVFP4-MLA with prior KV-cache quantization methods and quantized attention kernels. § 6 summarizes the findings.
Background
Multi-head latent attention
Low-rank compression creates shared KV state
MLA stores one compressed latent vector for each token [2 b]. For a hidden state , a down-projection produces
Learned up-projections recover head-specific keys and values from . During inference, matrix absorption folds these projections into the query and output projections. The runtime therefore caches instead of materializing a separate key and value for every head.
The shared latent is involved in both BMMs. The first, BMM1 (QK), reduces over latent channels to generate attention logits, and the second, BMM2 (PV), reduces over cached tokens to generate the output. The same cached values are therefore reduced along two directions, so their block scales must serve both BMMs.
As Appendix § 8 shows, MLA performs more computation per cached byte than a grouped-query attention (GQA) reference. Matrix absorption reduces cache traffic and reuses the latent cache for every query head. Model-native MTP raises arithmetic intensity further when the kernel reuses the cache across several query positions. MLA decoding therefore tends to be compute-bound.
NoPE and RoPE channels serve different roles
2.1.2. NoPE and RoPE channels serve different roles: Applying rotary position encoding (RoPE) [21] directly to the compressed keys would break matrix absorption, because the rotation depends on each token’s position [2]. MLA therefore carries RoPE on a decoupled path, with additional per-head queries and a single key shared by all heads. Each cached token thus holds a -channel compressed latent, which we call NoPE, and a -channel decoupled key, which we call RoPE. In BMM1, each head adds the NoPE and RoPE dot products to form its attention logit, whereas BMM2 aggregates only the NoPE latent. Kimi-K3 keeps this decoupled layout but applies no positional encoding to either path (Kimi Team, 2026); we still call its decoupled channels RoPE.
NVFP4 format and KV-cache quantization
KV-cache quantization reduces per-token storage by encoding cached values with fewer bits [13]. To keep accuracy, prior methods redistribute large outliers through rotations, preserve selected outliers separately, or assign different bit widths across tensors and layers [6, 11, 22, 26]. Beyond accuracy, using the quantized cache directly in low-precision BMMs requires a compatible hardware format. We therefore use block-scaled NVFP4 for both cache storage and attention computation.
NVFP4 and MXFP4 both encode values in 4-bit E2M1 and attach a shared scale to each block [20, 15]. NVFP4 uses an E4M3 scale per 16 values, whereas MXFP4 uses an E8M0 power-of-two scale per 32 values. NVFP4’s smaller blocks and non-power-of-two scales give finer control over local value ranges.
These formats have mostly been studied for weight and activation quantization [4]. For KV caches, their use also depends on how attention consumes the quantized values. When compute precision differs from storage precision, attention needs an extra dequantization step. Computing directly on the stored values avoids this step, but each BMM then requires the quantized values and scales in a compatible layout. This requirement links the block layout to the reduction axes of the attention BMMs. In GQA, separate key and value caches can adopt independent block layouts. In MLA, by contrast, both BMMs reduce the same latent cache along different axes, which couples their block layouts. A shared MLA cache intended for native FP4 computation therefore requires quantization blocks that support both reduction directions.
Methodology
NVFP4-MLA makes the shared latent a direct operand of both FP4 attention BMMs. Selective residual compensation reduces quantization error without mixed precision, two-dimensional quantization blocks align quantization with both reduction axes, the efficient kernel pipeline handles online quantization, and the corresponding cache system supports end-to-end integration.
Quantization of Q, K, P, and V
During decode in quantized layers, NVFP4-MLA quantizes all four attention operands: queries , keys , softmax scores , and values . holds stable, unnormalized softmax scores, which NVFP4-MLA quantizes online for BMM2.
denotes NVFP4 quantization with contiguous 16-element blocks along the reduction axis of the consuming BMM: latent channels for queries and the RoPE cache, and cached tokens for softmax scores. uses token-by-channel blocks, each with a single scale (Appendix § 7).
For one query head and cached tokens, let and . Let denote attention logits and stable, unnormalized softmax scores, respectively; is the single-head instance of . The normalized output is , and is the pre-absorption QK head width. NoPE uses because both BMMs consume it, whereas RoPE enters only BMM1 and uses along its channels. The quantized attention dataflow, evaluated as online softmax over successive KV tiles , is
Here is the -th KV tile and , with and ; , , and without a superscript are the running values, and is taken after the last tile. Both BMMs accumulate in FP32, and softmax keeps , , , and the final division by in FP32.
Selective residual compensation
FP4 quantization error in the MLA cache is disproportionately large in the RoPE partition (§ 4.2.2). NVFP4-MLA therefore keeps a single NVFP4 component for NoPE but adds a second NVFP4 component only for the RoPE quantization error. Because the RoPE partition holds 64 of the 576 latent channels, the residual adds little storage and computation; § 4.2 evaluates the accuracy of NVFP4-MLA.
The native kernel packs RoPE and its quantization error separately with independent NVFP4 block scales, logically concatenates them along the reduction dimension of BMM1, and duplicates the corresponding channels. The compensated logits, which replace in Equation (2), therefore take a single BMM1 reduction, without merging base and residual into a high-precision tensor.
Two-dimensional quantization blocks for shared keys and values
As § 2 shows, the shared MLA latent requires scales that work in both reduction directions. A conventional one-dimensional layout aligns its blocks with BMM1 but not with native NVFP4 BMM2 (Figure 2a). Converting this layout requires dequantizing the cached E2M1 values, transposing them, and requantizing them into BMM2-aligned blocks. Requantization introduces additional quantization error. It also adds overhead such as memory traffic, a scale reduction, and synchronization before BMM2.
Figure 2. Two-dimensional quantization blocks. Both BMMs consume the same FP4 values and scales without requantization. One-dimensional 1 × 16 blocks (a) serve only BMM1, whereas 16 × 16 blocks (b) serve both. NVFP4-MLA uses (b) for NoPE and (a) for RoPE, which only BMM1 consumes.
NVFP4-MLA therefore quantizes the shared NoPE latent in two-dimensional blocks over tokens and channels (Figure 2b). One scale covers every 16-element row and column segment of a block, so the block is valid in either reduction direction: both BMMs share the same NVFP4 values and scales without a high-precision intermediate or a requantization pass.
Kernel implementation
Kernel implementation
To turn FP4’s higher Tensor Core throughput into a faster attention kernel, NVFP4-MLA quantizes queries and softmax scores online (Figure 1b) and applies four optimizations: fusing online softmax with FP4 score quantization, balancing FP4 score construction and accumulator rescaling, reusing FP4 score operands across BMM2 output tiles, and shape-adaptive thread-block scheduling.
Fusing online softmax with FP4 score quantization. Online FP4 score quantization makes the logit-to-score path heavier than in FP8 MLA because the path must also derive block scales and pack the FP4 operand for BMM2. We algebraically fuse online softmax, FP4 score quantization, and scale generation. As in online softmax, let be the running maximum of one row of logits , and let , so that in Equation (2). Let ; for each 16-element block and ,
where and round to E2M1 and E4M3, and are the largest E2M1 and finite E4M3 magnitudes (Appendix § 7), is a constant tensor scale, and is quantized against the unrounded block scale. Because , the two arguments lie in and , so neither rounding saturates. The E2M1 value depends on and but not on , so the kernel evaluates the exponent with a fused multiply-add, converts the result to E2M1, and packs it before the cross-block reduction produces . Once is available, only the block scale remains (Appendix § 9).
Balancing FP4 score construction and accumulator rescaling. Fusion shortens the logit-to-score path, but this stage still carries more non-matrix work than its FP8 MLA counterpart. Rather than dedicate separate warp groups to FP4 score construction and online-softmax accumulator rescaling, we partition each logit tile across the warp groups. Each group constructs the FP4 score operand for its logit partition and also rescales a complementary partition of the resident BMM2 accumulator when the running maximum changes. This mixed assignment keeps the logit-to-score stage from serializing the pipeline.
Reusing FP4 score operands across BMM2 output tiles. Each packed FP4 score slice contributes to multiple BMM2 output tiles. The kernel uses the Tensor Core operand-reuse mechanism to retain this operand of BMM2 across these tiles instead of reloading it for each tile, so more execution bandwidth remains for FP4 score construction.
Shape-adaptive CTA scheduling. With MTP, each request has more queries, so the kernel can use larger matrix multiply-accumulate (MMA) tiles. It gives each cluster of thread blocks (cooperative thread arrays, CTAs) the packed query rows of one request; the CTAs of a cluster split the KV tiles, exchange the FP4 scores through distributed shared memory, and use the same FP4 operands and softmax. At long KV lengths, the kernel also splits the KV of a request into pieces that run on idle streaming multiprocessors (Figure 1c) and merges them with a reduction kernel.
End-to-end integration
For end-to-end integration, we adapt the cache system to be compatible with two-dimensional quantization blocks, the NVFP4 attention kernel, paged KV caches, and speculative decoding (Figure 1d). During prefill, a fused quantize-and-scatter kernel writes completed NVFP4 blocks into paged storage, which decode reads without requantization. Disaggregated serving transfers only these pages, namely the E2M1 values and their scales, plus the BF16 tail of the incomplete last block. During decode, a fused pre-attention kernel applies RoPE where the model uses it, updates a 16-token BF16 ring, quantizes the operands, and writes the cache. Each request owns a stable slot in the ring, and the runtime refreshes the active NVFP4 block from BF16 values and finalizes the block once it fills. It therefore never has to dequantize a block, append newly generated tokens, and requantize it, which would introduce additional quantization error. For speculative decoding such as MTP, the runtime snapshots the BF16 ring before speculative writes, commits accepted candidates, and restores locations overwritten by rejected candidates. Completed NVFP4 pages remain unchanged during rollback, so rejected tokens cannot contaminate persistent state. NVFP4-MLA has been integrated into TensorRT-LLM ([16]).
Evaluation
We evaluate NVFP4-MLA on DeepSeek-R1 and Kimi-K3 in three parts: accuracy (§ 4.2), KV-cache memory reduction (§ 4.3), and performance (§ 4.4).
Experimental setup
Accuracy. Both models use the same nine tasks, spanning reasoning, coding, instruction following, long context, and agentic tool use (Table 1). RULER and MRCR use a 128K-token context. We compare NVFP4-MLA with BF16 MLA, FP8 MLA, and vanilla NVFP4 MLA. BF16 MLA keeps the KV cache and attention BMM computation in BF16. FP8 MLA quantizes the KV cache and attention BMM computation to E4M3. Vanilla NVFP4 MLA quantizes them directly to NVFP4 without residual compensation.
| Configuration | GPQA | IFBench | AIME 2025 | LiveCode Bench v6 | SciCode | RULER | MRCR | AA-LCR | -Bench | Avg |
| DeepSeek-R1 | ||||||||||
| BF16 MLA | 80.07 | 39.53 | 87.14 | 71.31 | 42.60 | 92.97 | 36.35 | 52.52 | 35.75 | 59.80 |
| FP8 MLA | 80.16 | 37.93 | 87.60 | 71.75 | 42.27 | 92.67 | 37.21 | 52.62 | 35.42 | 59.74 |
| Vanilla NVFP4 MLA | 77.38 | 34.87 | 73.80 | 62.11 | 40.64 | 90.52 | 29.54 | 46.06 | 31.91 | 54.09 |
| NVFP4-MLA | 79.83 | 38.07 | 85.94 | 71.78 | 43.12 | 92.77 | 36.95 | 51.86 | 38.05 | 59.82 |
| Kimi-K3 | ||||||||||
| BF16 MLA | 93.66 | 73.96 | 96.20 | 57.49 | 58.48 | 97.00 | 85.02 | 81.75 | 78.84 | 80.27 |
| FP8 MLA | 93.50 | 73.69 | 95.63 | 57.32 | 58.19 | 97.06 | 84.38 | 82.19 | 77.49 | 79.94 |
| Vanilla NVFP4 MLA | 88.26 | 61.79 | 87.71 | 10.68 | 2.27 | 95.98 | 75.21 | 77.31 | 74.89 | 63.79 |
| NVFP4-MLA | 93.43 | 74.91 | 96.20 | 57.30 | 58.09 | 97.11 | 84.10 | 82.31 | 79.06 | 80.28 |
Table 1. Downstream accuracy on DeepSeek-R1 and Kimi-K3. NVFP4-MLA matches BF16 MLA average accuracy and greatly reduces the accuracy drop of vanilla NVFP4 MLA. Scores in %; Avg: nine-task mean. Repeats per task: 64 for GPQA [19], AIME 2025, and AA-LCR; 8 for LiveCodeBench [8], SciCode [23], and -Bench [1]; 5 for IFBench [17]; 3 for MRCR [24]; 1 for RULER [7].
Storage. We count the stored MLA cache. DeepSeek-R1 uses NVFP4 in all 61 MLA layers. Kimi-K3 uses NVFP4 in 22 of its 24 MLA layers and FP8 in the final two; the count excludes the recurrent state of its 69 Kimi Delta Attention (KDA) layers.
Performance. All performance experiments run on NVIDIA Rubin GPUs, and serving runs use TensorRT-LLM with the same runtime and the same set of parallel layouts for both precisions. On DeepSeek-R1 with MTP=3, MLPerf uses the evaluation prompts of the MLPerf Inference [18] DeepSeek-R1 benchmark, and 8K/16K Reasoning uses requests of about 8K input and 16K output tokens. Both workloads use disaggregated serving, with generation under attention data and expert parallelism (DEP) or attention tensor and expert parallelism (TEP). AgentX replays 393 recorded multi-turn Claude Code sessions from the SemiAnalysis InferenceX benchmark on both models, using distributed weight data parallelism (DWDP) [10] for Kimi-K3. Attention kernel measurements cover 39 shapes per model, at KV lengths of 2K–128K and per-GPU batches of 8–1,024, with MTP=3 on DeepSeek-R1 and MTP=7 on Kimi-K3.
Accuracy
NVFP4-MLA matches BF16 MLA average accuracy on both models (§ 4.2.1), and white-box analysis of the cache shows that direct FP4 quantization loses precision on significant outliers in the RoPE channels, which the NVFP4 residual encodes more accurately than FP8 (§ 4.2.2).
Downstream accuracy
With NVFP4 attention BMM computation, NVFP4-MLA matches BF16 MLA average accuracy on both models (59.82 vs. 59.80 and 80.28 vs. 80.27; Table 1), compared with −0.06 and −0.33 percentage points (pp) for FP8 MLA.
Quantization error analysis
Measurement. To locate FP4 error in the MLA cache, we analyze dumped BF16 cache tensors across layers of DeepSeek-R1 and MLA layers of Kimi-K3. We measure reconstruction error as the normalized mean squared error NMSE = . NoPE stays at NVFP4, and only the RoPE encoding varies: NVFP4, FP8 E4M3, or NVFP4 plus residual.
Error concentrates in RoPE. RoPE holds 64 of the 576 channels, yet compensating it alone lowers mean whole-cache NMSE by 60.3% on DeepSeek-R1 and 66.1% on Kimi-K3 (Figure 3a). Figure 4 contrasts the two partitions in the first few MLA layers of both models: the RoPE channels peak at 3.9–125× the NoPE maximum and vary sharply across tokens, whereas the NoPE latent stays flat.
Figure 3. Cache quantization error. A RoPE-only NVFP4 residual lowers whole-cache NMSE and is more accurate than FP8 RoPE in every layer. NoPE uses NVFP4; RoPE uses NVFP4, FP8 E4M3, or NVFP4 plus residual. (a) Whole-cache NMSE averaged over MLA layers. (b) Per-layer NMSE of each RoPE encoding (log scale).
Figure 4. RoPE and NoPE magnitudes. RoPE channels peak at 3.9–125× the NoPE maximum.
NVFP4 residual versus FP8 RoPE. The NVFP4 residual encodes RoPE more accurately than FP8 E4M3. Relative to plain NVFP4, it reduces mean RoPE NMSE by factors of 115.4 on DeepSeek-R1 and 116.1 on Kimi-K3, whereas FP8 RoPE leaves 9.5× and 9.1× as much error as the residual on average and 7.9–11.2× and 8.7–9.3× as much in every layer (Figure 3b).
Takeaway: NVFP4-MLA matches BF16 MLA average accuracy on both models while running both attention BMMs in NVFP4, whereas vanilla NVFP4 MLA drops by 5.71 and 16.48 pp. White-box analysis shows that this drop stems from significant outliers in the RoPE channels. Selective residual compensation therefore mitigates the accuracy drop with a small overhead (§ 4.3).
KV-cache memory reduction
Including the residual, NVFP4-MLA compresses the stored MLA cache by 3.20× and 3.05× relative to BF16 MLA and by 1.60× and 1.52× relative to FP8 MLA on DeepSeek-R1 and Kimi-K3 (Table 2). The RoPE residual accounts for only 10% of each NVFP4 layer’s cache.
| Cache representation | DeepSeek-R1 | Kimi-K3 | ||
| Bytes/token | Reduction | Bytes/token | Reduction | |
| BF16 MLA | 70,272 | 1.00× | 27,648 | 1.00× |
| FP8 MLA | 35,136 | 2.00× | 13,824 | 2.00× |
| NVFP4-MLA | 21,960 | 3.20× | 9,072 | 3.05× |
Table 2. Stored MLA cache size. NVFP4-MLA compresses the stored MLA cache by 3.20× and 3.05× relative to BF16 MLA and by 1.60× and 1.52× relative to FP8 MLA. Bytes per cached token, summed over all MLA layers.
On Kimi-K3, the MLA cache shares the KV budget with the recurrent state of the KDA layers; under the same budget, NVFP4-MLA holds 47–49% more cached tokens than FP8 MLA, depending on the share taken by KDA state. § 4.4.1 shows how this capacity turns into prefix reuse.
Performance
We evaluate performance at three levels. In agentic serving, the smaller cache of NVFP4-MLA keeps more session histories and lowers p90 time to first token by up to 3.5× on DeepSeek-R1 and by 2.1× on Kimi-K3 (§ 4.4.1). Kernel measurements show an attention kernel that runs up to 2.82× and 2.71× faster than FP8 MLA on DeepSeek-R1 and Kimi-K3 (§ 4.4.2). In disaggregated DeepSeek-R1 serving, the faster kernel raises per-GPU throughput by up to 2.30× (§ 4.4.3).
Prefix-cache reuse
With its smaller cache (§ 4.3), NVFP4-MLA delays cache eviction as the number of concurrent sessions grows. Agentic sessions benefit from this capacity because context workers retain earlier turns’ KV state and prefill only newly appended tokens.
DeepSeek-R1. We replay the AgentX sessions of § 4.1 under an equal KV-cache budget of 43 GiB per context GPU, in which NVFP4-MLA holds 60% more cached tokens than FP8 MLA (§ 4.3) and the trace allows a hit rate of about 94–96%. FP8 MLA starts evicting at 32 sessions (Table 3): from 32 to 64 sessions, its hit rate falls from 87.6% to 54.1% and its p90 time to first token (TTFT) rises from 11.9 to 178 s. NVFP4-MLA keeps a higher hit rate and therefore repeats less prefill: with 48 sessions, it raises the hit rate from 77.0% to 87.1% and lowers p90 TTFT from 108 to 30.9 s, a 3.5× reduction, and the reduction is 1.5–3.5× across all points where FP8 MLA evicts.
| Model | Concurrent sessions | p90 TTFT (s) FP8 MLA | p90 TTFT (s) NVFP4-MLA | Cache hit (%) FP8 MLA | Cache hit (%) NVFP4-MLA |
| DeepSeek-R1 | 16 | 2.98 | 1.99 | 95.2 | 95.1 |
| 24 | 2.68 | 2.41 | 93.6 | 94.4 | |
| 32 | 11.9 | 5.36 | 87.6 | 92.4 | |
| 40 | 36.6 | 23.7 | 82.3 | 87.9 | |
| 48 | 108 | 30.9 | 77.0 | 87.1 | |
| 64 | 178 | 79.8 | 54.1 | 79.6 | |
| Kimi-K3 | 128 | 11.0 | 5.24 | 90.8 | 94.4 |
Table 3. AgentX serving. At equal KV-cache memory, NVFP4-MLA keeps higher cache-hit rates and lower p90 TTFT once FP8 MLA starts evicting histories. AgentX replay on Rubin GPUs. Kimi-K3 KV-cache memory also holds KDA state. Concurrent sessions: agent sessions kept alive; Cache hit: fraction of input tokens reused from the prefix cache.
Kimi-K3. We replay the same sessions on Kimi-K3 under an equal KV-cache budget of 47.4 GiB per context GPU, in which NVFP4-MLA holds 47–49% more cached tokens than FP8 MLA (§ 4.3) and the trace allows a hit rate of at most 95.4%. FP8 MLA already evicts with 128 sessions (Table 3). NVFP4-MLA keeps a higher hit rate and therefore repeats less prefill: it raises the hit rate from 90.8% to 94.4% and lowers p90 TTFT from 11.0 to 5.24 s, a 2.1× reduction.
Kernel performance
Besides its smaller cache, NVFP4-MLA shortens the attention computation itself, because both BMMs run on FP4 Tensor Cores. Its speedup over FP8 MLA grows with KV length, reaching 2.82× on DeepSeek-R1 and 2.71× on Kimi-K3.
MTP and the roofline. MTP=3 reuses each cache read across four query positions (Appendix § 8), which raises the arithmetic intensity of FP8 MLA from 484 to 1,934 FLOP/s per cache byte, beyond the 911 FLOP/B ridge point of a Rubin GPU; FP8 MLA decoding with MTP is therefore compute-bound. A storage-only alternative that dequantizes an NVFP4 cache to FP8 operands performs the same FP8 MMAs and softmax as FP8 MLA and shortens only cache reads that no longer limit the kernel, so it cannot finish faster than FP8 MLA. NVFP4-MLA is faster because its FP4 BMMs shorten the computation itself.
KV-length and batch sweep. Each measurement profiles a single attention kernel under a CUDA graph, with equal KV lengths across requests, and includes the split-KV reduction. FP8 MLA uses its faster split-KV strategy per shape; NVFP4-MLA its best CTA scheduling per shape. NVFP4-MLA is faster at every measured shape (Figure 5), by – with a geometric mean of on the 39 DeepSeek-R1 shapes and by – with a geometric mean of on the 39 Kimi-K3 shapes. The speedup generally grows with KV length, from – at 2K, where fixed per-call cost dominates, to – at 128K, where per-token attention work dominates.
Figure 5. Kernel speedup over FP8 MLA on one Rubin GPU. (a) DeepSeek-R1. (b) Kimi-K3.
End-to-end serving
End-to-end serving: NVFP4-MLA improves the throughput–latency trade-off of disaggregated DeepSeek-R1 serving, with gains that grow with context length.
Pareto measurement. Closed-loop clients sweep the number of in-flight requests for each generation layout, and each precision’s Pareto frontier takes, at every per-user generation rate, the highest per-GPU throughput of any of its layouts.
Throughput. At the same per-user generation rate, NVFP4-MLA serves more output tokens per GPU over most of the measured range (Figure 6): 26.7% more on MLPerf and 39.9% more on 8K/16K Reasoning on average, and up to and as many.
Figure 6. Serving Pareto frontiers. NVFP4-MLA raises per-GPU throughput over most of the range. Pareto frontiers for DeepSeek-R1 disaggregated serving on Rubin GPUs with MTP=3: (a) 8K/16K Reasoning; (b) MLPerf.
Parallel layouts. Both precisions reach their highest throughput with DEP16 up to about 250 tokens/s/user and with TEP8 at higher per-user rates.
Takeaway: NVFP4-MLA improves performance through two effects. Its smaller cache keeps more session histories, lowering p90 TTFT by up to on DeepSeek-R1 and on Kimi-K3 when GPU memory bounds the prefix cache. Its FP4 BMMs shorten attention computation, with speedups that grow with context length, reaching and at the kernel level and up to in per-GPU throughput for DeepSeek-R1 serving.
Related work
KV-cache quantization. Most previous work focuses on quantizing the KV cache to reduce its footprint and memory traffic. KIVI ([13]) quantizes keys per channel and values per token, keeping the most recent tokens in full precision. KVQuant ([6]) quantizes keys per channel before RoPE and keeps outliers in a sparse format. KVTuner (Li et al., 2025) searches layer-wise mixed precision for keys and values. RotateKV [22] and TurboQuant [26] rotate cached vectors to spread outliers before low-bit quantization. Serving systems such as Atom [30] and QServe [12] also store 4-bit KV caches alongside low-bit weights and activations. These methods target separate key and value caches rather than MLA’s shared latent representation. They also execute the attention BMMs at higher precision, which adds dequantization overhead for the cached values and limits MLA decoding once MTP makes it compute-bound (§ 4.4.2). NVFP4-MLA instead stores a shared MLA cache in NVFP4 whose scale blocks match both reduction directions, so both BMMs read it in FP4 without dequantization and run at FP4 Tensor Core throughput.
Quantized attention kernels. Low-precision attention kernels quantize attention operands to run the attention BMMs on faster low-precision Tensor Cores. SageAttention [28] quantizes queries and keys to INT8 for the BMM and keeps the BMM mostly in FP16. SageAttention3 [27] runs both BMMs in NVFP4, quantizing FP16 queries, keys, and values inside each attention call, and mainly targets image and video generation. SnapMLA [29] pushes MLA computation to FP8 with a per-token FP8 latent cache and keeps the RoPE channels in BF16. NVFP4-MLA pushes it further to FP4 and combines low-precision storage and computation: both BMMs consume the persistent NVFP4 MLA cache without dequantization, and the RoPE channels are quantized to NVFP4 with residual compensation rather than kept in high precision.
Evaluation scope. Prior KV-cache quantization studies evaluate non-MLA models of limited size, mostly with perplexity, short-generation tasks, or long-context retrieval [13, 6, 11]. These settings do not test long reasoning traces or multi-turn agentic sessions, where quantization error can accumulate over thousands of generated tokens. We instead evaluate large MLA reasoning models with a broader set of tests, spanning downstream accuracy on reasoning and agentic tasks, cache footprint, and kernel and end-to-end serving performance, to understand how NVFP4-MLA affects both model quality and efficiency.
Conclusion
We have presented NVFP4-MLA, which shows that NVFP4 can serve as both the storage format and the compute format of MLA. Selective residual compensation targets quantization outliers and greatly reduces the accuracy drop of direct FP4 quantization, and two-dimensional quantization blocks let one shared latent cache supply both BMMs. We also design a corresponding cache system and an efficient kernel pipeline. NVFP4-MLA matches BF16 MLA average accuracy on DeepSeek-R1 and Kimi-K3, speeds up the attention kernel over FP8 MLA by up to 2.82× and 2.71×, lowers p90 time to first token in agentic multi-turn serving by up to 3.5× and 2.1×, and raises per-GPU throughput in disaggregated DeepSeek-R1 serving by 26.7% and 39.9% on average on two workloads and by up to 2.30×. By computing attention in FP4 while preserving accuracy, NVFP4-MLA turns KV-cache compression into real inference speedups.
NVFP4 quantization
Let be the set of E2M1 values, with largest magnitude , and let be the set of finite E4M3 values, with largest magnitude . Let and round a real number onto and , respectively, with saturation; rounds to nearest with ties to even. An NVFP4 tensor stores an FP32 tensor scale , one E4M3 scale for each block of 16 elements, and one E2M1 value for each element. For a real tensor and a partition of its indices into 16-element blocks, NVFP4 quantization is
with whenever . We identify an NVFP4 tensor with the real tensor that it represents, so sums and products of hatted (quantized) tensors denote exact real arithmetic. Block-scaled Tensor Core instructions evaluate these products from the stored values and scales with FP32 accumulation, without materializing a wider tensor. and apply Equation (4) with and partitions, respectively. The tensor scale is a per-layer constant for each operand; for queries and the cache in both models, is fixed offline from the largest magnitude observed during calibration, with margin .
Arithmetic intensity of MLA decoding
MLA’s compact shared cache and wider BMMs after matrix absorption raise arithmetic intensity and make math throughput relevant alongside memory bandwidth.
A shared cache supports wider BMMs
Let denote the cached-token count, the number of query heads, and the number of GQA KV groups. With key and value widths and , one GQA query performs FLOPs on distinct cache elements:
For MLA, BMM1 uses channels and BMM2 uses :
The NoPE latent contributes to both products but occupies storage only once.
For DeepSeek’s , , and [3], we compare a GQA reference with and . At equal payload bytes per element, ideal intensity is . Assuming one cold-cache read with reuse across heads and, for MLA, across BMM1 and BMM2 gives
MTP raises intensity through cache reuse
Let query positions share a long prefix in one invocation, including the ordinary next-token position. Neglecting newly appended positions gives . With as cache traffic relative to a single query, excluding query-dependent transfers gives
Full reuse gives and an approximately -fold increase; MTP=3 evaluates four query positions.
FP4 addresses both bandwidth and math throughput
For matrix-throughput ceiling , HBM bandwidth , and HBM traffic , roofline analysis gives the latency lower bound [25]
FP4 storage reduces the memory term, and native FP4 BMMs reduce the compute term. With MTP, FP8 MLA decoding is compute-bound on the evaluated GPU (§ 4.4.2).
Block-local softmax-score scaling
Section 3.4 computes the E2M1 score values and the block scales of Equation (3). Once is available, the kernel encodes the block scale with the tensor scale of Equation (4):
where rounds to nearest with ties to even, as the native E4M3 conversion does. Because the values assume the unrounded scale, scale rounding multiplies all scores of a block by the common factor . Rather than absorbing into every block scale, the kernel keeps this reciprocal off the BMM2 critical path and divides the BMM2 output by after BMM2, with accumulated in FP32 from the unquantized scores .
Range and underflow of the softmax-score scale. Because , the argument of in Equation (10) lies in : maps the row maximum to the largest finite E4M3 value, and no block scale saturates. In the normal E4M3 range, changes a block’s scores by a common factor within . A block falls into the subnormal range, where the scale spacing is , when . Round-to-nearest flushes the scale to zero when
that is, only when every score in the block is at most times the row maximum. Let be the set of tokens in flushed blocks. Because for and , flushing removes at most
of the normalized softmax mass. Within every retained block, E2M1 rounding already maps arguments of at or below to zero and thus removes any score at or below of its block maximum.
References
- [1]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ²-Bench: Evaluating conversational agents in a dual-control environment. In Proceedings of the 43rd International Conference on Machine Learning, 2026.arxiv.org/abs/2506.07982
- [2]DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a.arxiv.org/abs/2405.04434
- [3]DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024b.arxiv.org/abs/2412.19437
- [4]Vage Egiazarian, Roberto L. Castro, Denis Kuznedelev, Andrei Panferov, Eldar Kurtic, Shubhra Pandit, Alexandre Noll Marques, Mark Kurtz, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Bridging the gap between promise and performance for microscaling FP4 quantization. In International Conference on Learning Representations, 2026.arxiv.org/abs/2509.23202
- [5]Daya Guo et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638, 2025. doi: 10.1038/s41586-025-09422-z.DOI
- [6]Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2401.18079
- [7]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling, 2024.arxiv.org/abs/2404.06654
- [8]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025.arxiv.org/abs/2403.07974
- [10]Wangqian Li, Jintao Peng, Zongfei Jing, Tianyu Zhang, Ze Long, Xianjie Qiao, Xiaoming Chen, Dongxu Yang, Kefeng Duan, and June Yang. DWDP: Distributed weight data parallelism for high-performance LLM inference on NVL72. arXiv preprint arXiv:2604.01621, 2026.arxiv.org/abs/2604.01621
- [11]Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Yiwu Yao, Wulong Liu, Sinno Jialin Pan, and Mingxuan Yuan. KVTuner: Sensitivity-aware layer-wise mixed-precision KV cache quantization for efficient and nearly lossless LLM inference. In Proceedings of the 42nd International Conference on Machine Learning, 2025.arxiv.org/abs/2502.04420
- [12]Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. In Proceedings of Machine Learning and Systems, 2025.
- [13]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, 2024.DOI
- [15]NVIDIA. Pretraining large language models with NVFP4. arXiv preprint arXiv:2509.25149, 2025.arxiv.org/abs/2509.25149
- [16]NVIDIA. TensorRT-LLM. GitHub repository, 2026. URL https://github.com/NVIDIA/TensorRT-LLM.
- [17]Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Hajishirzi. Generalizing verifiable instruction following. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2507.02833
- [18]Vijay Janapa Reddi et al. MLPerf inference benchmark. In ACM/IEEE 47th Annual International Symposium on Computer Architecture, 2020.
- [19]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
- [20]Bita Darvish Rouhani et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.arxiv.org/abs/2310.10537
- [21]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/j.neucom.2023.127063.DOI
- [22]Zunhai Su, Hanyu Wei, Zhe Chen, Wang Shen, Linge Li, Huangqi Yu, and Kehong Yuan. RotateKV: Accurate and robust 2-bit KV cache quantization for LLMs via outlier-aware adaptive rotations. In Proceedings of the 34th International Joint Conference on Artificial Intelligence, 2025.arxiv.org/abs/2501.16383
- [23]Minyang Tian et al. SciCode: A research coding benchmark curated by scientists. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2407.13168
- [24]Kiran Vodrahalli et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024.arxiv.org/abs/2409.12640
- [25]Samuel Williams, Andrew Waterman, and David A. Patterson. Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009. doi: 10.1145/1498765.1498785.DOI
- [26]Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. TurboQuant: Online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations, 2026.arxiv.org/abs/2504.19874
- [27]Jintao Zhang, Jia Wei, Haoxu Wang, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Kai Jiang, Jianfei Chen, and Jun Zhu. SageAttention3: Microscaling FP4 attention for inference and an exploration of 8-bit training. In Advances in Neural Information Processing Systems, 2025a.arxiv.org/abs/2505.11594
- [28]Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. SageAttention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations, 2025b.arxiv.org/abs/2410.02367
- [29]Yifan Zhang, Zunhai Su, Shuhao Hu, Rui Yang, Wei Wu, Yulei Qian, Yuchen Xie, and Xunliang Cai. SnapMLA: Efficient long-context MLA decoding via hardware-aware FP8 quantized pipelining. arXiv preprint arXiv:2602.10718, 2026.arxiv.org/abs/2602.10718
- [30]Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate LLM serving. In Proceedings of Machine Learning and Systems, 2024.arxiv.org/abs/2310.19102





