betaFeedback

WorksComputer Science

NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent Attention

Abstract

Long-context reasoning and agentic workloads make key-value (KV) cache capacity and attention computation the dominant costs of serving frontier large language models. We present NVFP4-MLA, the first design to push both KV-cache storage and attention computation of multi-head latent attention (MLA) to FP4. NVFP4-MLA co-designs the quantization algorithm with the attention kernel and the cache system. Selective residual compensation preserves model accuracy by adding NVFP4 residuals only to the quantization-sensitive channels. Two-dimensional quantization blocks make the shared latent cache compatible with the different reduction directions of the two attention batched matrix multiplications (BMMs), removing requantization between them. On DeepSeek-R1 and Kimi-K3, NVFP4-MLA matches BF16 MLA average accuracy across nine benchmarks while compressing the MLA KV cache by 3.20x and 3.05x relative to BF16 MLA. On NVIDIA Rubin GPUs, NVFP4-MLA speeds up the attention kernel over FP8 MLA by up to 2.82x on DeepSeek-R1 and 2.71x on Kimi-K3, and lowers p90 time to first token in agentic multi-turn serving by up to 3.5x and 2.1x, respectively. In disaggregated DeepSeek-R1 serving, it also raises per-GPU throughput by up to 2.30x.

Introduction

Multi-head latent attention (MLA) represents keys and values in a shared low-rank latent space and is the common attention pattern of several popular open-source large language models, including DeepSeek-V3 ([3]), DeepSeek-R1 ([5]), Kimi-K2.6, Kimi-K3 ((Kimi Team, 2026)), and LongCat-Flash ((Meituan LongCat Team, 2025)), which use MLA in all or part of their attention layers. The low-rank latent shrinks the key-value (KV) cache, but the cache still grows with sequence length and serving concurrency, and every decoding step reads all of it, which puts pressure on both memory capacity and bandwidth. Agentic workloads sustain this pressure across turns and tool calls because they preserve and reuse growing contexts, which keeps the KV cache resident for longer. When GPU memory cannot hold these caches, the server evicts them and recomputes the prefix in the next turn, which increases time to first token. A smaller cache allows longer contexts, keeps more tokens and requests in GPU memory, reduces cache eviction and recomputation, and increases serving concurrency.

Beyond memory, computation can also be under pressure. In grouped-query attention (GQA), each KV head serves only the few query heads in its group, whereas all MLA heads share the latent cache during decoding, so MLA performs more computation per cached byte (Appendix § 8). Model-native multi-token prediction (MTP) verifies several queries against the same latent cache. Reusing the cache across these queries raises arithmetic intensity further and makes MLA decoding compute-bound (§ 4.4.2), so math throughput matters as much as cache bytes. Together, these pressures motivate us to apply NVFP4, a 4-bit format that recent NVIDIA Tensor Cores support natively, to both KV-cache storage and attention computation in MLA, so that MLA benefits from smaller storage and higher math throughput.

We introduce NVFP4-MLA, which brings unified NVFP4 storage and attention computation to MLA. Its design addresses the challenges of applying FP4 quantization directly to MLA. First, direct FP4 quantization degrades accuracy. Outliers in the MLA cache concentrate in the last few channels, so selective residual compensation adds an NVFP4 residual only to these channels and keeps all elements in FP4 (§ 4.2.2). Second, MLA shares one latent representation between keys and values. Default block-wise quantization groups this latent along channels, which matches the channel reduction of the first batched matrix multiplication (BMM1) but not the token reduction of BMM2. NVFP4-MLA introduces two-dimensional quantization blocks that are compatible with both reduction dimensions and therefore remove the need to requantize the shared latent cache for the second BMM (§ 3.3). Third, the higher arithmetic intensity makes math throughput a bottleneck. NVFP4-MLA therefore also quantizes queries and softmax scores online, so that both BMMs run on native FP4 Tensor Cores and exploit the higher FP4 math throughput. A corresponding cache system and kernel pipeline implement this design, and Figure 1 gives an overview.

Figure 1. Overview of NVFP4-MLA. (a) NVFP4 MLA cache: two-dimensional NoPE blocks shared by both BMMs, and RoPE with an NVFP4 residual. (b) NVFP4 MLA kernel: BMM1, online softmax fused with score quantization, and BMM2, with score construction and accumulator rescaling split across warp groups (WG). (c) Dataflow: score-operand reuse and split KV. (d) End-to-end integration in disaggregated serving: prefill, KV cache transfer, decode, and speculative decoding.

Our main contributions are:

  1. We design selective residual compensation, which keeps model accuracy by adding NVFP4 residuals to quantization-sensitive channels without mixed precision.

  2. We design two-dimensional quantization blocks and a corresponding cache system that let the shared quantized representation serve both BMMs without requantization.

  3. We build an efficient kernel pipeline. The pipeline quantizes queries and softmax scores online, so both BMMs benefit from the higher throughput of FP4 Tensor Cores.

We evaluate NVFP4-MLA on two frontier open MLA models, DeepSeek-R1 and Kimi-K3, across nine benchmarks that span reasoning, coding, instruction following, long context, and agentic tool use. NVFP4-MLA matches BF16 MLA average accuracy on both models while reducing the MLA KV cache by 3.20×3.20\times and 3.05×3.05\times relative to BF16 MLA. On NVIDIA Rubin GPUs, NVFP4-MLA speeds up the attention kernel over FP8 MLA by up to 2.82×2.82\times on DeepSeek-R1 and 2.71×2.71\times on Kimi-K3. In the AgentX benchmark, the smaller cache lowers p90 time to first token by up to 3.5×3.5\times and 2.1×2.1\times, respectively. In disaggregated DeepSeek-R1 serving on the MLPerf and 8K/16K Reasoning workloads, per-GPU throughput reaches up to 2.30×2.30\times that of FP8 MLA, and is 26.7%26.7\% and 39.9%39.9\% higher on average, respectively.

§ 2 reviews MLA and the NVFP4 format. § 3 presents selective residual compensation, two-dimensional quantization blocks, and the corresponding cache system and kernel pipeline. § 4 evaluates the accuracy, memory footprint, and performance of NVFP4-MLA on DeepSeek-R1 and Kimi-K3. § 5 compares NVFP4-MLA with prior KV-cache quantization methods and quantized attention kernels. § 6 summarizes the findings.

Background

Multi-head latent attention

Low-rank compression creates shared KV state

MLA stores one compressed latent vector for each token [2 b]. For a hidden state hth_t, a down-projection produces

ct=WDKVht,ct∈Rdc.(1)c_t = W^{\mathrm{DKV}}h_t,\qquad c_t \in\mathbb{R}^{d_c}. \tag*{(1)}

Learned up-projections recover head-specific keys and values from ctc_t. During inference, matrix absorption folds these projections into the query and output projections. The runtime therefore caches ctc_t instead of materializing a separate key and value for every head.

The shared latent is involved in both BMMs. The first, BMM1 (QK), reduces over latent channels to generate attention logits, and the second, BMM2 (PV), reduces over cached tokens to generate the output. The same cached values are therefore reduced along two directions, so their block scales must serve both BMMs.

As Appendix § 8 shows, MLA performs more computation per cached byte than a grouped-query attention (GQA) reference. Matrix absorption reduces cache traffic and reuses the latent cache for every query head. Model-native MTP raises arithmetic intensity further when the kernel reuses the cache across several query positions. MLA decoding therefore tends to be compute-bound.

NoPE and RoPE channels serve different roles

2.1.2. NoPE and RoPE channels serve different roles: Applying rotary position encoding (RoPE) [21] directly to the compressed keys would break matrix absorption, because the rotation depends on each token’s position [2]. MLA therefore carries RoPE on a decoupled path, with additional per-head queries and a single key shared by all heads. Each cached token thus holds a dc=512d_c = 512-channel compressed latent, which we call NoPE, and a dR=64d_R = 64-channel decoupled key, which we call RoPE. In BMM1, each head adds the NoPE and RoPE dot products to form its attention logit, whereas BMM2 aggregates only the NoPE latent. Kimi-K3 keeps this decoupled layout but applies no positional encoding to either path (Kimi Team, 2026); we still call its decoupled channels RoPE.

NVFP4 format and KV-cache quantization

KV-cache quantization reduces per-token storage by encoding cached values with fewer bits [13]. To keep accuracy, prior methods redistribute large outliers through rotations, preserve selected outliers separately, or assign different bit widths across tensors and layers [6, 11, 22, 26]. Beyond accuracy, using the quantized cache directly in low-precision BMMs requires a compatible hardware format. We therefore use block-scaled NVFP4 for both cache storage and attention computation.

NVFP4 and MXFP4 both encode values in 4-bit E2M1 and attach a shared scale to each block [20, 15]. NVFP4 uses an E4M3 scale per 16 values, whereas MXFP4 uses an E8M0 power-of-two scale per 32 values. NVFP4’s smaller blocks and non-power-of-two scales give finer control over local value ranges.

These formats have mostly been studied for weight and activation quantization [4]. For KV caches, their use also depends on how attention consumes the quantized values. When compute precision differs from storage precision, attention needs an extra dequantization step. Computing directly on the stored values avoids this step, but each BMM then requires the quantized values and scales in a compatible layout. This requirement links the block layout to the reduction axes of the attention BMMs. In GQA, separate key and value caches can adopt independent block layouts. In MLA, by contrast, both BMMs reduce the same latent cache along different axes, which couples their block layouts. A shared MLA cache intended for native FP4 computation therefore requires quantization blocks that support both reduction directions.

Methodology

NVFP4-MLA makes the shared latent a direct operand of both FP4 attention BMMs. Selective residual compensation reduces quantization error without mixed precision, two-dimensional quantization blocks align quantization with both reduction axes, the efficient kernel pipeline handles online quantization, and the corresponding cache system supports end-to-end integration.

Quantization of Q, K, P, and V

During decode in quantized layers, NVFP4-MLA quantizes all four attention operands: queries QQ, keys KK, softmax scores PP, and values VV. PP holds stable, unnormalized softmax scores, which NVFP4-MLA quantizes online for BMM2.

Q1×16Q_{1 \times16} denotes NVFP4 quantization with contiguous 16-element blocks along the reduction axis of the consuming BMM: latent channels for queries and the RoPE cache, and cached tokens for softmax scores. Q16×16Q_{16 \times16} uses 16×1616 \times16 token-by-channel blocks, each with a single scale (Appendix § 7).

For one query head and TT cached tokens, let C=[CN,CR]∈RT×(dc+dR)C = [C_N, C_R] \in\mathbb{R}^{T \times(d_c+d_R)} and q=[qN,qR]∈R1×(dc+dR)q = [q_N, q_R] \in\mathbb{R}^{1 \times(d_c+d_R)}. Let S,p~∈R1×TS,\tilde{p} \in\mathbb{R}^{1 \times T} denote attention logits and stable, unnormalized softmax scores, respectively; p~\tilde{p} is the single-head instance of PP. The normalized output is O∈R1×dcO \in\mathbb{R}^{1 \times d_c}, and dQKd_{QK} is the pre-absorption QK head width. NoPE uses Q16×16Q_{16 \times16} because both BMMs consume it, whereas RoPE enters only BMM1 and uses Q1×16Q_{1 \times16} along its channels. The quantized attention dataflow, evaluated as online softmax over successive KV tiles Tk\mathcal{T}_k, is

C^N=Q16×16(CN),C^R=Q1×16(CR),q^N=Q1×16(qN),q^R=Q1×16(qR),S=q^NC^NT+q^RC^RT,m(k)=max⁡(m(k−1),max⁡i∈TkSi),p~i=exp⁡((Si−m(k))/dQK),p^=Q1×16(p~),ℓ(k)=αkℓ(k−1)+∑i∈Tkp~i,O~(k)=αkO~(k−1)+∑i∈Tkp^iC^N,i,αk=exp⁡((m(k−1)−m(k))/dQK),O=O~/ℓ.(2)\begin{aligned} \widehat{C}_{\mathrm{N}} &= Q_{16\times16}(C_{\mathrm{N}}), & \widehat{C}_{\mathrm{R}} &= Q_{1\times16}(C_{\mathrm{R}}),\\ \widehat{q}_{\mathrm{N}} &= Q_{1\times16}(q_{\mathrm{N}}), & \widehat{q}_{\mathrm{R}} &= Q_{1\times16}(q_{\mathrm{R}}),\\ S &= \widehat{q}_{\mathrm{N}}\widehat{C}_{\mathrm{N}}^{\mathsf{T}}+\widehat{q}_{\mathrm{R}}\widehat{C}_{\mathrm{R}}^{\mathsf{T}}, & m^{(k)} &= \max\left(m^{(k-1)},\max_{i\in\mathcal{T}_k}S_i\right),\\ \widetilde{p}_i &= \exp\left((S_i-m^{(k)})/\sqrt{d_{\mathrm{QK}}}\right), & \widehat{p} &= Q_{1\times16}(\widetilde{p}),\\ \ell^{(k)} &= \alpha_k\ell^{(k-1)}+\sum_{i\in\mathcal{T}_k}\widetilde{p}_i, & \widetilde{O}^{(k)} &= \alpha_k\widetilde{O}^{(k-1)}+\sum_{i\in\mathcal{T}_k}\widehat{p}_i\widehat{C}_{\mathrm{N},i},\\ \alpha_k &= \exp\left((m^{(k-1)}-m^{(k)})/\sqrt{d_{\mathrm{QK}}}\right), & O &= \widetilde{O}/\ell. \tag*{(2)} \end{aligned}

Here Tk\mathcal{T}_k is the kk-th KV tile and i∈Tki\in\mathcal{T}_k, with m(0)=−∞m^{(0)}=-\infty and ℓ(0)=O~(0)=0\ell^{(0)}=\widetilde{O}^{(0)}=0; mm, ℓ\ell, and O~\widetilde{O} without a superscript are the running values, and OO is taken after the last tile. Both BMMs accumulate in FP32, and softmax keeps SS, mm, ℓ\ell, and the final division by ℓ\ell in FP32.

Selective residual compensation

FP4 quantization error in the MLA cache is disproportionately large in the RoPE partition (§ 4.2.2). NVFP4-MLA therefore keeps a single NVFP4 component for NoPE but adds a second NVFP4 component only for the RoPE quantization error. Because the RoPE partition holds 64 of the 576 latent channels, the residual adds little storage and computation; § 4.2 evaluates the accuracy of NVFP4-MLA.

The native kernel packs RoPE and its quantization error separately with independent NVFP4 block scales, logically concatenates them along the reduction dimension of BMM1, and duplicates the corresponding channels. The compensated logits, which replace SS in Equation (2), therefore take a single BMM1 reduction, without merging base and residual into a high-precision tensor.

Two-dimensional quantization blocks for shared keys and values

As § 2 shows, the shared MLA latent requires scales that work in both reduction directions. A conventional one-dimensional layout aligns its blocks with BMM1 but not with native NVFP4 BMM2 (Figure 2a). Converting this layout requires dequantizing the cached E2M1 values, transposing them, and requantizing them into BMM2-aligned blocks. Requantization introduces additional quantization error. It also adds overhead such as memory traffic, a scale reduction, and synchronization before BMM2.

Figure 2. Two-dimensional quantization blocks. Both BMMs consume the same FP4 values and scales without requantization. One-dimensional 1 × 16 blocks (a) serve only BMM1, whereas 16 × 16 blocks (b) serve both. NVFP4-MLA uses (b) for NoPE and (a) for RoPE, which only BMM1 consumes.

NVFP4-MLA therefore quantizes the shared NoPE latent in two-dimensional 16×1616\times16 blocks over tokens and channels (Figure 2b). One scale covers every 16-element row and column segment of a block, so the block is valid in either reduction direction: both BMMs share the same NVFP4 values and scales without a high-precision intermediate or a requantization pass.

Kernel implementation

Kernel implementation

To turn FP4’s higher Tensor Core throughput into a faster attention kernel, NVFP4-MLA quantizes queries and softmax scores online (Figure 1b) and applies four optimizations: fusing online softmax with FP4 score quantization, balancing FP4 score construction and accumulator rescaling, reusing FP4 score operands across BMM2 output tiles, and shape-adaptive thread-block scheduling.

Fusing online softmax with FP4 score quantization. Online FP4 score quantization makes the logit-to-score path heavier than in FP8 MLA because the path must also derive block scales and pack the FP4 operand for BMM2. We algebraically fuse online softmax, FP4 score quantization, and scale generation. As in online softmax, let mm be the running maximum of one row of logits SS, and let a=log⁡2(e)/dQKa = \log_{2}(e) / \sqrt{d_{\mathrm{QK}}}, so that p~i=2a(Si−m)\tilde{p}_i = 2^{a(S_i-m)} in Equation (2). Let mb=max⁡j∈bSj≤mm_b = \max_{j \in b} S_j \le m; for each 16-element block bb and i∈bi \in b,

p~i=2a(mb−m)2a(Si−mb),p^i=γPsbxi,xi=QE2M1(ME2M1−12a(Si−mb)),sb=QE4M3(ME4M3−12a(mb−m)),(3)\begin{aligned} \tilde{p}_i &= 2^{a(m_b-m)}2^{a(S_i-m_b)}, & \hat{p}_i &= \gamma_P s_b x_i, \\ x_i &= \mathcal{Q}_{\mathrm{E2M1}}\left(M_{\mathrm{E2M1}}^{-1}2^{a(S_i-m_b)}\right), \\ s_b &= \mathcal{Q}_{\mathrm{E4M3}}\left(M_{\mathrm{E4M3}}^{-1}2^{a(m_b-m)}\right), \tag*{(3)} \end{aligned}

where QE2M1\mathcal{Q}_{\mathrm{E2M1}} and QE4M3\mathcal{Q}_{\mathrm{E4M3}} round to E2M1 and E4M3, ME2M1=6M_{\mathrm{E2M1}} = 6 and ME4M3=448M_{\mathrm{E4M3}} = 448 are the largest E2M1 and finite E4M3 magnitudes (Appendix § 7), γP=1/(ME2M1ME4M3)\gamma_P = 1/(M_{\mathrm{E2M1}}M_{\mathrm{E4M3}}) is a constant tensor scale, and xix_i is quantized against the unrounded block scale. Because Si≤mb≤mS_i \le m_b \le m, the two arguments lie in (0,ME2M1](0, M_{\mathrm{E2M1}}] and (0,ME4M3](0, M_{\mathrm{E4M3}}], so neither rounding saturates. The E2M1 value xix_i depends on SiS_i and mbm_b but not on mm, so the kernel evaluates the exponent with a fused multiply-add, converts the result to E2M1, and packs it before the cross-block reduction produces mm. Once mm is available, only the block scale sbs_b remains (Appendix § 9).

Balancing FP4 score construction and accumulator rescaling. Fusion shortens the logit-to-score path, but this stage still carries more non-matrix work than its FP8 MLA counterpart. Rather than dedicate separate warp groups to FP4 score construction and online-softmax accumulator rescaling, we partition each logit tile across the warp groups. Each group constructs the FP4 score operand for its logit partition and also rescales a complementary partition of the resident BMM2 accumulator when the running maximum changes. This mixed assignment keeps the logit-to-score stage from serializing the pipeline.

Reusing FP4 score operands across BMM2 output tiles. Each packed FP4 score slice contributes to multiple BMM2 output tiles. The kernel uses the Tensor Core operand-reuse mechanism to retain this P^\hat{P} operand of BMM2 across these tiles instead of reloading it for each tile, so more execution bandwidth remains for FP4 score construction.

Shape-adaptive CTA scheduling. With MTP, each request has more queries, so the kernel can use larger matrix multiply-accumulate (MMA) tiles. It gives each cluster of thread blocks (cooperative thread arrays, CTAs) the packed query rows of one request; the CTAs of a cluster split the KV tiles, exchange the FP4 scores through distributed shared memory, and use the same FP4 operands and softmax. At long KV lengths, the kernel also splits the KV of a request into pieces that run on idle streaming multiprocessors (Figure 1c) and merges them with a reduction kernel.

End-to-end integration

For end-to-end integration, we adapt the cache system to be compatible with two-dimensional quantization blocks, the NVFP4 attention kernel, paged KV caches, and speculative decoding (Figure 1d). During prefill, a fused quantize-and-scatter kernel writes completed NVFP4 blocks into paged storage, which decode reads without requantization. Disaggregated serving transfers only these pages, namely the E2M1 values and their scales, plus the BF16 tail of the incomplete last block. During decode, a fused pre-attention kernel applies RoPE where the model uses it, updates a 16-token BF16 ring, quantizes the operands, and writes the cache. Each request owns a stable slot in the ring, and the runtime refreshes the active NVFP4 block from BF16 values and finalizes the block once it fills. It therefore never has to dequantize a block, append newly generated tokens, and requantize it, which would introduce additional quantization error. For speculative decoding such as MTP, the runtime snapshots the BF16 ring before speculative writes, commits accepted candidates, and restores locations overwritten by rejected candidates. Completed NVFP4 pages remain unchanged during rollback, so rejected tokens cannot contaminate persistent state. NVFP4-MLA has been integrated into TensorRT-LLM ([16]).

Evaluation

We evaluate NVFP4-MLA on DeepSeek-R1 and Kimi-K3 in three parts: accuracy (§ 4.2), KV-cache memory reduction (§ 4.3), and performance (§ 4.4).

Experimental setup

Accuracy. Both models use the same nine tasks, spanning reasoning, coding, instruction following, long context, and agentic tool use (Table 1). RULER and MRCR use a 128K-token context. We compare NVFP4-MLA with BF16 MLA, FP8 MLA, and vanilla NVFP4 MLA. BF16 MLA keeps the KV cache and attention BMM computation in BF16. FP8 MLA quantizes the KV cache and attention BMM computation to E4M3. Vanilla NVFP4 MLA quantizes them directly to NVFP4 without residual compensation.

ConfigurationGPQAIFBenchAIME 2025LiveCode Bench v6SciCodeRULERMRCRAA-LCRτ2\tau^{2}-BenchAvg
DeepSeek-R1
BF16 MLA80.0739.5387.1471.3142.6092.9736.3552.5235.7559.80
FP8 MLA80.1637.9387.6071.7542.2792.6737.2152.6235.4259.74
Vanilla NVFP4 MLA77.3834.8773.8062.1140.6490.5229.5446.0631.9154.09
NVFP4-MLA79.8338.0785.9471.7843.1292.7736.9551.8638.0559.82
Kimi-K3
BF16 MLA93.6673.9696.2057.4958.4897.0085.0281.7578.8480.27
FP8 MLA93.5073.6995.6357.3258.1997.0684.3882.1977.4979.94
Vanilla NVFP4 MLA88.2661.7987.7110.682.2795.9875.2177.3174.8963.79
NVFP4-MLA93.4374.9196.2057.3058.0997.1184.1082.3179.0680.28

Table 1. Downstream accuracy on DeepSeek-R1 and Kimi-K3. NVFP4-MLA matches BF16 MLA average accuracy and greatly reduces the accuracy drop of vanilla NVFP4 MLA. Scores in %; Avg: nine-task mean. Repeats per task: 64 for GPQA [19], AIME 2025, and AA-LCR; 8 for LiveCodeBench [8], SciCode [23], and τ2\tau^{2}-Bench [1]; 5 for IFBench [17]; 3 for MRCR [24]; 1 for RULER [7].

Storage. We count the stored MLA cache. DeepSeek-R1 uses NVFP4 in all 61 MLA layers. Kimi-K3 uses NVFP4 in 22 of its 24 MLA layers and FP8 in the final two; the count excludes the recurrent state of its 69 Kimi Delta Attention (KDA) layers.

Performance. All performance experiments run on NVIDIA Rubin GPUs, and serving runs use TensorRT-LLM with the same runtime and the same set of parallel layouts for both precisions. On DeepSeek-R1 with MTP=3, MLPerf uses the evaluation prompts of the MLPerf Inference [18] DeepSeek-R1 benchmark, and 8K/16K Reasoning uses requests of about 8K input and 16K output tokens. Both workloads use disaggregated serving, with generation under attention data and expert parallelism (DEP) or attention tensor and expert parallelism (TEP). AgentX replays 393 recorded multi-turn Claude Code sessions from the SemiAnalysis InferenceX benchmark on both models, using distributed weight data parallelism (DWDP) [10] for Kimi-K3. Attention kernel measurements cover 39 shapes per model, at KV lengths of 2K–128K and per-GPU batches of 8–1,024, with MTP=3 on DeepSeek-R1 and MTP=7 on Kimi-K3.

Accuracy

NVFP4-MLA matches BF16 MLA average accuracy on both models (§ 4.2.1), and white-box analysis of the cache shows that direct FP4 quantization loses precision on significant outliers in the RoPE channels, which the NVFP4 residual encodes more accurately than FP8 (§ 4.2.2).

Downstream accuracy

With NVFP4 attention BMM computation, NVFP4-MLA matches BF16 MLA average accuracy on both models (59.82 vs. 59.80 and 80.28 vs. 80.27; Table 1), compared with −0.06 and −0.33 percentage points (pp) for FP8 MLA.

Quantization error analysis

Measurement. To locate FP4 error in the MLA cache, we analyze dumped BF16 cache tensors across layers of DeepSeek-R1 and MLA layers of Kimi-K3. We measure reconstruction error as the normalized mean squared error NMSE = ∥X^−X∥F2/∥X∥F2\lVert\hat{X} - X \rVert_{F}^{2}/\lVert X \rVert_{F}^{2}. NoPE stays at NVFP4, and only the RoPE encoding varies: NVFP4, FP8 E4M3, or NVFP4 plus residual.

Error concentrates in RoPE. RoPE holds 64 of the 576 channels, yet compensating it alone lowers mean whole-cache NMSE by 60.3% on DeepSeek-R1 and 66.1% on Kimi-K3 (Figure 3a). Figure 4 contrasts the two partitions in the first few MLA layers of both models: the RoPE channels peak at 3.9–125× the NoPE maximum and vary sharply across tokens, whereas the NoPE latent stays flat.

Figure 3. Cache quantization error. A RoPE-only NVFP4 residual lowers whole-cache NMSE and is more accurate than FP8 RoPE in every layer. NoPE uses NVFP4; RoPE uses NVFP4, FP8 E4M3, or NVFP4 plus residual. (a) Whole-cache NMSE averaged over MLA layers. (b) Per-layer NMSE of each RoPE encoding (log scale).

Figure 4. RoPE and NoPE magnitudes. RoPE channels peak at 3.9–125× the NoPE maximum.

NVFP4 residual versus FP8 RoPE. The NVFP4 residual encodes RoPE more accurately than FP8 E4M3. Relative to plain NVFP4, it reduces mean RoPE NMSE by factors of 115.4 on DeepSeek-R1 and 116.1 on Kimi-K3, whereas FP8 RoPE leaves 9.5× and 9.1× as much error as the residual on average and 7.9–11.2× and 8.7–9.3× as much in every layer (Figure 3b).

Takeaway: NVFP4-MLA matches BF16 MLA average accuracy on both models while running both attention BMMs in NVFP4, whereas vanilla NVFP4 MLA drops by 5.71 and 16.48 pp. White-box analysis shows that this drop stems from significant outliers in the RoPE channels. Selective residual compensation therefore mitigates the accuracy drop with a small overhead (§ 4.3).

KV-cache memory reduction

Including the residual, NVFP4-MLA compresses the stored MLA cache by 3.20× and 3.05× relative to BF16 MLA and by 1.60× and 1.52× relative to FP8 MLA on DeepSeek-R1 and Kimi-K3 (Table 2). The RoPE residual accounts for only 10% of each NVFP4 layer’s cache.

Cache representationDeepSeek-R1Kimi-K3
Bytes/tokenReductionBytes/tokenReduction
BF16 MLA70,2721.00×27,6481.00×
FP8 MLA35,1362.00×13,8242.00×
NVFP4-MLA21,9603.20×9,0723.05×

Table 2. Stored MLA cache size. NVFP4-MLA compresses the stored MLA cache by 3.20× and 3.05× relative to BF16 MLA and by 1.60× and 1.52× relative to FP8 MLA. Bytes per cached token, summed over all MLA layers.

On Kimi-K3, the MLA cache shares the KV budget with the recurrent state of the KDA layers; under the same budget, NVFP4-MLA holds 47–49% more cached tokens than FP8 MLA, depending on the share taken by KDA state. § 4.4.1 shows how this capacity turns into prefix reuse.

Performance

We evaluate performance at three levels. In agentic serving, the smaller cache of NVFP4-MLA keeps more session histories and lowers p90 time to first token by up to 3.5× on DeepSeek-R1 and by 2.1× on Kimi-K3 (§ 4.4.1). Kernel measurements show an attention kernel that runs up to 2.82× and 2.71× faster than FP8 MLA on DeepSeek-R1 and Kimi-K3 (§ 4.4.2). In disaggregated DeepSeek-R1 serving, the faster kernel raises per-GPU throughput by up to 2.30× (§ 4.4.3).

Prefix-cache reuse

With its smaller cache (§ 4.3), NVFP4-MLA delays cache eviction as the number of concurrent sessions grows. Agentic sessions benefit from this capacity because context workers retain earlier turns’ KV state and prefill only newly appended tokens.

DeepSeek-R1. We replay the AgentX sessions of § 4.1 under an equal KV-cache budget of 43 GiB per context GPU, in which NVFP4-MLA holds 60% more cached tokens than FP8 MLA (§ 4.3) and the trace allows a hit rate of about 94–96%. FP8 MLA starts evicting at 32 sessions (Table 3): from 32 to 64 sessions, its hit rate falls from 87.6% to 54.1% and its p90 time to first token (TTFT) rises from 11.9 to 178 s. NVFP4-MLA keeps a higher hit rate and therefore repeats less prefill: with 48 sessions, it raises the hit rate from 77.0% to 87.1% and lowers p90 TTFT from 108 to 30.9 s, a 3.5× reduction, and the reduction is 1.5–3.5× across all points where FP8 MLA evicts.

ModelConcurrent sessionsp90 TTFT (s) FP8 MLAp90 TTFT (s) NVFP4-MLACache hit (%) FP8 MLACache hit (%) NVFP4-MLA
DeepSeek-R1162.981.9995.295.1
242.682.4193.694.4
3211.95.3687.692.4
4036.623.782.387.9
4810830.977.087.1
6417879.854.179.6
Kimi-K312811.05.2490.894.4

Table 3. AgentX serving. At equal KV-cache memory, NVFP4-MLA keeps higher cache-hit rates and lower p90 TTFT once FP8 MLA starts evicting histories. AgentX replay on Rubin GPUs. Kimi-K3 KV-cache memory also holds KDA state. Concurrent sessions: agent sessions kept alive; Cache hit: fraction of input tokens reused from the prefix cache.

Kimi-K3. We replay the same sessions on Kimi-K3 under an equal KV-cache budget of 47.4 GiB per context GPU, in which NVFP4-MLA holds 47–49% more cached tokens than FP8 MLA (§ 4.3) and the trace allows a hit rate of at most 95.4%. FP8 MLA already evicts with 128 sessions (Table 3). NVFP4-MLA keeps a higher hit rate and therefore repeats less prefill: it raises the hit rate from 90.8% to 94.4% and lowers p90 TTFT from 11.0 to 5.24 s, a 2.1× reduction.

Kernel performance

Besides its smaller cache, NVFP4-MLA shortens the attention computation itself, because both BMMs run on FP4 Tensor Cores. Its speedup over FP8 MLA grows with KV length, reaching 2.82× on DeepSeek-R1 and 2.71× on Kimi-K3.

MTP and the roofline. MTP=3 reuses each cache read across four query positions (Appendix § 8), which raises the arithmetic intensity of FP8 MLA from 484 to 1,934 FLOP/s per cache byte, beyond the 911 FLOP/B ridge point of a Rubin GPU; FP8 MLA decoding with MTP is therefore compute-bound. A storage-only alternative that dequantizes an NVFP4 cache to FP8 operands performs the same FP8 MMAs and softmax as FP8 MLA and shortens only cache reads that no longer limit the kernel, so it cannot finish faster than FP8 MLA. NVFP4-MLA is faster because its FP4 BMMs shorten the computation itself.

KV-length and batch sweep. Each measurement profiles a single attention kernel under a CUDA graph, with equal KV lengths across requests, and includes the split-KV reduction. FP8 MLA uses its faster split-KV strategy per shape; NVFP4-MLA its best CTA scheduling per shape. NVFP4-MLA is faster at every measured shape (Figure 5), by 1.311.31–2.82×2.82\times with a geometric mean of 1.87×1.87\times on the 39 DeepSeek-R1 shapes and by 1.351.35–2.71×2.71\times with a geometric mean of 1.99×1.99\times on the 39 Kimi-K3 shapes. The speedup generally grows with KV length, from 1.31.3–1.5×1.5\times at 2K, where fixed per-call cost dominates, to 2.22.2–2.8×2.8\times at 128K, where per-token attention work dominates.

Figure 5. Kernel speedup over FP8 MLA on one Rubin GPU. (a) DeepSeek-R1. (b) Kimi-K3.

End-to-end serving

End-to-end serving: NVFP4-MLA improves the throughput–latency trade-off of disaggregated DeepSeek-R1 serving, with gains that grow with context length.

Pareto measurement. Closed-loop clients sweep the number of in-flight requests for each generation layout, and each precision’s Pareto frontier takes, at every per-user generation rate, the highest per-GPU throughput of any of its layouts.

Throughput. At the same per-user generation rate, NVFP4-MLA serves more output tokens per GPU over most of the measured range (Figure 6): 26.7% more on MLPerf and 39.9% more on 8K/16K Reasoning on average, and up to 2.30×2.30\times and 2.25×2.25\times as many.

Figure 6. Serving Pareto frontiers. NVFP4-MLA raises per-GPU throughput over most of the range. Pareto frontiers for DeepSeek-R1 disaggregated serving on Rubin GPUs with MTP=3: (a) 8K/16K Reasoning; (b) MLPerf.

Parallel layouts. Both precisions reach their highest throughput with DEP16 up to about 250 tokens/s/user and with TEP8 at higher per-user rates.

Takeaway: NVFP4-MLA improves performance through two effects. Its smaller cache keeps more session histories, lowering p90 TTFT by up to 3.5×3.5\times on DeepSeek-R1 and 2.1×2.1\times on Kimi-K3 when GPU memory bounds the prefix cache. Its FP4 BMMs shorten attention computation, with speedups that grow with context length, reaching 2.82×2.82\times and 2.71×2.71\times at the kernel level and up to 2.30×2.30\times in per-GPU throughput for DeepSeek-R1 serving.

Related work

KV-cache quantization. Most previous work focuses on quantizing the KV cache to reduce its footprint and memory traffic. KIVI ([13]) quantizes keys per channel and values per token, keeping the most recent tokens in full precision. KVQuant ([6]) quantizes keys per channel before RoPE and keeps outliers in a sparse format. KVTuner (Li et al., 2025) searches layer-wise mixed precision for keys and values. RotateKV [22] and TurboQuant [26] rotate cached vectors to spread outliers before low-bit quantization. Serving systems such as Atom [30] and QServe [12] also store 4-bit KV caches alongside low-bit weights and activations. These methods target separate key and value caches rather than MLA’s shared latent representation. They also execute the attention BMMs at higher precision, which adds dequantization overhead for the cached values and limits MLA decoding once MTP makes it compute-bound (§ 4.4.2). NVFP4-MLA instead stores a shared MLA cache in NVFP4 whose scale blocks match both reduction directions, so both BMMs read it in FP4 without dequantization and run at FP4 Tensor Core throughput.

Quantized attention kernels. Low-precision attention kernels quantize attention operands to run the attention BMMs on faster low-precision Tensor Cores. SageAttention [28] quantizes queries and keys to INT8 for the QKTQK^{\mathsf{T}} BMM and keeps the PVPV BMM mostly in FP16. SageAttention3 [27] runs both BMMs in NVFP4, quantizing FP16 queries, keys, and values inside each attention call, and mainly targets image and video generation. SnapMLA [29] pushes MLA computation to FP8 with a per-token FP8 latent cache and keeps the RoPE channels in BF16. NVFP4-MLA pushes it further to FP4 and combines low-precision storage and computation: both BMMs consume the persistent NVFP4 MLA cache without dequantization, and the RoPE channels are quantized to NVFP4 with residual compensation rather than kept in high precision.

Evaluation scope. Prior KV-cache quantization studies evaluate non-MLA models of limited size, mostly with perplexity, short-generation tasks, or long-context retrieval [13, 6, 11]. These settings do not test long reasoning traces or multi-turn agentic sessions, where quantization error can accumulate over thousands of generated tokens. We instead evaluate large MLA reasoning models with a broader set of tests, spanning downstream accuracy on reasoning and agentic tasks, cache footprint, and kernel and end-to-end serving performance, to understand how NVFP4-MLA affects both model quality and efficiency.

Conclusion

We have presented NVFP4-MLA, which shows that NVFP4 can serve as both the storage format and the compute format of MLA. Selective residual compensation targets quantization outliers and greatly reduces the accuracy drop of direct FP4 quantization, and two-dimensional quantization blocks let one shared latent cache supply both BMMs. We also design a corresponding cache system and an efficient kernel pipeline. NVFP4-MLA matches BF16 MLA average accuracy on DeepSeek-R1 and Kimi-K3, speeds up the attention kernel over FP8 MLA by up to 2.82× and 2.71×, lowers p90 time to first token in agentic multi-turn serving by up to 3.5× and 2.1×, and raises per-GPU throughput in disaggregated DeepSeek-R1 serving by 26.7% and 39.9% on average on two workloads and by up to 2.30×. By computing attention in FP4 while preserving accuracy, NVFP4-MLA turns KV-cache compression into real inference speedups.

NVFP4 quantization

Let FE2M1\mathbb{F}_{\mathrm{E2M1}} be the set of E2M1 values, with largest magnitude ME2M1=6M_{\mathrm{E2M1}}=6, and let FE4M3\mathbb{F}_{\mathrm{E4M3}} be the set of finite E4M3 values, with largest magnitude ME4M3=448M_{\mathrm{E4M3}}=448. Let QE2M1\mathcal{Q}_{\mathrm{E2M1}} and QE4M3\mathcal{Q}_{\mathrm{E4M3}} round a real number onto FE2M1\mathbb{F}_{\mathrm{E2M1}} and FE4M3\mathbb{F}_{\mathrm{E4M3}}, respectively, with saturation; QE2M1\mathcal{Q}_{\mathrm{E2M1}} rounds to nearest with ties to even. An NVFP4 tensor stores an FP32 tensor scale γ>0\gamma>0, one E4M3 scale sbs_b for each block bb of 16 elements, and one E2M1 value xix_i for each element. For a real tensor XX and a partition B\mathcal{B} of its indices into 16-element blocks, NVFP4 quantization Q(⋅;B)\mathcal{Q}(\cdot;\mathcal{B}) is

sb=QE4M3(max⁡j∈b∣Xj∣ME2M1γ),xi=QE2M1(Xiγsb),Q(X;B)i=γsbxi,i∈b,(4)s_b=\mathcal{Q}_{\mathrm{E4M3}}\left(\frac{\max_{j\in b}|X_j|}{M_{\mathrm{E2M1}}\gamma}\right),\qquad x_i=\mathcal{Q}_{\mathrm{E2M1}}\left(\frac{X_i}{\gamma s_b}\right),\qquad\mathcal{Q}(X;\mathcal{B})_i=\gamma s_bx_i,\qquad i\in b, \tag*{(4)}

with xi=0x_i=0 whenever sb=0s_b=0. We identify an NVFP4 tensor with the real tensor that it represents, so sums and products of hatted (quantized) tensors denote exact real arithmetic. Block-scaled Tensor Core instructions evaluate these products from the stored values and scales with FP32 accumulation, without materializing a wider tensor. Q1×16\mathcal{Q}_{1\times16} and Q16×16\mathcal{Q}_{16\times16} apply Equation (4) with 1×161\times16 and 16×1616\times16 partitions, respectively. The tensor scale γ\gamma is a per-layer constant for each operand; for queries and the cache in both models, γ=ηXmax⁡/(ME2M1ME4M3)\gamma=\eta X_{\max}/(M_{\mathrm{E2M1}}M_{\mathrm{E4M3}}) is fixed offline from the largest magnitude Xmax⁡X_{\max} observed during calibration, with margin η>1\eta>1.

Arithmetic intensity of MLA decoding

MLA’s compact shared cache and wider BMMs after matrix absorption raise arithmetic intensity and make math throughput relevant alongside memory bandwidth.

A shared cache supports wider BMMs

Let TT denote the cached-token count, nhn_h the number of query heads, and ngn_g the number of GQA KV groups. With key and value widths dKd_K and dVd_V, one GQA query performs FF FLOPs on NN distinct cache elements:

FGQA=2nhT(dK+dV),NGQA=ngT(dK+dV).(5)F_{\mathrm{GQA}}=2n_hT(d_K+d_V),\qquad N_{\mathrm{GQA}}=n_gT(d_K+d_V). \tag*{(5)}

For MLA, BMM1 uses dc+dRd_c+d_R channels and BMM2 uses dcd_c:

FMLA=2nhT(2dc+dR),NMLA=T(dc+dR).(6)F_{\mathrm{MLA}}=2n_hT(2d_c+d_R),\qquad N_{\mathrm{MLA}}=T(d_c+d_R). \tag*{(6)}

The NoPE latent contributes to both products but occupies storage only once.

For DeepSeek’s nh=128n_h=128, dc=512d_c=512, and dR=64d_R=64 [3], we compare a GQA reference with ng=8n_g=8 and dK=dV=128d_K=d_V=128. At equal payload bytes β\beta per element, ideal intensity is I⋆=F/(βN)I^\star=F/(\beta N). Assuming one cold-cache read with reuse across heads and, for MLA, across BMM1 and BMM2 gives

FMLAFGQA=4.25,NGQANMLA≈3.56,IMLA⋆IGQA⋆=1369≈15.1.(7)\frac{F_{\mathrm{MLA}}}{F_{\mathrm{GQA}}}=4.25,\qquad\frac{N_{\mathrm{GQA}}}{N_{\mathrm{MLA}}}\approx3.56,\qquad\frac{I^\star_{\mathrm{MLA}}}{I^\star_{\mathrm{GQA}}}=\frac{136}{9}\approx15.1. \tag*{(7)}

MTP raises intensity through cache reuse

Let nqn_q query positions share a long prefix in one invocation, including the ordinary next-token position. Neglecting newly appended positions gives F(nq)≈nqF(1)F(n_q)\approx n_qF(1). With κ(nq)=Dcache(nq)/Dcache(1)\kappa(n_q)=D_{\mathrm{cache}}(n_q)/D_{\mathrm{cache}}(1) as cache traffic relative to a single query, excluding query-dependent transfers gives

I(nq)≈nqκ(nq)I(1).(8)I(n_q)\approx\frac{n_q}{\kappa(n_q)}I(1). \tag*{(8)}

Full reuse gives κ(nq)=1\kappa(n_q)=1 and an approximately nqn_q-fold increase; MTP=3 evaluates four query positions.

FP4 addresses both bandwidth and math throughput

For matrix-throughput ceiling πmat\pi_{\mathrm{mat}}, HBM bandwidth ωHBM\omega_{\mathrm{HBM}}, and HBM traffic DHBMD_{\mathrm{HBM}}, roofline analysis gives the latency lower bound [25]

t≥max⁡(Fπmat,DHBMωHBM).(9)t\ge\max\left(\frac{F}{\pi_{\mathrm{mat}}},\frac{D_{\mathrm{HBM}}}{\omega_{\mathrm{HBM}}}\right). \tag*{(9)}

FP4 storage reduces the memory term, and native FP4 BMMs reduce the compute term. With MTP, FP8 MLA decoding is compute-bound on the evaluated GPU (§ 4.4.2).

Block-local softmax-score scaling

Section 3.4 computes the E2M1 score values xix_i and the block scales sbs_b of Equation (3). Once mm is available, the kernel encodes the block scale with the tensor scale γP=1/(ME2M1ME4M3)\gamma_P = 1/(M_{\mathrm{E2M1}}M_{\mathrm{E4M3}}) of Equation (4):

sb=QE4M3(ME4M32a(mb−m)),p^i=γPsbxi,i∈b,(10)s_b = \mathcal{Q}_{\mathrm{E4M3}}\left(M_{\mathrm{E4M3}} 2^{a(m_b-m)}\right), \qquad\hat{p}_i = \gamma_P s_b x_i,\quad i \in b, \tag*{(10)}

where QE4M3\mathcal{Q}_{\mathrm{E4M3}} rounds to nearest with ties to even, as the native E4M3 conversion does. Because the values xix_i assume the unrounded scale, scale rounding multiplies all scores of a block by the common factor sb/(ME4M32a(mb−m))s_b/(M_{\mathrm{E4M3}}2^{a(m_b-m)}). Rather than absorbing 1/ℓ1/\ell into every block scale, the kernel keeps this reciprocal off the BMM2 critical path and divides the BMM2 output by ℓ\ell after BMM2, with ℓ\ell accumulated in FP32 from the unquantized scores p~i\tilde{p}_i.

Range and underflow of the softmax-score scale. Because mb≤mm_b \le m, the argument of QE4M3\mathcal{Q}_{\mathrm{E4M3}} in Equation (10) lies in (0,ME4M3](0,M_{\mathrm{E4M3}}]: γP\gamma_P maps the row maximum to the largest finite E4M3 value, and no block scale saturates. In the normal E4M3 range, QE4M3\mathcal{Q}_{\mathrm{E4M3}} changes a block’s scores by a common factor within 1±2−41 \pm2^{-4}. A block falls into the subnormal range, where the scale spacing is 2−92^{-9}, when 2a(mb−m)<2−6/ME4M3≈3.5×10−52^{a(m_b-m)} < 2^{-6}/M_{\mathrm{E4M3}} \approx3.5 \times10^{-5}. Round-to-nearest flushes the scale to zero when

2a(mb−m)≤τ=2−10ME4M3≈2.2×10−6,(11)2^{a(m_b-m)} \le\tau= \frac{2^{-10}}{M_{\mathrm{E4M3}}} \approx2.2 \times10^{-6}, \tag*{(11)}

that is, only when every score in the block is at most τ\tau times the row maximum. Let F\mathcal{F} be the set of tokens in flushed blocks. Because p~i≤τ\tilde{p}_i \le\tau for i∈Fi \in\mathcal{F} and ℓ≥1\ell\ge1, flushing removes at most

1ℓ∑i∈Fp~i≤∣F∣τ(12)\frac{1}{\ell}\sum_{i\in\mathcal{F}}\tilde{p}_i \le|\mathcal{F}|\tau \tag*{(12)}

of the normalized softmax mass. Within every retained block, E2M1 rounding already maps arguments of QE2M1\mathcal{Q}_{\mathrm{E2M1}} at or below 0.250.25 to zero and thus removes any score at or below 1/241/2^4 of its block maximum.

References

  1. [1]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ²-Bench: Evaluating conversational agents in a dual-control environment. In Proceedings of the 43rd International Conference on Machine Learning, 2026.arxiv.org/abs/2506.07982
  2. [2]DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a.arxiv.org/abs/2405.04434
  3. [3]DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024b.arxiv.org/abs/2412.19437
  4. [4]Vage Egiazarian, Roberto L. Castro, Denis Kuznedelev, Andrei Panferov, Eldar Kurtic, Shubh​​ra Pandit, Alexandre Noll Marques, Mark Kurtz, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Bridging the gap between promise and performance for microscaling FP4 quantization. In International Conference on Learning Representations, 2026.arxiv.org/abs/2509.23202
  5. [5]Daya Guo et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638, 2025. doi: 10.1038/s41586-025-09422-z.DOI
  6. [6]Coleman Hooper, Sehoon Kim, Hiva Mohammad​​zadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2401.18079
  7. [7]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling, 2024.arxiv.org/abs/2404.06654
  8. [8]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025.arxiv.org/abs/2403.07974
  9. [10]Wangqian Li, Jintao Peng, Zongfei Jing, Tianyu Zhang, Ze Long, Xianjie Qiao, Xiaoming Chen, Dongxu Yang, Kefeng Duan, and June Yang. DWDP: Distributed weight data parallelism for high-performance LLM inference on NVL72. arXiv preprint arXiv:2604.01621, 2026.arxiv.org/abs/2604.01621
  10. [11]Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Yiwu Yao, Wulong Liu, Sinno Jialin Pan, and Mingxuan Yuan. KVTuner: Sensitivity-aware layer-wise mixed-precision KV cache quantization for efficient and nearly lossless LLM inference. In Proceedings of the 42nd International Conference on Machine Learning, 2025.arxiv.org/abs/2502.04420
  11. [12]Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. In Proceedings of Machine Learning and Systems, 2025.
  12. [13]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, 2024.DOI
  13. [15]NVIDIA. Pretraining large language models with NVFP4. arXiv preprint arXiv:2509.25149, 2025.arxiv.org/abs/2509.25149
  14. [16]NVIDIA. TensorRT-LLM. GitHub repository, 2026. URL https://github.com/NVIDIA/TensorRT-LLM.
  15. [17]Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Hajishirzi. Generalizing verifiable instruction following. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2507.02833
  16. [18]Vijay Janapa Reddi et al. MLPerf inference benchmark. In ACM/IEEE 47th Annual International Symposium on Computer Architecture, 2020.
  17. [19]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
  18. [20]Bita Darvish Rouhani et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.arxiv.org/abs/2310.10537
  19. [21]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/j.neucom.2023.127063.DOI
  20. [22]Zunhai Su, Hanyu Wei, Zhe Chen, Wang Shen, Linge Li, Huangqi Yu, and Kehong Yuan. RotateKV: Accurate and robust 2-bit KV cache quantization for LLMs via outlier-aware adaptive rotations. In Proceedings of the 34th International Joint Conference on Artificial Intelligence, 2025.arxiv.org/abs/2501.16383
  21. [23]Minyang Tian et al. SciCode: A research coding benchmark curated by scientists. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2407.13168
  22. [24]Kiran Vodrahalli et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024.arxiv.org/abs/2409.12640
  23. [25]Samuel Williams, Andrew Waterman, and David A. Patterson. Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009. doi: 10.1145/1498765.1498785.DOI
  24. [26]Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. TurboQuant: Online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations, 2026.arxiv.org/abs/2504.19874
  25. [27]Jintao Zhang, Jia Wei, Haoxu Wang, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Kai Jiang, Jianfei Chen, and Jun Zhu. SageAttention3: Microscaling FP4 attention for inference and an exploration of 8-bit training. In Advances in Neural Information Processing Systems, 2025a.arxiv.org/abs/2505.11594
  26. [28]Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. SageAttention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations, 2025b.arxiv.org/abs/2410.02367
  27. [29]Yifan Zhang, Zunhai Su, Shuhao Hu, Rui Yang, Wei Wu, Yulei Qian, Yuchen Xie, and Xunliang Cai. SnapMLA: Efficient long-context MLA decoding via hardware-aware FP8 quantized pipelining. arXiv preprint arXiv:2602.10718, 2026.arxiv.org/abs/2602.10718
  28. [30]Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate LLM serving. In Proceedings of Machine Learning and Systems, 2024.arxiv.org/abs/2310.19102

Paper details

vlnc:W_7s8neh6b
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents