betaFeedback

WorksComputer Science

TraceKV: Policy-Preserving Low-Rank KV Cache Compression for Reasoning Vision-Language Models

Abstract

Reasoning vision-language models (VLMs) impose substantial KV cache demands from both large multimodal contexts and growing reasoning traces. Low-rank KV compression has emerged as a promising paradigm that stores low-dimensional latents of KV states and can be combined with quantization. However, existing low-rank methods often induce growing predictive policy drift from the dense model when compressing reasoning KV states, resulting in reasoning failures. We identify two key causes of this degradation: calibration-state mismatch between static contexts and on-policy reasoning states, and unequal error sensitivity across KV states. To address these challenges, we propose TraceKV, a training-free low-rank KV compression framework that unifies on-policy calibration, sensitivity-aware SVD, and global rank allocation under a single policy-preserving principle, substantially reducing policy-KL divergence during reasoning. Experiments across multiple VLMs and visual reasoning benchmarks demonstrate superior accuracy–memory trade-offs, achieving near-lossless reasoning accuracy with 5.64×\times KV cache compression when combined with quantization. Optimized low-rank attention kernels further deliver up to 1.83×\times end-to-end decoding speedup. Code will be available at https://github.com/AK-DREAM/TraceKV.

Introduction

Vision-language models (VLMs) are advancing beyond conventional visual understanding and direct answering toward complex multi-step tasks, such as visual mathematical and scientific reasoning [18, 19, 30]. Reasoning VLMs leverage extended chain-of-thought generation to address these challenges [3, 33], but at the cost of substantial inference overhead. In particular, their KV cache must store numerous visual and textual tokens from multimodal input contexts and continuously grow as new reasoning states are generated during autoregressive decoding. The resulting memory storage and bandwidth demands pose significant bottlenecks to efficient long-context inference, necessitating effective KV cache compression techniques.

Existing KV cache compression methods reduce cache size along different axes. Token eviction selectively retains past tokens [6, 14], while quantization lowers numerical precision to reduce per-token storage cost, but risks substantial accuracy loss at low bit-widths [11, 17]. Recently, low-rank compression has emerged as a promising paradigm, factorizing KV projection matrices to cache low-dimensional latents instead of full KV states [5, 7, 25]. Moreover, suitable transformations of these latents can further improve their quantization fidelity, enabling a more favorable accuracy–memory trade-off [7]. Existing VLM low-rank methods typically derive low-rank projections by minimizing KV reconstruction error over static multimodal calibration sequences, with evaluations primarily conducted on short-form or direct-answer tasks [23, 26].

Although previous low-rank methods effectively compress multimodal input states, we notice substantial degradation when applying them to reasoning VLMs, where lengthy generated states must also be compressed. As shown in Figure 1 (right), when evaluated on dense-model-generated prefixes, the next-token policy-KL divergence between the dense and compressed models grows sharply as generation length increases. End-to-end generation also reveals reasoning failures, such as logical errors and infinite loops. These findings suggest that static, reconstruction-centric compression does not necessarily preserve the model’s predictive policy throughout reasoning, raising a central question: How can low-rank KV compression preserve model’s reasoning policy while compressing KV states of both large multimodal contexts and growing reasoning traces under a constrained cache budget?

Figure 1. Comparison of compression paradigms and policy-KL divergence during reasoning. Left: Static, reconstruction-centric low-rank KV compression versus our on-policy, sensitivity-aware paradigm. Right: Next-token policy-KL divergence between dense and compressed models along dense-model-generated reasoning traces, evaluated on MMMU dataset using Qwen2.5-VL-7B-Instruct.

To understand this degradation, we conduct empirical analyses and identify two key issues: (1) Calibration-state mismatch: Static calibration on input multimodal contexts may not adequately cover self-generated reasoning KV states. We observe substantially higher reconstruction errors during reasoning than on input contexts, while incorporating ground-truth solution traces only partially alleviates this mismatch. (2) Unequal error sensitivities: Equal-magnitude errors at different KV states can have substantially different impacts on the model’s policy. In particular, many visual-token states exhibit lower policy-KL sensitivity than certain textual and reasoning states, with considerable variation even among reasoning tokens. Consequently, uniform reconstruction may fail to allocate limited low-rank capacity to the most policy-sensitive states.

Motivated by these observations, we propose TraceKV, a training-free framework guided by a unified policy-preserving principle that minimizes policy-KL. Based on Observation 1, we collect calibration activations from both multimodal contexts and on-policy traces generated by the dense VLM, improving alignment with inference-time states. Based on Observation 2, we estimate KV state sensitivities using the local Fisher information of policy-KL and formulate an on-policy, sensitivity-aware surrogate objective solvable by SVD. We further estimate the policy-KL importance of resulting singular components to globally allocate ranks across KV projections and layers. Together, our method addresses both state alignment and sensitivity-aware protection, as illustrated in Figure 1 (left), substantially reducing policy-KL divergence during reasoning (right).

Our contributions are summarized as follows:

  • Analysis. We analyze low-rank KV compression on reasoning VLMs and identify two sources of policy drift: calibration-state mismatch and unequal error sensitivity.

  • Method. We propose TraceKV, a training-free low-rank KV compression framework that unifies on-policy calibration, sensitivity-aware SVD, and global rank allocation under a single policy-preserving principle.

  • Results. Extensive experiments across multiple reasoning VLMs and visual reasoning benchmarks demonstrate superior accuracy over existing compression baselines at comparable memory budgets. TraceKV achieves 5.64×5.64\times KV-cache compression with near-lossless accuracy at 70% rank ratio with quantization, and up to 1.83×1.83\times end-to-end decoding speedup with optimized kernels.

Preliminaries

Low-Rank KV Cache Compression. Low-rank KV compression alleviates substantial KV cache memory overhead by factorizing the key and value projection matrices and caching low-dimensional latents. For a KV projection matrix W∈Rdin×doutW \in\mathbb{R}^{d_{\mathrm{in}}\times d_{\mathrm{out}}}, we construct a low-rank approximation W^=AB\widehat{W}=AB, where A∈Rdin×rA \in\mathbb{R}^{d_{\mathrm{in}}\times r} and B∈Rr×doutB \in\mathbb{R}^{r\times d_{\mathrm{out}}}, with r<doutr<d_{\mathrm{out}}. Given a token activation xtx_t, only the low-dimensional latent zt=xtAz_t=x_tA is cached, rather than the full KV state yt=xtWy_t=x_tW. This reduces the KV cache memory footprint by storing rr instead of doutd_{\mathrm{out}} elements per token. During attention computation, the pre-RoPE KV state is approximately reconstructed as y^t=ztB=xtW^\hat{y}_t=z_tB=x_t\widehat{W}.

Reconstruction-Based SVD. Existing SVD-based compression methods [7, 25] often formulate low-rank approximation as minimizing the KV reconstruction error over calibration activations. Let X∈RN×dinX \in\mathbb{R}^{N\times d_{\mathrm{in}}} denote the activation matrix collected from a calibration set, where NN is the total calibration tokens. The reconstruction objective is

min⁡rank⁡(W^)≤r∥X(W−W^)∥F2.(1)\min_{\operatorname{rank}(\widehat{W})\le r} \left\lVert X(W-\widehat{W})\right\rVert_F^2. \tag*{(1)}

This objective admits an optimal rank-rr solution via whitened SVD [25]. Specifically, a whitening transform SS is obtained from the Cholesky decomposition of the activation covariance matrix XTXX^{\mathsf{T}}X, followed by SVD of SW=UΣVTSW=U\Sigma V^{\mathsf{T}}. The optimal rank-rr approximation is then given by retaining the first rr singular components: A=S−1UrΣr1/2A=S^{-1}U_r\Sigma_r^{1/2} and B=Σr1/2VrTB=\Sigma_r^{1/2}V_r^{\mathsf{T}}, yielding W^=AB\widehat{W}=AB.

Policy-KL Divergence. We consider the discrepancy between the original model policy pp and the compressed model policy pδp_\delta over autoregressive reasoning states. Given a fixed prefix cc, the policy-KL divergence is measured by DKL(p(⋅∣c)∥pδ(⋅∣c))D_{\mathrm{KL}}(p(\cdot\mid c)\parallel p_\delta(\cdot\mid c)), where δ\delta denotes a perturbation to an internal model variable θ\theta. Since the policy-KL divergence vanishes at the original model (δ=0\delta=0) and its first-order derivative is zero, a second-order Taylor expansion around this point yields

DKL(p∥pδ)=12δ⊤Fθδ+o(∥δ∥2),(2)D_{\mathrm{KL}}(p\parallel p_\delta)=\frac{1}{2}\delta^\top F_\theta\delta+o(\lVert\delta\rVert^2), \tag*{(2)}

where FθF_\theta is the Fisher information matrix with respect to θ\theta:

Fθ=Ea∼p(⋅∣c)[∇θlog⁡p(a∣c)∇θlog⁡p(a∣c)⊤].(3)F_\theta=\mathbb{E}_{a\sim p(\cdot\mid c)}\left[\nabla_\theta\log p(a\mid c)\nabla_\theta\log p(a\mid c)^\top\right]. \tag*{(3)}

This establishes Fisher information as a local measure of policy-KL increase to perturbations, providing a principled basis for evaluating the impact of compression.

Motivation

As illustrated in Figure 1, we observe a progressively increasing policy-KL divergence between low-rank-compressed and dense models during reasoning. We investigate two underlying causes of this degradation through controlled analyses on Qwen2.5-VL-7B-Instruct [4].

Observation 1: Static calibration yields larger errors on reasoning KV states. Existing VLM low-rank compression methods typically construct calibration activation matrices from static input multimodal contexts to learn low-rank projections for accurate reconstruction. While effective for short-answer tasks, reasoning VLMs generate lengthy reasoning KV states throughout decoding, which are compressed using the same low-rank projections and repeatedly accessed in subsequent steps. This creates a potential mismatch between the calibrated low-rank subspaces and the actual inference-time reasoning states.

To examine this mismatch, Figure 2(a) visualizes the KV reconstruction error, et=∥xt(W−W^)∥2e_t=\lVert x_t(W-\widehat{W})\rVert_2, averaged over KV projections across all layers at each token position. Token positions are aligned at the input-generation boundary (yellow line), separating the input context tokens and generated tokens. As shown, while errors remain relatively low on input KV states, they increase sharply upon entering the reasoning phase and remain elevated throughout generation. As a result, these errors may accumulate during reasoning and lead to policy drift from the dense model.

Figure 2. Analysis of KV states on 32 MMMU samples (rows). (a) KV reconstruction errors at 50% rank ratio under input-context calibration, and (b) KV state sensitivities estimated using Fisher information, averaged over all KV projections. Token positions are aligned at the input-generation boundary (yellow line).

We further compare calibration strategies in Table 1. Adding ground-truth solution traces to the calibration activations reduces reasoning-state error from 10.27 to 9.67. However, such static traces remain insufficient to capture

Calibration DataInput ContextReasoning TraceOverall
Input Context8.0210.279.21
+ GT Solutions8.019.678.89
+ On-policy Traces7.968.888.45

Table 1. Average reconstruction error on MMMU at 50% rank ratio under different strategies. Calibration data is from ScienceQA.

the model’s actual reasoning distribution. Instead, we augment calibration with activations from self-generated reasoning traces of the dense model (i.e., on-policy traces), further reducing the error to 8.88 with negligible changes in input-context error. These findings motivate on-policy calibration to better preserve reasoning KV states.

Observation 2: KV states have unequal error sensitivities. While incorporating on-policy reasoning traces mitigates the calibration distribution mismatch, the conventional reconstruction objective in Eq. (1) still weights errors across all KV states uniformly. However, equal-magnitude KV perturbations may have vastly different impacts on subsequent reasoning: errors in redundant visual states may be inconsequential, whereas those in reasoning-critical states can substantially alter the generation trajectory. To investigate this heterogeneity, we quantify KV state sensitivities using estimated Fisher information (formally defined in § 4.2), which approximates the impact of local KV perturbations on the subsequent cumulative policy-KL divergence. As shown in Figure 2(b), sensitivity varies substantially across token positions: many visual input states, corresponding to the large central region on the left, exhibit low sensitivity, while certain textual input states and generated reasoning states on the right are considerably more sensitive. Moreover, sensitivity also varies notably within reasoning states. These findings suggest that the uniform reconstruction objective overlooks the unequal sensitivities of KV states, motivating sensitivity-aware protection.

Method

The above analyses suggest that the conventional KV reconstruction objective may be suboptimal. Motivated by knowledge distillation [10], we instead seek to preserve the dense model’s predictive policy under low-rank compression. For a KV projection matrix WW with rank budget rr, we formulate the ideal policy-KL objective as

min⁡rank⁡(W^)≤r Es∼dp[DKL(p(⋅∣s)∥pW^(⋅∣s))](4)\min_{\operatorname{rank}(\widehat{W})\le r}\ \mathbb{E}_{s\sim d_p}\left[D_{\mathrm{KL}}\left(p(\cdot\mid s)\mathbin\Vert p_{\widehat{W}}(\cdot\mid s)\right)\right] \tag*{(4)}

where dpd_p denotes the distribution of calibration prefixes along the dense model’s on-policy reasoning traces, and pW^p_{\widehat{W}} is the policy induced by replacing WW with its low-rank approximation W^\widehat{W}. Since this ideal objective admits no closed-form solution, we instead develop an on-policy, sensitivity-aware objective as a tractable surrogate solvable via SVD. We then extend the policy-preserving principle to global rank allocation across KV projections and layers. The overall framework is illustrated in Figure 3.

Figure 3. Overview of TraceKV. Left: The dense VLM generates on-policy traces to construct calibration data covering both multimodal contexts and reasoning states. Top: Monte Carlo sampling of next-token predictions provide score gradients for estimating policy-KL sensitivities. Bottom: KV-state sensitivities guide weighted SVD to obtain singular components, which are further evaluated by their policy-KL importance and selected under a shared global rank budget to construct the final low-rank projections.

On-Policy Calibration

Observation 1 reveals that calibration with on-policy reasoning traces yields lower reconstruction errors than using static input contexts, even when augmented with ground-truth solution traces. This suggests a distribution mismatch between predefined calibration data and the states encountered during the model’s own reasoning process. Motivated by this finding, we let the dense VLM autoregressively generate reasoning traces for image-question pairs in the calibration set. We then collect activations from both the multimodal input contexts and the self-generated reasoning states, forming the on-policy calibration matrix XonX_{\mathrm{on}} for each KV projection. Replacing the static calibration matrix XX in Eq. (1) with XonX_{\mathrm{on}} yields

min⁡rank⁡(W^)≤r∥Xon(W−W^)∥F2.(5)\min_{\operatorname{rank}(\widehat{W})\le r}\left\|X_{\mathrm{on}}(W-\widehat{W})\right\|_{F}^{2}. \tag*{(5)}

The generated prefixes are also retained for subsequent policy-KL sensitivity estimation. This procedure aligns calibration with the policy-KL objective in Eq. (4), as the next-token policy depends on both the multimodal input and the evolving reasoning prefix, where the collected on-policy prefixes provide an empirical approximation to dpd_p.

Sensitivity-aware SVD

Observation 2 reveals substantial heterogeneity in how KV-state perturbations affect the reasoning policy. We first formalize the sensitivity measure used in the analysis. A perturbation to a KV state can affect predictions at all subsequent positions. Building on the second-order policy-KL approximation in Eqs. (2) and (3), let $p_j = p(\cdot\mid cjc_j denote the dense policy at prefix cjc_j, and ht,j(a)=∇ytlog⁡pj(a)h_{t,j}(a)=\nabla_{y_t}\log p_j(a) denote the score gradient with respect to a KV state yty_t. We define its policy-KL sensitivity sts_t as

Ft=∑j∈J(t)Ea∼pj[ht,j(a)ht,j(a)⊤],st=12doutTr⁡(Ft),(6)F_t=\sum_{j\in\mathcal{J}(t)}\mathbb{E}_{a\sim p_j}\left[h_{t,j}(a)h_{t,j}(a)^{\top}\right],\qquad s_t=\frac{1}{2d_{\mathrm{out}}}\operatorname{Tr}(F_t), \tag*{(6)}

where J(t)\mathcal{J}(t) denotes all prediction positions after yty_t. Intuitively, since st=12doutTr⁡(Ft)=E∥u∥=1[12u⊤Ftu]s_t=\frac{1}{2d_{\mathrm{out}}}\operatorname{Tr}(F_t)=\mathbb{E}_{\lVert u\rVert=1}\left[\frac{1}{2}u^{\top}F_tu\right], according to Eq. (2), sts_t quantifies the expected second-order increase in cumulative policy-KL under an isotropic, unit-energy perturbation uu to yty_t.

Since explicitly computing the Fisher matrices FtF_t is prohibitively expensive, we employ Monte Carlo estimation. For each fixed on-policy prefix cjc_j, we independently sample aj(k)∼pja_j^{(k)}\sim p_j and construct L(k)=−∑jlog⁡pj(aj(k))\mathcal{L}^{(k)}=-\sum_j\log p_j(a_j^{(k)}). The resulting gradients gt(k)=∇ytL(k)g_t^{(k)}=\nabla_{y_t}\mathcal{L}^{(k)} yield

Tr⁡(Ft)=E∥gt∥22,s^t=12Kdout∑k=1K∥gt(k)∥22,(7)\operatorname{Tr}(F_t)=\mathbb{E}\lVert g_t\rVert_2^2,\qquad\hat{s}_t=\frac{1}{2Kd_{\mathrm{out}}}\sum_{k=1}^{K}\lVert g_t^{(k)}\rVert_2^2, \tag*{(7)}

where KK is the number of Monte Carlo samples. This estimator is unbiased since independently sampled score gradients across prediction positions have zero mean, causing cross-position terms to vanish in expectation.

The pronounced sensitivity variation revealed by Observation 2 motivates prioritizing fidelity for reasoning-critical KV states. Accordingly, we extend Eq. (5) to the final on-policy, sensitivity-aware objective:

min⁡rank⁡(W^)≤r∥D1/2Xon(W−W^)∥F2,(8)\min_{\operatorname{rank}(\widehat{W})\le r}\left\lVert D^{1/2}X_{\mathrm{on}}(W-\widehat{W})\right\rVert_F^2, \tag*{(8)}

where D=diag⁡(s^1,…,s^N)/(∑s^)D=\operatorname{diag}(\hat{s}_1,\ldots,\hat{s}_N)/(\sum\hat{s}) weights error of each KV state by its normalized policy-KL sensitivity. Similar to reconstruction-based SVD in § 2, this objective admits an optimal rank-rr solution. We first construct the weighted covariance C=Xon⊤DXonC=X_{\mathrm{on}}^{\top}DX_{\mathrm{on}} and obtain its Cholesky factor S⊤S=CS^{\top}S=C. We then perform SVD on the whitened weight matrix SW=UΣV⊤SW=U\Sigma V^{\top}, yielding a set of singular components ΔWi=S−1uiσivi⊤\Delta W_i=S^{-1}u_i\sigma_i v_i^{\top}. The leading rr components provide an optimal rank-rr solution to the above objective.

We further show Eq. (8) as a tractable surrogate of the ideal policy-KL objective in Eq. (4) and analyze the associated approximation errors in Sec. A.3.

Global Rank Allocation

The preceding method yields a fixed-rank surrogate solution for each KV projection by retaining its leading singular components. However, prior studies have shown that using uniform ranks across layers and KV projections under a global budget can be suboptimal, as their impact on model performance varies substantially [1, 7].

Following the same policy-preserving principle, we again leverage the Fisher quadratic approximation in Eqs. (2) and (3) to estimate the second-order policy-KL increase of removing a singular component as its importance. Let mm index KV projections across all layers, and ii index the singular components ΔWm,i\Delta W_{m,i} of projection WmW_m. Reusing the sampled NLL L(k)\mathcal{L}^{(k)} in § 4.2, we compute Gm(k)=∇WmL(k)G_m^{(k)}=\nabla_{W_m}\mathcal{L}^{(k)} and estimate the policy-KL importance of each singular component as

Im,i=12K∑k=1K⟨Gm(k),ΔWm,i⟩F2.(9)I_{m,i}=\frac{1}{2K}\sum_{k=1}^{K}\left\langle G_m^{(k)},\Delta W_{m,i}\right\rangle_F^2. \tag*{(9)}

We then globally select singular components with the highest importance scores under the total rank budget. Let Im\mathcal{I}_m denote the selected component indices for projection WmW_m, determining its rank rm=∣Im∣r_m=\lvert\mathcal{I}_m\rvert. The final low-rank factors are Am=Sm−1UmΣm1/2A_m=S_m^{-1}U_m\Sigma_m^{1/2} and Bm=Σm1/2Vm⊤B_m=\Sigma_m^{1/2}V_m^{\top}.

Low-Bit Quantization Integration

To further reduce KV cache memory, we apply low-bit quantization to the low-rank latents. The uneven singular values from SVD can concentrate energy in leading latent dimensions, introducing outliers that hinder quantization. Following prior works [7, 23], we apply an orthogonal Hadamard transform to balance latent magnitudes, similar to QuaRot [2]. This transformation can be fused into the low-rank factors without additional inference overhead. Further quantization details are provided in Sec. D.1.

System Implementation

Naively reconstructing and materializing the full KV cache at every decoding step would incur substantial computation and memory traffic. Thanks to our regular latent-cache structure and fixed computation pipeline, we develop an efficient low-rank attention implementation that translate memory savings into practical decoding acceleration.

For values, we aggregate low-rank latents using attention weights before reconstruction, avoiding full-sequence value-cache reconstruction. For keys, reconstruction must precede applying RoPE. We therefore implement custom Triton kernels that fuse key reconstruction, RoPE, and attention-logit computation, avoiding intermediate key-cache materialization and reducing memory traffic in bandwidth-bound decoding. For 4-bit quantization, dequantization is further fused into these kernels. Additional implementation details are provided in Sec. D.2.

Experiments

Experimental Setup

Models and Benchmarks. We evaluate our method on 4 reasoning-capable VLMs: Qwen2.5-VL-7B-Instruct [4], InternVL3-8B [33], and Qwen3-VL-8B/32B-Thinking [3]. We evaluate on three multimodal reasoning benchmarks: ScienceQA [18] for scientific reasoning, MathVista [19] for mathematical reasoning, and MMMU [30] for multidisciplinary reasoning over visual contexts. All evaluations are conducted using the lmms-eval framework [31] with reasoning-oriented prompts. Further details of the reasoning evaluation protocols are provided in Sec. B.

Baselines. We compare against general-purpose SVD-based compression methods, including SVD-LLM [25] and ZS-SVD [1], as well as low-rank KV cache compression methods, including Palu [7] and WSVD [23]. For a fair comparison, we restrict all methods to compressing only the key and value projection matrices. We also compare against two KV cache quantization baselines: token-wise RTN quantization and KIVI [17]. Token eviction methods are excluded as they are largely orthogonal.

Implementation Details. For baselines, we follow the official implementation of WSVD [23], which randomly samples 256 examples from the ScienceQA train split, collects calibration activations on input image-text contexts, and estimates calibration loss or Fisher information using ground-truth solutions. Our method uses the same calibration samples with the proposed on-policy, sensitivity-aware calibration. Following Palu [7], we adopt a head-group size of 4 for low-rank decomposition. When combined with quantization, our method employs simple per-token-latent asymmetric quantization without further grouping. All experiments are conducted on NVIDIA L40S GPUs.

Main Results

Low-Rank KV Compression. Table 2 compares low-rank-only KV compression methods. At 70% rank ratio, our method achieves nearly lossless compression, retaining 73.33% overall accuracy compared to 73.58% for the dense models. At 50% rank ratio, our method maintains 70.28% accuracy, surpassing the best baseline, Palu, by 7.09 p.p. The larger degradation of SVD-LLM and Palu suggests that static calibration and reconstruction-centric objectives are insufficient to preserve reasoning performance. WSVD suffers even more severe degradation, potentially due to its per-head SVD limiting cross-head component sharing [7]. Additional results under alternative baseline calibration protocols are provided in Sec. C.1.

MethodRankQwen2.5-VL-7BInternVL3-8BQwen3-VL-8BOverall Avg.
SciQAMathVMMMUSciQAMathVMMMUSciQAMathVMMMU
Dense100%87.8567.353.0093.1670.158.5693.6077.061.6773.58
SVD-LLM70%87.5662.446.6793.4565.754.8889.4971.054.1169.47
ZS-SVD70%85.1858.045.4493.9068.557.6791.1372.154.3369.58
Palu70%83.5463.850.8993.6967.658.2290.9874.159.1171.33
WSVD70%85.1861.047.2291.2763.856.3393.4674.758.1170.12
Ours70%87.8567.751.3393.6069.959.1193.9576.160.4473.33
SVD-LLM50%82.7554.637.8986.8651.144.6679.3859.440.2259.65
ZS-SVD50%83.0949.942.5689.0958.551.3366.2441.921.8956.06
Palu50%74.1755.143.7792.6159.952.8983.3961.345.5663.19
WSVD50%71.6440.037.3381.4136.136.4484.7851.538.8953.12
Ours50%87.4063.249.1192.9665.054.6792.7672.654.7870.28

Table 2. Accuracy (%) comparison of low-rank only KV compression methods on multimodal reasoning benchmarks. The rank ratio denotes the fraction of retained KV dimensions across all heads and layers.

Integration with Quantization. Table 3 compares KV quantization and low-rank KV compression combined with quantization. Naive token-wise RTN quantization at 4-bit leads to substantial accuracy degradation on Qwen2.5-VL-7B and InternVL3-8B. In contrast, our low-rank projections combined with Hadamard transform produce quantization-friendly KV latents, where 4-bit quantization introduces negligible additional accuracy loss over low-rank compression alone. At 70% rank ratio with 4-bit KV quantization, our method achieves 73.41% overall accuracy compared to 73.58% for the dense model, while reducing KV cache size by 5.64×5.64\times. This demonstrates the complementary benefits of low-rank compression and quantization. Even under aggressive 3-bit quantization, our method maintains competitive reasoning performance. Compared to KIVI, we achieve higher accuracy at comparable KV memory budgets without requiring per-channel key quantization or fine-grained element grouping.

MethodBitsComp.Qwen2.5-VL-7BInternVL3-8BQwen3-VL-8BOverall Avg.
SciQAMathVMMMUSciQAMathVMMMUSciQAMathVMMMU
Dense161.00×87.8567.353.0093.1670.158.5693.6077.061.6773.58
RTN-KV4 (gs32)43.20×46.6047.828.4477.2444.946.3393.1676.958.7857.79
KIVI-2 (gs32, r32)25.33×86.9665.851.1194.0569.056.3391.7776.058.5672.18
Palu (70%, KV4)45.64×86.0761.548.5693.1567.057.3391.2773.856.6770.59
Ours (70%, KV4)45.64×88.1067.551.7893.5569.658.6793.6576.761.1173.41
Ours (70%, KV3)37.44×88.2065.549.6793.6068.158.3393.8074.758.2272.24
Palu (50%, KV4)47.76×76.9552.941.1192.4061.152.1183.4961.645.3363.00
Ours (50%, KV4)47.76×87.4162.749.8993.4664.853.5692.4172.053.1169.93
Ours (50%, KV3)310.24×87.3662.547.5692.7664.352.3391.6767.550.4468.49

Table 3. Accuracy (%) comparison of KV cache compression methods with quantization. Comp. denotes the KV cache compression factor relative to the original BF16 cache (including quantization metadata). Overall Avg. is averaged across all models and benchmarks.

Evaluation on Larger Models. We further evaluate representative methods on Qwen3-VL-32B-Thinking [3], as

shown in Table 4. Notably, with 50% rank ratio and 4-bit latent quantization, TraceKV achieves an average accuracy of 78.98%, only 1.31 percentage points below the dense model. Interestingly, the accuracy degradation is smaller than that observed on Qwen3-VL-8B-Thinking under the same configuration, suggesting that larger reasoning VLMs may tolerate more aggressive KV compression. These results highlight our method’s scalability for efficient inference at larger model scales.

MethodComp.SciQAMathVMMMUAvg.
Dense1.00×1.00\times93.8080.466.6780.29
SVD-LLM (50%)2.00×2.00\times93.9570.859.4474.73
Palu (50%)2.00×2.00\times91.2775.955.5674.24
Ours (50%)2.00×2.00\times94.0078.664.0078.87
Palu (50%, KV4)7.76×7.76\times90.2375.556.1173.95
Ours (50%, KV4)7.76×7.76\times93.9579.263.7878.98

Table 4. Results on Qwen3-VL-32B-Thinking. Comp. denotes the KV cache compression factor.

Ablation and Analysis

Component Ablation. Table 5 evaluates the contribution of each component of our method. To isolate the effect of rank allocation, we use uniform ranks across projections for all variants except the full method. Incorporating ground-truth solution traces into calibration statistics only moderately improves average accuracy from 58.41% to 60.61%. In contrast, calibrating on on-policy reasoning traces achieves a larger improvement to 62.09%, highlighting the importance of capturing the actual inference-time reasoning distribution. Sensitivity-aware SVD further increases accuracy to 63.68% by prioritizing reasoning-critical KV states during compression. Finally, our global rank allocation brings the full method to 66.57% accuracy, outperforming the baseline by 8.16 p.p.

VariantSciQAMathVMMMUAvg.
Baseline (SVD-LLM)82.7554.6037.8958.41
+ GT Solutions83.7457.2040.8960.61
+ On-policy Traces85.5757.6043.1162.09
+ On-policy & Sensitivity86.8158.8045.4463.68
Ours (Full)87.4063.2049.1166.57

Table 5. Component ablation on Qwen2.5-VL-7B-Instruct at 50% rank ratio without KV quantization.

Compression Error Analysis. To intuitively understand the advantages of our proposed methods, Figure 4 visualizes the policy-KL divergence and token-wise reconstruction error under different compression strategies. As shown in (a), the SVD-LLM baseline exhibits low reconstruction error on the input context but a sharp increase upon entering the reasoning phase, accompanied by rapidly growing policy-KL throughout generation. Incorporating ground-truth solution traces in (b) partially alleviates these issues. By calibrating on on-policy reasoning traces, (c) substantially reduces both policy-KL divergence and reconstruction error during later generation steps. Finally, incorporating sensitivity-aware SVD and global rank allocation in (d) redistributes fidelity: errors increase on less sensitive visual states, while critical KV states are preserved with lower errors (deeper blue). Notably, we observe that initial sink tokens consistently exhibit very low errors due to their high sensitivity, consistent with prior observations and methods that explicitly preserve sink tokens [11, 28]. Consequently, our full method exhibits the slowest growth in policy-KL divergence, better preserving the dense model’s reasoning policy.

Figure 4. Visualization of policy-KL divergence and reconstruction error under different compression strategies. Evaluated using Qwen2.5-VL-7B-Instruct at 50% rank ratio on MMMU.

Calibration Domain Shift. To assess robustness to calibration domain shift, we further evaluate the use of an alternative calibration dataset. We sample 256 examples from a held-out split of MathVista and follow the same calibration procedure, with results reported in Table 6. As shown, in-domain calibration yields higher accuracy compared to cross-domain calibration, improving our method by 2.33 and 2.10 percentage points on ScienceQA and MathVista, respectively. Nevertheless, our method consistently outperforms the baseline across all three evaluation benchmarks regardless of the calibration dataset, demonstrating its effectiveness under domain shift. This cross-domain effectiveness is further supported by the component ablation in Table 5 and the visualization analysis in Figure 4, which also demonstrate improvements on MathVista and MMMU despite using ScienceQA for calibration.

Calib. SetMethodSciQAMathVMMMUAvg.
ScienceQASVD-LLM82.7554.6037.8958.41
Ours87.4063.2049.1166.57
MathVistaSVD-LLM81.7059.6039.3360.21
Ours85.0765.3048.1166.16

Table 6. Accuracy (%) with different calibration sets on Qwen2.5-VL-7B-Instruct at 50% rank ratio.

System Efficiency

We measure decoding latency on Qwen2.5-VL-7B-Instruct with a batch size of 16 and context lengths ranging from 4K to 32K. The dense model uses the default Hugging Face implementation with FlashAttention-2 [9], while our method employs 50% low-rank KV compression with the optimized kernels described in § 4.5. Figure 5 presents the attention-module and end-to-end speedups across different context lengths. At 16K context length, our method achieves 1.53×1.53\times attention speedup and 1.38×1.38\times end-to-end speedup, further increasing to 2.42×2.42\times and 1.83×1.83\times with 4-bit quantization, while reducing peak GPU memory usage from 32.13 GB to 18.57 GB. These gains arise from reduced KV cache memory traffic and efficient low-rank attention execution, highlighting the potential of our approach for efficient long-context and long-reasoning inference.

Figure 5. Decoding speedup on Qwen2.5-VL-7B (batch size 16). At 32K, dense model runs out of memory; dashed lines and hollow markers denote speedups estimated by linear extrapolation.

Related Work

KV Cache Compression. KV cache compression has been explored through architectural designs and post-training techniques. Multi-Head Latent Attention [15] (MLA) incorporates learned low-rank KV bottlenecks into model architectures. In contrast, post-training methods reduce KV memory through token eviction [6, 14], low-bit quantization [11, 17], or low-rank projection [7, 23].

Low-Rank Compression Methods. SVD-LLM [25] improves over previous SVD-based methods [12, 29] through activation whitening, while subsequent methods further explore adaptive rank allocation [1, 7, 26]. However, their static calibration and reconstruction-centric objectives do not necessarily preserve reasoning policy. Recent and concurrent methods explore alternative objectives, such as supervised task loss [24] or attention-level output [16]. We instead leverage policy-KL sensitivity to guide low-rank factorization and rank allocation during VLM reasoning.

Calibration for Post-Training Compression. While conventional methods typically rely on static calibration data, Self-calibration [27] and RAC [20] incorporate model-generated data and on-policy traces into calibration for model pruning. Separately, FWSVD [12] and MBQ [13] leverage loss-based sensitivities for weight compression and VLM quantization, respectively. Our work investigates related challenges in the context of low-rank KV compression for reasoning VLMs, using a unified policy-preserving principle to explain and guide our method.

Conclusion

We introduce TraceKV, a training-free low-rank KV cache compression framework for reasoning VLMs that employs on-policy, sensitivity-aware compression for preserving model policy. Extensive experiments demonstrate superior accuracy–memory trade-offs, effective integration with quantization, and practical decoding speedup, highlighting its potential for efficient reasoning VLM inference.

Theoretical Analysis

KV-State Sensitivity

We revisit the derivation of KV-state sensitivity in § 4.2 and establish the unbiasedness of Monte Carlo estimation. Consider a perturbation δyt\delta y_t to a KV state yty_t. Let pj=p(⋅∣cj)p_j = p(\cdot\mid c_j) and pj,δp_{j,\delta} denote the dense and perturbed policies at prefix cjc_j, respectively. Aggregating the policy-KL divergence over subsequent prediction positions and applying the second-order approximation in Eq. (2), we obtain

Dt(δyt)=∑j∈J(t)DKL(pj∥pj,δ)=12δyt⊤Ftδyt+o(∥δyt∥2),D_t(\delta y_t) = \sum_{j\in\mathcal{J}(t)} D_{\mathrm{KL}}(p_j\Vert p_{j,\delta}) = \frac{1}{2}\delta y_t^{\top} F_t \delta y_t + o(\lVert\delta y_t\rVert^2),
Ft=∑j∈J(t)Eaj∼pj[ht,j(aj)ht,j(aj)⊤],(10)F_t = \sum_{j\in\mathcal{J}(t)} \mathbb{E}_{a_j\sim p_j}\left[h_{t,j}(a_j)h_{t,j}(a_j)^{\top}\right], \tag*{(10)}

where J(t)\mathcal{J}(t) denotes all prediction positions after yty_t, ht,j(a)=∇ytlog⁡pj(a)h_{t,j}(a) = \nabla_{y_t}\log p_j(a). For an isotropic, unit-energy perturbation uu to yty_t, uniformly sampled from the unit sphere, we have E[uu⊤]=I/dout\mathbb{E}[uu^{\top}] = I/d_{\mathrm{out}}. The expected second-order policy-KL increase is therefore

E∥u∥=1[12u⊤Ftu]=12doutTr⁡(Ft)=st,(11)\mathbb{E}_{\lVert u\rVert=1}\left[\frac{1}{2}u^{\top}F_tu\right] = \frac{1}{2d_{\mathrm{out}}}\operatorname{Tr}(F_t) = s_t, \tag*{(11)}

recovering the sensitivity definition in Eq. (6). To estimate sts_t, we independently sample aj(k)∼pja_j^{(k)}\sim p_j at each fixed on-policy prefix and compute the NLL gradient gt(k)=−∑jht,j(aj(k))g_t^{(k)} = -\sum_j h_{t,j}(a_j^{(k)}). Since score gradients have zero expectation, Eaj∼pj[ht,j(aj)]=0\mathbb{E}_{a_j\sim p_j}[h_{t,j}(a_j)] = 0, independence across sampled tokens eliminates the cross-position terms:

E[∥gt∥2]=∑j∈J(t)E[∥ht,j(aj)∥2]=Tr⁡(Ft).(12)\mathbb{E}[\lVert g_t\rVert^2] = \sum_{j\in\mathcal{J}(t)}\mathbb{E}[\lVert h_{t,j}(a_j)\rVert^2] = \operatorname{Tr}(F_t). \tag*{(12)}

Consequently, the estimator in Eq. (7) satisfies

E[s^t]=12Kdout∑k=1KE[∥gt∥2]=st,(13)\mathbb{E}[\hat{s}_t] = \frac{1}{2Kd_{\mathrm{out}}}\sum_{k=1}^{K}\mathbb{E}[\lVert g_t\rVert^2] = s_t, \tag*{(13)}

proving that the Monte Carlo estimator is unbiased.

Singular-Component Importance

We derive the policy-KL importance of singular components used for global rank allocation in Eq. (9). For a KV projection WmW_m, removing a singular component ΔWm,i\Delta W_{m,i} corresponds to a weight perturbation −ΔWm,i-\Delta W_{m,i}. Following the second-order policy-KL approximation, the resulting policy divergence over calibration prefixes is

Dm,i=∑jDKL(pj∥pj,−ΔWm,i)≈12vec⁡(ΔWm,i)⊤Fmvec⁡(ΔWm,i),(14)D_{m,i} = \sum_j D_{\mathrm{KL}}(p_j\Vert p_{j,-\Delta W_{m,i}}) \approx\frac{1}{2}\operatorname{vec}(\Delta W_{m,i})^{\top}F_m\operatorname{vec}(\Delta W_{m,i}), \tag*{(14)}

where pj,−ΔWm,ip_{j,-\Delta W_{m,i}} denotes the perturbed policy and FmF_m is the Fisher matrix with respect to WmW_m. Let hm,j(aj)=∇Wmlog⁡pj(aj)h_{m,j}(a_j) = \nabla_{W_m}\log p_j(a_j). Expanding FmF_m using Eq. (3) yields the component importance

Im,i=12∑jEaj∼pj[⟨hm,j(aj),ΔWm,i⟩F2].(15)I_{m,i} = \frac{1}{2}\sum_j\mathbb{E}_{a_j\sim p_j}\left[\langle h_{m,j}(a_j),\Delta W_{m,i}\rangle_F^2\right]. \tag*{(15)}

As in KV-state sensitivity estimation, we use independently sampled next-token actions and the aggregated NLL gradients Gm(k)=−∑jhm,j(aj(k))G_m^{(k)} = -\sum_j h_{m,j}(a_j^{(k)}) to obtain

I^m,i=12K∑k=1K⟨Gm(k),ΔWm,i⟩F2.(16)\hat{I}_{m,i} = \frac{1}{2K}\sum_{k=1}^{K}\left\langle G_m^{(k)},\Delta W_{m,i}\right\rangle_F^2. \tag*{(16)}

The estimation is unbiased by the same zero-mean score property and independence argument established in Sec. A.1. Thus, Im,iI_{m,i} measures the second-order policy-KL impact of removing a singular component. Although jointly removing multiple singular components may introduce cross terms in the second-order policy-KL approximation, we adopt individual importance scores as a tractable additive proxy for global rank allocation.

Policy-KL Surrogate and Approx Errors

While the main paper develops our on-policy, sensitivity-aware objective in Eq. (8) based on our two observations, in this section, we reversely derive how this objective serves as a surrogate of the ideal policy-KL objective in Eq. (4) for each KV projection matrix, and characterize the associated approximation errors. Consider a KV projection WW and its low-rank approximation W~=W+ΔW\widetilde{W} = W + \Delta W. Using the collected on-policy calibration prefixes {cj}j=1N\{c_j\}_{j=1}^{N} to approximate the prefix distribution dpd_p, we define the empirical policy-KL divergence as D(W′)=∑jDKL(pj∥pjW′)\mathcal{D}(W') = \sum_j D_{\mathrm{KL}}(p_j\Vert p_j^{W'}). Expanding around the dense weights yields

D(W+ΔW^)=12vec⁡(ΔW^)⊤FWvec⁡(ΔW^)+o(∥ΔW^∥F2)=12∑t,uδyt⊤Ftuδyu+o(∥ΔW^∥F2),(17)\begin{aligned} \mathcal{D}(W+\Delta\widehat{W})=\frac{1}{2}\operatorname{vec}(\Delta\widehat{W})^{\top}F_W\operatorname{vec}(\Delta\widehat{W})+o(\lVert\Delta\widehat{W}\rVert_F^2) \\ &=\frac{1}{2}\sum_{t,u}\delta y_t^{\top}F_{tu}\delta y_u+o(\lVert\Delta\widehat{W}\rVert_F^2), \tag*{(17)} \end{aligned}

where δyt=(xtΔW^)⊤\delta y_t=(x_t\Delta\widehat{W})^{\top} is the perturbation to KV state yty_t, FWF_W is the Fisher matrix with respect to WW, and FtuF_{tu} denotes the (t,u)(t,u) block of the joint Fisher matrix over KV states. The second equality expresses the weight perturbation through its induced KV-state perturbations.

To derive Eq. (8) from this form, two approximations are introduced. First, we neglect cross-state interactions by retaining only the diagonal Fisher blocks FttF_{tt}. Second, we approximate each block isotropically as 12Ftt≈stI\frac{1}{2}F_{tt}\approx s_t I, where st=Tr⁡(Ftt)/(2dout)s_t=\operatorname{Tr}(F_{tt})/(2d_{\mathrm{out}}). Consequently,

D(W+ΔW^)≈12∑tδyt⊤Fttδyt≈∑tst∥δyt∥22=∥D1/2Xon(W−W^)∥F2,(18)\begin{aligned} \mathcal{D}(W+\Delta\widehat{W})&\approx\frac{1}{2}\sum_t\delta y_t^{\top}F_{tt}\delta y_t\\ &\approx\sum_t s_t\lVert\delta y_t\rVert_2^2 \\ &=\lVert D^{1/2}X_{\mathrm{on}}(W-\widehat{W})\rVert_F^2, \tag*{(18)} \end{aligned}

where D=diag⁡(s1,…,sN)D=\operatorname{diag}(s_1,\ldots,s_N). This recovers our on-policy, sensitivity-aware objective in Eq. (8) as a surrogate.

More precisely, denoting this surrogate by R(W^)\mathcal{R}(\widehat{W}), the approximation error terms are

D(W+ΔW^)−R(W^)=12∑t≠uδyt⊤Ftuδyu⏟Ecross+12∑tδyt⊤(Ftt−2stI)δyt⏟Eanis+o(∥ΔW^∥F2).(19)\begin{aligned} \mathcal{D}(W+\Delta\widehat{W})-\mathcal{R}(\widehat{W}) &=\underbrace{\frac{1}{2}\sum_{t\ne u}\delta y_t^{\top}F_{tu}\delta y_u}_{E_{\mathrm{cross}}}\\ &\quad+\underbrace{\frac{1}{2}\sum_t\delta y_t^{\top}(F_{tt}-2s_tI)\delta y_t}_{E_{\mathrm{anis}}}\\ &\quad+o(\lVert\Delta\widehat{W}\rVert_F^2). \tag*{(19)} \end{aligned}

The approximation errors arise from cross-state Fisher interactions, directional anisotropy within individual Fisher blocks, and higher-order effects beyond the local second-order Taylor approximation. For simplicity, we omit the Monte Carlo estimation error of sts_t here.

Optimality of Sensitivity-Weighted SVD

We show that Eq. (8) admits an optimal rank-rr solution via SVD. Let C=Xon⊤DXon=S⊤SC=X_{\mathrm{on}}^{\top}DX_{\mathrm{on}}=S^{\top}S, where SS is the Cholesky factor of the weighted covariance CC. The objective can be rewritten as

∥D1/2Xon(W−W^)∥F2=∥S(W−W^)∥F2=∥SW−SW^∥F2.(20)\lVert D^{1/2}X_{\mathrm{on}}(W-\widehat{W})\rVert_F^2 =\lVert S(W-\widehat{W})\rVert_F^2 =\lVert SW-S\widehat{W}\rVert_F^2. \tag*{(20)}

The first equality follows from the trace representation of the Frobenius norm. Assuming SS is invertible with damping factor, let Z=SW^Z=S\widehat{W}, which preserves the rank constraint. By the Eckart–Young–Mirsky theorem, the optimal rank-rr approximation to SW=UΣV⊤SW=U\Sigma V^{\top} is Z⋆=UrΣrVr⊤Z^\star=U_r\Sigma_rV_r^{\top}. Therefore,

W^⋆=S−1UrΣrVr⊤=∑i=1rΔWi.(21)\widehat{W}^{\star}=S^{-1}U_r\Sigma_rV_r^{\top}=\sum_{i=1}^{r}\Delta W_i. \tag*{(21)}

where ΔWi=S−1uiσivi⊤\Delta W_i=S^{-1}u_i\sigma_i v_i^{\top} denotes the ii-th singular component. Retaining the leading rr components with the largest singular values minimizes this objective for a fixed rank.

Thus, solving this surrogate objective yields a set of candidate singular components, whose squared singular values quantify their contributions to the surrogate error. Our global rank allocation then directly estimates the second-order impact of removing each component on the policy-KL divergence and uses these importance scores to determine which components to retain under a global budget. Since the surrogate error relies on simplifying approximations, the squared singular values and the policy-KL importance scores are not necessarily equivalent.

Calibration and Evaluation Protocols

Calibration Protocol. We use 256 fixed samples from the ScienceQA train split for calibration. For each model, we generate its own on-policy reasoning traces using greedy decoding, with a maximum of 512 generated tokens per sample. We collect all activations processed by each KV projection, including system prompts, multimodal input contexts, and generated reasoning tokens. For Fisher estimation, we independently sample next-token actions from the dense model’s predictive distributions at all generation positions, using K=8K=8 Monte Carlo samples by default, with a fixed random seed of 0.

The calibration overhead primarily comes from generating on-policy reasoning traces for 256 examples and performing K=8K=8 backward passes, taking approximately 30 minutes for a 7–8B model on a single NVIDIA L40S GPU.

Evaluation Protocol. We evaluate on the test split of ScienceQA (2017 samples), the testmini split of MathVista (1000 samples), and the validation split of MMMU (900 samples). All models use greedy decoding with a maximum of 4,096 generated tokens. We adopt the answer extraction and scoring procedures of lmms-eval [31], which extract answers from the final <answer>...</answer> tags. when available and apply the benchmark-specific fallback procedures otherwise. The same prompting protocol is used for calibration and evaluation.

Reasoning Prompts. All models share the following system prompt, with minor modifications to the default reasoning prompt of lmms-eval:

You are a helpful assistant. When the user asks a question, your response must include two parts: first, the reasoning process enclosed in <think>...</think> tags, then the final answer enclosed in <answer>...</answer> tags. For multiple-choice questions, put only the option letter inside <answer>, for example: <answer>B</answer>. Otherwise, provide a clear, concise response within <answer> </answer> tags that directly addresses the question.

For Qwen3-VL-8B/32B-Thinking, we additionally append the following instruction to the user prompt:

Reason concisely in a few sentences. After </think>, directly output the requested <answer>...</answer> with no additional explanation.

Additional Results and Analysis

Alternative Baseline Calibration

In our main experiments, we adopt a example-matched calibration protocol, where all methods use the same 256 ScienceQA train split examples. To further isolate the effects of calibration token count and reasoning-state coverage, we evaluate representative baselines under two additional protocols for collecting calibration activations: (a) Token-matched calibration, which adds more input-context examples until the total calibration token count matches our on-policy calibration; and (b) GT-solution calibration, which augments the original input contexts with ground-truth solution traces from the same 256 examples. For reference, the original 256 ScienceQA examples contain 171.7K tokens with input-only calibration, 190.9K tokens with GT solutions, and 200.3K tokens with on-policy reasoning traces.

As shown in Table 7, simply increasing the number of calibration tokens does not yield consistent improvement, while incorporating ground-truth solutions provides moderate gains. Nevertheless, our method consistently outperforms these enhanced baselines, highlighting the importance of on-policy calibration beyond increased token coverage. The component ablation in Table 5 and visualization analysis in Figure 4 further support that aligning calibration with on-policy reasoning states contributes the most to preserving reasoning performance.

MethodSciQAMathVMMMUAvg.
SVD-LLM82.7554.637.8958.41
SVD-LLM†81.3654.339.2258.29
SVD-LLM‡83.7457.240.8960.61
Palu74.1755.143.7757.68
Palu†75.7653.742.5657.34
Palu‡84.8658.546.1163.16
Ours87.4063.249.1166.57

Table 7. Alternative calibration protocols on Qwen2.5-VL-7B at 50% rank ratio. † denotes token-matched input-context calibration; ‡ denotes calibration with ground-truth solutions.

Monte Carlo Estimation Stability

We examine the stability of our Monte Carlo estimators with K∈{1,2,4,8,16}K \in\{1, 2, 4, 8, 16\}, using an independent estimate with Kref=64K_{\mathrm{ref}} = 64 as reference. For sensitivity-weighted covariance, we compute the relative Frobenius error for each KV projection and average across all projections. For singular-component importance, we measure the global Spearman rank correlation across all components, with the SVD bases held fixed. Results are averaged over three random seeds with standard deviation.

As shown in Figure 6, the covariance estimation error decreases steadily as KK increases, reaching approximately 2.4% at the default K=8K = 8. Meanwhile, component importance rankings remain highly consistent, with Spearman correlations exceeding 0.998 even at K=1K = 1 and approaching 1 as KK increases. These results demonstrate the stability of our Monte Carlo estimators and support the use of a modest sample count (K=8K = 8) in practice.

Figure 6. Monte Carlo estimation stability on Qwen2.5-VL-7B-Instruct. (a) Mean relative Frobenius error of sensitivity-weighted covariance matrices. (b) Global Spearman correlation of singular-component importance scores. Results show mean ± standard deviation over three random seeds, using Kref=64K_{\mathrm{ref}} = 64 as reference.

Generation Length on Thinking Models

We analyze the generation lengths and truncation rates of Qwen3-VL-8B-Thinking in Table 8. Unlike general-purpose VLMs such as Qwen2.5-VL-7B-Instruct, we observe that errors on difficult questions in this thinking-tuned model often manifest as repetitive reasoning without producing a certain answer, rather than terminating with an incorrect answer [21]. Non-truncated generations are typically much shorter (e.g., only avg. 680 tokens for the dense model on MMMU), and increasing the token limit to 8,192 yields little improvements, suggesting that such failures are not merely caused by an insufficient token budget.

MethodScienceQAMathVistaMMMU
Dense214 / 1.59%700 / 6.50%1,280 / 17.56%
SVD-LLM443 / 10.91%1,343 / 18.00%1,943 / 33.44%
Palu396 / 7.59%1,456 / 20.50%1,871 / 30.89%
Ours260 / 2.93%1,121 / 10.40%1,559 / 23.22%
Palu (KV4)407 / 7.93%1,463 / 20.80%1,959 / 31.33%
Ours (KV4)276 / 3.07%1,170 / 11.70%1,637 / 25.67%

Table 8. Generation length and truncation analysis on Qwen3-VL-8B-Thinking at 50% rank ratio. Each entry reports average generated tokens / truncation rate under a 4,096-token limit.

KV compression can further exacerbate such behavior. On MMMU at 50% rank ratio, SVD-LLM increases the average generation length from 1,280 to 1,943 tokens, with the truncation rate rising from 17.56% to 33.44%. In contrast, TraceKV limits the average length to 1,559 tokens and the truncation rate to 23.22%, substantially mitigating this issue. Similar trends hold across other benchmarks and quantized settings. These findings suggest that our method better preserves the original reasoning behavior, while mitigating the additional per-sample decoding overhead associated with prolonged reasoning failures.

Analysis on KV-State Sensitivity

Figure 7 provides a qualitative visualization of policy-KL sensitivities of KV states. As shown, high sensitivity among visual tokens is concentrated in a few regions containing question-relevant numerical information, while most visual regions exhibit substantially lower sensitivity. In contrast, textual tokens generally receive higher sensitivity scores, particularly reasoning-critical tokens in both the input context and generated reasoning trace. Notably, the initial attention sink token exhibits exceptionally high sensitivity, consistent with prior findings on attention sinks [11, 28]. In practice, the sink token has an identical activation across different samples, and its KV state can theoretically be reconstructed with a single rank-one component, allowing the remaining rank budget to accommodate other tokens despite the sink token’s disproportionately high sensitivity.

Figure 7. Visualization of KV-state sensitivity for a ScienceQA example on Qwen2.5-VL-7B-Instruct, measured at the value projection of layer 14.

Analysis on Rank Allocation

Figure 8 visualizes the layer-wise rank allocation for Qwen2.5-VL-7B-Instruct under a global rank ratio of 50%. The allocated ranks exhibit substantial variation across both layers and projection types. In particular, value projections receive larger rank budgets than key projections in most layers, while both generally receive more capacity in the middle-to-late layers, especially around layers 16–21. These observations reveal the heterogeneous importance of singular components across KV projections, highlighting the need for importance-guided global rank allocation rather than uniform rank assignment.

Figure 8. Visualization of global rank allocation on Qwen2.5-VL-7B-Instruct at 50% rank retention.

Additional Implementation Details

Quantization Integration

To facilitate Hadamard rotation, we align the allocated ranks to multiples of 16. We apply an orthogonal Hadamard transform to the latent dimensions, which can be absorbed into the low-rank factors without additional inference overhead. Inspired by QSVD [26], we further redistribute the singular values between the two factors. Specifically, for each KV projection Wm≈AmBmW_m \approx A_mB_m, we construct

Am=Sm−1UΣmΣmαHm,A_m = S_m^{-1} U_{\Sigma_m} \Sigma_m^\alpha H_m,
Bm=Hm⊤Σm1−αVΣm⊤,(22)B_m = H_m^\top\Sigma_m^{1-\alpha} V_{\Sigma_m}^\top, \tag*{(22)}

where HmH_m is a normalized Hadamard matrix. Although this α\alpha-redistribution leaves AmBmA_mB_m largely unchanged in BF16 precision, the symmetric choice α=0.5\alpha= 0.5 is not necessarily optimal under latent quantization. Specifically, the down-projection factor AmA_m controls the magnitude of latent activations, thereby affecting their quantization error, while the up-projection factor BmB_m determines how this error is amplified during KV-state reconstruction.

Empirically, we find that α=0.7\alpha= 0.7 consistently reduces the quantization-induced normalized mean squared error (NMSE) of reconstructed KV states by 4–8% relative to α=0.5\alpha= 0.5 across our evaluated models. We therefore fix α=0.7\alpha= 0.7 for all experiments without model-specific tuning. Table 9 further examines the downstream effects of Hadamard rotation and singular-value redistribution under 4-bit and 3-bit latent quantization.

BitsVariantSciQAMathVMMMUAvg.
KV4Ours (default)87.4162.749.8966.67
KV4w/o Hadamard87.4162.448.5666.12
KV4α=0.5\alpha= 0.588.1562.147.6765.97
KV3Ours (default)87.3662.547.5665.80
KV3w/o Hadamard86.0761.848.5665.47
KV3α=0.5\alpha= 0.587.1161.646.6765.13

Table 9. Ablation of Hadamard rotation and singular-value redistribution on Qwen2.5-VL-7B-Instruct at 50% rank ratio, with 4-bit and 3-bit latent quantization. The default configuration uses Hadamard rotation and α=0.7\alpha= 0.7.

Efficient Low-Rank Attention

We implement a two-stage decode pipeline for Qwen2.5-VL-7B-Instruct that avoids materializing full historical K/V cache in global memory. A fused Triton QK kernel reconstructs key tiles on-chip, applies MRoPE using cached position IDs, and reuses each key tile across its associated query heads under grouped-query attention (GQA). Attention weights then aggregate values directly in the latent space, followed by the value up-projection, so value reconstruction is performed only after reduction over the sequence dimension. Both BF16 and INT4 caches follow this pipeline; the INT4 kernels unpack codes on-chip and factor affine quantization corrections out of the inner products. Prefill still uses standard attention with reconstructed K/V.

Figure 9 presents the decoding latency breakdown under different compression settings. Under the default Hugging Face implementation, TraceKV substantially reduces KV-cache update overhead by storing compact latent cache, with 4-bit quantization providing further reductions in memory traffic. Despite requiring KV reconstruction for attention computation, our optimized low-rank attention remains efficient, with the combined QK and PV kernel latency comparable to or lower than dense FlashAttention-2.

Figure 9. Per-step decoding latency breakdown on Qwen2.5-VL-7B-Instruct with batch size 16 and 16K context length. Reported latencies may differ slightly from end-to-end measurements due to the overhead of kernel-level profiling.

Limitations and Discussion

Compression Objective Alignment. TraceKV is guided by a unified policy-preserving principle that minimizes policy-KL divergence over reasoning traces generated by the dense model, following the widely adopted objective of knowledge distillation. However, this objective does not necessarily guarantee optimal performance of the compressed model, which encounters its own autoregressive states during inference rather than those of the dense model. Such distribution mismatch may lead to accumulated errors over long reasoning trajectories. Exploring compression objectives that explicitly account for compressed-model traces is a promising direction for future work.

Calibration Overhead and Generalization. TraceKV introduces additional calibration overhead from on-policy trace generation and Monte Carlo estimation. Moreover, although our experiments demonstrate generalization across several downstream benchmarks, calibration still relies on a limited set of task-specific examples and their model-generated reasoning traces. Whether the resulting compression strategy generalizes to broader domains or other tasks involving long-form generation beyond reasoning remains an interesting direction for future research.

Comparison with Online Low-Rank Methods. Another line of work explores online low-rank KV compression by directly factorizing prefill KV cache matrices on-the-fly [8, 22]. However, the resulting low-rank bases may not adequately cover newly generated states during reasoning, while updating the bases and historical caches online remains expensive [32]. In contrast, TraceKV calibrates fixed low-rank projection using on-policy reasoning traces for compressing both input and generated KV states without additional runtime cost.

References

  1. [1]Ali Abbasi, Chayne Thrash, Haoran Qin, Shansita Sharma, Sepehr Seifi, and Soheil Kolouri. Zero sum SVD: Balancing loss sensitivity for low rank LLM compression. In Proceedings of the 43rd International Conference on Machine Learning, pages 84–100, 2026. 5, 6, 8arxiv.org/abs/2602.02848
  2. [2]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240, 2024. 5arxiv.org/abs/2404.00456
  3. [3]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen2.5-vl technical report. arXiv preprint arXiv:2511.21631, 2025. 1, 6, 7
  4. [4]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 3, 5arxiv.org/abs/2502.13923
  5. [5]Priyansh Bhatnagar, Ashkan Moradifrouzabadi, Se-Hyun Yang, Seungjae Lee, Jungwook Choi, and Mingu Kang. STAR-KV: Low-rank KV cache compression via soft thresholding for adaptive rank control. In Proceedings of the 43rd International Conference on Machine Learning, pages 7941–7960. PMLR, 2026. 1arxiv.org/abs/2606.08382
  6. [6]Zefan Cai, Wen Xiao, Hanshi Sun, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li-Wen Chang, Jiuxiang Gu, Zhen Dong, et al. R-kv: Redundancy-aware kv cache compression for reasoning models. Advances in neural information processing systems, 38:60980–61005, 2026. 1, 8arxiv.org/abs/2505.24133
  7. [7]Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. PaLu: Kv-cache compression with low-rank projection. In International Conference on Learning Representations, 2025. 1, 2, 5, 6, 7, 8arxiv.org/abs/2407.21118
  8. [8]Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Hung-Yueh Chiang, Yash Akhauri, Xilai Dai, Huiqiang Jiang, Yucheng Li, Luis Ceze, Kai-Chiang Wu, and Mohamed S. Abdelfattah. xKV: Cross-layer KV-cache compression via aligned singular vector extraction. In Forty-third International Conference on Machine Learning, 2026. 5arxiv.org/abs/2503.18893
  9. [9]Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, pages 35549–35562, 2024. 8arxiv.org/abs/2307.08691
  10. [10]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 4arxiv.org/abs/1503.02531
  11. [11]Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270–1303, 2024. 1, 8, 4arxiv.org/abs/2401.18079
  12. [12]Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022. 8arxiv.org/abs/2207.00112
  13. [13]Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xichun Li, Yaqi Yan, Pei Ran, Guohao Dai, et al. Mbq: Modality-balanced quantization for large vision-language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4167–4177. IEEE, 2025. 8arxiv.org/abs/2412.19509
  14. [14]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyi Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947–22970, 2024. 1, 8arxiv.org/abs/2404.14469
  15. [15]Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. 8arxiv.org/abs/2405.04434
  16. [16]Yaoyang Liu, Shuwei Shen, Jincheng Tang, Jinguang Liu, Jinbao Xue, Hailong Yang, Ke Zhang, and Yinjun Wu. Belt: Behavior-aligned low-rank kv cache compression with learned top-k rank allocation. arXiv preprint arXiv:2610.17966, 2026. 8arxiv.org/abs/2610.17966
  17. [17]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, pages 32332–32344, 2024. 1, 6, 8DOI
  18. [18]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, 2022. 1, 6
  19. [19]Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyu­an Li, Hannah Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jiangfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024. 1, 6arxiv.org/abs/2310.02255
  20. [20]Ryan Lucas, Kayhan Behdin, Zhipeng Wang, Qingquan Song, Rahul Mazumder, et al. Reasoning models can be accurately pruned via chain-of-thought reconstruction. In International Conference on Learning Representations, pages 142840–142852, 2026. 8arxiv.org/abs/2509.12464
  21. [21]Charilaos Pipis, Shivam Garg, Vasilis Kontonis, Vaishnavi Shrivastava, Akshay Krishnamurthy, and Dimitris Papailiopoulos. Wait, wait, wait... why do reasoning models loop? In Forty-third International Conference on Machine Learning, 2026. 4arxiv.org/abs/2512.12895
  22. [22]Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. In Forty-second International Conference on Machine Learning, 2025. 5arxiv.org/abs/2410.21465
  23. [23]Haiyu Wang, Yutong Wang, Jack Qian Zhang. Wsvd: Weighted low-rank approximation for fast and efficient execution of low-precision vision-language models. In International Conference on Learning Representations, 2026. 1, 5, 6, 8arxiv.org/abs/2604.02570
  24. [24]Haiyu Wang, Yutong Wang, Leshu Li, Yihui Ren, and Sai Qian Zhang. Laser: Loss-aware singular-value decomposition and rank allocation for efficient low-precision vision-language models. arXiv preprint arXiv:2606.00573, 2026. 8
  25. [25]Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. In International Conference on Learning Representations, 2025. 1, 2, 6, 8arxiv.org/abs/2403.07378
  26. [26]Yutong Wang, Haiyu Wang, and Sai Qian Zhang. Qsvd: Efficient low-rank approximation for unified query-key-value weight compression in low-precision vision-language models. Advances in Neural Information Processing Systems, 38: 1789–1820, 2026. 1, 8, 4arxiv.org/abs/2510.16292
  27. [27]Miles Williams, George Chrysostomou, and Nikolaos Aletras. Self-calibration for language model quantization and pruning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10149–10167, 2025. 8DOI
  28. [28]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, pages 21875–21895, 2024. 8, 4arxiv.org/abs/2309.17453
  29. [29]Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023. 8arxiv.org/abs/2312.05821
  30. [30]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024. 1, 6arxiv.org/abs/2311.16502
  31. [31]Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyu­an Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 881–916, 2025. 6, 2DOI
  32. [32]Yiren Zhao, Guanghui Song, Tianrui Qin, Kejiang Ye, Cheng-zhong Xu, and Xitong Gao. is-kv: Online low-rank kv cache compression via block-incremental svd. arXiv preprint arXiv:2610.02815, 2026. 5
  33. [33]Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen-glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiah­ao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Liujun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 1, 6arxiv.org/abs/2504.10479

Paper details

vlnc:W_86d8dpyu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents