betaFeedback

WorksComputer Science

BasisKV: Value Latents for Communication and Key Routing

Abstract

Long-context large language model (LLM) inference is increasingly constrained by the cost of maintaining an ever-growing KV-cache. Low-rank K/V projections provide a promising way to compress the KV-cache with minimal quality loss, but prior work has largely optimized KV-cache compression in isolation, leaving distributed inference, especially with tensor parallelism where attention computation is partitioned across multiple GPUs, largely underexplored. We make two observations: (i) tensor-parallel communication is an important cost when deploying low-rank KV-cache compression in distributed inference; and (ii) Value compression can be associated with the output projection, allowing the same low-rank Value representation to reduce both Value-cache width and tensor-parallel communication. Based on these observations, we propose BasisKV, a low-rank KV-cache compression method for communication-intensive, long-context LLM inference. BasisKV jointly optimizes the low-rank factors across per-GPU attention partitions against the complete attention output, rather than fitting each partition in isolation. The compressed Value cache is further reused to help identify which Key pages should be accessed: it predicts part of the Key information needed for routing, while a Key-specific residual stores what cannot be recovered from Value. The original Keys are then retrieved only for the selected pages to compute exact QK scores. Across multiple LLM families and system settings, BasisKV achieves a better quality–efficiency tradeoff than existing low-rank baselines, reaching up to 1.42x end-to-end speedup under 8-GPU tensor-parallel inference. Our code is available at https://github.com/alexz949/BasisKV

Introduction

Long-context large language model (LLM) inference is increasingly constrained by the cost of maintaining an ever-growing KV cache [4, 15]. A promising approach is to factorize or transform the Key and Value projection matrices such that inference directly produces and caches low-dimensional K/V representations [4, 18]. This broader family of low-rank KV-cache methods has attracted growing interest, including trainable feature-space projections [14] and cross-layer low-rank cache representations [5]. However, prior low-rank KV-cache methods largely optimize cache efficiency in isolation from tensor-parallel communication. Under tensor parallelism (TP), attention computation is partitioned across multiple GPUs, and the resulting partial attention outputs must additionally be communicated across accelerators [7, 19]. This raises a natural question: can low-rank KV-cache compression provide additional benefits under distributed inference beyond reducing the cache itself? Our key observation is that low-rank Value compression can also reduce the width of the attention outputs communicated across GPUs, allowing the same latent representation to reduce both KV-cache size and tensor-parallel traffic. This motivates BasisKV, a low-rank KV-cache compression method designed for distributed long-context inference.

Figure 1. Overview of BasisKV. BasisKV compresses each TP-local attention output into a low-rank Value latent, communicates the source-specific latents with AllGather, and reconstructs the complete output after communication. The same Value latent is retained in the cache and reused for Key routing where a Key-specific residual provides the remaining routing information.

BasisKV first moves tensor-parallel communication into a low-rank space by projecting each per-GPU attention output (source) to a lower dimension before communication, preserving these source-specific latents through a source-preserving AllGather, and reconstructing them afterward. Rather than fitting each source independently, BasisKV jointly optimizes all low-rank factors against the complete attention output, accounting for the cross-source interactions that contribute to the reconstruction error.

For attention blocks where the Value path remains linear, the low-rank compression projection applied before communication (the encoder) can be moved to the Value side and folded into the Value projection. This allows the model to directly produce and cache the same compressed Values used for TP communication, with no additional approximation beyond the offline derived low-rank model. We further reuse the compressed Value cache for Key routing. Compressed Values provide part of the information needed to select Key pages, while a Key-specific residual captures the remaining routing information. Original Keys are retrieved only for selected pages to recompute exact QK scores.

Across multiple LLM families, model scales, and inference settings, BasisKV consistently preserves stronger downstream quality than matched low-rank baselines while simultaneously reducing tensor-parallel communication. On eight NVIDIA L40S GPUs, BasisKV reaches up to 1.42× end-to-end speedup under tensor-parallel inference. For long-context decoding, the Value-assisted Key routing avoids accessing full Keys over most of the context; at 128K context on a single L40S GPU, BasisKV achieves 1.63× attention-block speedup and 1.25× full-model decode speedup.

Our contributions are threefold. First, to our knowledge, BasisKV is the first post-training method to jointly reduce persistent Value-cache width and tensor-parallel attention-output communication using the same low-rank Value representation for standard attention. Second, we connect low-rank Value compression with tensor-parallel communication and jointly optimize the resulting low-rank factors against the complete attention output. Third, we further reuse the compressed Value cache for sparse Key routing and demonstrate improved quality–efficiency tradeoffs and end-to-end speedups across multiple models and inference settings.

Background

Background

Tensor-Parallel Attention Output Projection

We consider standard tensor parallelism (TP) (Hansen-Palmus et al., [7]) over PP GPUs. Let TT denote the sequence length, dd the model width, HqH_q the number of query heads, and dhd_h the head dimension. We denote the concatenated attention-output width by da=Hqdhd_a = H_qd_h. For both Multi-Head Attention (MHA) and Grouped-Query Attention (GQA), let Z(h)∈RT×dhZ^{(h)} \in\mathbb{R}^{T \times d_h} denote the output of query head hh. The concatenated attention output before the output projection is

Z=Concat⁡(Z(1),…,Z(Hq))∈RT×da.Z = \operatorname{Concat}(Z^{(1)}, \ldots, Z^{(H_q)}) \in\mathbb{R}^{T \times d_a}.

Under TP, the QKV projections are column-parallel, so the query heads are partitioned across GPUs. Let ZpZ_p denote the concatenated outputs of the query heads assigned to GPU pp. The output projection WOW_O is partitioned along its input dimension to match this sharding. With Zp∈RT×(da/P)Z_p \in\mathbb{R}^{T \times(d_a/P)} and WO,p∈R(da/P)×dW_{O,p} \in\mathbb{R}^{(d_a/P) \times d}, the corresponding partitions are

Z=[Z1 ⋯ ZP],WO=[WO,1⋮WO,P].Z = [Z_1 \ \cdots\ Z_P], \qquad W_O = \begin{bmatrix} W_{O,1}\\ \vdots\\ W_{O,P} \end{bmatrix}.

Each GPU holds only a da/Pd_a/P-dimensional shard of the attention output, but its row-parallel block WO,pW_{O,p} still maps this local shard to a full dd-dimensional contribution,

Yp=ZpWO,p∈RT×d.Y_p = Z_pW_{O,p} \in\mathbb{R}^{T \times d}.

The partial contributions are then summed across GPUs using AllReduce, yielding the complete attention output on every GPU:

Y=AllReduce⁡(Yp)=∑p=1PYp=∑p=1PZpWO,p.Y = \operatorname{AllReduce}(Y_p) = \sum_{p=1}^{P} Y_p = \sum_{p=1}^{P} Z_pW_{O,p}.

Thus, although the attention output remains sharded at width da/Pd_a/P before WOW_O, the standard row-parallel execution projects each local shard to a full-width dd-dimensional contribution before communication. Our goal is to move the communication boundary earlier by communicating a lower-dimensional local representation and reconstructing the full-width output only after the collective.

Low-Rank TP Communication

Consider a rank-rr factorization of the local output-projection matrix,

WO,p≈W^O,p=EpDp,Ep∈R(da/P)×r,Dp∈Rr×d.W_{O,p} \approx\widehat{W}_{O,p} = E_pD_p, \qquad E_p \in\mathbb{R}^{(d_a/P) \times r}, \qquad D_p \in\mathbb{R}^{r \times d}.

If both factors are evaluated locally, the partial output ZpEpDp∈RT×dZ_pE_pD_p \in\mathbb{R}^{T \times d} remains full width. Thus, low-rank factorization of WO,pW_{O,p} alone does not reduce TP communication.

To reduce communication, we place the collective between the two factors. GPU pp first forms Up=ZpEp∈RT×rU_p = Z_pE_p \in\mathbb{R}^{T \times r}. If UpU_p is communicated using AllReduce, the TP-source representations are summed before reconstruction:

Y^AR=AllReduce⁡(Up)D=(∑p=1PZpEp)D,(1)\widehat{Y}_{\mathrm{AR}} = \operatorname{AllReduce}(U_p)D = \left(\sum_{p=1}^{P} Z_pE_p\right)D, \tag*{(1)}

where the shared decoder D∈Rr×dD \in\mathbb{R}^{r \times d} reconstructs the reduced representation. If AllGather is used instead, the TP-source representations remain separate through communication:

Y^AG=AllGather⁡(Up)[D1⋮DP]=∑p=1PZpEpDp.\widehat{Y}_{\mathrm{AG}} = \operatorname{AllGather}(U_p) \begin{bmatrix} D_1 \\ \vdots\\ D_P \end{bmatrix} = \sum_{p=1}^{P} Z_p E_p D_p.

The individual representations remain available after communication, allowing a separate decoder DpD_p for each TP source.

Collective-Aware Low-Rank Attention

Building on the Low-Rank AllGather structure in § 3.2, we jointly fit the TP-local low-rank factors against the complete attention output. For linear Value paths, the compression factors can be folded into the Value projections, so the same low-dimensional representation reduces both TP communication and Value-cache width. We further allocate ranks across layers under a fixed global budget using forward-only Kullback–Leibler (KL) sensitivity profiling.

Joint Fitting

Although each TP-local block can be approximated separately, the complete attention output is the sum of all source contributions. Let ϵp=ZpWO,p−ZpEpDp\epsilon_p = Z_p W_{O,p} - Z_p E_p D_p denote the approximation error from source pp. Then

∥Y−Y^∥F2=∑p=1P∥ϵp∥F2+2∑p<q⟨ϵp,ϵq⟩F.\left\|Y-\widehat{Y}\right\|_{F}^{2}=\sum_{p=1}^{P}\left\|\epsilon_p\right\|_{F}^{2}+2\sum_{p<q}\langle\epsilon_p,\epsilon_q\rangle_{F}.

Independent fitting does not account for the cross-source terms that appear in the reconstruction error of the complete TP output. We therefore optimize the complete output directly.

min⁡{Ep,Dp}p=1P∥Y−∑p=1PZpEpDp∥F2(2)\min_{\{E_p,D_p\}_{p=1}^{P}}\left\|Y-\sum_{p=1}^{P}Z_pE_pD_p\right\|_{F}^{2} \tag*{(2)}

where EpE_p is the compression factor for source pp, and DpD_p maps the resulting latent representation to its full-width output contribution. The objective in equation 1 is bilinear in {Ep}\{E_p\} and {Dp}\{D_p\}, but becomes a linear least-squares problem in either factor family when the other is fixed. We therefore optimize it by alternating least squares.

DD update. With {Ep}\{E_p\} fixed, we denote U=[Z1E1 ⋯ ZPEP]U=[Z_1E_1\ \cdots\ Z_PE_P] and let DD be the corresponding stacked matrix of DpD_p factors. Then equation 1 becomes

D⋆=arg min⁡D∥Y−UD∥F2.D^\star=\underset{D}{\operatorname{arg\,min}}\left\|Y-UD\right\|_{F}^{2}.

We solve this least-squares problem through the normal equations UTUD⋆=UTYU^{\mathsf{T}}UD^\star=U^{\mathsf{T}}Y. The coupled system is assembled from calibration covariance statistics and solved jointly.

EE update. With {Dp}\{D_p\} fixed, we update one EpE_p at a time. We first define Rp=Y−∑q≠pZqEqDqR_p=Y-\sum_{q\ne p}Z_qE_qD_q as the residual excluding source pp. With the remaining factors fixed, equation 1 becomes

Ep⋆=arg min⁡Ep∥Rp−ZpEpDp∥F2.E_p^\star=\underset{E_p}{\operatorname{arg\,min}}\left\|R_p-Z_pE_pD_p\right\|_{F}^{2}.

The corresponding normal equation is (Zp⊤Zp)Ep⋆(DpDp⊤)=Zp⊤RpDp⊤(Z_p^{\top}Z_p)E_p^{\star}(D_pD_p^{\top}) = Z_p^{\top}R_pD_p^{\top}. Both updates are therefore convex linear least-squares subproblems of equation 1. In practice, the DD system is assembled and solved by Cholesky factorization. We solve the EpE_p system by matrix-free conjugate gradient, avoiding explicit formation of the large Kronecker-structured system. The complete derivation and implementation details are given in Appendix § 9.§ 1.

Value Compression

The compression factor learned from the complete attention-output objective can also be applied directly to the Value path. Let ShS_h and VhV_h denote the attention matrix and Value state for head hh, respectively, so that Zh=ShVhZ_h = S_hV_h. Since ShS_h is independent of VhV_h, the fitted approximation satisfies

Y^=∑h=1HqShVhEhDh=∑h=1HqSh(VhEh)Dh.\widehat{Y} = \sum_{h=1}^{H_q} S_hV_hE_hD_h = \sum_{h=1}^{H_q} S_h(V_hE_h)D_h.

Hence the compression factor EhE_h may be applied to the Value state before attention without changing the fitted approximation. Defining V^h=VhEh\widehat{V}_h = V_hE_h, the compressed Value can therefore be cached directly. For GQA, all query heads sharing the same physical KV head also share the same Value encoder EgE_g, while retaining head-specific decoders DhD_h. This ensures that only one compressed Value latent is stored per physical KV head. Since Vh=XWV,hV_h = XW_{V,h}, we have V^h=XWV,hEh\widehat{V}_h = XW_{V,h}E_h. Thus EhE_h can be folded into the Value projection by setting W^V,h=WV,hEh\widehat{W}_{V,h} = W_{V,h}E_h. The same latent representation therefore determines both the communicated width and the cached Value width, without introducing any approximation beyond the fitted low-rank output model.

Rank Allocation

A uniform rank assignment ignores that layers have different sensitivity to compression. We estimate this sensitivity around a reference rank r0r_0 using one lower-rank and one higher-rank probe for each layer. Let eℓ(r)e_\ell(r) denote the reconstruction error of layer ℓ\ell at rank rr, and let ΔKℓ±\Delta K_\ell^{\pm} denote the change in output KL relative to the reference model when only that layer is changed from r0r_0 to r±r^{\pm}. We define the two one-sided sensitivities as

sℓ±=max⁡(0,ΔKℓ±eℓ(r±)−eℓ(r0)).s_\ell^{\pm} = \max\left(0,\frac{\Delta K_\ell^{\pm}}{e_\ell(r^{\pm}) - e_\ell(r_0)}\right).

The profiling requires only forward evaluation and two single-layer probes per layer, so its cost grows linearly with model depth. For each candidate rank, we weight its reconstruction-error change by sℓ−s_\ell^{-} below r0r_0 and by sℓ+s_\ell^{+} above r0r_0 to obtain a layerwise rank cost. We then use dynamic programming to find the minimum-cost discrete rank assignment under the fixed global budget. The derivation and dynamic-programming details are given in Appendix § 9.§ 3.

Key Routing with Value Latents

We next examine whether the compressed Value representation can also be reused for Key Routing. We find that the compressed Values can predict a nontrivial component of the corresponding pre-RoPE Keys through a low-rank representation. We then use this Value-derived prediction as part of the routing signal and store a low-rank residual for the remaining Key-specific information. When compressed Values are already retained for attention, this reuse reduces the amount of dedicated per-token routing state. The resulting approximate scores are used only to select KV-cache pages and the original Keys are then accessed to recompute QK scores on the selected pages.

Key Prediction from Value

Let V^c\widehat{V}_c and KcK_c denote the centered compressed Value latent and pre-RoPE Key. We fit the rank-rbr_b reduced-rank regression

M⋆=arg⁡min⁡rank⁡(M)≤rb ∥Kc−V^cM∥F2(3)M^\star= \underset{\operatorname{rank}(M) \le r_b}{\arg\min}\ \lVert K_c - \widehat{V}_c M \rVert_F^2 \tag*{(3)}

With G=V^c⊤V^cG = \widehat{V}_c^\top\widehat{V}_c and H=V^c⊤KcH = \widehat{V}_c^\top K_c, let UΣV⊤U\Sigma V^\top be the rank-rbr_b truncated SVD of G†/2HG^{\dagger/2}H. Then

M⋆=G†/2UΣV⊤M^\star= G^{\dagger/2}U\Sigma V^\top

Therefore the predicted Key is K^=μK+V^cM⋆\widehat{K} = \mu_K + \widehat{V}_cM^\star. The model’s native RoPE transformation is applied after prediction.

Routing-aligned Value basis. Evaluating the predictor directly would require reading the full compressed Value state for every cached token. However, since M⋆=G†/2UΣV⊤M^\star= G^{\dagger/2}U\Sigma V^\top, the prediction depends on V^c\widehat{V}_c only through the rbr_b-dimensional quantity V^cG†/2UΣ\widehat{V}_cG^{\dagger/2}U\Sigma. We therefore align this predictive subspace with explicit coordinates of the Value cache. We compute the thin QR factorization

G†/2UΣ=QbRb,M⋆=QbRbV⊤.G^{\dagger/2}U\Sigma= Q_bR_b,\qquad M^\star= Q_bR_bV^\top.

Completing QbQ_b to an orthogonal matrix Q=[Qb Q⊥]Q = [Q_b\ Q_\perp], we rotate the Value latent and decoder as V‾=V^Q\overline{V} = \widehat{V}Q and D‾=Q⊤D\overline{D} = Q^\top D. This change of basis leaves the Value reconstruction unchanged and places all information required by the Key predictor in the first rbr_b coordinates of V‾\overline{V}. The rotation is folded into the Value encoder offline, so full-context routing reads only these rbr_b coordinates. The derivation is given in Appendix § 11.

Residual Routing Component

After applying RoPE, let ktk_t and k^t\widehat{k}_t denote the exact and predicted Keys for token tt, and define the residual et=kt−k^te_t = k_t - \widehat{k}_t. The Value-based predictor does not recover the full Key, so we represent the remaining score contribution with a low-rank residual component. We factor this contribution into a query-side factor UhU_h and a residual-side factor FhF_h,

s^ht=1dh[qhk^t⊤+(qhUh)(etFh)⊤].(4)\widehat{s}_{ht} = \frac{1}{\sqrt{d_h}}\left[q_h\widehat{k}_t^\top+ (q_hU_h)(e_tF_h)^\top\right]. \tag*{(4)}

Thus, only the rqr_q-dimensional residual representation etFhe_tF_h is stored for each cached token.

Page-Fisher fitting. The residual correction is used to select pages rather than to reconstruct the Key itself. Let ℓ\ell and ℓ^\widehat{\ell} denote the exact and approximate page logits. We measure the routing error by

DKL(softmax⁡(ℓ) ∥ softmax⁡(ℓ^)),D_{\mathrm{KL}}\left(\operatorname{softmax}(\ell)\,\middle\|\,\operatorname{softmax}(\widehat{\ell})\right),

which measures how the approximation redistributes attention mass among pages. Direct optimization of this objective is nonlinear in the residual factors. However, a second-order expansion around the exact page logits gives a quadratic objective for the residual correction.

Fix one calibration query and head. Let πp\pi_p denote the exact attention mass of page pp and eˉp\bar{e}_p be the attention-weighted average Key residual within page pp. Define

eˉ=∑pπpeˉp,Γ=∑pπp(eˉp−eˉ)⊤(eˉp−eˉ).\bar{e} = \sum_p \pi_p\bar{e}_p,\qquad\Gamma= \sum_p \pi_p(\bar{e}_p-\bar{e})^\top(\bar{e}_p-\bar{e}).

Here eˉ\bar{e} is the mean residual over pages, and Γ\Gamma is the corresponding covariance. The second-order expansion gives

LPF=12dhqh(UhFhT−I)Γ(UhFhT−I)TqhT,(5)\mathcal{L}_{\mathrm{PF}} = \frac{1}{2d_h}q_h(U_hF_h^{\mathsf{T}}-I)\Gamma(U_hF_h^{\mathsf{T}}-I)^{\mathsf{T}}q_h^{\mathsf{T}}, \tag*{(5)}

II is the identity matrix corresponding to the exact residual score, while UhFhTU_hF_h^{\mathsf{T}} gives its low-rank approximation. We sum this loss over calibration queries and heads. For fixed FhF_h, equation (5) is a convex quadratic problem in UhU_h, and vice versa. We alternate between the two factors and solve the resulting positive-semidefinite systems by matrix-free conjugate gradient. The KL expansion and normal equations are given in Appendix § 11.

Query selection. Fitting the residual at every query position is expensive and provides many redundant observations. We therefore select a small set of representative positions.

For each query head hh, we first compute Hh=E[qhqhT]H_h = \mathbb{E}[q_hq_h^{\mathsf{T}}] over the calibration queries. The factor Hh†H_h^{\dagger} balances query directions with different scales when comparing positions. Let QhQ_h be the stacked matrix of candidate queries for head hh. We then define the position Gram matrix

G=1Hq∑h=1HqE[QhHh†QhT].(6)G = \frac{1}{H_q}\sum_{h=1}^{H_q}\mathbb{E}\left[Q_hH_h^{\dagger}Q_h^{\mathsf{T}}\right]. \tag*{(6)}

Thus, GG measures similarity between candidate positions after accounting for the scale of each query direction. We apply diagonal-pivoted Cholesky to GG and retain the selected pivot positions for residual calibration. The construction and its equivalence to column-pivoted QR (CPQR) are given in Appendix § 11.

Page Selection and Exact Key Retrieval

The approximate scores in equation (4) rank tokens without scanning the original Key cache. For each query head, we assign page Pj\mathcal{P}_j the score log⁡∑t∈Pjexp⁡(s^ht)\log\sum_{t\in\mathcal{P}_j}\exp(\hat{s}_{ht}) and retain the highest-scoring pages under a fixed budget. The original post-RoPE Keys are then accessed only for the selected pages, where their QK scores are recomputed exactly. The resulting exact scores are used to attend to the corresponding compressed Values. Thus, full-dimensional Keys need not be read for unselected pages during decoding and may reside in a slower memory tier.

Experiment Results

Experimental Setup. We evaluate BasisKV on Llama-3.1, Qwen3, and Qwen3.5. All Value-compression methods use the same 256 C4 calibration sequences of 2048 tokens. Key routing uses 32 128K-token windows, equally split between C4 and synthetic long-range retrieval data, plus 16 held-out C4 validation windows. The default router combines a 16-dimensional Value-derived Base and a 16-dimensional residual (B16R16), with page size 4. We evaluate routing on RULER-128K ([9]) and nine LongBench ([1]) tasks with inputs longer than 4K tokens, selecting 2048 and 256 routed tokens, respectively, plus 64 recent tokens. Appendix C.§ 1 details calibration, baselines, and measurement protocols.

Collective Choice. Figure 2 compares AllGather and AllReduce on Qwen3-8B-Base under 4-way tensor parallelism (TP=4) at matched ideal ring traffic. Both are jointly fitted against the full attention output. AllGather retains source-specific latents and decoders; AllReduce sums latents before a shared decoder. AllGather achieves lower WikiText-2 perplexity throughout, especially at low traffic. Both approach dense quality as traffic increases, supporting source-specific latents under constrained communication.

Figure 2. Perplexity at matched TP traffic. Qwen3-8B-Base, WikiText-2, TP4. Gray dotted line: dense quality.

Value Compression across Models. Table 1 compares BasisKV with PaLU M-LRD and G-LRD at matched global Value-retention budgets of 75% and 50%. Each method uses its own rank-allocation procedure. M-LRD is the matched-efficiency PaLU baseline; G-LRD groups 4 heads for stronger quality at 4× the low-rank decoding cost. BasisKV achieves the highest average accuracy among compressed methods on all three models at both budgets, while reducing WOW_O communication. It also gives the lowest WikiText-2 and C4 perplexity on both Qwen3 models; PaLU G-LRD is more competitive on Llama-3.1-8B-Instruct.

ModelBudgetMethodWikiT.C4BoolQARC-EARC-CWinoG.PIQAHellaS.OBAvg.Relative
Qwen3-8B Base75%Dense7.009.170.8310.8020.5690.7280.7940.7860.4200.7041.00
Qwen3-8B Base75%PaLU M-LRD8.6610.750.7940.7400.5220.7190.7790.7680.3940.6741.00
Qwen3-8B Base75%PaLU G-LRD8.2410.400.7930.7710.5560.7180.7960.7870.4080.6901.00
Qwen3-8B Base75%BasisKV7.289.340.8180.7880.5440.7300.7930.7720.4240.6960.375
Qwen3-8B Base50%PaLU M-LRD14.8416.870.7630.6260.4210.6700.7390.7000.3560.6111.00
Qwen3-8B Base50%PaLU G-LRD10.6512.870.7200.6980.5020.7100.7670.7540.4060.6511.00
Qwen3-8B Base50%BasisKV8.2610.060.8040.7710.5190.7190.7880.7350.3980.6760.250
Llama-3.1 8B-Instruct75%Dense7.229.770.8410.7960.5520.7410.8100.7920.4300.7091.00
Llama-3.1 8B-Instruct75%PaLU M-LRD8.6011.250.8320.7780.5280.7470.7950.7750.4300.6981.00
Llama-3.1 8B-Instruct75%PaLU G-LRD7.9810.660.8370.7790.5290.7360.7920.7820.4300.6981.00
Llama-3.1 8B-Instruct75%BasisKV8.6110.660.8460.7870.5340.7480.7950.7760.4400.7040.375
Llama-3.1 8B-Instruct50%PaLU M-LRD16.0917.160.7250.5970.3640.7020.7380.6790.3760.5971.00
Llama-3.1 8B-Instruct50%PaLU G-LRD10.6112.900.8130.7120.4620.7100.7640.7380.3960.6561.00
Llama-3.1 8B-Instruct50%BasisKV11.5012.450.8310.7350.4900.7210.7730.7390.4220.6730.250
Qwen3 32B75%Dense7.6110.810.8650.8300.6060.7390.8220.8260.4600.7351.00
Qwen3 32B75%PaLU M-LRD8.4811.460.8590.8140.6110.7150.8080.8160.4460.7241.00
Qwen3 32B75%PaLU G-LRD8.2411.270.8660.8080.6240.7360.8050.8190.4480.7291.00
Qwen3 32B75%BasisKV7.7610.920.8640.8320.6230.7160.8080.8250.4800.7350.600
Qwen3 32B50%PaLU M-LRD10.2913.350.7420.7400.5210.7100.7850.7690.4300.6711.00
Qwen3 32B50%PaLU G-LRD9.1612.070.8740.7490.5740.7400.7930.7950.4340.7081.00
Qwen3 32B50%BasisKV8.6211.620.8410.8090.5930.7150.8070.8080.4460.7170.400

Table 1. Value compression across models. PaLU M-LRD uses per-head decomposition; G-LRD groups 4 heads. Comm. is WOW_O traffic relative to dense. Bold marks the best compressed result within each budget.

Tensor-Parallel Inference. Communication-compression gains grow with batch size on both models (Figure 3). V64 crosses over earlier and provides larger gains than V96, reaching 1.42× end-to-end speedup on Llama-3.1-8B-Instruct and 1.31× on Qwen3-32B at batch size 256. Time-to-first-token (TTFT) follows the same trend.

Figure 3. TP inference. Llama-3.1-8B-Instruct and Qwen3-32B, 8× L40S; 4K-token prompts, 128 output tokens. Left: end-to-end speedup; right: TTFT. V96/V64 retain Value widths of 96/64.

Long-Context Routing Quality. In Table 2, Full-K uses full attention. Support is approximately matched for LRQK and Loki as described in Appendix C.§ 1, while ShadowKV retains its native support. BasisKV remains close to Full-K on RULER and among the strongest sparse methods. Its retrieval results support combining Value-derived prediction with residual correction. LongBench differences are smaller and model-dependent.

ModelMethodS1S2MK1MK2MQMVQA1QA2VTFWEAvg.NQAMQAHQAMQueDReadGRepSAMPRetrLCCAvg.
Llama-3.1 8B-InstructFull-K100.00100.0099.0083.0099.0095.0074.0048.0076.8052.6782.7528.0650.2552.2331.3232.3432.7644.5099.5052.2247.02
Loki79.0096.0097.0018.0055.5053.2548.0032.0025.0048.3355.2127.7248.4852.0431.2329.5432.3646.0699.5051.9546.54
ShadowKV99.0099.0098.0061.0097.0091.7574.0048.0028.4058.0075.4129.2251.0652.6431.4931.1230.9943.7499.0051.1846.72
LRQK100.00100.0099.0076.0099.0093.2575.0047.0053.6048.6779.1527.9050.0052.1331.4630.6331.0045.3799.5051.6146.62
BasisKV100.00100.0098.0078.0099.2594.5075.0047.0064.8046.6780.3229.0551.0252.5130.7028.9531.4744.9099.5054.0546.91
Qwen3-8BFull-K100.00100.0085.0061.0093.5096.5055.0036.0093.4088.0080.8425.3551.4751.6127.3417.8933.5346.52100.0062.0646.20
Loki100.0099.0085.0038.0091.0090.7543.0027.0078.4073.3372.5526.4751.6951.1127.6922.5832.9444.9498.0055.0945.61
ShadowKV100.0095.0084.0044.0083.7586.7553.0028.0082.0088.6774.5226.1250.4751.9427.3219.9333.8244.6791.0055.1044.49
LRQK†100.0098.0086.0057.0093.7596.5051.0036.0091.2081.3379.0826.7649.5052.9326.8616.3633.8246.64100.0059.8445.79
BasisKV100.0099.0086.0053.0094.0094.0054.0033.0093.0080.3378.6326.9750.9451.4726.6316.6832.8644.6098.5058.8045.27
Qwen3.5-9BFull-K100.00100.00100.00100.00100.00100.0086.0056.00100.00100.0094.2033.8553.9561.7147.0224.6131.0424.2729.4265.8041.30
Loki100.00100.00100.0097.0099.75100.0080.0055.00100.0098.6793.0432.5254.5161.2747.1422.9631.1024.4530.1764.1540.92
ShadowKV100.0099.0099.00100.0099.0095.0084.0058.0094.8099.6792.8533.1454.9661.8746.4623.0931.0423.8440.7561.3541.83
LRQK100.00100.00100.00100.00100.0098.7585.0055.0099.2099.0093.7032.8953.2061.8547.1523.9830.9024.6430.9264.1741.08
BasisKV100.00100.00100.00100.00100.00100.0083.0056.00100.0098.0093.7033.0254.2961.0046.3124.7830.7423.7735.0862.8741.32

Table 2. Sparse attention across models. RULER (10 tasks) and LongBench (9 tasks). Avg. is the unweighted task mean. Bold marks the best sparse score.

Sparse Decoding. Table 3 evaluates B16R16 at 24K–128K, excluding prefill. Steady-state attention-block and full-model speedups grow with context, reaching 1.63× and 1.25× at 128K. All exact Keys and Values remain on GPU, without CPU offload.

ContextDenseBasisKV×DenseBasisKV×
24K5.275.171.0228.6328.331.01
32K6.705.851.1530.1129.001.04
48K9.617.311.3133.0230.431.09
64K12.588.701.4536.0131.931.13
96K18.3311.771.5641.7834.981.19
128K23.8514.671.6347.3137.871.25

Table 3. Sparse decoding across context lengths. TP1/B1, one L40S; median ms over three trials. ×: speedup over dense.

Long-Context Throughput and Memory. Figure 4 shows throughput gains across batch sizes and context lengths, reaching 2.06×2.06 \times for Llama-3.1-8B-Instruct and 1.99×1.99 \times for Qwen3-8B at comparable measured decode memory footprints.

Figure 4. Decode throughput and GPU memory. Llama-3.1-8B-Instruct (left) and Qwen3-8B (right), TP8 on 8× L40S. Memory: peak per-rank PyTorch allocation during decode. Circles/squares: BasisKV/dense; solid/dashed: 64K/128K.

Related Work

Tensor-parallel communication. Tensor parallelism partitions Transformer operators across accelerators and uses collective communication to combine their partial results [19]. Prior work reduces this communication through activation quantization [7] or improves parallel execution for decomposed models [10]. BasisKV instead preserves the individual TP sources through a low-rank AllGather and jointly fits their factors against the complete attention output. The resulting Value representation is also reused for caching and Key routing.

KV-cache compression. KV-cache compression has been studied through quantization [8, 11, 15], token and head eviction [3, 6, 13] (Tang et al. 2025; Xiao et al. 2025; Zhang et al. 2023), and low-dimensional representations [2, 4, 14]. PaLU stores low-rank latent Key and Value states obtained from decomposed projection matrices and absorbs Value reconstruction into the output projection [4]. STAR-KV learns adaptive low-rank truncation through differentiable soft-thresholding and additional distillation-based training [2]. BasisKV instead derives its Value representation post hoc by fitting the complete TP attention output, so the cached representation is simultaneously optimized for the communication path.

Sparse attention and KV routing. Sparse-attention methods reduce decode-time memory traffic by selecting a subset of cached states for attention [3, 13] (Zhang et al. 2023). SparQ selectively fetches cached history [17], while Quest scores KV-cache pages using query-dependent Key summaries (Tang et al., 2024). Loki ranks tokens using a low-dimensional Key representation (Singhania et al., 2024), and ShadowKV retains a low-rank Key representation on GPU while retrieving selected KV states (Sun et al., 2025). Recent low-rank approaches similarly construct compact Query–Key representations for sparse retrieval, including LRQK [12] and SALS [16]. BasisKV instead reuses routing information already present in the compressed Value representation and combines it with a Key-specific residual.

Conclusion

We presented BasisKV, a low-rank KV-cache compression method for communication-intensive long-context LLM inference. BasisKV jointly fits low-rank factors against the complete tensor-parallel attention output, allowing the resulting Value representation to reduce both TP communication and Value-cache width. The same Value latent is further reused for Key routing, with a Key-specific residual capturing information not recovered from Value. Across multiple model families and system settings, BasisKV improves the quality–efficiency tradeoff of low-rank compression while providing end-to-end gains in both tensor-parallel and long-context sparse decoding. These results show that a shared Value representation can serve multiple inference boundaries without optimizing each component in isolation.

Appendix

Collective-Aware Fitting

For the calibration set, let ZhZ_h denote the activation matrix of source block hh, and let YY denote the corresponding complete attention output. For compactness, define the Gram and cross-product statistics

Chk=ZhTZk,Bh=ZhTY.C_{hk}=Z_h^{\mathsf{T}}Z_k,\qquad B_h=Z_h^{\mathsf{T}}Y.

Decoder update. With the encoders fixed, the decoder blocks are solved jointly. For each hh, stationarity with respect to DhD_h gives

0=EhTZhT(∑k=1HZkEkDk−Y)=∑k=1HEhTChkEkDk−EhTBh.0=E_h^{\mathsf{T}}Z_h^{\mathsf{T}}\left(\sum_{k=1}^{H}Z_kE_kD_k-Y\right)=\sum_{k=1}^{H}E_h^{\mathsf{T}}C_{hk}E_kD_k-E_h^{\mathsf{T}}B_h.

Thus, the hhth stationarity condition couples DhD_h to all decoder blocks {Dk}k=1H\{D_k\}_{k=1}^{H}. The resulting normal equations are

∑k=1H(EhTChkEk)Dk=EhTBh,h=1,…,H.(7)\sum_{k=1}^{H}\left(E_h^{\mathsf{T}}C_{hk}E_k\right)D_k=E_h^{\mathsf{T}}B_h,\qquad h=1,\ldots,H. \tag*{(7)}

We assemble this coupled block system from the calibration statistics and solve for all decoder blocks jointly by Cholesky factorization.

Encoder update. With the decoders fixed, we update one encoder at a time. For encoder EhE_h, we first subtract the contributions of all other source blocks and define the conditional target

Th=Y−∑k≠hZkEkDk.T_h = Y - \sum_{k \ne h} Z_k E_k D_k.

With the remaining encoders fixed, the joint objective therefore reduces to the least-squares problem

min⁡Eh∥Th−ZhEhDh∥F2.\min_{E_h} \left\|T_h - Z_h E_h D_h\right\|_{F}^{2}.

Its first-order optimality condition gives the normal equation

ChhEhDhDhT=ZhTThDhT.(8)C_{hh} E_h D_h D_h^{\mathsf{T}} = Z_h^{\mathsf{T}} T_h D_h^{\mathsf{T}}. \tag*{(8)}

Using the definition of ThT_h, the right-hand side can be written entirely in terms of the calibration statistics as

ChhEhDhDhT=BhDhT−∑k≠hChkEkDkDhT.C_{hh} E_h D_h D_h^{\mathsf{T}} = B_h D_h^{\mathsf{T}} - \sum_{k \ne h} C_{hk} E_k D_k D_h^{\mathsf{T}}.

The left-hand side defines the linear operator Hh(X)=ChhXDhDhT\mathcal{H}_h(X) = C_{hh} X D_h D_h^{\mathsf{T}}. It is positive semidefinite since

⟨X,Hh(X)⟩F=∥ZhXDh∥F2≥0.\langle X, \mathcal{H}_h(X) \rangle_F = \left\|Z_h X D_h\right\|_F^2 \ge0.

Vectorizing equation (8) would produce a Kronecker-structured linear system with coefficient matrix (DhDhT)⊗Chh(D_h D_h^{\mathsf{T}}) \otimes C_{hh}. Rather than form this matrix explicitly, we solve the normal equation by matrix-free conjugate gradient using Hh(X)=ChhXDhDhT\mathcal{H}_h(X) = C_{hh} X D_h D_h^{\mathsf{T}} for each matrix–vector product.

Because the factorization is invariant to a change of latent basis, we orthonormalize each updated encoder by a thin QR factorization Eh=QhRhE_h = Q_h R_h and set Eh←QhE_h \leftarrow Q_h and Dh←RhDhD_h \leftarrow R_h D_h. This leaves EhDhE_h D_h unchanged and avoids scale drift between the two factors. After the final encoder sweep, we solve equation (7) once more for the decoder blocks.

Algorithm 1 Collective-Aware ALS

  1. function Collective-Aware-ALS({Chk}\{C_{hk}\}, {Bh}\{B_h\}, {Eh}\{E_h\}, SS)

  2. Solve equation (7) for {Dh}\{D_h\} &nbsp;&nbsp;&nbsp;&nbsp; ▷ joint Cholesky solve

  3. for s=1,…,Ss = 1, \ldots, S do

  4. &nbsp;&nbsp;&nbsp;&nbsp;for h=1,…,Hh = 1, \ldots, H do

  5. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Solve equation (8) for EhE_h &nbsp;&nbsp;&nbsp;&nbsp; ▷ matrix-free CG

  6. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Qh,Rh]←QR⁡(Eh)[Q_h, R_h] \leftarrow\operatorname{QR}(E_h)

  7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Eh←QhE_h \leftarrow Q_h

  8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Dh←RhDhD_h \leftarrow R_h D_h

  9. &nbsp;&nbsp;&nbsp;&nbsp;end for

  10. &nbsp;&nbsp;&nbsp;&nbsp;Solve equation (7) for {Dh}\{D_h\} &nbsp;&nbsp;&nbsp;&nbsp; ▷ joint Cholesky solve

  11. end for

  12. return {Eh},{Dh}\{E_h\}, \{D_h\}

  13. end function

Rank Allocation

Let ℓ=1,…,L\ell= 1, \ldots, L index the compressed layers, and let eℓ(r)e_\ell(r) denote the reconstruction error of layer ℓ\ell at rank rr. The two-sided perturbations in equation 3.3 relate changes in eℓ(r)e_\ell(r) to changes in end-to-end KL divergence.

Using the measured sensitivities sℓ−s_{\ell}^{-} and sℓ+s_{\ell}^{+}, we estimate the KL change at a candidate rank rr by

ΔK^ℓ(r)={sℓ−[eℓ(r)−eℓ(r0)],r<r0,0,r=r0,sℓ+[eℓ(r)−eℓ(r0)],r>r0.\widehat{\Delta\mathcal{K}}_{\ell}(r)= \begin{cases} s_{\ell}^{-}\left[e_{\ell}(r)-e_{\ell}(r_{0})\right], & r<r_{0},\\ 0, & r=r_{0},\\ s_{\ell}^{+}\left[e_{\ell}(r)-e_{\ell}(r_{0})\right], & r>r_{0}. \end{cases}

Thus the reconstruction curve of each layer is converted into an estimated end-to-end KL change using only the two measured perturbations. We use the separable approximation

K(r)−K(r0)≈∑ℓ=1LΔK^ℓ(rℓ).\mathcal{K}(\mathbf{r})-\mathcal{K}(\mathbf{r}_{0})\approx\sum_{\ell=1}^{L}\widehat{\Delta\mathcal{K}}_{\ell}(r_{\ell}).

For a total rank budget BB, the allocation is therefore

min⁡r1,…,rL∈R∑ℓ=1LΔK^ℓ(rℓ)subject to∑ℓ=1Lrℓ=B.\min_{r_{1},\ldots,r_{L}\in\mathbb{R}}\sum_{\ell=1}^{L}\widehat{\Delta\mathcal{K}}_{\ell}(r_{\ell}) \qquad\text{subject to}\qquad \sum_{\ell=1}^{L}r_{\ell}=B.

Let F(ℓ,b)F(\ell,b) denote the minimum estimated KL change for the first ℓ\ell layers with total rank bb. We solve the exact-budget allocation by dynamic programming with

F(ℓ,b)=min⁡r∈Rr≤b{F(ℓ−1,b−r)+ΔK^ℓ(r)}.F(\ell,b)=\min_{\substack{r\in\mathbb{R}\\r\leq b}}\left\{F(\ell-1,b-r)+\widehat{\Delta\mathcal{K}}_{\ell}(r)\right\}.

The resulting dynamic program gives the rank schedule under the budget BB.

Architecture Extensions

The output-side BasisKV factorization only requires a linear output projection. It therefore applies without modification to other sequence-model blocks with the same structure, including Gated DeltaNet and Mamba blocks. In these blocks, we apply BasisKV to the output projection WOW_O and no Value-cache factorization is introduced.

Value-Based Key Routing

Gated attention. For standard attention, the Value reconstruction can be reassociated with the output projection, allowing the jointly fitted factors to also define the compressed Value cache. An elementwise output gate breaks this reassociation. For a gated attention block, we therefore use a separate Value-side factorization

Y^=∑h=1HQ[(ZhEhVRhV)⊙Gh]WO,h,\widehat{Y}=\sum_{h=1}^{H_Q}\left[\left(Z_hE_h^VR_h^V\right)\odot G_h\right]W_{O,h},

where GhG_h is the output gate. The Value-side factors EhVE_h^V and RhVR_h^V are fitted before the gate and are not part of the joint output-side fit. The encoder EhVE_h^V is folded into the Value projection, so V^h=VhEhV\widehat{V}_h=V_hE_h^V is stored in the cache. After the gate, the resulting activations are treated as the sources of the output projection, and the standard collective-aware BasisKV factorization is applied to WOW_O. Thus gated attention uses a separate Value-cache factorization followed by the same jointly fitted output-side compression used elsewhere.

Value-Based Key Routing

Reduced-Rank Key Prediction

Let G=V^c⊤V^cG = \widehat{V}_c^{\top}\widehat{V}_c and H=V^c⊤KcH = \widehat{V}_c^{\top}K_c. Since range⁡(H)⊆range⁡(G)\operatorname{range}(H) \subseteq\operatorname{range}(G), the objective in equation (3) can be written as

∥Kc−V^cM∥F2=∥Kc∥F2−∥G†/2H∥F2+∥G1/2M−G†/2H∥F2\lVert K_c-\widehat{V}_cM\rVert_F^2 = \lVert K_c\rVert_F^2-\lVert G^{\dagger/2}H\rVert_F^2 + \lVert G^{1/2}M-G^{\dagger/2}H\rVert_F^2

The first two terms are independent of MM, so it remains to minimize

∥G1/2M−G†/2H∥F2\lVert G^{1/2}M-G^{\dagger/2}H\rVert_F^2

subject to rank⁡(M)≤rb\operatorname{rank}(M) \leq r_b. Let UΣV⊤U\Sigma V^{\top} be the rank-rbr_b truncated singular value decomposition of G†/2HG^{\dagger/2}H. Since G†/2HG^{\dagger/2}H lies in the range of GG, its truncated SVD is attainable as G1/2MG^{1/2}M and gives the best rank-rbr_b approximation. Thus one minimizer is

M⋆=G†/2UΣV⊤.M^\star= G^{\dagger/2}U\Sigma V^{\top}.

For the orthogonal matrix Q=[Qb Q⊥]Q = [Q_b\ Q_\perp] defined in § 5.1, we have QQ⊤=IQQ^{\top} = I. Hence

(V^Q)(Q⊤D)=V^QQ⊤D=V^D.(\widehat{V}Q)(Q^{\top}D)=\widehat{V}QQ^{\top}D=\widehat{V}D.

The change of basis leaves the Value reconstruction unchanged. Moreover,

V^cM⋆=(V^cQ):,:rbRbV⊤,\widehat{V}_cM^\star=(\widehat{V}_cQ)_{:,:r_b}R_bV^{\top},

so the Key predictor depends only on the first rbr_b coordinates of the rotated Value latent.

Low-Rank Residual Correction

Residual approximation. Let et=kt−k^te_t = k_t-\widehat{k}_t denote the post-RoPE Key residual. We approximate the exact residual score qhet⊤/dhq_he_t^{\top}/\sqrt{d_h} by the rank-rqr_q form (qhUh)(etFh)⊤/dh(q_hU_h)(e_tF_h)^{\top}/\sqrt{d_h}, where Uh,Fh∈Rdh×rqU_h,F_h \in\mathbb{R}^{d_h\times r_q}. The resulting token-score perturbation is

Δsht=1dhqh(UhFh⊤−I)et⊤.\Delta s_{ht}=\frac{1}{\sqrt{d_h}}q_h(U_hF_h^{\top}-I)e_t^{\top}.

The corresponding residual representation etFh∈Rrqe_tF_h \in\mathbb{R}^{r_q} is stored for each cached token.

Page-Fisher Objective

Fix one calibration query and head, and suppress these indices. Let sts_t denote the exact token score and pt=softmax⁡(s)tp_t=\operatorname{softmax}(s)_t be the corresponding attention probability over the routable prefix. Since routing is performed over pages, for each page Pp\mathcal{P}_p we define its exact attention mass as

πp=∑t∈Pppt.\pi_p=\sum_{t\in\mathcal{P}_p}p_t.

Let Δst\Delta s_t denote the perturbation of token score sts_t induced by the low-rank residual approximation. The page score is the log-sum-exp of its token scores, so its first-order perturbation is

δp=1πp∑t∈PpptΔst+O(∥Δs∥2).\delta_p=\frac{1}{\pi_p}\sum_{t\in\mathcal{P}_p}p_t\Delta s_t+O(\lVert\Delta s\rVert^2).

Using Δst=q(UF⊤−I)et⊤/dh\Delta s_t = q(UF^{\top} - I)e_t^{\top}/\sqrt{d_h}, this becomes

δp=1dhq(UF⊤−I)(1πp∑t∈Ppptet)⊤+O(∥Δs∥2).\delta_p = \frac{1}{\sqrt{d_h}}q(UF^{\top} - I)\left(\frac{1}{\pi_p}\sum_{t \in\mathcal{P}_p}p_t e_t\right)^{\top} + O(\lVert\Delta s\rVert^2).

We therefore define the attention-weighted mean residual within page pp as

eˉp=1πp∑t∈Ppptet,\bar{e}_p = \frac{1}{\pi_p}\sum_{t \in\mathcal{P}_p}p_t e_t,

so that

δp=1dhq(UF⊤−I)eˉp⊤+O(∥Δs∥2).\delta_p = \frac{1}{\sqrt{d_h}}q(UF^{\top} - I)\bar{e}_p^{\top} + O(\lVert\Delta s\rVert^2).

After perturbing the page scores by δ\delta, the resulting page distribution is

π~p=πpexp⁡(δp)∑jπjexp⁡(δj).\tilde{\pi}_p = \frac{\pi_p\exp(\delta_p)}{\sum_j \pi_j\exp(\delta_j)}.

The KL divergence from the exact page distribution can therefore be written as

DKL(π∥π~)=log⁡(∑pπpexp⁡(δp))−∑pπpδp=12[∑pπpδp2−(∑pπpδp)2]+O(∥δ∥3).\begin{aligned} D_{\mathrm{KL}}(\pi\Vert\tilde{\pi}) = \log\left(\sum_p \pi_p\exp(\delta_p)\right) - \sum_p \pi_p\delta_p \\ &= \frac{1}{2}\left[\sum_p \pi_p\delta_p^2 - \left(\sum_p \pi_p\delta_p\right)^2\right] + O(\lVert\delta\rVert^3). \end{aligned}

Let eˉ=∑pπpeˉp\bar{e} = \sum_p \pi_p\bar{e}_p denote the mean residual over pages. Substituting the first-order expression for δp\delta_p into the quadratic term gives

12dhq(UF⊤−I)[∑pπpeˉp⊤eˉp−eˉ⊤eˉ](UF⊤−I)⊤q⊤.\frac{1}{2d_h}q(UF^{\top} - I)\left[\sum_p \pi_p\bar{e}_p^{\top}\bar{e}_p - \bar{e}^{\top}\bar{e}\right](UF^{\top} - I)^{\top}q^{\top}.

The matrix in brackets is the covariance of the page residuals under the page-attention distribution. We denote it by

Γ=∑pπp(eˉp−eˉ)⊤(eˉp−eˉ).\Gamma= \sum_p \pi_p(\bar{e}_p - \bar{e})^{\top}(\bar{e}_p - \bar{e}).

Dropping higher-order terms gives the local quadratic objective

12dhq(UF⊤−I)Γ(UF⊤−I)⊤q⊤.\frac{1}{2d_h}q(UF^{\top} - I)\Gamma(UF^{\top} - I)^{\top}q^{\top}.

Restoring the calibration-query and head indices and summing over them gives

LPF=12dh∑a,hqa,h(UhFh⊤−I)Γa,h(UhFh⊤−I)⊤qa,h⊤.(9)\mathcal{L}_{\mathrm{PF}} = \frac{1}{2d_h}\sum_{a,h}q_{a,h}(U_hF_h^{\top} - I)\Gamma_{a,h}(U_hF_h^{\top} - I)^{\top}q_{a,h}^{\top}. \tag*{(9)}

Alternating Updates

For fixed residual-side factors, differentiating equation (9) with respect to UhU_h gives

∑a(qa,h⊤qa,h)Uh(Fh⊤Γa,hFh)=∑aqa,h⊤qa,hΓa,hFh.\sum_a(q_{a,h}^{\top}q_{a,h})U_h(F_h^{\top}\Gamma_{a,h}F_h) = \sum_a q_{a,h}^{\top}q_{a,h}\Gamma_{a,h}F_h.
StatePersistent widthFull-context routingSelected attention
Compressed ValuerVr_Vrbr_brVr_V
Residual coderqr_qrqr_q–
Exact Keydhd_h–dhd_h

caption: Per-token state and logical read width for one physical KV head.

For fixed query-side factors, differentiating with respect to FhF_h gives

∑aΓa,hFh(UhTqa,hTqa,hUh)=∑aΓa,hqa,hTqa,hUh.\sum_a \Gamma_{a,h}F_h\left(U_h^\mathsf{T}q_{a,h}^\mathsf{T}q_{a,h}U_h\right)=\sum_a \Gamma_{a,h}q_{a,h}^\mathsf{T}q_{a,h}U_h.

Both updates are positive-semidefinite linear systems. We solve them by matrix-free preconditioned conjugate gradient and alternate between the two factor families.

Query Position Selection

For query head hh, let Hh=E[qhTqh]H_h = \mathbb{E}[q_h^\mathsf{T}q_h] be the uncentered query second moment over the calibration set. Define

q~w,i,h=qw,i,hHh†/2,ϕi=1WHqconcat⁡w,hq~w,i,h.\widetilde{q}_{w,i,h}=q_{w,i,h}H_h^{\dagger/2},\qquad\phi_i=\frac{1}{\sqrt{WH_q}}\operatorname{concat}_{w,h}\widetilde{q}_{w,i,h}.

Stacking these whitened features by row gives Φ\Phi and the position Gram matrix G=ΦΦTG=\Phi\Phi^\mathsf{T}. We apply diagonal-pivoted Cholesky to GG independently within positional strata, selecting 64 representative query positions for Page-Fisher calibration. In exact arithmetic, this selection is equivalent to column-pivoted QR on the corresponding whitened feature matrix ΦT\Phi^\mathsf{T} within each stratum.

Runtime and Memory Accounting

After the routing-aligned basis transformation in Appendix § 11.§ 1, the first rbr_b coordinates of the compressed Value contain all information required by the Value-to-Key predictor. Thus, no separate per-token representation is needed for the base prediction. The persistent routing state consists of the rVr_V-dimensional compressed Value and the rqr_q-dimensional residual code, while the original Key is retained separately for exact QK computation on selected pages.

Let TT be the context length and SS the number of tokens retained after page selection. If bVb_V, bRb_R, and bKb_K denote the corresponding element sizes, the context-dependent cache traffic per layer is

Bdecode=Hkv[T(rbbV+rqbR)+S(dhbK+rVbV)].(10)B_{\mathrm{decode}}=H_{kv}\left[T\left(r_bb_V+r_qb_R\right)+S\left(d_hb_K+r_Vb_V\right)\right]. \tag*{(10)}

The first term is the full-context routing scan. The second is paid only for selected pages, where the original Keys are retrieved to recompute exact QK scores and the resulting attention weights are applied to the full compressed Value states.

Without routing-aligned Value coordinates, the first term would require reading all rVr_V compressed-Value coordinates rather than only rbr_b. Thus the basis transformation reduces the context-wide Value read from rVr_V to rbr_b without changing the Value approximation. Exact Keys may remain outside GPU memory because unselected Key pages are never accessed.

Experimental Setup

Models and Calibration. We evaluate Llama-3.1-8B-Instruct, Qwen3-8B-Base, Qwen3-32B, Qwen3-8B, and Qwen3.5-9B; Qwen3-8B uses static YaRN with factor 4 for 128K routing. Value compression uses 256 C4 sequences of 2,048 tokens, activation-aware initialization, FP32 fitting, and BF16 factors. Llama-3.1 and Qwen3 use six ALS sweeps, 16 conjugate-gradient iterations per encoder update, covariance damping 10−710^{-7}, and joint decoder refits. The 75% and 50% Value-retention settings use global rank allocation (§ 4.3). Qwen3.5-9B uses 12 sweeps and 25% compression of Gated-DeltaNet output projections. Routing uses V96 for Llama-3.1-8B and Qwen3-8B, and V192 on Qwen3.5-9B full-attention layers.

Routing calibration uses 16 C4 and 16 synthetic retrieval 128K fitting windows, with 16 held-out C4 diagnostic windows. Each C4 window concatenates 32 document-disjoint 4,096-token excerpts. Synthetic windows insert multi-key, multi-value, variable-tracking, and aggregation records into C4 across eight seeded positional strata, followed by a 9–13K-token segment of 48–96 deterministic retrieval queries; identifiers are absent from the haystack and templates are independent of the evaluation benchmarks. Page-Fisher fitting uses 64 representative query positions (Appendix § 11.§ 4), 40 block-coordinate sweeps, and preconditioned conjugate gradients with at most 100 iterations and relative damping 10−510^{-5}. Unless stated otherwise, B16R16 denotes a 16-dimensional Value-derived Base and a 16-dimensional Key-derived residual, with Page4 routing and 64 recent tokens excluded from the routed prefix and retained exactly.

Evaluation Protocols. The main RULER-128K evaluation reports 10 tasks with 100 prompts per task, generated by the official implementation with seed 42 and a maximum length of 131,072 tokens. LongBench uses nine tasks: narrativeqa, multifieldqa_en, hotpotqa, musique, dureader, gov_report, samsum, passage_retrieval_en, and lcc. Rendered prompts must exceed 4,096 tokens under the evaluated tokenizer, leaving 1,543/1,549/1,564 prompts for Llama-3.1-8B/Qwen3-8B/Qwen3.5-9B. BasisKV selects 2,048 routed tokens per KV group on RULER and 256 on LongBench, plus 64 recent tokens. Sparse baselines share the compressed-Value checkpoint of Full-K and use the original selected Keys for exact QK computation after routing. LRQK and Loki perform routing independently per query head. To approximately match BasisKV’s 2,048-token group-level support, we set their per-head top-kk values to 832 and 856, respectively. After deduplicating selections across query heads within each physical KV group, the average support is 2,111.68 tokens for LRQK and 1,985.09 tokens for Loki, closely matching BasisKV’s 2,048-token budget. ShadowKV retains its native chunk, outlier, and local-support configuration. Controlled ablations specify their exceptions below. These top-kk values are chosen based on empirical support size rather than downstream task accuracy.

System Measurements. TP8 experiments use eight 48 GB NVIDIA L40S GPUs connected by PCIe 4.0×164.0 \times16 without NVLink, with NCCL through vLLM’s PyNCCL backend; serving uses 4K-token prompts and 128 generated tokens at the reported batch sizes. Sparse decode measurements use one L40S at TP1/B1 and report medians over three trials of steady-state attention-block and full-model decoding, excluding prefill.

Collective-Aware Compression Ablations

Unless stated otherwise, compression ablations use Qwen3-8B-Base with dense Keys and dense attention.

Calibration robustness. Refitting BasisKV and matched PaLU baselines on the same WikiText-2 training windows preserves the ordering of the compressed methods at both ranks (Table 5); BasisKV remains close to dense quality at 25% compression.

BudgetMethodPPL ↓\downarrowBoolQARC-EARC-CWinoG.PIQAHellaS.OBAvg.
Dense7.00250.8310.8020.5690.7280.7940.7860.4200.704
75%PaLU M9.06490.7520.6820.4800.7040.7660.7550.3940.647
75%PaLU G49.00870.7900.7450.5330.7080.7870.7930.4140.681
75%BasisKV7.11810.8210.7920.5430.7340.7900.7660.4220.695
50%PaLU M10.97820.4810.5370.3780.6650.7180.6310.3880.543
50%PaLU G49.97100.6900.5960.4430.7140.7500.7430.4280.623
50%BasisKV7.65900.7910.7060.4740.6940.7610.7060.3960.647

Table 5. WikiText-2 calibration robustness. PPL and 7-task MCQ at matched Value rank; bold marks the best compressed method per rank.

With compression fixed within each model, changing the calibration domain affects structured-generation quality on Qwen3.5-9B and Qwen3-32B, with no single corpus preferred across all tasks (Table 6). For both models, Mixed consists of 128 C4, 64 mathematics, and 64 code calibration windows, whereas Reasoning consists of 128 mathematics and 128 code windows. Each window contains 2,048 tokens. Mathematics and code prompts are drawn from OpenThoughts3, with source datasets ai2-adapt-dev/openmath-2-math and nvidia/OpenCodeReasoning, respectively; responses are generated by the corresponding target model rather than taken from the dataset’s teacher answers.

ModelCalibrationGSM8KMATH500MBPP+IFEval
Qwen3.5-9BDense93.2591.0072.7583.92
Qwen3.5-9BC486.8175.6051.5969.32
Qwen3.5-9BMixed89.2383.0069.8477.82
Qwen3.5-9BReasoning88.9385.6072.4972.64
Qwen3-32BDense89.0879.6045.7729.21
Qwen3-32BC489.0158.0035.4523.66
Qwen3-32BMixed90.6780.4042.5924.77
Qwen3-32BReasoning91.3683.0040.7426.99

Table 6. Calibration-domain transfer. Full-test-set task scores (%); bold marks the best calibration corpus within each model. Dense rows are reference scores, not calibration variants.

ALS convergence. Activation-aware initialization converges faster than random orthogonal initialization (Figure 5). Six sweeps are the fixed production budget, with a final decoder refit for every evaluated checkpoint; PPL is not strictly monotonic with additional sweeps.

Figure 5. ALS initialization and convergence. Relative MSE (left) and WikiText-2 perplexity (right) at R64 and R96. The dotted line marks six ALS sweeps.

Joint fitting. We compare BasisKV with Group-SVD, which fits each physical KV group independently using an activation-aware decomposition. BasisKV instead fits the low-rank factors against the complete attention output. As shown in Table 7, the joint fit improves held-out reconstruction error, perplexity, and average MCQ accuracy at both ranks, although some individual tasks favor Group-SVD.

RankMethodRel. MSE ↓PPL ↓BoolQARC-EARC-CWinoG.PIQAHellaS.OBAvg.
R64Group-SVD0.168069.25470.8010.7170.4640.7280.7770.7100.3880.655
R64BasisKV S60.136818.67470.7990.7470.4930.7180.7830.7270.4000.667
R96Group-SVD0.069327.37830.8060.8080.5290.7240.7910.7590.4160.690
R96BasisKV S60.055237.26790.7990.8110.5690.7210.7930.7680.4120.696

Table 7. Group-SVD versus BasisKV. WikiText-2 PPL and seven-task MCQ at R64/R96; bold marks the better method per rank and metric.

Key Routing Ablations

Routing ablations use Llama-3.1-8B-Instruct with V96 unless stated otherwise; the page-granularity comparison also includes Qwen3-8B. Calibration and default support follow Appendix C.§ 1.

Routing-calibration robustness. The synthetic routing-calibration windows are constructed independently of the evaluation benchmarks. Each 128K window consists of C4 text with synthetic records inserted across eight positional strata, followed by retrieval questions with deterministic answers. Record and query templates are written independently of RULER and LongBench, and all identifiers and values are freshly sampled. The generator does not import or reuse benchmark-specific prompts, templates, or evaluation code. The 16 synthetic windows consist of 8 multi-key, 4 multi-value, 2 variable-tracking, and 2 aggregation windows. For example, a multi-key window may contain records such as

The registry entry for ABXZ-4821 is QM-7A2F91.

[C4 text and unrelated records] Identifier RTKP-0936 has authorization code ND-C840B2.

[C4 text and unrelated records] Record: asset LWFS-7150 maps to token HK-19DE63.

[additional C4 text] Question: Retrieve the codes associated with RTKP-0936, ABXZ-4821 and LWFS-7150, in that order. Answer: The codes are ND-C840B2, QM-7A2F91, HK-19DE63.

The answers are included in the calibration sequence because these windows are used only for forward-pass statistic collection rather than as evaluation prompts. As an additional robustness check, Table 8 reports an earlier B16R16 router fitted using only C4 windows. Despite receiving no synthetic retrieval calibration, BasisKV remains competitive with the sparse routing baselines on the full 11-task RULER-128K suite. This experiment uses Page32 and a different routing configuration from the Page4 setup in the main results, so we treat it as a robustness check rather than a controlled calibration-domain ablation.

MethodS1S2S3MK1MK2MQMVVTFWEQA1QA2Avg.
Full-K100.0099.0099.0099.0091.0097.7589.7566.8065.6776.0045.0084.45
Loki80.0094.000.0092.0041.0055.7556.5020.4040.6753.0026.0050.85
ShadowKV96.0099.0096.0098.0084.0093.2587.5026.6054.3378.0045.0077.97
LRQK100.0099.0099.0099.0088.0097.5091.2544.4044.3377.0045.0080.41
BasisKV B16R16100.00100.0099.0099.0075.0097.0088.5056.8056.3376.0042.0080.88

Table 8. C4-only routing robustness on RULER-128K. Llama-3.1-8B-Instruct with uniform V96; all methods are evaluated on 11 tasks with 100 prompts per task. Full-K uses the same compressed V96 checkpoint. BasisKV B16R16 is fitted using 32 purely C4 128K windows.

Value–Key predictability. Affine predictors fitted on the calibration windows and evaluated held-out measure centered explained pre-RoPE Key energy, with targets centered by the held-out mean. These diagnostics use local or concatenated all-group Values and are not restricted to the router’s rank-16 Base. Local compressed Values retain substantial predictive Key information, and compressed-Value curves remain close to raw-Value curves across depth (Figure 6). All-group Values provide a higher diagnostic reference; routing uses only local Values.

Figure 6. Key predictability across layers. Held-out centered explained pre-RoPE Key energy on Llama-3.1-8B-Instruct and Qwen3-8B.

Base and residual components. Table 9 is a controlled 11-task RULER-128K ablation with 30 shared prompts per task, separate from the main 10-task evaluation. Base alone is insufficient, whereas R16 provides substantial routing quality. Adding the Value-derived Base improves over R16, with a paired 95% bootstrap interval of [0.80, 5.17] points, and approaches R32 while using only 16 Key-derived residual dimensions.

RouterS1S2S3MK1MK2MQMVVTFWEQA1QA2Avg.
Full-K100.00100.00100.0096.6786.6798.3394.1778.0057.7860.0036.6782.57
B1670.0086.6716.6786.6723.3360.0025.0027.3332.2253.3333.3346.78
R16100.00100.0096.6796.6750.0097.5095.8364.6743.3360.0036.6776.48
R32100.00100.0096.6796.6773.3399.1794.1768.0050.0060.0036.6779.52
B16R16100.00100.0096.6796.6776.6799.1795.8366.0043.3363.3336.6779.48

Table 9. Base and Residual contributions on RULER-128K. Llama-3.1-8B-Instruct with compressed V96. All methods use the same 30 prompts per task. Sparse routers use Page4, 2048 routed tokens and 64 recent tokens.

Page granularity. We compare Page1/4/8/32 on both models (Table 10); Page1 is token-level routing, the limiting case of Page-Fisher. Finer pages favor MK2, while coarser pages can favor FWE. Page4 has the strongest LongBench point estimate on Llama, with paired differences from Page1 and Page8 not statistically resolved; Qwen’s highest mean is at Page1. We adopt Page4 as a balanced operating point across task types rather than optimizing for a single retrieval pattern.

Table 10: Page granularity across models and tasks. Both models use V96 and B16R16 routing. LongBench reports nine tasks and their unweighted mean over 1,543/1,549 samples for Llama/Qwen; RULER-128K reports five common tasks with 100 shared prompts per task. Higher scores are better. Shading marks the adopted Page4 configuration.

ModelPage sizeNQAMQAHQAMQueDReadGRepSAMPRetrLCCAvg.MK2FWEVTQA1QA2
Llama-3.1 8B-InstructFull-K28.0650.2552.2331.3232.3432.7644.5099.5052.2247.0283.0052.6776.8074.0048.00
Llama-3.1 8B-Instruct128.7950.1552.1829.9330.7732.1845.2399.5051.2446.6680.0046.3366.0075.0048.00
Llama-3.1 8B-Instruct429.0551.0252.5130.7028.9531.4744.9099.5054.0546.9178.0046.6764.8075.0047.00
Llama-3.1 8B-Instruct827.9650.2451.4531.1530.0031.2944.4399.5053.4646.6175.0050.6763.4075.0047.00
Llama-3.1 8B-Instruct3229.1651.6251.6631.1529.5829.1043.3499.5048.9546.0166.0060.3366.8074.0046.00
Qwen3-8BFull-K25.3551.4751.6127.3417.8933.5346.52100.0062.0646.2061.0088.0093.4055.0036.00
Qwen3-8B125.4050.7851.7326.5616.7232.7445.44100.0059.4445.4257.0079.0093.2055.0034.00
Qwen3-8B426.9750.9451.4726.6316.6832.8644.6098.5058.8045.2753.0080.3393.0054.0033.00
Qwen3-8B825.4750.4751.2825.8216.0332.8844.6398.0059.6444.9153.0082.6792.4052.0031.00
Qwen3-8B3225.5148.2250.8025.1415.0331.1944.1688.5059.8043.1544.0088.6792.0049.0031.00

Table 10.

Dense-Value RULER-128K. To isolate routing quality from Value compression, we repeat the RULER-128K comparison with the original dense Values and output projection. Table 11 compares B16R16, ShadowKV, LRQK, and Loki with dense attention using their evaluated sparse configurations; support budgets are not strictly matched. BasisKV remains close to dense and competitive with the sparse baselines, with task-dependent differences: it is stronger on variable tracking and frequent-word extraction than ShadowKV and LRQK, while LRQK is stronger on multi-key, multi-query, and multi-value retrieval.

Table 11: RULER-128K with dense Values. Llama-3.1-8B-Instruct, original V128 and WOW_O; 100 shared prompts per task. Scores are percentages; the mean weights all 11 tasks equally. Bold marks the best sparse score; ties involving BasisKV highlight only BasisKV.

MethodS1S2S3MK1MK2MQMVVTFWEQA1QA2Avg.
Dense100.00100.0099.0099.0089.0098.7594.2576.2074.6778.0046.0086.81
ShadowKV100.00100.00100.0099.0076.0097.2589.5070.2069.3379.0046.0084.21
LRQK100.00100.00100.0099.0083.0098.5092.2564.2058.0080.0046.0083.72
Loki84.0097.000.0098.0031.0045.2548.0039.0033.6764.0034.0052.17
BasisKV100.00100.00100.0099.0080.0097.7588.5073.0069.6780.0047.0084.99

Table 11.

Compatibility with Quantization

Value compression. Applying KVQuant to Keys and compressed Value latents adds only small PPL changes at both evaluated ranks (Table 12), supporting compatibility with Value compression.

ModelDatasetAvg. V rankBF16 PPL ↓KVQuant (4-bit) PPL ↓ΔPPL
Qwen3-8B-BaseWikiText-2967.2837.338+0.055
Qwen3-8B-BaseWikiText-2648.2598.277+0.019
Llama-3.1-8B InstructC49610.66210.727+0.064
Llama-3.1-8B InstructC46412.45412.595+0.141

Table 12. Value compression with KVQuant. 4-bit NUQ4 with outlier handling for Keys and Value latents, simulated by quantize–dequantize on each BF16 checkpoint. Ranks are average retained Value dimensions; Δ\DeltaPPL is relative to BF16.

Table 12: Value compression with KVQuant. 4-bit NUQ4 with outlier handling for Keys and Value latents, simulated by quantize–dequantize on each BF16 checkpoint. Ranks are average retained Value dimensions; Δ\DeltaPPL is relative to BF16.

Sparse routing. BasisKV’s LongBench average stays close to BF16 with FP8 and decreases slightly with KVQuant (Table 13). Routing remains effective when the Base is computed from dequantized Value latents; residual codes remain BF16. Both tables use quantize–dequantize quality simulations.

MethodKV formatNQAMQAHQAMQueDReadGRepSAMPRetrLCCAvg.
Full-KBF1628.0650.2552.2331.3232.3432.7644.5099.5052.2247.02
Full-KFP828.7350.3752.3231.2933.0633.0645.0599.0051.3447.14
Full-KKVQuant (4-bit)28.6250.0552.7831.4931.8732.6345.3799.5052.8347.24
BasisKVBF1629.0551.0252.5130.7028.9531.4744.9099.5054.0546.91
BasisKVFP828.7650.9852.6230.7130.4131.4544.5599.5052.6246.85
BasisKVKVQuant (4-bit)28.1449.7952.8130.5228.8331.4843.9699.5053.4346.50

Table 13. Sparse routing with quantized KV caches. Llama-3.1-8B-Instruct, uniform V96, 1,543 shared LongBench samples. Full-K: full attention; BasisKV: B16R16 Page4. FP8: E4M3; KVQuant: NUQ4 with outlier handling. Avg.: unweighted nine-task mean; higher is better.

Table 13: Sparse routing with quantized KV caches. Llama-3.1-8B-Instruct, uniform V96, 1,543 shared LongBench samples. Full-K: full attention; BasisKV: B16R16 Page4. FP8: E4M3; KVQuant: NUQ4 with outlier handling. Avg.: unweighted nine-task mean; higher is better.

References

  1. [1]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119–3137. Association for Computational Linguistics, 2024.arxiv.org/abs/2308.14508
  2. [2]Priyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang, SeungJae Lee, Jungwook Choi, and Mingu Kang. STAR-KV: Low-rank KV cache compression via soft thresholding for adaptive rank control. arXiv preprint arXiv:2606.08382, 2026.arxiv.org/abs/2606.08382
  3. [3]Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024.arxiv.org/abs/2406.02069
  4. [4]Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: KV-cache compression with low-rank projection. In International Conference on Learning Representations, 2025.arxiv.org/abs/2407.21118
  5. [5]Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Hung-Yueh Chiang, Yash Akhauri, Xilai Dai, Huiqiang Jiang, Yucheng Li, Luis Ceze, Kai-Chiang Wu, and Mohamed S. Abdelfattah. XKV: Cross-layer KV-cache compression via aligned singular vector extraction. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2503.18893
  6. [6]Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.01801
  7. [7]Jan Hansen-Palmus, Michael Truong Le, Oliver Hausdörfer, and Alok Verma. Communication compression for tensor parallel llm inference. arXiv preprint arXiv:2411.09510, 2024.
  8. [8]Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2401.18079
  9. [9]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024.arxiv.org/abs/2404.06654
  10. [10]You-Liang Huang, Xinhao Huang, Chengxi Liao, and Zeyi Wen. Deinfer: Efficient parallel inferencing for decomposed large language models. arXiv preprint arXiv:2604.17709, 2026.DOI
  11. [11]Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: An efficient KV cache compression recipe for near-lossless generative inference of LLM. arXiv preprint arXiv:2403.05527, 2024.arxiv.org/abs/2403.05527
  12. [12]Tenghui Li, Guoxu Zhou, Xuyang Zhao, Yuning Qiu, and Qibin Zhao. Efficient low rank attention for long-context inference in large language models. In Advances in Neural Information Processing Systems, volume 38, pp. 43048–43074, 2025.arxiv.org/abs/2510.23649
  13. [13]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2404.14469
  14. [14]Bokai Lin, Zihao Zeng, Zipeng Xiao, Siqi Kou, TianQi Hou, Xiaofeng Gao, Hao Zhang, and Zhijie Deng. MatryoshkaKV: Adaptive KV compression via trainable orthogonal projection. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.14731
  15. [15]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 32332–32344. PMLR, 2024.DOI
  16. [16]Junlin Mu, Hantao Huang, Jihang Zhang, Minghui Yu, Tao Wang, and Yidong Li. SALS: Sparse attention in latent space for KV cache compression. In Advances in Neural Information Processing Systems, volume 38, pp. 428–447, 2025.
  17. [17]Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. SparQ attention: Bandwidth-efficient LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42558–42583. PMLR, 2024.arxiv.org/abs/2312.04985
  18. [18]Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention in low-rank space for kv cache compression. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 15332–15344, 2024.
  19. [19]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.

Paper details

vlnc:W_mtwmh5zw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents