The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

Introduction

Many large language model (LLM) applications repeatedly query a fixed context such as a document, patient record, or codebase. The key–value cache (KV-cache) for such a context grows linearly with its length and occupies a large fraction of the memory used during long-context inference. Every attention layer reads this cache for every generated token, so every request pays for its size in both memory and latency. Offline compaction reduces this cost by processing the context once and compressing the resulting cache, which is then used to serve all later requests ([6, 17, 34]).

Unlike serving-time eviction, which discards cache entries within the forward pass of a request ([33, 19]), offline compaction can spend a larger one-time budget that is amortized over future requests. In particular, it can use calibration queries, i.e., query vectors the model produces while re-reading or discussing the context, to decide which entries to retain. Eyuboglu et al. [6] introduce this regime with Cartridges, which trains a new latent cache per context. More recent work makes offline compaction much cheaper by replacing end-to-end training with eviction-inspired selection and efficient optimization of the remaining cache (Zweiger et al. [34]).

The central problem in cache compaction is deciding which cache entries can be removed while preserving the resulting attention outputs. For one attention head, a cache of $T$ entries contains a key $k_j \in\mathbb{R}^{d_k}$ and a value $v_j \in\mathbb{R}^{d_v}$ at each position $j = 1,\ldots,T$. A query $q \in\mathbb{R}^{d_k}$ produces

where $P_j$ is the attention weight of entry $j$ and $o$ is the resulting output. Removing cache entry $j$ does not simply subtract $P_jv_j$, since softmax renormalization changes the attention weights of all remaining entries. The effect of one deletion therefore depends on which other entries remain, making selection a combinatorial problem rather than a ranking problem. Scores based only on received attention, as in attention matching Zweiger et al., [34], additionally overlook value redundancy: if an entry’s value is close to the output the head already produces, i.e., $\lVert v_j-o\rVert_2$ is small, removing it might change the output little, even when its attention weight is large. Good selection must therefore account not only for the contribution of each entry, but also for how deletions interact.

Recent output-aware methods instead estimate the output error caused by deleting a single entry [10, 2, 13, 7]. CAOTE Goel et al., [10] derives the exact change caused by deleting one entry for a single calibration query, and OBCache Gu et al., [13] uses a Taylor approximation of the attention output error aggregated across queries. These methods nevertheless evaluate the deletion of one entry while the rest of the cache is held fixed. Attention matching Zweiger et al., [34] refits the retained cache to minimize the error of the compacted output, but only after the retained set has been chosen. What is missing is compensation-aware selection: each candidate deletion should be scored by the error remaining after the surviving entries have been adjusted to compensate for it, so that selection and refitting are optimized jointly rather than in sequence. This is difficult under softmax attention because deletion changes the normalization itself, so the error after both deletion and compensation has no obvious closed form.

In this work, we substantially advance the cost–accuracy Pareto frontier for offline KV-cache compaction. We derive a quadratic surrogate that makes compensation-aware selection tractable. We introduce KV-SURGEON, which uses this surrogate for greedy, compensation-aware elimination. For each candidate deletion, the surrogate admits closed-form expressions for both the optimal compensating update and the resulting increase in surrogate loss. Each candidate entry is assigned a score after the remaining entries have been adjusted to compensate for its removal, and the update is then applied to the cache, so the next deletion is chosen against the compensated entries, as Optimal Brain Surgeon (OBS) Hassibi & Stork, [15] does for network weights.

We evaluate KV-SURGEON on QuALITY, LongHealth, and Long Code Arena, using dense and mixture-of-experts models from 4B to 30B parameters and compaction ratios from 5× to 100×. Across these settings, KV-SURGEON improves the cost–accuracy frontier at every ratio, achieving higher mean accuracy at comparable cost. Mean question-answering accuracy across our experiments exceeds the state of the art by 4.3 percentage points at 100× compaction and 3.0 percentage points at 50×. The gains are less pronounced at 5× and 10×, where the strongest methods already suffer almost no deterioration relative to the uncompacted cache. At $50 \times$ compaction the state of the art, attention matching with its AM-HAK selection rule [34], reaches 62.0% accuracy in 138 seconds² per context. KV-SURGEON reaches 65.1% in 145 seconds, and KV-SURGEON-fast reaches 63.9% in 54 seconds. Figure 1 shows the resulting frontier at $50 \times$ compaction on the largest model we evaluate.

Our contributions are the following.

1. _Quadratic surrogate._ We derive a quadratic surrogate to the exact attention-output error, which allows us to design efficient methods for approximately solving the compaction problem.

2. _Compensation-aware selection._ Based on this surrogate, we introduce KV-SURGEON, which iteratively removes entries while adjusting the remaining entries in closed form after each deletion, then jointly refits the retained cache. Unlike prior output-aware scores or post-selection refitting, compensation directly influences which entries are retained.

3. _A new cost–accuracy Pareto frontier._ Across QuALITY, LongHealth, and Long Code Arena, dense and mixture-of-experts models from 4B to 30B parameters, and $5 \times$ to $100 \times$ compaction, KV-SURGEON substantially advances the frontier, with gains at every evaluated ratio and the largest improvements at $50 \times$ and $100 \times$. We show empirically that accounting for compensation during selection substantially improves accuracy over exact single-deletion scoring.

Related work

Other forms of KV-cache compression. KV-caches can be compressed along several axes. Quantization lowers the precision of stored keys and values [22, 16], and low-rank methods reduce the feature dimension [26, 4]. Both are complementary to reducing the number of entries. A different route, _LLM summarization_, shortens the text input itself by prompting an LLM to summarize it. This approach transfers across models, but gives no direct control over cache size. We study compaction along the sequence dimension, where a fixed budget of entries is retained from an already computed cache.

Offline compaction. Offline methods exploit contexts that are processed once and queried repeatedly. Cartridges [6] trains a new latent cache per context from self-generated data. KVzip [17] removes cache entries based on the attention they receive while the model repeats the context verbatim. Attention matching [34] also keeps a subset of the original keys, then fits a scalar logit bias per retained key and refits the stored values to reproduce the original attention outputs on reference queries. Its AM-HAK variant selects entries by root-mean-square attention, and AM-OMP, based on orthogonal matching pursuit (OMP), greedily selects keys so that their reweighted attention reproduces the attention mass of the full cache. In both, the support is chosen from attention statistics before the values are refit, so the ability of the retained entries to reproduce the output does not influence selection.

Eviction by proxy scores. Serving-time eviction discards entries within a single request, ranking each one by a score that ignores the rest of the cache: accumulated attention, as in H2O [33] and related methods [24, 19, 21], or a query-free proxy such as position [30], key norm [5], alignment with a principal query direction [9], deviation from neighboring keys [20], or value norm [14]. Merging methods soften the same decision by folding evicted entries into retained ones. KVMerger [29] merges similar entries into attention-weighted pivots, and Zhang et al. [32]; Wan et al. [28]; Nawrot et al. [23] use related rules. In all of these, the score of an entry is fixed before any other entry is removed.

Output-aware eviction. More recent methods score entries directly through the attention-output error induced by their removal [10, 2, 13, 7]. CAOTE [10] and ReST-KV [2] derive the exact effect of deleting one entry for one query. OBCache [13] derives a second-order approximation in the style of Optimal Brain Damage LeCun et al., [18]. CriticalKV Feng et al., [7] considers deletion of an arbitrary set for one query and derives an additive upper bound on the resulting output error for selection. These methods differ in how they estimate deletion error, but share a common restriction: candidates are evaluated with the retained cache held fixed.

Our work starts from the same output-reconstruction objective as CAOTE, ReST-KV, and OBCache but changes how it is used for selection. Instead of scoring one deletion against a fixed cache, we derive a quadratic surrogate of the output error over arbitrary deletions and reweightings of the retained entries, aggregated over many calibration queries, and let the retained entries compensate during greedy deletion of cache entries.

KV-Surgeon

In this section, we present our method for offline KV-cache compaction, KV-Surgeon. It compacts each attention head independently by selecting which original cache entries to delete, accounting for how the remaining entries can compensate for deleted ones, and then refits the retained entries to preserve the behavior of the original cache.

Problem formulation

We consider one layer and one attention head with $T$ cache entries. Cache entry $j$ consists of a key $k_j \in\mathbb{R}^{d_k}$ and a value $v_j \in\mathbb{R}^{d_v}$. We describe the behavior of the head on $N$ calibration queries $q_1,\ldots,q_N \in\mathbb{R}^{d_k}$ recorded from the model itself as described in § 4.1. For query $i$ and cache entry $j$, the corresponding logit is $Z_{ij} \triangleq q_i^{\mathsf{T}}k_j/\sqrt{d_k}$. The attention weight and output are

Let $V \in\mathbb{R}^{T \times d_v}$ collect the values as rows and $P \in\mathbb{R}^{N \times T}$ the attention weights. The corresponding output matrix is then $O = PV \in\mathbb{R}^{N \times d_v}$.

Compacted cache. Given a budget $t < T$, which is allocated per head, we retain $t$ cache entries corresponding to some index set $S \subset\{1,\ldots,T\}$ with $|S| = t$, which we call the support. Each retained entry $j \in S$ keeps its original key $k_j$ and is assigned an adapted value $\widehat{v}_j \in\mathbb{R}^{d_v}$ and a positive attention multiplier $w_j > 0$, which rescales the contribution of its entry before the softmax normalization. Together, the remaining keys $k_j$, adapted values $\widehat{v}_j$, and attention multipliers $w_j$ form the compacted cache. For query $i$ and entry $j$, the attention weights and output of the compacted cache are given by

Here,

is the ratio between the compacted and original attention normalization terms. Let $P_S \in\mathbb{R}^{N \times|S|}$ contain the columns of $P$ indexed by $S$, let $w \in\mathbb{R}^{|S|}$ collect the multipliers, and let $\widehat{V} \in\mathbb{R}^{|S| \times d_v}$ collect the new values as rows. The vector of the normalization ratios is then $\mu\triangleq(\mu_1,\ldots,\mu_N)^{\mathsf{T}} = P_Sw$ and the output of the compacted cache is $\widehat{O} = \operatorname{diag}(P_Sw)^{-1}P_S\operatorname{diag}(w)\widehat{V}$.

Compaction objective. Our goal is for the compacted cache to preserve two properties of the original cache. First, its attention outputs $\widehat{O}$ should be close to the outputs $O$ of the original, uncompacted cache. We measure the output error by

Second, when combined with other context, such as a question, a system prompt, or other compacted chunks, its entries should collectively receive approximately the same share of attention as the original cache entries, i.e., $\sum_{j\in S} w_j e^{Z_{ij}} \approx\sum_{j=1}^{T} e^{Z_{ij}}$. Output preservation alone does not ensure this: scaling all $w_j$ by the same constant leaves $\hat{O}$ unchanged but changes this sum. For a more detailed discussion, see Appendix § 14.

On the calibration queries, we express these two goals as follows, using the ratio $\mu$ of the compacted and original normalization terms defined in (3)

Tractable surrogate

Choosing the optimal support $S$ is difficult even if we consider only output preservation. For a fixed support $S$, the output error $\mathcal{L}(S,w,\hat{V})$ depends on how well the multipliers $w$ and values $\hat{V}$ can be adjusted, so evaluating all $\binom{T}{t}$ possible supports is infeasible. In order to approximate this objective, we start from a larger support and sequentially delete entries. However, even assessing the cost of deleting one entry $j\in S$ requires re-optimizing $w$ and $\hat{V}$ on $S\setminus\{j\}$. Doing that for every candidate deletion at every step is still prohibitively expensive.

We therefore seek a surrogate objective that makes it inexpensive to compare candidate deletions while accounting for how the remaining entries can compensate. After removing an entry, we also want to reuse previous computations to evaluate the next deletions efficiently. Starting from (5), we derive a quadratic surrogate with these properties.

Attention as a ratio of two linear terms. We introduce the variable $U \triangleq\operatorname{diag}(w)\hat{V}\in\mathbb{R}^{|S|\times d_v}$, whose row corresponding to entry $j\in S$ is $u_j = w_j\hat{v}_j^{\mathsf{T}}$ and which absorbs the attention multipliers $w$ into the values $\hat{V}$. This is an exact change of variables: since $w_j>0$, we can recover the original values as $\hat{v}_j=u_j/w_j$. Substituting into (2), the output of the compacted cache can be written as the ratio

The numerator $n_i$ and the denominator $\mu_i$ are linear in $U$ and $w$, respectively.

For the original cache, the numerator is $\sum_{j=1}^{T}P_{ij}v_j=o_i$ and the denominator is $\sum_{j=1}^{T}P_{ij}=1$, since every row of $P$ sums to one. Instead of searching for an $\hat{o}_i$ matching this ratio, we ask the compacted cache to reproduce numerator and denominator separately, $n_i\approx o_i$ and $\mu_i\approx1$. The second condition is the mass requirement of (5), and together they imply $\hat{o}_i\approx o_i$. Define the residuals $r_i\triangleq n_i-o_i$ and $\varepsilon_i\triangleq\mu_i-1$,

The output error is thus small whenever both residuals are small: a small $\varepsilon_i$ also keeps $\mu_i$ close to one.

Quadratic surrogate. We construct a surrogate of (4) by summing the squared residuals from (7) over all queries. In matrix form, the $r_i$ are the rows of $P_S U-O\in\mathbb{R}^{N\times d_v}$, and the $\varepsilon_i$ are the entries of $\mu-1=P_Sw-1\in\mathbb{R}^{N}$. A weight $\gamma^2>0$ controls the relative importance of mass preservation (see Appendix § 7.§ 3), giving

The original loss (4) couples $w$ and $\hat{V}$ through their products and the division by $\mu_i$, whereas for a fixed support $S$, the quadratic surrogate separates the optimization of $U$ and $w$ into two linear least-squares problems with the same design matrix $P_S\in\mathbb{R}^{N\times|S|}$. The right-hand side stacks the targets into $[O\ \ \gamma1]\in\mathbb{R}^{N\times(d_v+1)}$ and the coefficients into $[U\ \ \gamma w]\in\mathbb{R}^{|S|\times(d_v+1)}$, yielding a single linear least-squares problem with $d_v+1$ target columns.

Support-selection score. In (8), $U$ is unconstrained, since any $U$ is attained by $\widehat{V} = \operatorname{diag}(w)^{-1}U$, but the last column of $[U\ \gamma w]$ is constrained by $w > 0$. To obtain an unconstrained quadratic problem, we remove this constraint and replace $[U\ \gamma w]$ with a free matrix $B \in\mathbb{R}^{|S| \times(d_v+1)}$. Each row of $B$ corresponds to a retained cache entry $j$: its first $d_v$ coefficients replace the weighted value $w_j\widetilde{v}_j$, and its last coefficient replaces the scaled attention multiplier $\gamma w_j$. These coefficients need not correspond to a valid compacted cache because the last coefficient can be negative. In practice, fewer than half a percent of the retained entries end with a negative coefficient in the last column (Appendix § 15), and valid multipliers are restored by the final refit (§ 3.4).

Finally, we add a small ridge penalty so that the regression has a unique solution for every support. With $Y \triangleq[O\ \gamma1] \in\mathbb{R}^{N \times(d_v+1)}$, we define the support score

$Y$ and $\lambda$ stay fixed during support selection: $Y$ depends only on the original cache, and $\lambda$ is set once per head (Appendix § 8). For a fixed support $S$, $\mathcal{F}(S)$ is the minimum value of this quadratic objective after optimizing all coefficients in $B$. Changing $S$ changes the available columns of $P_S$ and therefore also the optimal coefficients: removing an entry requires the remaining coefficients to compensate for its absence.

Greedy elimination with closed-form compensation

With the support score $\mathcal{F}(S)$ in place, we now develop a greedy elimination algorithm that uses it to choose which cache entries to retain. Starting from a support larger than the target budget $t$, we repeatedly remove the entry whose deletion causes the smallest increase in $\mathcal{F}(S)$, allowing the remaining coefficients to compensate at each step. The quadratic structure of (9) lets us compute both these loss increases and the compensating updates in closed form, using an approach developed in the pruning literature ([15, 8]). We first describe the optimal coefficients on the current support, then show how to evaluate and apply a deletion.

Optimal coefficients on the current support. Each column of $B$ contains the coefficients used to reconstruct one target column of $Y$: either an output coordinate or the attention mass. In (9), these linear least-squares problems all have the same Hessian $2(P_S^\mathsf{T}P_S + \lambda I)$ (see Appendix § 7.§ 2). A single $|S| \times|S|$ inverse $G$ therefore suffices for all columns. Writing $B$ and $G$ for the optimal coefficients and shared inverse on the current support $S$, we have

The current $B$ and $G$ suffice to evaluate every candidate deletion and obtain the optimal coefficients afterward, without solving another regression problem. For an entry $j \in S$, we define its saliency $s_j$ as the increase in the optimal surrogate objective when that entry is removed:

The saliency $s_j$ measures the surrogate loss increase after the remaining coefficients have been re-optimized. An entry can make a large contribution to the original outputs yet have low saliency if other retained entries can compensate for its removal.

Once an entry $j$ is selected for removal, we obtain the optimal coefficients and inverse for the reduced support directly from the current matrices. Writing $-j$ for the indices in $S \setminus\{j\}$, the updates are

The updated $B$ is the exact minimizer of (9) on $S \setminus\{j\}$, and $G$ is the corresponding inverse Gram matrix. We repeatedly remove the entry with the smallest current saliency, update $B$ and $G$, and recompute the remaining saliencies until $|S| = t$, without solving another regression problem from scratch. A detailed derivation is given in Appendix § 7.§ 4.

Implementation details

KV-Surgeon builds on the greedy elimination process described above, with a few practical additions. To reduce computational cost, we restrict the initial set of candidates, delete entries in blocks, and use a cheaper selection rule for heads that retain a relatively large fraction of their original cache entries. After selecting the support, we refit the attention multipliers $w$ and values $\hat{V}$ to obtain a valid compacted cache. Algorithm 1 summarizes the complete method. We describe these additions below, with further implementation details in Appendix § 8 and ablations in Appendix § 11.§ 3.

Algorithm 1 KV-Surgeon for one attention head

Require: keys $K \in\mathbb{R}^{T \times d_k}$, values $V \in\mathbb{R}^{T \times d_v}$, calibration queries $Q \in\mathbb{R}^{N \times d_k}$, cache budget $t$, pool factor $c$, block size $n_b$, relative ridge $\lambda_0$, mass weight $\gamma_0$, fallback threshold $\tau$, multiplier bound $\beta$

1. $P \gets\operatorname{softmax}(QK^\top/\sqrt{d_k})$ row-wise, $O \gets PV$

2. $s_j^{\mathrm{LOO}} \gets\sum_{i=1}^{N} \frac{P_{ij}^{2}}{(1-P_{ij})^{2}}\lVert v_j-o_i\rVert_{2}^{2}$ for $j=1,\ldots,T$

3. if $t > \tau T$ then ▷ mild-compaction fallback

4. $S \gets$ indices of the $t$ largest $s_j^{\mathrm{LOO}}$

5. return $S$, $w=1$, $\hat{V}=V_{S,:}$

6. end if

7. $m \gets\min\{T,\lceil ct\rceil\}$, $S \gets$ indices of the $m$ largest $s_j^{\mathrm{LOO}}$ ▷ select candidate pool

8. $\gamma\gets\gamma_0\operatorname{RMS}_i\lVert o_i\rVert_2$, $Y \gets[O\ \gamma\mathbf{1}]$

9. $\lambda\gets\lambda_0\operatorname{tr}(P_S^\top P_S)/m$

10. $G \gets(P_S^\top P_S+\lambda I)^{-1}$, $B \gets GP_S^\top Y$

11. while $|S|>t$ do ▷ greedy elimination

12. $s_j \gets\lVert B_{:,j}\rVert_2^2/G_{jj}$ for $j\in S$

13. $J \gets$ the $\min\{n_b,|S|-t\}$ entries of $S$ with the smallest $s_j$

14. $R \gets S\setminus J$

15. $B \gets B_{R,:}-G_{R,J}G_{J,J}^{-1}B_{J,:}$

16. $G \gets G_{R,R}-G_{R,J}G_{J,J}^{-1}G_{J,R}$

17. $S \gets R$

18. end while

19. $w \gets\operatorname*{arg\,min}_{e^{-\beta}\leq w_j\leq e^\beta}\lVert\mathbf{1}-P_Sw\rVert_2^2$ ▷ final refit of attention multipliers

20. $\hat{P} \gets\operatorname{diag}(P_Sw)^{-1}P_S\operatorname{diag}(w)$

21. $\hat{V} \gets\operatorname*{arg\,min}_{\hat{V}}\lVert\hat{P}\hat{V}-O\rVert_F^2$ ▷ final refit of values

22. return $S,w,\hat{V}$

Candidate pool. To limit the size of $G$ and the number of deletion steps, we first rank entries by a leave-one-out (LOO) score. We obtain it by removing entry $j$ in (2), keeping unit multipliers and the remaining values unchanged, and evaluating the output-preservation loss (4):

LOO measures the exact output error on the original cache without adjusting the remaining entries, whereas the saliency in (11) measures the surrogate loss increase accounting for compensation. We keep the $m=\min\{T,\lceil ct\rceil\}$ entries with the largest $s_j^{\mathrm{LOO}}$ and perform greedy elimination only on this pool, using $c=4$. We also use (13) directly as the score of the LOO baseline (§ 4.1). The score derivation and pool-size ablation are given in Appendix § 7.§ 5 and Appendix § 11.§ 3, respectively.

Block deletion. We select a block $J\subset S$ containing the entries with the smallest current saliencies. Analogously to (12), the optimal coefficients and inverse for the reduced support $R=S\setminus J$ are obtained by

For any selected block $J$, these updates give the exact minimizer of the surrogate on $S\setminus J$. Selecting $J$ by individual saliencies, however, need not minimize the joint deletion cost. Empirically, deleting multiple entries at once improves efficiency without hurting performance. Further implementation details are given in Appendix § 8.

Final refit. The support-selection procedure determines which entries to retain, but the unconstrained coefficients $B$ need not represent a valid compacted cache. We therefore discard $B$ and $G$ and refit the multipliers $w$ and values $\hat{V}$ on the selected support $S$. Following Zweiger et al. [34], we first minimize $\lVert1-P_{S}w\rVert_{2}^{2}$ subject to box constraints on $w$, obtaining positive multipliers that preserve the original attention mass as closely as possible. We then fix $w$ and optimize $\hat{V}$ to minimize the output-preservation loss (4). The resulting cache retains the original keys with refitted multipliers and values. See Appendix § 8 for further details.

Mild-compaction fallback. For heads that retain a relatively large fraction of their original cache entries, we skip greedy elimination and the final refit, retaining the $t$ entries with the largest LOO scores, their original values, and unit multipliers. Since these heads undergo little compaction, this simpler procedure reduces computation with negligible accuracy loss. Further implementation details are given in Appendix § 8 and the corresponding ablation in Appendix § 11.§ 3.

Experiments

We compare KV-SURGEON to existing methods and show that it improves the cost–accuracy frontier across different models, benchmarks, and compaction ratios. We then analyze which part of the algorithm is the source of this improvement through controlled ablations. Finally, we study how much of the gain is retained when self-study calibration is replaced by a single repeat pass. Our code is available at github.com/ZIB-IOL/KV-Surgeon.

Experimental setup

Benchmarks and models. We evaluate offline KV-cache compaction on long-context multiple-choice question-answering tasks from QuALITY [25] and LongHealth [1], and code completion tasks from Long Code Arena [3], where each compacted context is reused across many questions. We evaluate five models, Qwen3-4B, Qwen3-4B-Instruct-2507, Llama-3.1-8B-Instruct, Gemma-3-12B, and Qwen3-30B-A3B [31, 11, 27], spanning three model families, dense and mixture-of-experts architectures, and different attention variants. Qwen3-4B-Instruct-2507 replaces Qwen3-4B on LongHealth and Long Code Arena because it supports longer contexts. We consider $\{100\times,50\times,20\times,10\times,5\times\}$ compaction, with $50\times$ as the primary operating point. Dataset construction and evaluation details are given in Appendix § 9.

Protocol. All accuracies are means over five random seeds, with one standard error reported in the tables. All methods shown in the main comparison, except LLM summarization, use heterogeneous allocation of the global cache budget across heads. The allocation is computed once per model on data disjoint from the evaluation benchmarks following Zweiger et al. [34] and kept fixed between methods. All methods using calibration additionally share the attention-multiplier parameterization and final cache refit of § 3, so their comparisons isolate differences in support selection. KV-SURGEON hyperparameters are fixed once on Qwen3-4B with QuALITY at 50× compaction and reused for all other models, benchmarks, and compaction ratios, i.e., KV-SURGEON does not require per-model or per-benchmark tuning.

Calibration and cost. To collect calibration queries, we use two different approaches. Cheap calibration queries are obtained by a repeat pass, which concatenates the context with a “repeat verbatim” prompt and another copy of the context (Kim et al., [17]). For self-study, the more expensive approach, we prompt the model to discuss the context multiple times and record the queries produced during answer generation (Eyuboglu et al., [6]). The suffix -fast denotes repeat-pass calibration for KV-SURGEON and AM-HAK. AM-OMP-CS is the cheap-solver setting of Zweiger et al. [34], named AM-OMP-fast in the original work, and uses self-study queries. Pipeline time covers calibration-query generation and compaction, but excludes the context prefill shared by all methods and preprocessing performed once per model. Further timing and calibration details are given in Appendix § 9.

Baselines. Our main prior baseline is attention matching (Zweiger et al., [34]), for which we evaluate both AM-HighestAttentionKeys (AM-HAK) and AM-OMP using the authors’ implementation and hyperparameters. We also adapt several serving-time eviction methods to the offline setting (Appendix § 13) and include our adaptation of OBCache in the main experiments. LOO ranks entries by the exact single-deletion score of (13) and shares KV-SURGEON’s calibration queries, head-budget allocation, and final refit, providing a controlled comparison to independent selection. We additionally include LLM summarization and query-free baselines (Appendix § 10).

KV-Surgeon improves the cost–accuracy frontier

At 50× compaction, both KV-SURGEON and KV-SURGEON-fast exceed the prior state of the art (Table 1). KV-SURGEON-fast reaches 63.9% mean accuracy in 54 seconds per context, compared with 63.2% for AM-OMP in 763 seconds, a 14.1× reduction in pipeline time. Self-study calibration improves KV-SURGEON further to 65.1% in 145 seconds, 1.9 percentage points above AM-OMP at 5.3× lower time. At comparable cost, KV-SURGEON-fast is 3.3 percentage points ahead of adapted OBCache, while KV-SURGEON is 3.0 points ahead of AM-HAK.

Method	Time (s)	Qwen3-4B QuAL.	Qwen3-4B LongH.	Llama-3.1-8B QuAL.	Llama-3.1-8B LongH.	Gemma-3-12B QuAL.	Gemma-3-12B LongH.	Qwen3-30B QuAL.	Qwen3-30B LongH.	Mean
Repeat-pass
KV-SURGEON-fast (ours)	54	65.9 ±0.4	66.9 ±0.5	61.0 ±0.4	61.5 ±0.6	68.1 ±0.5	52.1 ±0.4	71.4 ±0.3	64.1 ±0.2	63.9 ±0.15
OBCache (adapted)	58	62.4 ±0.7	62.8 ±0.9	59.8 ±0.3	52.4 ±0.2	66.6 ±0.4	50.2 ±0.8	67.3 ±0.4	62.8 ±0.4	60.5 ±0.20
AM-HAK-fast	54	58.1 ±0.5	53.5 ±1.3	58.7 ±0.6	53.5 ±1.1	62.2 ±0.4	45.6 ±1.2	65.8 ±0.5	59.8 ±0.6	57.1 ±0.30
Self-study
KV-SURGEON (ours)	145	66.8 ±0.3	67.8 ±0.8	62.3 ±0.5	62.1 ±0.3	67.5 ±0.7	56.4 ±0.7	72.5 ±0.6	65.2 ±0.4	65.1 ±0.20
LOO	139	65.6 ±0.2	64.2 ±1.2	61.2 ±0.7	59.1 ±0.6	65.1 ±0.5	49.6 ±0.7	69.6 ±0.3	62.3 ±0.7	62.1 ±0.24
AM-HAK	138	64.1 ±0.9	65.0 ±0.8	60.4 ±0.5	58.2 ±0.5	65.1 ±0.3	52.7 ±1.4	69.5 ±0.3	61.4 ±0.4	62.0 ±0.26
AM-OMP-CS	266	65.7 ±0.4	65.9 ±1.1	60.6 ±0.8	55.8 ±0.9	67.2 ±0.2	50.9 ±0.8	69.7 ±0.5	63.1 ±0.2	62.4 ±0.24
AM-OMP	763	66.4 ±0.2	65.5 ±0.6	61.3 ±0.6	57.3 ±0.9	67.7 ±0.3	54.5 ±0.9	69.2 ±0.4	63.9 ±0.6	63.2 ±0.22
Q-Filters (query-free)	1.6	52.4 ±0.5	43.6 ±0.4	47.7 ±0.8	36.1 ±1.1	55.9 ±0.3	39.0 ±0.7	50.9 ±0.3	43.2 ±0.5	46.1 ±0.22
Summarization	5.1	51.2 ±0.5	28.0 ±0.7	46.7 ±0.3	31.5 ±0.7	50.2 ±0.4	39.4 ±0.4	54.2 ±0.6	47.2 ±0.6	43.6 ±0.19
Full context	–	72.1 ±0.3	80.5 ±0.8	62.7 ±0.5	73.0 ±0.8	70.6 ±0.4	74.5 ±0.4	80.7 ±0.4	78.3 ±0.5	74.0 ±0.19

These gains are consistent across models and benchmarks. For both calibration methods, KV-SURGEON is best or tied for best within uncertainty on all eight model–benchmark pairs, while KV-SURGEON-fast is ahead of every other repeat-pass method beyond uncertainty. Thus, the more expensive AM-OMP variants no longer lie on the cost–accuracy frontier at our primary operating point (Figure 1).

The advantage of KV-SURGEON also persists across compaction ratios. Figure 2 shows mean accuracy over the eight model–benchmark pairs relative to KV-SURGEON. All prior methods are worse by more than two standard errors at every compaction ratio. The gap is largest between 100× and 20× compaction and narrows at 10× and 5×, where all methods approach the performance of the full-context ceiling. We therefore focus the analysis below on large compaction ratios, where support selection has the largest effect.

The same compute-time advantage transfers to project-level code completion on Long Code Arena (Table 12, Appendix § 10.§ 1). At 50×, KV-SURGEON and KV-SURGEON-fast match AM-OMP within uncertainty while using 9.7 and 21.5 times less compute time, respectively. At 20×, KV-SURGEON has the highest point estimate, with AM-OMP-CS the only prior method within uncertainty at 6.7 times higher cost.

Table 12: Code completion accuracy (%) and pipeline time vs. compaction ratio on Long Code Arena project-level code completion with Qwen3-4B-Instruct-2507 (20 Python repository snapshots, $n = 650$ next-line completions, exact match of the first generated line). Each entry is the mean over five seeds; the grey number is the standard error over seeds. Time is average query-generation plus compaction wall-clock per repository, in seconds. Full-context ceiling: 40.6%. The best accuracy per ratio is bold; underlined entries are within $2\sqrt{\mathrm{se}^{2}+\mathrm{se}_{\mathrm{best}}^{2}}$ of the best. ‘–’ marks ratios the AM-OMP solvers do not reach.

Method	Accuracy (%)					Time (s)
	100×	50×	20×	10×	5×	100×	50×	20×	10×	5×
StreamingLLM	22.6 ±0.2	23.3 ±0.2	22.0 ±0.1	24.5 ±0.1	32.5 ±0.1	7.62	8.07	7.78	8.18	8.39
L2 key-norm	23.2 ±0.1	23.4 ±0.1	23.4 ±0.2	23.4 ±0.1	27.6 ±0.2	7.45	8.55	8.77	7.70	8.12
Q-Filters	22.8 ±0.1	22.4 ±0.2	23.8 ±0.1	27.4 ±0.1	34.7 ±0.2	7.11	7.41	7.50	8.02	7.82
KVzip	20.5 ±0.2	22.4 ±0.2	28.6 ±0.4	36.2 ±0.5	38.3 ±0.2	217	232	292	379	596
H2O	22.5 ±0.1	24.1 ±0.3	30.9 ±0.2	36.4 ±0.2	39.4 ±0.2	221	230	307	385	572
KVMerger	13.5 ±0.2	15.1 ±0.2	24.4 ±0.1	31.7 ±0.2	35.9 ±0.2	490	554	688	962	1293
OBCache	22.0 ±0.2	25.4 ±0.3	32.7 ±0.2	37.2 ±0.2	39.2 ±0.3	235	243	309	391	554
AM-HAK-fast	22.2 ±0.4	23.7 ±0.3	32.4 ±0.1	37.4 ±0.1	39.1 ±0.3	222	245	287	384	536
AM-HAK	22.7 ±0.3	24.6 ±0.5	31.0 ±0.2	37.2 ±0.3	39.1 ±0.2	437	448	505	628	846
AM-OMP-CS	23.1 ±0.4	25.5 ±0.3	33.3 ±0.3	–	–	649	1110	5128	–	–
AM-OMP	23.5 ±0.3	27.0 ±0.4	–	–	–	1646	5037	–	–	–
LOO	22.6 ±0.3	25.3 ±0.2	32.5 ±0.2	37.2 ±0.2	38.8 ±0.4	444	469	534	661	872
KV-SURGEON-fast	22.8 ±0.2	26.5 ±0.2	33.1 ±0.3	36.9 ±0.2	39.8 ±0.1	204	234	297	407	542
KV-SURGEON	23.6 ±0.3	26.8 ±0.4	33.9 ±0.2	38.1 ±0.3	39.7 ±0.2	474	519	771	875	1322

Sources of the performance gain

The gain over independent selection comes from compensating the surviving cache entries as the support changes. To isolate this effect, we keep the surrogate, candidate pool, calibration queries, and final refit fixed and disable only the compensating update: we delete entries based on the saliency ((11)) but do not apply the update ((12)) to the remaining entries, so the saliencies stay fixed and the elimination reduces to a single ranking of the candidate pool. At 50× compaction on Qwen3-4B with QuALITY, this reduces accuracy from 66.9% to 63.9% (Table 2). KV-SURGEON without compensation also performs similarly to LOO, which uses the different deletion rule ((13)) and does not adapt the remaining entries at all. Replacing the saliency score alone does not explain the improvement. Rather, the gain comes from updating the remaining entries during selection. Additional controls that remove the opportunity for iterative compensation lead to the same conclusion (Appendices § 11 and § 12).

Method	Selection rule	Compensation	Accuracy (%)
KV-SURGEON	(11)	✓	66.9
KV-SURGEON w/o compensation	(11)	✗	63.9
LOO	(13)	✗	64.7

The improvement is also not an artifact of the optimized allocation of the global cache budget across heads inherited from [34]. Under uniform allocation, KV-SURGEON remains 3.0 accuracy points ahead of AM-HAK (Appendix § 11). Compensation-aware support selection therefore provides an improvement that is orthogonal to that of the offline-compaction pipeline proposed by Zweiger et al. [34].

Cheap calibration preserves most of the gain

KV-SURGEON is comparatively insensitive to how calibration queries are collected. Replacing self-study with a single repeat pass reduces pipeline time from 145 to 54 seconds per context at 50× compaction while reducing mean accuracy by only 1.2 points. The same change is substantially more costly for AM-HAK, which loses 4.9 points (Table 1). This makes KV-SURGEON-fast not only cheaper, but sufficient to outperform all prior methods at the primary operating point.

This robustness persists across compaction ratios. Replacing self-study with repeat-pass calibration costs KV-SURGEON at most 1.2 points of mean accuracy at any compaction ratio, with gap shrinking as more cache entries are retained. Detailed results across ratios are reported in Appendix § 10; Appendix § 12 offers a hypothesis for this robustness.

Conclusion

Prior offline KV-cache compaction methods first deleted cache entries and then adapted the remaining ones. We showed that deletion and adaptation can be coupled efficiently, and that doing so improves accuracy. We derived a quadratic surrogate of the attention-output error that allows for an efficient greedy algorithm with compensation-aware selection: every candidate deletion is scored after the retained entries have compensated for it. The resulting method, KV-SURGEON, improves the cost–accuracy frontier at every compaction ratio on QuALITY, LongHealth, and Long Code Arena with models from 4B to 30B parameters, and its fast variant exceeds all prior methods at two fifths of the cost of the strongest baseline. A controlled comparison attributes the gain over independent selection to compensation during selection, not to the surrogate.

Acknowledgment

This research was partially supported by the DFG Cluster of Excellence MATH+ (EXC-2046/2, project id 390685689) funded by the Deutsche Forschungsgemeinschaft (DFG) as well as by the German Federal Ministry of Research, Technology and Space (research campus Modal, fund number 05M14ZAM, 05M20ZBM) and the VDI/VDE Innovation + Technik GmbH (fund number 16IS23025B).

We would also like to thank the Speinshart Scientific Center for AI and SuperTech, where part of this work was conducted, for providing such a stimulating research environment.

Derivations

The surrogate as a second-order Taylor expansion

An alternative route to the same surrogate starts with the exact output loss after substituting $U = \operatorname{diag}(w)\widehat{V}$:

Replacing $\mu_i^{-2}$ by its value of one at the original cache gives the quadratic

Dropping the cross term between the output and mass residuals separates the two terms:

Replacing the query-dependent weights $\lVert o_i\rVert_2$ by the shared $\gamma$ then gives (8). The quadratic $Q$ in (16) is the second-order Taylor expansion of $\mathcal{L}$ around the original cache. This supplies an alternative derivation of the surrogate in § 3.2. The method represents a deletion by removing an index from $S$, and a support $S$ and the original cache have parameters of different sizes. To place them in a common coordinate system we use, in this proof only, a full-length multiplier vector $m\in\mathbb{R}^{T}$ with $m_j=w_j$ for $j\in S$ and $m_j=0$ otherwise, and $\widetilde{U}\in\mathbb{R}^{T\times d_v}$ with rows $m_jv_j$, which vanish outside $S$. Then $\sum_{j\in S}P_{ij}u_j=P_i\widetilde{U}$ and $\mu_i=P_im$, where $P_i$ is row $i$ of $P$, so that

are functions of $(\widetilde{U},m)\in\mathbb{R}^{T\times d_v}\times\mathbb{R}^{T}$ whose domain contains the original cache $(\widetilde{U},m)=(V,1)$. There the vector inside the norms is $P_iV-(P_i1)o_i=o_i-o_i=0$ for every query, because $P_iV=o_i$ by the definition of $O$ and $P_i1=1$ since every row of $P$ is a softmax; hence $\mathcal{L}$ and $Q$ vanish together with their gradients. Write $(\widetilde{U},m)=(V+\Delta_U,1+\Delta_m)$. Substituting, the vector inside the norms becomes $P_i(V+\Delta_U)-(P_i(1+\Delta_m))o_i=(P_iV-o_i)+P_i\Delta_U-(P_i\Delta_m)o_i=P_i\Delta_U-(P_i\Delta_m)o_i$, again by $P_iV=o_i$ and $P_i1=1$. So its squared norm is a homogeneous quadratic in the perturbation, while $(P_im)^{-2}=1-2P_i\Delta_m+O(\lVert\Delta_m\rVert_2^2)$. Hence $\mathcal{L}=Q+O(\lVert(\Delta_U,\Delta_m)\rVert^3)$, and $Q$, itself a quadratic, is the second-order Taylor polynomial of $\mathcal{L}$ at the original cache. As in OBS (Hassibi & Stork, [15]), the expansion is then used at deletions, which move $m_j$ from one to zero and row $j$ of $\widetilde{U}$ from $v_j$ to zero and are not small perturbations.

For unchanged values, $\widetilde{U}=\operatorname{diag}(m)V$, the two terms inside the norms are $P_i\operatorname{diag}(m)V=\sum_jm_jP_{ij}v_j$ and $(P_im)o_i=\sum_jm_jP_{ij}o_i$, so the vector inside the norms is $\sum_jm_jP_{ij}(v_j-o_i)$ and the frozen quadratic is the quadratic form

Its off-diagonal entries record whether the errors of deleting $j$ and $k$ reinforce or cancel. Its diagonal $H_{jj}=\sum_iP_{ij}^2\lVert v_j-o_i\rVert_2^2$ is the second-order truncation of the exact single-deletion loss, the score of (13) without the factor $(1-P_{ij})^{-2} {xref}(Appendix A.5). Finally, $H1=0$ because $\sum_kP_{ik}(v_k-o_i)=0$. This is the null direction of the Hessian along uniform scalings of all weights, which leave the ratio of {xref}(2) unchanged; the frozen quadratic alone therefore does not determine the scale of the multipliers, and the mass term of {xref}(8) does {xref}(Appendices A.3 and H.1`.

Hessian structure of the surrogate

We compare the quadratic objectives of § 3.2 through their Hessians in the unknowns $(U,w)$. Stack the unknowns as $x \triangleq(U_{:,1},\ldots,U_{:,d_v},w) \in\mathbb{R}^{|S|(d_v+1)}$, where $U_{:,c}$ is column $c$ of $U$, and write $O_{:,c}$ for column $c$ of $O$.

After freezing the normalization. The frozen objective (16) is $\|P_SU-\operatorname{diag}(P_Sw)O\|_F^2$. Column $c$ of $\operatorname{diag}(P_Sw)O$ is $\operatorname{diag}(O_{:,c})P_Sw$, so

This is a quadratic in $x$. Its Hessian has the diagonal blocks $2P_S^{\mathsf T}P_S$ for each column $U_{:,c}$, the diagonal block $2P_S^{\mathsf T}\operatorname{diag}(\|o_1\|_2^2,\ldots,\|o_N\|_2^2)P_S$ for $w$, and the off-diagonal blocks $-2P_S^{\mathsf T}\operatorname{diag}(O_{:,c})P_S$ between $U_{:,c}$ and $w$. The off-diagonal blocks are the Hessian of the cross term in (16): expanding the square gives $2\sum_i\sum_j P_{ij}u_j-o_i,\ (1-\mu_i)o_i\rangle$, whose bilinear part in $(U,w)$ is $-2\sum_c U_{:,c}^{\mathsf T}P_S^{\mathsf T}\operatorname{diag}(O_{:,c})P_Sw$. Because of these blocks, the columns of $U$ and the multipliers are not separate regressions on $P_S$. Scoring a deletion, or computing the coefficients after one, requires the inverse of the full Hessian of dimension $|S|(d_v+1)$ instead of an $|S|\times|S|$ matrix, which multiplies the memory by $(d_v+1)^2$ and the cost of every update by at least the same factor.

After dropping the cross term. The objective (17) is a function of $U$ plus a function of $w$, so its Hessian is block diagonal, with the blocks $2P_S^{\mathsf T}P_S$ for the columns of $U$ and the block $2P_S^{\mathsf T}\operatorname{diag}(\|o_1\|_2^2,\ldots,\|o_N\|_2^2)P_S$ for $w$. Every column can be optimized on its own. The multipliers, however, have a different Gram matrix from the values, so greedy elimination would carry two inverses and update both after every deletion.

After replacing the weights by $\gamma$. In (8) the mass term is $\gamma^2\|1-P_Sw\|_2^2=\|\gamma1-P_S(\gamma w)\|_2^2$, a regression of the column $\gamma1$ on $P_S$ in the unknown $\gamma w$, with Gram matrix $P_S^{\mathsf T}P_S$. All $d_v+1$ diagonal blocks of the Hessian are now $2P_S^{\mathsf T}P_S$ and there are no off-diagonal blocks. The ridge in (9) adds $2\lambda I$ to each block, so the Hessian of (9) in $B$ consists of $d_v+1$ identical blocks $2(P_S^{\mathsf T}P_S+\lambda I)$.

Closed form. Expanding (9) gives

The matrix $P_S^{\mathsf T}P_S+\lambda I$ is positive definite for $\lambda>0$, since $x^{\mathsf T}(P_S^{\mathsf T}P_S+\lambda I)x=\|P_Sx\|_2^2+\lambda\|x\|_2^2>0$ for $x\ne0$. The objective is therefore strictly convex in $B$, and its unique minimizer is the stationary point $(P_S^{\mathsf T}P_S+\lambda I)B=P_S^{\mathsf T}Y$, that is, $B=GP_S^{\mathsf T}Y$ with $G=(P_S^{\mathsf T}P_S+\lambda I)^{-1}$, which is (10). Column $c$ of $B$ is $GP_S^{\mathsf T}Y_{:,c}$, so one inverse serves all columns. Inserting the minimizer and using $(P_S^{\mathsf T}P_S+\lambda I)B=P_S^{\mathsf T}Y$ for the first equality and $G(P_S^{\mathsf T}P_S+\lambda I)G=G$ for the second,

Scale of the mass term

In (17) the mass error of query $i$ is weighted by $\|o_i\|_2^2$: a query with a large output is hurt more by a given relative mass deficit. These weights depend neither on $w$ nor on $S$, so the mass term of (17) could be minimized as it stands, with the Gram matrix $P_S^{\mathsf T}\operatorname{diag}(\|o_1\|_2^2,\ldots,\|o_N\|_2^2)P_S$. Carrying it through the elimination would require a second inverse next to $G$, updated after every deletion. Replacing $\|o_i\|_2$ by one common scalar $\gamma$ gives the mass term the Gram matrix $P_S^{\mathsf T}P_S$ of the output term, so that one inverse serves all $d_v+1$ columns (Appendix § 7.§ 2).

We set $\gamma\triangleq\gamma_0\operatorname{RMS}_i\|o_i\|_2=\gamma_0\left(N^{-1}\sum_i\|o_i\|_2^2\right)^{1/2}$ with a dimensionless $\gamma_0>0$. The RMS keeps the mass column of $Y$ on the scale of the output columns: for a uniform relative mass deficit $1-\mu_i\equiv\epsilon$, the mass term of (17) is $\epsilon^2\sum_i\|o_i\|_2^2$ and that of (8) is $N\gamma_0^2\epsilon^2$, and the two agree for every $\epsilon$ if and only if $\gamma_0=1$. The factor $\gamma_0$ therefore sets the weight of the mass term relative to the output term, with $\gamma_0=1$ matching the cost that a uniform deficit has under (17). We use $\gamma_0=1$ throughout (Appendix § 9). Setting $\gamma_0=0$ removes the mass term from the surrogate and changes accuracy without one standard error (Appendix § 11.§ 3).

Deletion saliencies and compensation

We derive the saliency (11) and the updates (12) and (14) from the block structure of the inverse $G$. Throughout, $S$ is the current support, $A \coloneqq P_S^{\mathsf T}P_S+\lambda I$, $G=A^{-1}$, and $B=GP_S^{\mathsf T}Y$ as in (10). Rows and columns of $A$, $G$, and $B$ are indexed by the elements of $S$, so that an entry keeps its index when other entries are deleted. For $j\in S$, $-j$ denotes the index set $S\setminus\{j\}$: $G_{-j,-j}$ is $G$ without row and column $j$, $G_{-j,j}$ is column $j$ of $G$ without its $j$-th entry, and $B_{-j,:}$ is $B$ without row $j$. We write $p_j\in\mathbb{R}^N$ for column $j$ of $P$, the attention weights of all queries on entry $j$.

Deleting an entry removes a row and a column of $A$. Entry $(k,l)$ of $P_S^{\mathsf T}P_S$ is $p_k^{\mathsf T}p_l$ and involves only the columns $k$ and $l$. Removing column $j$ from $P_S$ removes row and column $j$ from $P_S^{\mathsf T}P_S$ and changes no other entry, and $\lambda I$ loses the same row and column. The Gram matrix of the reduced support is therefore the principal submatrix

This requires the same $\lambda$ on both supports: a ridge that changes with $S$ would change the remaining entries as well.

Block inverse. The block equations of $AG=I$ for the partition $\left(\{j\},-j\right)$ are

$A_{-j,-j}$ is invertible as a principal submatrix of a positive definite matrix, and $G_{jj}=e_j^{\mathsf T}A^{-1}e_j>0$. The second equation gives $A_{-j,j}=-A_{-j,-j}G_{-j,j}/G_{jj}$, and substituting this into the third gives $A_{-j,-j}\left(G_{-j,-j}-G_{-j,j}G_{j,-j}/G_{jj}\right)=I$. Hence

where the second identity follows from the first block equation after inserting $\left(A_{-j,-j}\right)^{-1}A_{-j,j}=-G_{-j,j}/G_{jj}$. Equivalently, the matrix $M_j\coloneqq G-G_{:,j}G_{j,:}/G_{jj}$ has zero row $j$ and zero column $j$, and its remaining block is $\left(A_{-j,-j}\right)^{-1}$.

The update of $G$. By (19), the inverse Gram matrix of $S\setminus\{j\}$ is $\left(A_{-j,-j}\right)^{-1}$, which by (20) is the second update in (12).

The update of $B$. Since $G_{j,:}P_S^{\mathsf T}Y=B_{j,:}$,

Row $j$ of this matrix is zero because row $j$ of $M_j$ is zero. Because column $j$ of $M_j$ is zero as well, the rows $-j$ of $M_jP_S^{\mathsf T}Y$ equal $(M_j)_{-j,-j}P_{S\setminus\{j\}}^{\mathsf T}Y=\left(A_{-j,-j}\right)^{-1}P_{S\setminus\{j\}}^{\mathsf T}Y$, which by (19) is the minimizer (10) on the reduced support. These rows are $B_{-j,:}-G_{-j,j}B_{j,:}/G_{jj}$, the first update in (12). The updated coefficients are thus the exact minimizer of (9) on $S\setminus\{j\}$, not an approximation to it.

The saliency. Applying (18) on both supports, the term $\operatorname{tr}(Y^{\mathsf T}Y)$ cancels in $F(S\setminus\{j\})-F(S)$. Since $M_j$ vanishes on row and column $j$, $P_{S\setminus\{j\}}\left(A_{-j,-j}\right)^{-1}P_{S\setminus\{j\}}^{\mathsf T}=P_SM_jP_S^{\mathsf T}$, and therefore

which is (11).

Interpretation. A coefficient matrix with $B'_{j,:}=0$ has the same objective value on $S$ as its rows $-j$ have on $S\setminus\{j\}$, since row $j$ multiplies column $j$ of $P_S$ and contributes nothing to the ridge penalty, and every coefficient matrix on $S\setminus\{j\}$ arises in this way. Hence $F(S\setminus\{j\})=\min\{\|Y-P_SB'\|_F^2+\lambda\|B'\|_F^2:B'_{j,:}=0\}\geq F(S)$, and $s_j$ is the smallest increase of the objective over all coefficient matrices with zero row $j$. The objective is a quadratic in $B$ with Hessian $2A$ and zero gradient at the minimizer, so $\lVert Y-P_S(B+\Delta)\rVert_F^2+\lambda\lVert B+\Delta\rVert_F^2=\mathcal{F}(S)+\operatorname{tr}(\Delta^{\mathsf{T}}A\Delta)$ for every $\Delta$. Setting row $j$ to zero without changing the other rows, $\Delta=-e_jB_{j,:}$, increases the objective by $A_{jj}\lVert B_{j,:}\rVert_2^2$. Since $A_{jj}G_{jj}\ge1$ for positive definite $A$ (Cauchy–Schwarz in the inner product defined by $A$: $1=(e_j^{\mathsf{T}}e_j)^2\le(e_j^{\mathsf{T}}Ae_j)(e_j^{\mathsf{T}}A^{-1}e_j)$), the compensated cost $s_j$ is at most the uncompensated one, with equality if and only if $p_j$ is orthogonal to all other retained columns. By (20) and (18) with the target $p_j$,

the ridge-regularized residual of regressing the attention column of entry $j$ on the attention columns of the other retained entries. The saliency is therefore the squared coefficient magnitude of entry $j$ times the part of its attention column that the other retained columns do not reproduce, plus the ridge. An entry with a large coefficient is cheap to delete when its attention column lies close to the span of the other columns, and an entry with a small coefficient is expensive to delete when its attention column lies far from that span.

Deleting a block. Let $J\subseteq S$ and $R\triangleq S\setminus J$. The block equations of $AG=I$ on the partition $(J,R)$ read $A_{R,J}G_{J,J}+A_{R,R}G_{R,J}=0$ and $A_{R,J}G_{J,R}+A_{R,R}G_{R,R}=I$, and $G_{J,J}$ is positive definite as a principal submatrix of $G$. Eliminating $A_{R,J}=-A_{R,R}G_{R,J}G_{J,J}^{-1}$ from the second equation gives $A_{R,R}(G_{R,R}-G_{R,J}G_{J,J}^{-1}G_{J,R})=I$, so $(A_{R,R})^{-1}=G_{R,R}-G_{R,J}G_{J,J}^{-1}G_{J,R}$, and the matrix $M_J\triangleq G-G_{:,J}G_{J,J}^{-1}G_{J,:}$ vanishes on the rows and columns in $J$. The argument for one entry applies verbatim with $G_{J,:}P_S^{\mathsf{T}}Y=B_{J,:}$: the rows $R$ of $M_JP_S^{\mathsf{T}}Y$, which are $B_{R,:}-G_{R,J}G_{J,J}^{-1}B_{J,:}$, are the minimizer of (9) on $R$, and $(A_{R,R})^{-1}$ is its inverse Gram matrix, which is (14). The exact cost of the joint deletion is

Since the minimizer on $R$ does not depend on the order in which $R$ is reached, (14) gives the same coefficients and inverse as $\lvert J\rvert$ successive single deletions of the elements of $J$. The block cost (21) equals $\sum_{j\in J}s_j$ when $G_{J,J}$ is diagonal and differs from it otherwise: it is the sum of the sequential saliencies, each evaluated after the preceding deletions. KV-SURGEON selects $J$ by the individual saliencies, computed before any entry of $J$ is removed. This choice of $J$ is the only approximation of the block step; the updates (14) are exact for the surrogate.

The single-deletion score

The score of (13) is the one case in which the exact loss of (4) has a closed form: a single deletion with every other cache parameter at its original value, $S=\{1,\ldots,T\}\setminus\{j\}$, $w=1$, and $\hat{v}_k=v_k$ for $k\in S$. Then $U=V_S$, $\mu_i=1-P_{ij}$, and $\sum_{k\in S}P_{ik}v_k-\mu_io_i=(o_i-P_{ij}v_j)-(1-P_{ij})o_i=-P_{ij}(v_j-o_i)$, so (2) and (4) give

Goel et al. [10] and An et al. [2] derive the per-query error for serving-time eviction, with queries read from the prompt suffix; summing its squared norm over the calibration queries carries it to the offline setting. The error is the product of two factors. The renormalization factor $P_{ij}/(1-P_{ij})$ is large for a heavy hitter (Zhang et al., [33]) with $P_{ij}$ near one, because the mass of the deleted entry is redistributed over retained entries that hold little mass themselves. The value novelty $v_j-o_i$ is small when the entry carries what the head already retrieves, so even a heavily attended entry can be deleted at no cost if its value is close to the current output. KV-SURGEON uses the score in three roles: as the prefilter that forms the candidate pool, as the selection rule of the mild-compaction fallback, and, followed by the same final refit, as the controlled baseline LOO of § 4.3. Its second-order truncation, the diagonal $H_{jj}$ of Appendix § 7.§ 1, drops the factor $(1-P_{ij})^{-2}$.

Algorithm and implementation

Algorithm 1 in § 3.4 gives the complete procedure for one attention head. This section provides the hyperparameters, solvers, and computational details.

Multipliers as logit biases. For every retained entry $j \in S$, the compacted cache stores the original key $k_j$, the refit value $\hat{v}_j$, and the logarithm $\log w_j$ of its multiplier. At inference, $\log w_j$ is added to the logit of entry $j$ before the softmax, so that the entry contributes $e^{Z_{ij}+\log w_j}=w_j e^{Z_{ij}}$ to the unnormalized attention of query $i$, which is (2). The bound $e^{-\beta} \le w_j \le e^\beta$ of the final refit is the bound $\lvert\log w_j\rvert\le\beta$ on the logit bias of any single entry; it bounds the shift that compaction introduces and leaves the original logit $Z_{ij}$ unchanged. Mass preservation, $\mu_i \approx1$, keeps the total unnormalized attention of the compacted cache at that of the original cache, so that the compacted cache draws the same share of attention when it is served next to other tokens (Appendix § 14).

Candidate pool and ridge. The pool consists of the $m=\min\{T,\lceil ct\rceil\}$ entries with the largest scores (13), with $c=4$. The score is evaluated with a floor, $P_{ij}^{2}/((1-P_{ij})^{2}+10^{-3})$, which bounds the weight of entries that receive almost all attention of a query; the LOO baseline uses the same evaluation. The minimum matters for heads with $T/c < t \le\tau T$, whose pool is the whole cache. The ridge is set relative to the scale of the Gram matrix on the pool, $\lambda=\lambda_0\operatorname{tr}(P_S^{\mathsf T}P_S)/m$ with $\lambda_0=0.01$, computed once when the pool is formed and kept fixed for all later supports. A fixed $\lambda$ is required for the updates (12) and (14) to be exact, because the Gram matrix of a reduced support is a principal submatrix of the current one only if both carry the same ridge (Appendix § 7.§ 4). Since $m$ can exceed $N$, $P_S^{\mathsf T}P_S$ is singular on the pool in general, and the ridge is what makes $G$ well defined there. The inverse $G$ and the coefficients $B=GP_S^{\mathsf T}Y$ are computed explicitly once, on the pool, from a Cholesky factorization of $P_S^{\mathsf T}P_S+\lambda I$; if the factorization fails, $\lambda$ is increased tenfold, at most three times.

Block deletions. Each elimination step deletes the $n_b=32$ entries with the smallest saliencies; the last step deletes $\min\{n_b,\lvert S\rvert-t\}$ entries, so that the loop ends at $\lvert S\rvert=t$ exactly. The updates (14) require solving linear systems with the $n_b \times n_b$ matrix $G_{J,J}$, which is positive definite as a principal submatrix of $G$. Compared with single deletions, the block step computes the saliencies once per block rather than once per deletion and runs the updates as matrix products with $n_b$ columns; the arithmetic count is the same.

Final refit. After the elimination, $B$ and $G$ are discarded, and the multipliers and values are solved on $S$ in two ordered steps, following the refit of attention matching ([34]). The multipliers solve the box-constrained least-squares problem

The box restores the positivity dropped in § 3.2 and bounds the logit bias of every entry. The solver is that of Zweiger et al. [34] and solves (22) approximately. It works with the unnormalized attention $E=\exp(QK^{\mathsf T}/\sqrt{d_k}-\max)$, the row maximum of the logits subtracted, and its row sums $s$, so that $P=\operatorname{diag}(s)^{-1}E$, and minimizes $\lVert E_Sw-s\mathbf{1}\rVert_2^2$, which is (22) with row $i$ scaled by $s_i$. The unconstrained least-squares solution is computed by a QR solve, clamped to the box, and then refined by two projected gradient steps with step size $1/\lVert E_S\rVert_2^2$, clamping after each step; the spectral norm is estimated by three power iterations. With $w$ fixed, the normalized attention matrix of the compacted cache is $\hat{P}\triangleq\operatorname{diag}(P_Sw)^{-1}P_S\operatorname{diag}(w)$, cf. (2), and the values solve the linear least-squares problem

which is $\min_{\hat{V}}\mathcal{L}(S,w,\hat{V})$, the exact output-preservation loss (4) for the chosen support and multipliers. All $d_v$ columns of $\hat{V}$ share the Gram matrix $\hat{P}^{\mathsf T}\hat{P}$ and are solved at once. They are computed by a least-squares solve; if the solve fails numerically, the normal equations $(\hat{P}^{\mathsf T}\hat{P}+10^{-6}\bar{a}I)\hat{V}=\hat{P}^{\mathsf T}O$ are solved by a Cholesky factorization instead, where $\bar{a}$ is the mean diagonal of $\hat{P}^{\mathsf T}\hat{P}$. The order of the two solves is forced: $\hat{P}$ depends on $w$ through $\operatorname{diag}(w)$ and through $P_Sw$, so the values have to be solved after the multipliers. The minimizer of (23) is unique if and only if $P_S$ has full column rank, which cannot hold for $t>N$.

Mild-compaction fallback. Heads whose budget exceeds the fraction $\tau= 0.3$ of their cache, $t > \tau T$, skip the pool, the elimination, and the final refit: they retain the $t$ entries with the largest scores (13), their original values, and unit multipliers. Eviction is near-lossless at such mild compaction, the $O(t^3)$ solves are largest for exactly these heads, and for $t > N$ the value solve (23) is underdetermined. Disabling the fallback leaves accuracy unchanged and adds 37% to pipeline time (Appendix § 11.§ 3).

Complexity. Forming the Gram matrix of a pool of $m$ candidates costs $O(Nm^2)$ and its inverse $O(m^3)$; the right-hand side $P_S^\mathsf{T}Y$ costs $O(Nmd_v)$ and the initial coefficients $B = GP_S^\mathsf{T}Y$ cost $O(m^2d_v)$. In each elimination step, the saliencies cost $O(|S|(d_v + 1))$ given $B$ and the diagonal of $G$, the update of $B$ costs $O(|S|n_b(d_v + 1) + n_b^2)$, and the update of $G$ costs $O(|S|^2n_b)$. Summed over the $(m-t)/n_b$ steps, the loop costs $O(m^3 + m^2d_v)$, the same count as $m-t$ single deletions. The elimination on a pool of $m$ entries thus costs $O(Nm^2 + m^3 + Nmd_v + m^2d_v)$ per head, and its memory is dominated by the $m \times m$ matrix $G$. The pool bounds $m$ by $ct$, so these costs grow with the budget $t$ and not with the cache size $T$; eliminating from the full cache instead changes accuracy within noise at 3.8× the compaction time (Appendix § 11.§ 3). The fallback removes the $O(t^3)$ terms for the heads with the largest budgets.

Hyperparameters. All hyperparameters were fixed once on Qwen3-4B with QuALITY at 50× and reused unchanged for every other model, benchmark, and ratio (Appendix § 9): the mass weight $\gamma_0 = 1$ (Appendix § 7.§ 3), the relative ridge $\lambda_0 = 0.01$, the pool factor $c = 4$, the block size $n_b = 32$, the fallback threshold $\tau= 0.3$, and the multiplier bound $\beta= 3$. Appendix § 11.§ 3 varies $\gamma_0$, $c$, and $\tau$.

Experimental details

This section records what is needed to reproduce and read the experiments; § 4.1 gives the summary. Appendix § 9.§ 1 states the protocol and conventions and Appendix § 9.§ 2 breaks down where the pipeline time goes.

Protocol and conventions

Task and benchmarks. The model reads a context once, the resulting cache is compacted once by a target factor between 5× and 100×, and all benchmark questions about that context are answered from the compacted cache, so calibration and compaction are one-time costs per context. QuALITY uses the dev split, articles 0–49, with $n = 894$ multiple-choice questions; one article is one context. LongHealth consists of 20 fictional patient records, grouped into four contexts of five patients with roughly 60k tokens each; each record is compacted as its own chunk at the same ratio and the compacted chunks are concatenated (Appendix § 14), and all $n = 400$ multiple-choice questions are evaluated against their contexts. Accuracy is the percentage of correctly answered questions. Answers are extracted from the generated text, and questions whose answer cannot be parsed count as incorrect; parse rates are checked per run and are uniformly high in the reported cells, so parsing does not drive the reported differences. All eight model and benchmark combinations are reported.

Compaction ratio and per-head budgets. The compaction factor, 5× to 100×, is the nominal ratio of the original cache size to the compacted one. Cache-side methods are reported at the nominal ratio because measured effective ratios agree with it to within one percent. Heads differ widely in how much compaction they tolerate, so the compaction factor is a global budget rather than a per-head prescription. Head $(\ell, h)$ receives its own retained fraction $r_{\ell h}$, subject to the constraint that $\operatorname{mean}_{\ell h} r_{\ell h}$ equals the reciprocal of the compaction factor; each head is additionally capped at $r_{\ell h} \le0.75$, and the budget a capped head cannot use is redistributed to the other heads in proportion to their shares, so the global ratio is preserved. The allocation is computed once per model on contexts disjoint from the benchmark contexts, following the procedure introduced by Zweiger et al. [34]. We measure per head how the log-perplexity of model-generated reference answers degrades as that head alone is compressed, and we allocate the budget along these influence curves. The solver is ratio-agnostic: it finds one set of per-head proportions that works across all target ratios, and this single allocation is rescaled to each compaction factor. Every method in a table consumes the same allocation, so differences between rows are attributable to selection and refit alone; the published configuration of KVzip, reported only in Table 16, is the deliberate exception. Uniform per-head budgets serve as an ablation (Appendix § 11).

Method	as published	calib. queries	+ refit	+ budgets
H2O	$47.4 \pm0.5$	$48.7 \pm0.4$	$56.1 \pm0.4$	$62.3 \pm0.4$
OBCache	$43.2 \pm0.2$	$53.0 \pm0.4$	$58.1 \pm0.3$	$62.4 \pm0.7$
KVMerger	$47.7 \pm0.5$	$47.6 \pm0.2$	$51.5 \pm0.5$	$57.2 \pm0.5$
KVzip	$36.4 \pm0.3$	$42.6 \pm0.3$	$43.5 \pm0.6$	$49.8 \pm0.4$

Calibration tiers. All implemented methods score positions against calibration queries recorded from the model itself and, except AM-OMP, come in two tiers that differ mainly in how these queries are generated. The fast tier prompts the model to reproduce the context and records the query vectors of that single pass. The self-study tier additionally generates self-study conversations about the context, which cost more time but match the future workload better. AM-HAK-fast and KV-SURGEON-fast are the fast variants of AM-HAK and KV-SURGEON. AM-OMP, AM-OMP-CS, and KV-SURGEON are self-study methods; AM-OMP-CS is the cheap-solver setting of Zweiger et al. [34], named AM-OMP-fast there, which selects several keys per greedy step and refits less often. After selecting which positions to keep, every calibrated method applies the same support-fixed refit of the retained multipliers and values (§ 3.4), so accuracy differences between them reflect the selection rule. AM-OMP is evaluated only at the most aggressive ratios because its runtime grows steeply with the retention budget.

Calibration queries. Self-study generates five conversations per context with vLLM: a verbatim repetition of the context, a summary, an aggregation of key facts, a JSON structuring of the context, and three self-written questions with answers. The repetition is teacher-forced, with the context itself as the answer; the summary, aggregation, and JSON conversations are answered without thinking (at most 2048 tokens, temperature 0.7, top-$p$ 0.8, top-$k$ 20); the question conversation uses thinking for both writing and answering the questions (at most 8192 and 4096 tokens, temperature 0.6, top-$p$ 0.95, top-$k$ 20). The calibration queries are the query vectors of the answer tokens, capped at 50,000 per attention head by random subsampling. The repeat pass is the repetition conversation alone, that is, a single forward pass over the context, the repeat prompt, and a second copy of the context, without any generation. Queries are extracted on-policy, following Zweiger et al. [34]: layer 0 is compacted with the original queries, and for each later layer the same sequences are run through the model with the already compacted earlier layers in place, so every layer is compacted against the queries it receives at inference.

Replication and uncertainty. Each cell is the mean over five replicate runs with explicit seeds, and the grey number is the standard error over these runs; a single run is noisy to about $\pm1.5$ points at $n = 894$ and $\pm2.0$ at $n = 400$, comparable to many method gaps, so we only trust differences that survive the seed error bars. These standard errors measure generation noise under fixed contexts and questions, not uncertainty over the population of contexts, and LongHealth contributes only four composite contexts; generalization rests on the spread of the margins across the eight combinations in Appendix § 10. Full-context ceilings come from seed-replicated runs without compaction under the same protocol.

Timing conventions. Pipeline time is the average wall-clock per context spent on calibration query generation plus compaction, in seconds. It excludes prefilling the context to extract its cache, answer generation, and answer extraction, all of which are common to all methods. Query-free methods therefore report compaction time alone. All runs are single-job SLURM runs on machines with H200 GPUs, launched with the same tooling, so times are comparable within a table; comparisons across models should be treated as indicative. Every table splits calibrated methods by calibration tier, repeat-pass above self-study; AM-OMP-CS uses self-study queries and sits in the self-study group.

Defaults. The method defaults were fixed once on Qwen3-4B with QuALITY at $50\times$ and reused unchanged for every other model, benchmark, and ratio: the mass-term scale $\gamma= 1$, so that $\gamma$ is the RMS norm of the full-cache outputs; the ridge $\lambda_0 = 0.01$, applied relative to the mean diagonal of $P_S^{\mathsf{T}} P_S$ on the candidate pool rather than as an absolute penalty; the block size $n_b = 32$ selections per elimination step; the candidate pool of $m = \min\{T, [4T]\}$ entries; the threshold $\tau= 0.3$ on the retained fraction $t/T$ above which a head is left to plain eviction; and, in the final refit, the bound $|\log w_j| \le\beta= 3$ on the served logit biases, that is, $w_j \in[e^{-3}, e^3]$, the interval that Table 18 refers to. Appendix § 11 varies $\gamma_0$, the pool, and the threshold; Appendix § 8 describes how each hyperparameter enters the algorithm.

Ratio	$\widetilde{w}<0$	median head	heads w/ any	rel. move	corr.	at bounds
100×	0.42%	0.00%	13.6%	0.81	0.39	1.3% / 14.7%
50×	0.33%	0.00%	19.4%	0.70	0.51	1.3% / 9.4%
20×	0.24%	0.00%	24.6%	0.57	0.62	1.8% / 5.9%
10×	0.24%	0.00%	28.0%	0.54	0.68	2.9% / 5.5%
5×	0.28%	0.00%	33.0%	0.58	0.70	4.7% / 4.1%

Where the time goes

Query generation sets the cost of the self-study tier. Self-study query generation is a sampled generation pass over the context and costs the same whichever method consumes the queries: 29 to 33 seconds per QuALITY article and 193 to 341 per LongHealth context on the dense models, 3.4 to 5.8 ms per token, and 162 and 1062 seconds on Qwen3-30B-A3B, whose mixture-of-experts decoding runs four to five times slower per token under vLLM. A repeat pass, including re-extraction of its query vectors, costs 5 to 160 seconds at 50× on the dense models and on Qwen3-30B-A3B with QuALITY, and about 300 seconds on Qwen3-30B-A3B with LongHealth; the query-free methods need no pass at all. Table 3 splits the self-study pipeline for LOO, KV-SURGEON, and AM-HAK, which consume the same queries. At 50×, compaction is 3 to 25% of the pipeline for all three, so the spread of their pipeline times in Appendix § 10, 2 to 12%, comes entirely from the compaction step, while the order-of-magnitude gaps are between tiers.

Model, benchmark	Query gen. (s)	LOO 50×	LOO 5×	KV-SURGEON 50×	KV-SURGEON 5×	AM-HAK 50×	AM-HAK 5×
Qwen3-4B, QuALITY	33	7.7	29	10.5	53	7.0	28
Llama-3.1-8B, LongHealth	274	66	284	93	578	57	277
Gemma-3-12B, QuALITY	29	1.2	4.8	1.7	11	1.1	4.8
Qwen3-30B-A3B, QuALITY	162	5.5	20	–	–	4.6	20
Qwen3-30B-A3B, LongHealth	1062	47	–	65	–	–	–

Compaction grows with the retained fraction. The compaction step of KV-SURGEON costs 1.4× that of LOO at 50× and 1.8 to 2.3× at 5×, because the block solve grows with the candidate pool: the log-log slope of compaction time against the retained fraction is 0.45 to 0.6 for LOO and AM-HAK and 0.65 to 0.95 for KV-SURGEON, and since query generation does not depend on the compaction factor, the compaction share rises from about 20% to 50–60% across the grid on the dense models. Gemma-3-12B’s compaction is small because five of every six layers use sliding-window attention and only the global layers are compacted.

Full results

The tables that follow cover one model and benchmark combination each and report QA accuracy and pipeline time for every method at all five compaction ratios, under the protocol of Appendix § 9. The conventions are the same throughout and are stated once here. Rows are grouped as in Table 1, query-free, repeat-pass, and self-study, separated by midrules, and the best accuracy per ratio within each group is bold; an entry is underlined when it trails the bold entry by at most two standard errors of the difference, computed from the two seed standard errors, so that bold without any underlined entry in the same group and column marks a separation beyond seed noise. Time is pipeline time per context in seconds; LongHealth contexts are about eight times longer than QuALITY articles, so every method costs roughly an order of magnitude more there. ‘–’ marks ratios a method does not reach: AM-OMP and AM-OMP-CS are capped because their solver cost grows with the retention budget. Summarization replaces the context by a model-generated summary of it. We use three prompts that lead to summaries of different lengths; the summary length is not a size parameter, so summarization has no table row, and Table 1 and Figure 1 report the prompt whose resulting cache size was closest to 50× compaction.

Observations across the grids. Three regularities are not visible in Table 1. First, mild compaction saturates: averaged over the eight combinations, the self-study methods span 54.7 to 59.3 points at $100 \times$ but only 72.9 to 73.7 at $5 \times$ against a ceiling of 74.0, and Llama-3.1-8B on QuALITY reaches its ceiling of 62.7% at $20 \times$, so that combination is informative only at $50 \times$ and $100 \times$. Second, KV-SURGEON is never behind beyond noise at the cell level: it is bold in 32 of the 40 self-study cells and within noise of the best in the other eight, and its largest deficit anywhere is 1.3 points, to an AM-OMP variant costing far more. Third, KV-SURGEON-fast is bold or within noise in 37 of the 40 repeat-pass cells and separated from every other repeat-pass method in all 16 cells at $50 \times$ and $100 \times$; its three losses beyond noise are all at $10 \times$ and $5 \times$ and at most 1.8 points.

Query-free baselines. Q-Filters, StreamingLLM, and the $L_2$ key-norm heuristic use the same optimized per-head budgets as every other method and need no calibration pass. Q-Filters is the best of them at every ratio, with a mean of 46.5% at $50 \times$ over the seven combinations that report all three, where KV-SURGEON-fast reaches 63.8% on the same seven for about forty more seconds per context, so at this compaction, calibration is necessary.

Long Code Arena

Table 12 reports project-level code completion on Long Code Arena [3] with Qwen3-4B-Instruct-2507, the experiment behind the last paragraph of § 4.2. The benchmark is the large-context configuration of its project-level code completion task: each datapoint is a repository snapshot at a fixed commit, one held-out completion file, and a list of target lines in that file.

Contexts and questions. We walk the test split in order and keep the first snapshot of each not-yet-seen repository whose Python files plus completion file fit 120,000 Gemma-3 tokens, until 20 repositories are collected; the dataset order and commit hashes are recorded, so the selection is deterministic. The context of a repository is the concatenation of its non-empty .py files in snapshot order, each preceded by a header with its path; the completion file itself is not part of the context. The median context is 73,298 Qwen tokens (range about 45k to 95k). Every listed target line with non-blank content is one question, $n = 650$ in total. A question shows the path of the completion file and its prefix, all lines before the target, in a code fence, and asks for exactly the next line; prefixes longer than 16,000 characters are truncated to their tail at a line boundary, which affects two questions. The prefix is part of the question, not of the compacted context, so compaction is tested through the repository context alone. The answer is generated inside an open code fence with at most 64 new tokens.

Compaction and scoring. As for LongHealth (Appendix § 14), each repository is compacted in chunks: consecutive files are packed into chunks of at most 10,000 tokens without crossing file boundaries, about nine per repository, each chunk is compacted at the target ratio, and the compacted chunks are concatenated. Head budgets, calibration, final refit, method configurations, and the five-seed protocol are identical to the LongHealth cells; seeds affect calibration-query sampling and generation, not the fixed set of repositories and questions. A completion is correct when its first generated code line, after dropping empty lines and fence lines and stripping surrounding whitespace, matches the reference line exactly; accuracy is the mean over the 650 questions. Time is query generation plus compaction per repository. The full-context ceiling under the same questions and scorer is 40.6%, and the query-free baselines reach 22 to 23% from the file prefix alone, so the effective range of the benchmark is about 22 to 41 points.

Results. At $100 \times$ no calibrated method is more than 0.4 points above the best query-free baseline, so the budget rather than the selection rule limits accuracy there. From $50 \times$ on, the calibrated methods separate. KV-SURGEON and KV-SURGEON-fast match AM-OMP within uncertainty at $50 \times$ with 9.7$\times$ and 21.5$\times$ less pipeline time, KV-SURGEON has the highest point estimate at $20 \times$ and $10 \times$, and at $5 \times$ every calibrated method except KVzip and KVMerger is within 1.8 points of the ceiling. OBCache is again the strongest ported method, and AM-OMP-CS is the only prior method within uncertainty of KV-SURGEON at $20 \times$, at 6.7$\times$ its cost.

Method	Accuracy (%) 100×	Accuracy (%) 50×	Accuracy (%) 20×	Accuracy (%) 10×	Accuracy (%) 5×	Pipeline time (s) 100×	Pipeline time (s) 50×	Pipeline time (s) 20×	Pipeline time (s) 10×	Pipeline time (s) 5×
$L_2$ key-norm	37.0 ±0.3	37.9 ±0.4	41.2 ±0.6	51.1 ±0.3	64.2 ±0.3	0.7	0.6	0.5	0.6	0.6
Q-Filters	47.1 ±0.7	52.4 ±0.5	62.0 ±0.2	66.5 ±0.3	69.9 ±0.2	0.5	0.6	0.5	0.5	0.6
StreamingLLM	40.4 ±0.6	44.7 ±0.4	52.0 ±0.4	58.4 ±0.4	64.2 ±0.2	0.7	0.5	0.5	0.8	0.7
KVzip	48.0 ±0.4	49.8 ±0.4	61.6 ±0.4	69.1 ±0.5	70.5 ±0.2	16	17	21	26	40
H2O	60.0 ±0.5	62.3 ±0.4	<u>68.2</u> ±0.4	<u>70.3</u> ±0.4	71.3 ±0.2	16	17	20	27	40
OBCache	60.1 ±0.3	62.4 ±0.7	<u>68.3</u> ±0.6	<u>70.2</u> ±0.3	71.3 ±0.1	16	17	22	27	41
KVMerger	53.7 ±0.4	57.2 ±0.5	63.7 ±0.4	68.6 ±0.1	70.3 ±0.2	36	42	55	76	106
AM-HAK-fast	56.7 ±0.3	58.1 ±0.5	66.5 ±0.3	<u>70.8</u> ±0.5	71.9 ±0.1	15	17	20	28	38
KV-SURGEON-fast	62.4 ±0.1	65.9 ±0.4	68.9 ±0.3	71.3 ±0.5	71.9 ±0.2	14	16	22	27	28
AM-HAK	59.9 ±0.2	64.1 ±0.9	69.0 ±0.2	71.5 ±0.2	<u>71.7</u> ±0.4	35	37	41	61	65
AM-OMP-CS	<u>62.2</u> ±0.6	65.7 ±0.4	70.6 ±0.2	71.8 ±0.3	–	51	84	277	919	–
AM-OMP	<u>62.4</u> ±0.4	66.4 ±0.2	<u>70.1</u> ±0.2	–	–	117	314	1700	–	–
LOO	61.5 ±0.5	65.6 ±0.2	<u>70.2</u> ±0.2	<u>71.4</u> ±0.2	72.3 ±0.3	37	38	43	61	66
KV-SURGEON	63.4 ±0.4	66.8 ±0.3	<u>70.2</u> ±0.2	<u>71.6</u> ±0.5	72.3 ±0.5	37	41	50	68	91

Method	Accuracy (%) 100×	Accuracy (%) 50×	Accuracy (%) 20×	Accuracy (%) 10×	Accuracy (%) 5×	Pipeline time (s) 100×	Pipeline time (s) 50×	Pipeline time (s) 20×	Pipeline time (s) 10×	Pipeline time (s) 5×
$L_2$ key-norm	38.2 ±0.8	37.5 ±0.4	36.0 ±0.6	51.8 ±0.5	64.8 ±0.8	7.0	4.9	4.9	6.0	5.3
Q-Filters	<u>37.6</u> ±0.4	43.6 ±0.4	56.2 ±0.6	69.1 ±0.5	76.5 ±0.4	4.8	4.8	5.1	5.4	5.2
StreamingLLM	34.6 ±0.5	39.4 ±0.7	42.9 ±0.8	55.4 ±0.3	64.2 ±0.4	5.3	5.6	5.2	4.6	5.6
KVzip	31.8 ±1.0	39.9 ±0.8	64.6 ±0.6	75.9 ±0.3	77.7 ±0.4	206	222	278	363	530
H2O	49.6 ±0.6	58.4 ±0.4	71.2 ±0.4	76.2 ±0.9	<u>78.5</u> ±0.2	203	222	277	370	529
OBCache	52.8 ±0.6	62.8 ±0.9	71.2 ±0.4	76.2 ±0.4	<u>77.9</u> ±1.0	222	233	277	371	555
KVMerger	35.2 ±0.4	45.5 ±0.4	65.6 ±0.8	76.2 ±0.6	<u>78.1</u> ±0.8	406	461	598	772	1122
AM-HAK-fast	43.5 ±0.9	53.5 ±1.3	73.0 ±0.8	78.5 ±0.4	79.3 ±0.5	204	215	263	357	530
KV-SURGEON-fast	57.5 ±0.8	66.9 ±0.5	<u>72.5</u> ±0.6	76.7 ±0.6	<u>78.8</u> ±0.5	193	220	297	415	628
AM-HAK	53.7 ±0.4	65.0 ±0.8	73.7 ±0.8	77.2 ±0.5	<u>78.3</u> ±0.4	331	349	409	512	696
AM-OMP-CS	57.8 ±1.0	<u>65.9</u> ±1.1	74.5 ±0.9	79.1 ±0.5	–	586	1091	4518	16653	–
AM-OMP	59.3 ±0.3	65.5 ±0.6	76.0 ±0.5	–	–	1678	4884	29489	–	–
LOO	53.6 ±0.4	64.2 ±1.2	74.2 ±0.5	77.5 ±0.5	<u>78.5</u> ±0.5	353	359	426	551	701
KV-SURGEON	<u>59.1</u> ±0.5	67.8 ±0.8	<u>74.8</u> ±0.7	<u>77.8</u> ±0.7	<u>78.8</u> ±0.4	359	387	523	749	1227

Method	Accuracy (%) 100×	Accuracy (%) 50×	Accuracy (%) 20×	Accuracy (%) 10×	Accuracy (%) 5×	Pipeline time (s) 100×	Pipeline time (s) 50×	Pipeline time (s) 20×	Pipeline time (s) 10×	Pipeline time (s) 5×
$L_2$ key-norm	38.0 ±0.4	40.2 ±0.5	48.1 ±0.5	56.4 ±0.5	60.6 ±0.1	0.5	0.5	0.5	0.5	0.6
Q-Filters	40.8 ±0.3	47.7 ±0.8	56.7 ±1.0	60.9 ±0.4	62.5 ±0.3	0.6	0.6	0.6	0.6	0.6
StreamingLLM	38.7 ±0.5	43.2 ±0.4	49.0 ±0.7	54.1 ±0.4	59.2 ±0.2	0.5	0.5	0.6	0.6	0.6
KVzip	46.0 ±0.6	50.5 ±0.6	58.9 ±0.1	<u>64.1</u> ±0.6	<u>63.4</u> ±0.3	14	15	18	24	34
H2O	55.7 ±0.5	59.1 ±0.4	<u>63.1</u> ±0.3	65.5 ±0.5	<u>64.2</u> ±0.2	14	15	18	25	34
OBCache	56.9 ±0.4	59.8 ±0.3	<u>63.2</u> ±0.4	63.7 ±0.1	<u>64.1</u> ±0.7	15	16	19	25	35
KVMerger	49.4 ±0.6	54.0 ±0.7	59.6 ±0.4	62.4 ±0.3	<u>63.4</u> ±0.4	32	37	48	62	92
AM-HAK-fast	54.9 ±0.7	58.7 ±0.6	<u>63.0</u> ±0.4	63.8 ±0.3	64.5 ±0.4	14	15	18	24	34
KV-SURGEON-fast	58.5 ±0.6	61.0 ±0.4	63.4 ±0.6	63.8 ±0.6	<u>64.3</u> ±0.8	13	15	20	22	29
AM-HAK	57.0 ±0.7	60.4 ±0.5	63.2 ±0.4	<u>63.9</u> ±0.2	63.8 ±0.2	28	29	33	40	53
AM-OMP-CS	57.1 ±0.6	<u>60.6</u> ±0.8	62.8 ±0.8	64.6 ±0.4	–	42	74	283	934	–
AM-OMP	58.6 ±0.5	61.3 ±0.6	63.1 ±0.7	–	–	101	288	1573	–	–
LOO	58.0 ±0.4	<u>61.2</u> ±0.7	<u>63.1</u> ±0.3	63.1 ±0.4	<u>63.2</u> ±0.3	27	28	32	39	51
KV-SURGEON	60.3 ±0.7	62.3 ±0.5	63.2 ±0.6	<u>63.9</u> ±0.3	63.5 ±0.5	28	31	39	50	68

Method	Accuracy (%) 100×	50×	20×	10×	5×	Pipeline time (s) 100×	50×	20×	10×	5×
$L_2$ key-norm	33.0 $\pm$1.2	34.1 $\pm$1.6	37.6 $\pm$0.7	50.5 $\pm$0.3	62.5 $\pm$0.6	5.6	5.9	5.6	5.6	6.2
Q-Filters	36.0 $\pm$0.8	36.1 $\pm$1.1	39.9 $\pm$0.6	49.5 $\pm$0.8	61.4 $\pm$0.2	7.0	5.7	6.0	6.6	5.7
StreamingLLM	37.1 $\pm$1.1	37.2 $\pm$0.8	42.2 $\pm$1.1	45.4 $\pm$0.5	56.3 $\pm$0.8	5.9	5.2	5.7	5.6	5.7
KVzip	23.3 $\pm$1.1	41.0 $\pm$1.4	63.5 $\pm$0.7	71.3 $\pm$0.4	72.7 $\pm$0.5	173	179	229	302	437
H2O	43.5 $\pm$0.7	50.6 $\pm$0.9	66.2 $\pm$0.6	70.2 $\pm$0.5	73.2 $\pm$0.6	173	189	220	307	437
OBCache	44.7 $\pm$0.9	52.4 $\pm$0.2	66.6 $\pm$0.4	71.5 $\pm$1.0	73.2 $\pm$0.5	193	202	232	326	457
KVMerger	34.5 $\pm$0.8	42.8 $\pm$0.6	57.3 $\pm$0.2	67.8 $\pm$0.6	72.5 $\pm$0.5	337	397	505	643	929
AM-HAK-fast	44.4 $\pm$1.2	53.5 $\pm$1.1	68.0 $\pm$0.5	73.5 $\pm$1.4	74.3 $\pm$0.3	165	180	221	303	447
KV-SURGEON-fast	53.9 $\pm$1.0	61.5 $\pm$0.6	68.8 $\pm$0.4	71.4 $\pm$0.5	73.0 $\pm$0.5	153	181	262	287	403
AM-HAK	48.4 $\pm$1.1	58.2 $\pm$0.5	70.2 $\pm$0.7	72.8 $\pm$0.3	70.5 $\pm$1.0	291	311	350	456	614
AM-OMP-CS	47.9 $\pm$0.5	55.8 $\pm$0.9	66.1 $\pm$0.5	–	–	564	1209	5867	–	–
AM-OMP	49.2 $\pm$0.6	57.3 $\pm$0.9	69.0 $\pm$0.6	–	–	1650	5292	35590	–	–
LOO	44.5 $\pm$1.7	59.1 $\pm$0.6	66.8 $\pm$0.7	70.8 $\pm$0.9	73.4 $\pm$0.6	295	323	360	453	620
KV-SURGEON	54.0 $\pm$0.8	62.1 $\pm$0.3	71.5 $\pm$1.2	73.2 $\pm$0.5	74.1 $\pm$0.8	313	349	487	654	908

Method	Accuracy (%) 100×	50×	20×	10×	5×	Pipeline time (s) 100×	50×	20×	10×	5×
$L_2$ key-norm	41.3 $\pm$0.4	42.5 $\pm$0.6	45.9 $\pm$0.4	52.4 $\pm$0.2	68.3 $\pm$0.3	0.3	0.3	0.3	0.3	0.4
Q-Filters	46.3 $\pm$0.6	50.9 $\pm$0.3	59.9 $\pm$0.7	70.6 $\pm$0.4	77.3 $\pm$0.1	0.3	0.4	0.4	0.4	0.3
StreamingLLM	44.7 $\pm$0.5	47.5 $\pm$0.3	53.8 $\pm$0.1	60.9 $\pm$0.4	70.2 $\pm$0.3	0.3	0.3	0.3	0.3	0.3
KVzip	48.3 $\pm$0.2	53.0 $\pm$0.4	60.4 $\pm$0.3	70.3 $\pm$0.7	78.7 $\pm$0.2	56	63	66	96	78
H2O	60.6 $\pm$0.4	66.5 $\pm$0.3	73.7 $\pm$0.7	78.3 $\pm$0.2	80.5 $\pm$0.4	53	53	59	64	76
OBCache	61.7 $\pm$0.2	67.3 $\pm$0.4	73.1 $\pm$0.5	78.4 $\pm$0.3	80.0 $\pm$0.3	62	63	61	67	88
KVMerger	58.2 $\pm$0.6	63.9 $\pm$0.6	72.2 $\pm$0.2	75.1 $\pm$0.3	79.2 $\pm$0.1	76	82	94	100	122
AM-HAK-fast	60.3 $\pm$0.3	65.8 $\pm$0.5	71.2 $\pm$0.3	77.3 $\pm$0.4	79.5 $\pm$0.3	51	52	56	63	77
KV-SURGEON-fast	66.5 $\pm$0.2	71.4 $\pm$0.3	76.5 $\pm$0.2	78.4 $\pm$0.4	80.2 $\pm$0.3	53	54	61	65	76
AM-HAK	63.5 $\pm$0.5	69.5 $\pm$0.3	74.0 $\pm$0.3	78.2 $\pm$0.3	80.3 $\pm$0.4	156	162	168	172	194
AM-OMP-CS	63.5 $\pm$0.6	69.7 $\pm$0.5	75.2 $\pm$0.4	78.5 $\pm$0.4	–	165	183	283	616	–
AM-OMP	63.6 $\pm$0.6	69.2 $\pm$0.4	75.5 $\pm$0.2	–	–	199	300	996	–	–
LOO	64.7 $\pm$0.3	69.6 $\pm$0.3	74.9 $\pm$0.3	78.3 $\pm$0.7	79.8 $\pm$0.1	163	165	170	173	214
KV-SURGEON	67.6 $\pm$0.4	72.5 $\pm$0.6	77.4 $\pm$0.3	79.2 $\pm$0.2	80.5 $\pm$0.3	164	159	166	181	198

Method	Accuracy (%) 100×	50×	20×	10×	5×	Pipeline time (s) 100×	50×	20×	10×	5×
Q-Filters	–	43.2 $\pm$0.5	–	–	–	–	–	–	–	–
OBCache	52.7 $\pm$0.4	62.8 $\pm$0.4	68.6 $\pm$0.5	74.4 $\pm$0.1	76.7 $\pm$0.3	391	395	402	409	424
AM-HAK-fast	50.4 $\pm$1.1	59.8 $\pm$0.4	66.9 $\pm$0.4	74.3 $\pm$0.5	77.2 $\pm$0.2	358	361	362	372	389
KV-SURGEON-fast	54.7 $\pm$0.3	64.1 $\pm$0.2	70.2 $\pm$0.5	75.0 $\pm$0.3	77.1 $\pm$0.2	363	366	373	386	413
AM-HAK	51.4 $\pm$0.4	61.4 $\pm$0.4	69.8 $\pm$0.5	73.7 $\pm$0.1	76.8 $\pm$0.3	1121	1129	1130	1159	1162
AM-OMP-CS	52.6 $\pm$0.5	63.1 $\pm$0.2	70.0 $\pm$0.3	–	–	1271	1569	3521	–	–
AM-OMP	53.3 $\pm$0.2	63.9 $\pm$0.6	–	–	–	2147	4011	–	–	–
LOO	52.1 $\pm$0.3	62.3 $\pm$0.7	69.3 $\pm$0.4	74.2 $\pm$1.2	77.0 $\pm$0.3	1109	1112	1131	1143	1159
KV-SURGEON	55.9 $\pm$0.3	65.2 $\pm$0.4	71.4 $\pm$0.4	75.8 $\pm$0.6	77.6 $\pm$0.3	1124	1135	1149	1177	1211

Method	Accuracy (%) 100×	50×	20×	10×	5×	Pipeline time (s) 100×	50×	20×	10×	5×
$L_2$ key-norm	$47.4 \pm0.5$	$48.5 \pm0.3$	$49.3 \pm0.2$	$50.8 \pm0.1$	$55.3 \pm0.5$	0.3	0.3	0.3	0.3	0.3
Q-Filters	$53.2 \pm0.5$	$55.9 \pm0.3$	$61.7 \pm0.4$	$65.4 \pm0.4$	$68.2 \pm0.4$	0.3	0.3	0.3	0.3	0.3
StreamingLLM	$49.3 \pm0.6$	$50.2 \pm0.3$	$52.4 \pm0.5$	$55.7 \pm0.3$	$61.4 \pm0.4$	0.3	0.3	0.3	0.3	0.3
KVzip	$56.0 \pm0.2$	$57.6 \pm0.8$	$63.1 \pm0.4$	$67.2 \pm0.5$	$69.1 \pm0.4$	6.0	6.2	6.7	7.4	9.1
H2O	$61.5 \pm0.3$	$66.2 \pm0.2$	$68.1 \pm0.2$	$71.1 \pm0.4$	$71.3 \pm0.4$	6.0	6.1	6.6	7.4	9.0
OBCache	$62.7 \pm0.3$	$66.6 \pm0.4$	$68.0 \pm0.1$	$70.5 \pm0.4$	$71.2 \pm0.2$	6.19	6.47	6.84	7.54	9.39
KVMerger	$56.5 \pm0.2$	$61.1 \pm0.2$	$64.8 \pm0.4$	$67.6 \pm0.3$	$69.3 \pm0.2$	12	12	15	19	25
AM-HAK-fast	$59.5 \pm0.2$	$62.2 \pm0.4$	$66.8 \pm0.5$	$69.3 \pm0.4$	$69.5 \pm0.3$	6.1	6.2	6.7	7.4	8.9
KV-SURGEON-fast	$65.2 \pm0.3$	$68.1 \pm0.5$	$70.4 \pm0.8$	$71.0 \pm0.6$	$70.6 \pm0.2$	6.0	6.5	7.4	8.8	11
AM-HAK	$61.5 \pm0.6$	$65.1 \pm0.3$	$68.8 \pm0.3$	$70.0 \pm0.3$	$70.9 \pm0.4$	29	30	30	31	34
AM-OMP-CS	$64.2 \pm0.3$	$67.2 \pm0.2$	$69.2 \pm0.3$	$70.3 \pm0.3$	–	32	37	56	125	–
AM-OMP	$64.7 \pm0.4$	$67.7 \pm0.3$	$69.8 \pm0.4$	–	–	38	59	180	–	–
LOO	$60.5 \pm0.3$	$65.1 \pm0.5$	$68.6 \pm0.3$	$70.5 \pm0.4$	$70.4 \pm0.4$	29	29	30	31	33
KV-SURGEON	$65.5 \pm0.2$	$67.5 \pm0.7$	$70.0 \pm0.4$	$70.9 \pm0.4$	$71.1 \pm0.2$	30	31	33	35	39

Method	Accuracy (%) 100×	50×	20×	10×	5×	Pipeline time (s) 100×	50×	20×	10×	5×
$L_2$ key-norm	$37.6 \pm0.5$	$38.9 \pm1.2$	$37.8 \pm0.6$	$40.4 \pm0.4$	$41.0 \pm0.6$	6.2	6.2	6.3	6.1	6.0
Q-Filters	$38.4 \pm0.2$	$39.0 \pm0.7$	$39.4 \pm0.9$	$44.5 \pm0.8$	$53.9 \pm0.7$	6.0	6.1	6.3	6.1	6.1
StreamingLLM	$40.9 \pm0.3$	$39.5 \pm0.9$	$39.3 \pm0.6$	$42.0 \pm0.3$	$44.4 \pm0.6$	6.1	5.9	6.0	6.1	6.3
KVzip	$38.1 \pm0.5$	$39.8 \pm0.7$	$44.5 \pm0.6$	$52.9 \pm0.8$	$60.7 \pm0.6$	55	59	64	74	95
H2O	$38.5 \pm0.6$	$47.3 \pm0.5$	$55.1 \pm0.2$	$63.5 \pm0.6$	$68.8 \pm0.7$	54	60	64	74	91
OBCache	$41.3 \pm0.5$	$50.2 \pm0.8$	$56.1 \pm0.4$	$64.0 \pm0.8$	$69.7 \pm0.5$	59.2	59.9	66.4	75.1	95.3
KVMerger	$33.7 \pm0.6$	$36.1 \pm0.8$	$42.5 \pm0.6$	$50.1 \pm0.9$	$59.2 \pm0.6$	97	106	140	170	239
AM-HAK-fast	$41.5 \pm1.0$	$45.6 \pm1.2$	$54.6 \pm0.8$	$61.8 \pm0.6$	$66.0 \pm0.7$	57	59	80	73	96
KV-SURGEON-fast	$45.6 \pm0.6$	$52.1 \pm0.4$	$61.1 \pm0.3$	$66.5 \pm0.7$	$71.2 \pm1.1$	53	59	71	97	138
AM-HAK	$44.7 \pm0.7$	$52.7 \pm1.4$	$62.5 \pm0.6$	$67.2 \pm0.2$	$71.0 \pm0.5$	198	207	210	227	291
AM-OMP-CS	$43.6 \pm0.9$	$50.9 \pm0.8$	$61.2 \pm0.9$	$66.2 \pm0.7$	–	225	287	624	2048	–
AM-OMP	$45.5 \pm1.2$	$54.5 \pm0.9$	$61.9 \pm1.0$	–	–	353	689	2831	–	–
LOO	$42.5 \pm0.7$	$49.6 \pm0.7$	$60.6 \pm0.6$	$67.2 \pm0.7$	$69.8 \pm0.7$	206	205	213	218	249
KV-SURGEON	$48.4 \pm1.1$	$56.4 \pm0.7$	$63.0 \pm0.7$	$68.8 \pm0.8$	$71.8 \pm0.5$	194	201	224	263	346

Component ablations

This section holds every ablation of the shared pipeline and of KV-SURGEON behind § 4.3 and Table 2: the shared pipeline components in Appendix § 11.§ 1, compensation in Appendix § 11.§ 2, and the efficiency and design choices in Appendix § 11.§ 3. Every ablation runs on Qwen3-4B with QuALITY at $50 \times$, changes one component of the pipeline of § 3 at a time, and reports the mean over five seeds with one standard error over seeds. Two protocols are used and must not be mixed. The 50-article protocol (Table 13 and Table 15) is that of the main tables, 894 questions; its self-study controls are the KV-SURGEON, LOO, and AM-HAK entries of Table 1, and its time is pipeline time, calibration-query generation plus compaction. The 25-article protocol (Table 14) uses the first 25 articles, so its accuracies are compared only against the reference row of the same panel and never against the main-table numbers, and its time is compaction only, because query generation is identical across arms. The fast-tier row of Table 13 was run with the fast configuration in use at the time of the ablation, whose reference values (65.5%, 5.4 s) differ from the KV-SURGEON-fast entry of Table 1; it too is read only against its own control. Defaults are those of Appendix § 9.

Ablation	Method	Default Acc. (%)	Default Time (s)	Ablated Acc. (%)	Ablated Time (s)
Final refit off (plain eviction on the selected support)	LOO	65.6 ±0.2	38	60.2 ±0.4	35
Mass term off ($\gamma_0 = 0$)	KV-SURGEON	66.8 ±0.3	41	67.0 ±0.4	43
Mild-compaction fallback off (threshold 0.3 $\to$ 1.0)	KV-SURGEON-fast	65.5 ±0.2	5.4	65.5 ±0.2	7.4

Method	Queries	uniform budgets		optimized budgets
		no refit	refit	+ refit
AM-HAK	self-study	54.1 $\pm0.2$	57.2 $\pm0.5$	64.1 $\pm0.9$
LOO	self-study	53.2 $\pm0.5$	57.8 $\pm0.6$	65.6 $\pm0.2$
KV-SURGEON	self-study	58.2 $\pm0.4$	60.2 $\pm0.3$	66.8 $\pm0.3$
AM-OMP	self-study	–	59.0 $\pm0.5$	66.4 $\pm0.2$
AM-HAK-fast	repeat	48.1 $\pm0.5$	53.2 $\pm0.3$	58.1 $\pm0.5$
KV-SURGEON-fast	repeat	56.3 $\pm0.5$	58.6 $\pm0.5$	65.9 $\pm0.4$
AM-OMP-CS	self-study	–	58.1 $\pm0.4$	65.7 $\pm0.4$

Shared pipeline components: head budgets and final refit

Both components are inherited from attention matching ([34]) and applied to every calibrated method, so their effect is measured per method. Table 15 switches them independently for our methods and the attention-matching baselines; the ported eviction methods are walked through the same switches in Table 16.

Head budgets. The ablation replaces the optimized per-head budgets of Appendix § 9 by uniform per-layer budgets at the same global ratio. It is the largest single lever, and it is not specific to KV-SURGEON: with the refit in place, KV-SURGEON loses 6.6 points, AM-HAK 6.9, and LOO 7.8. Under uniform budgets KV-SURGEON remains 2.4 points ahead of LOO and 3.0 ahead of AM-HAK, so the selection-rule margins of § 4.3 do not depend on the optimized budgets. KVzip, the one baseline that sets its own head allocation from a global attention threshold, leads to the same conclusion: ported with the shared budgets and refit, it collapses under strong compaction, a median of 15.3 points below KV-SURGEON-fast at $50 \times$ and $100 \times$, yet sits a median of 1.6 points below the ceiling at $5 \times$ (Appendix § 10); the deficit tracks how hard the compaction is, not how good the score is.

Final refit. The refit is ablated on the LOO support, where it can be switched off without touching selection (Table 13). Serving the selected keys with their original multipliers and values, which is plain eviction on the score of (13), costs 5.4 points against LOO with the refit. Across methods the two switches compound: the budgets are the larger lever, worth +4 to +8 points on top of the refit, and the refit adds +1 to +7 on top of raw compaction (Table 15 and Table 16). The improvements are not the same for all methods, but KV-SURGEON is ahead in all three settings. Panel (c) of Table 14 decomposes the refit into its two solves on the KV-SURGEON support; it is kept as supporting evidence for Appendix § 14.§ 1 and is not built on elsewhere.

KV-SURGEON-specific component: compensation

Panel (b) of Table 14 isolates compensation. The surrogate, the $4t$ candidate pool, and the final refit are held fixed and only the compensating update of (12) is disabled, so that elimination proceeds greedily on the same surrogate saliencies. This costs $2.9 \pm0.8$ points (seed-paired; 3.0 between the seed means of Table 2), and leaves greedy elimination no better than LOO selection on the same articles (64.7%). Two controls show that the gain is not a better standalone objective. Greedy elimination on the exact set loss of (4) instead of the surrogate reaches 65.8%, 1.1 $\pm$ 1.3 below KV-SURGEON and too noisy on 25 articles to separate from either side. Shrinking the candidate pool to the support itself (pool factor 1 in panel (a)), which leaves the elimination nothing to eliminate or compensate, costs the same 2.9 points. The loss-level counterpart, that the compensated support moves toward an exact-refit oracle and that the compensation generalizes to held-out queries, is in Appendix § 12.

Efficiency and design choices

The remaining choices change accuracy by at most one standard error at their defaults and mainly control cost.

Mass term. Setting $\gamma_{0}=0$ removes the mass term from the surrogate (Table 13). Accuracy moves by +0.2 points, within one standard error, at 2 s more pipeline time.

Candidate pool. The elimination runs on the $\min\{T,\lceil4t\rceil\}$ best entries per head by the score of (13) (panel (a) of Table 14). Widening the pool to the full cache changes accuracy by 0.2 points (66.9 against 66.7) and multiplies compaction time by 3.8 (10 against 38 s); halving it to $2t$ costs 1.0 point.

Mild-compaction fallback. Heads whose budget $t$ exceeds 30% of their cache are served by plain eviction on the score of (13), without pool, elimination, or final refit: eviction is near-lossless there, the $O(t^{3})$ value solve is the dominant cost and is largest exactly for these heads, and for $t>N$ the value solve of (23) is underdetermined. Raising the threshold to 1.0 disables the fallback (Table 13). The accuracy of KV-SURGEON-fast is unchanged to the reported precision, and its pipeline time rises from 5.4 to 7.4 s, so the fallback saves 27% of the pipeline. The fallback was not ablated on the self-study tier.

Surrogate refresh. The frozen surrogate can be re-solved at the current normalization ratios $\mu$ every 256 or 64 eliminations, either reweighting the residuals of (17) (panel (a) of Table 14) or consistently, holding that mass fixed so that the refreshed objective agrees with the served output (panel (b)). At the default pool the reweighted refresh costs 1.6 and 1.9 points at 1.3× and 2.2× the compaction time, and the consistent refresh costs 1.3 ± 0.7 and 0.7 ± 0.2 points. Across the pool sweep the refreshed arm is never better than the frozen one beyond one standard error of the difference (the largest apparent gain is +1.0 at pool factor 1, with standard errors of 1.0 and 0.7), while its cost grows to 11× at the full cache. At the loss level, measured as in Appendix § 12, the refresh changes which positions survive, Jaccard 0.55 to 0.85 against the frozen support, without changing the exact loss after the final refit on calibration or held-out queries (within ±2% on every head where a refresh occurs). Freezing is therefore kept for cost, not because it regularizes.

Pool factor	Refresh off Acc. (%)	Refresh off s	Refresh every 256 Acc. (%)	Refresh every 256 s	Refresh every 64 Acc. (%)	Refresh every 64 s
1 (LOO selection)	64.0 ±1.0	7.9	65.0 ±0.7	7.8	64.1 ±0.6	8.0
2	65.9 ±1.0	8.7	66.0 ±0.5	9.2	65.8 ±0.7	10
4 (default)	66.9 ±0.3	10	65.3 ±0.3	13	65.0 ±0.5	22
8	66.5 ±0.7	15	65.5 ±0.4	28	66.0 ±0.3	74
16	66.8 ±0.4	22	66.0 ±0.7	59	64.5 ±0.5	181
50 (full cache)	66.7 ±0.5	38	65.9 ±0.4	126	66.7 ±0.5	416

Variant	Acc. (%)	$\Delta$ vs. KV-SURGEON
surrogate with compensation (KV-SURGEON)	66.9 ±0.6	—
surrogate, greedy, no compensation	63.9 ±1.6	−2.9 ±0.8
exact loss (4), greedy, no compensation	65.8 ±2.8	−1.1 ±1.3
LOO selection (same articles)	64.7 ±0.9	−2.1 ±0.4
consistent refresh every 256	65.6 ±1.5	−1.3 ±0.7
consistent refresh every 64	66.1 ±0.4	−0.7 ±0.2

Final refit re-solves	Refresh off Acc. (%)	Refresh off s	Refresh every 256 Acc. (%)	Refresh every 256 s
values and multipliers (default)	66.9 ±0.3	10	65.3 ±0.3	13
values only, multipliers from the elimination	65.5 ±0.5	9.7	64.2 ±0.5	12
multipliers only, values from the elimination	52.7 ±0.8	12	42.7 ±1.6	12
none, both from the elimination	64.7 ±1.1	8.5	60.9 ±0.5	11

tbl:14: Ablations under the 25-article protocol on Qwen3-4B with QuALITY at 50×, five seeds, one standard error over seeds; accuracies are compared only within this table. The three panels are separate runs, so the same configuration can appear in more than one panel with slightly different numbers. Time is compaction only. (a) The pool holds the $\lceil\text{pool factor} \times t \rceil$ best entries per head by the score of (13); pool factor 1 is LOO selection with the shared final refit and 50 saturates at the full cache. The reweighted refresh re-solves the surrogate at the current normalization ratios $\mu$ every 256 or 64 eliminations. (b) Greedy rows eliminate on the stated objective without the compensating update of (12); the exact-loss row evaluates (4) on the current support at every step; the consistent refresh re-solves the surrogate at the current normalization ratios $\mu$ while holding that mass fixed, so that the refreshed objective agrees with the served output. Differences are seed-paired; run times were not recorded for this panel. (c) Whatever the final refit does not re-solve is kept from the coefficients $B$ that the elimination itself leaves on the final support.

Compensation at the loss level and calibration robustness

This section measures compensation directly on the attention-output loss rather than through accuracy, and then asks why repeat-pass calibration costs KV-SURGEON less than it costs the exact single-deletion score; § 4.3 and § 4.4 cite the results. All measurements use Qwen3-4B and Llama-3.1-8B, five QuALITY articles, and every head, that is, 1440 head–article pairs on Qwen3-4B and 1280 on Llama-3.1-8B. Scores are computed from the self-study calibration queries, and the exact loss of (4) is evaluated on held-out benchmark questions the scores never saw; where the same measurement on the calibration queries differs, it is given in parentheses.

Compensation moves the support toward the oracle and generalizes. Against an oracle that runs the same greedy backward elimination but scores every candidate deletion by the exact loss (4) after refitting the remaining entries, on the 546 heads where this is affordable ($t \le57$, Qwen3-4B), LOO selection sits 46% above the oracle’s loss and KV-SURGEON 25% (median relative loss 0.213, 0.183, and 0.146; random selection 0.243), closing 45% of the gap, and on the typical head the closed fraction is 39%; the oracle is itself greedy, so these gaps are to exact-loss greedy elimination rather than to the optimal support. KV-SURGEON beats LOO on 84% of heads and still shares only 44% of its keys with the oracle set, because many near-equivalent supports exist. On the supports KV-SURGEON selects at 50×, the final refit leaves 25% of the raw-eviction loss on Qwen and 7.5% on Llama on held-out questions, against 25% and 7.7% on the calibration queries: no head overfits, and on no head is the stored cache worse than eviction. On Llama raw eviction at 50× leaves a loss larger than the output itself, so the final refit, which every calibrated method shares, removes most of the loss there, and selection decides the remainder. The coefficients the elimination itself produces already achieve most of this (31% and 8%), but they cannot be served raw: on 1% of Qwen heads and 5% of Llama heads they exceed the eviction loss, and the final refit never does, which is the loss-level counterpart of the final-refit ablation of Appendix § 11.§ 1.

Why repeat-pass calibration costs KV-SURGEON less than the exact score. Averaged over the eight combinations at 50×, replacing self-study queries with a single repeat pass costs KV-SURGEON 1.2 points (Table 1); the drop is 4.3 points on LongHeal with Gemma-3-12B and at most 1.3 elsewhere. At the level of rankings, recomputing the scores from a single repeat pass drops the Spearman correlation between the surrogate and the exact loss over random deletion sets, computed within one head and one set size, from 0.905 to 0.786 on Qwen and from 0.929 to 0.833 on Llama, and that of the score of (13) from 0.881 to 0.690 on both, so under cheap queries the surrogate loses about half the ranking fidelity the exact score loses. Cross-head comparability, which is what the per-head budget allocation consumes, is unaffected (0.81 and 0.91 at 98% deletion), so budget allocation does not need self-study queries. The surrogate is therefore less sensitive to the calibration queries than the exact score, not insensitive.

Hypothesis, not tested as a mechanism. The factor $(1 - P_{ij})^{-2}$ in the exact deletion score amplifies exactly the queries a repeat pass gets wrong: a query for which one token receives unusually large attention contributes disproportionately to that token’s score, so an atypical repeat-pass attention pattern moves the exact ranking more than it moves the frozen surrogate, which contains no such amplification. This is consistent with the measurements above and is a candidate mechanism for the fast-tier robustness of § 4.4, but it does not establish why AM-HAK degrades more strongly under the same change: AM-HAK ranks tokens by attention mass rather than by either of these objectives, so its larger calibration drop may have a different cause.

Ported baselines: from the published regime to ours

H2O, OBCache, KVMerger, and KVzip are serving-time or self-allocating methods; this section describes how each is ported into the offline pipeline. Each port replaces the method’s own queries by the shared repeat-pass calibration queries, keeps its selection score, and serves the selected support with the shared per-head budgets and final refit of § 3.4. For OBCache, the published observation window of the last 64 prompt tokens is replaced by the calibration queries, and its second-order score is aggregated over them. Table 16 validates the port by walking each method from its published configuration to our pipeline one component at a time, at $50 \times$ on Qwen3-4B with QuALITY under the 50-article protocol of Appendix § 9.

The pipeline we use, inherited from Zweiger et al. [34], is worth 9 to 19 points to each of the eviction methods we ported to the offline compaction regime; the budgets are the largest single step for KVMerger and KVzip, the refit for H2O, and the calibration queries for OBCache. Even with every component, the best ported rule, OBCache at 62.4, stays 3.5 points below KV-SURGEON-fast (65.9) at the same cost, and H2O at 62.3 is within noise of OBCache. The two hybrid configurations with their own queries but our budgets (H2O $58.7 \pm0.2$, KVMerger $57.5 \pm0.5$) confirm that budgets alone do not close the gap.

Cache construction and the role of mass matching

Chunked contexts. Long structured contexts, for example records of several patients, are compacted one chunk at a time. At inference the compacted blocks sit side by side in one cache, and the softmax normalizes across all of them. Block $b$ therefore competes for the attention of query $i$ through its total unnormalized mass $m_i^b = \sum_{j\in S_b} w_j e^{Z_{ij}}$. A compaction that shrinks a block’s mass re-routes attention away from that block and toward its neighbors, even when the reconstruction within the block is perfect. In this setting the absolute mass of a block is meaningful, and it must be preserved alongside the outputs. Lemma 1 makes this precise, and Appendix § 14.§ 1 discusses what mass matching can and cannot tell selection.

Proof. The softmax over all tokens has the denominator $\sum_{j\in S} w_j e^{Z_{ij}} + c_i \sum_{\ell} e^{Z_{i\ell}}$ and the numerator $\sum_{j\in S} w_j e^{Z_{ij}}\hat{v}_j + c_i o'_i \sum_{\ell} e^{Z_{i\ell}}$. Dividing both by $\sum_{\ell} e^{Z_{i\ell}}$ and using $\sum_{j\in S} w_j p_{ij} = \mu_i$ and $\sum_{j\in S} w_j p_{ij}\hat{v}_j = \mu_i\hat{o}_i$ from (2) and (3) gives the first expression. The original cache is the case $S = \{1,\ldots,T\}$, $w = 1$, $\hat{V} = V$, for which $\mu_i = 1$ and $\hat{o}_i = o_i$. If $\hat{o}_i = o_i$ and $\mu_i = 1$, the two expressions coincide. Conversely, $c_i = 0$ gives $\hat{o}_i = o_i$; then both expressions are convex combinations of $o_i$ and $o'_i$ with the weights $\mu_i/(\mu_i+c_i)$ and $1/(1+c_i)$ on $o_i$, which agree for $c_i > 0$ and $o'_i \ne o_i$ only if $\mu_i = 1$. □

The role and limits of mass matching

Attention matching ([34]), the strongest prior offline method, builds its selection and refit around reproducing each query’s attention mass on the compacted block. Two consequences of § 3.2 matter here: for how the refit must be run, and for what mass matching can tell selection.

Per-position mass fitting is harmful alone and misleading as a selection signal. Non-uniform multipliers change the retained attention pattern, so the original values no longer reconstruct $o_i$ under the reweighted pattern. Fitting multipliers to mass targets without the subsequent value solve is therefore worse than no refit at all, which is why the refit of § 3.4 applies the two solves only as an ordered unit. This is measured on the KV-Surgeon support: re-solving the multipliers alone scores 12.0 points below serving the elimination’s own coefficients with no refit at all, and 14.2 below the full final refit (panel (c) of Table 14). The mechanism is the reweighting rather than the mass objective in particular, so the ablation bounds what mass fitting costs when it is run alone and does not separate it from other multiplier objectives. Mass matching also misleads selection. Weighting the mass term heavily, formally $\gamma\to\infty$ in (9), reduces the saliency (11) to its mass component, so that entries are ranked by their fractional contribution to reproducing the attention mass alone, as in attention matching, and their values play no role in selection. This discards both factors that the exact loss (13) says matter, the value novelty $\lVert v_j-o_i\rVert^2$ and the heavy-hitter weight $(1-P_{ij})^{-2}$. At mild compaction the difference is small, because retained masses stay near one and all criteria agree. At aggressive compaction the two criteria part ways, and the exact loss sides with the output: replacing mass ranking by KV-Surgeon selection inside an otherwise identical pipeline is worth 1.3 points at $20\times$, 3.0 at $50\times$, and 4.3 at $100\times$ (Figure 2).

None of this contradicts the published attention-matching results, because all of those results combine mass matching with a value refit.

Served multipliers in chunked caches. Lemma 1 says that a compacted block must preserve its attention mass when it is served next to other tokens. We test this on LongHealth with KV-Surgeon-fast on Qwen3-4B-Instruct-2507, where every context holds five patient records compacted as separate chunks. We vary only what multipliers the compacted entries are served with: the box-constrained refit of § 3.4, the multipliers that the elimination itself leaves on the support, clamped to the same box, or unit multipliers. Values are refit in all three cases. Each setting is run in two regimes. In the first, all five records are compacted. In the second, two or three records are kept uncompacted next to the compacted ones; the two complementary masks {1, 3} and {0, 2, 4} are both run, so every record is served once compacted and once uncompacted, and accuracy is split by whether a question asks about a compacted or an uncompacted record.

Three things follow from Table 17. First, unit multipliers are harmless when everything is compacted the same but harmful when compacted cache chunks are served next to uncompacted ones. With all records compacted they cost 1.6 and 1.7 points. With uncompacted neighbors the same compacted records lose 12.2 points at 100× and 8.9 at 50×, while the uncompacted neighbors gain 1 to 3 points. This is the re-routing of Lemma 1: a block whose mass falls below the original hands attention to its neighbors. When every block is compacted, the deficits are similar and largely cancel. Second, mass-preserving multipliers make a block’s accuracy independent of its neighbors: for both fitted settings the compacted records score the same whether their neighbors are compacted or not. Third, the multipliers that the elimination leaves on the support, clamped to the box of (22), match the refit within seed noise, so the final multiplier solve can be skipped at no cost. Removing the mass term from the surrogate ($\gamma_{0}=0$) changes every cell within two standard errors, consistent with Appendix § 11.§ 3: mass preservation is carried by the served multipliers, not by the selection.

Multipliers	100× All compacted	100× Mixed compacted	100× Mixed uncompacted	50× All compacted	50× Mixed compacted	50× Mixed uncompacted
Refit (default)	57.5 ±0.8	58.0 ±0.7	78.2 ±0.6	66.9 ±0.5	67.9 ±0.6	79.4 ±0.5
From the elimination	56.7 ±0.4	56.5 ±0.7	79.2 ±0.5	68.5 ±0.7	67.8 ±0.3	78.7 ±0.5
Unit	55.9 ±0.6	45.8 ±0.8	81.3 ±0.2	65.2 ±0.6	59.0 ±0.9	80.6 ±0.5

Multiplier diagnostics for the unconstrained elimination

Table 18 shows the impact of a relaxation that makes closed-form compensation possible, i.e., relaxing the multiplier coefficients to real numbers instead of positive numbers. Fewer than half a percent of kept coefficients end negative at any ratio, the median head has no negative coefficients, and the numbers are unchanged between the repeat and self-study query tiers. The refit is calibration, not sign repair: it rescales substantially (median relative movement 0.5 to 0.8, correlation 0.4 to 0.7) and its pinning at the upper bound grows from 4% of multipliers at 5× to 15% at 100×, where restoring the attention mass requires the largest multipliers. Negative coefficients concentrate in early layers (1–6), and refit movement is largest in late layers (25–34).