WorksComputer Science
LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time
Abstract
Autoregressive decoding relies on the KV cache, whose memory footprint grows linearly with context length and caps batch size and throughput. Existing post-training methods exploit redundancy across tokens, layers, and feature dimensions, or lower numerical precision. We identify a complementary axis: within each attention head, values are largely predictable from their corresponding keys. We exploit this with value cache amortisation, which fits a lightweight per-head predictor at test time and stores residuals only for the tokens it reconstructs worst, giving near-lossless value compression. The predictor can be initialised by meta-learning for substantially reduced prefill overhead, or fitted from scratch in settings that allow a one-off cost, such as multi-query inference. We pair value amortisation with cross-layer low-rank key compression to produce LowRAM, an asymmetric compression pipeline that stays close to lossless across three model families. By reconstructing only a selected subset of KV pairs at each decoding step, LowRAM can keep decoding overhead minimal and achieve up to higher peak decode throughput at compression. LowRAM further composes with quantisation and eviction, reaching compression at of baseline performance on long contexts. These results show that treating keys and values as structurally distinct enables efficient, aggressive, and composable KV cache compression.
Introduction
Transformers (Vaswani et al., 2017) have become the dominant architecture for language modelling, but their deployment at long context lengths is fundamentally constrained by the compute cost of autoregressive decoding. Modern inference stacks address this via a KV cache: the key and value projections of each preceding token are computed once and reused at every subsequent decoding step. This turns decoding from a quadratic to a linear operation in sequence length, but shifts the bottleneck from compute to memory: the cache grows linearly with context length and batch size, and at long contexts comes to dominate the model’s activation footprint.
A wide range of architectural responses to this bottleneck have been proposed, including linear attention and state-space alternatives that replace the cache with a fixed-size recurrent hidden state (Katharopoulos et al., 2020; Gu & Dao, [17]; Dao & Gu, [9]), and sparse, sliding-window, grouped-query, and multi-head latent attention variants that reduce per-token cache size architecturally (Ainslie et al., [1]; DeepSeek-AI et al., [10]). However, these approaches require pretraining or substantial finetuning, and most open-weight models still rely on grouped-query attention with a full KV cache (Grattafiori et al., [16]; Jiang et al., 2023; Team et al., 2024; Yang et al., 2025a). This has motivated a parallel line of post-training KV cache compression methods operating on frozen pretrained models. Existing methods fall into three families: eviction, which discards unimportant entries (Xiao et al., 2024; Zhang et al., 2023; Li et al., 2024; Cai et al., [6]); quantisation, which lowers the precision of stored entries ([12] (Liu et al. 2024; Hooper et al. 2024)); and low-rank decomposition, which approximates KV matrices with smaller, factorised representations ([7] (Yu et al. 2024; Saxena et al. 2024)). These families are largely orthogonal, exploiting redundancies along different axes of the cache. This leaves the redundancy between keys and values largely unexplored. Keys and values are computed from the same representation, and although they serve different functions, their shared origin induces a predictable relationship between them. We exploit this redundancy by using the keys as a source from which the value cache can be recovered.
Figure 1. Mistral-7B compression on LongBench contexts longer than 10K tokens across different KV cache compression methods: eviction (adapted from Yang et al. (2025b)), quantisation (Zandieh et al., 2026), LowRAM (our proposed method), and LowRAM-EQ (LowRAM combined with eviction and quantisation). Left: recovered performance against compression ratio. Right: maximum evaluated compression retaining at least 90% of baseline performance. LowRAM-EQ combines LowRAM, eviction, and quantisation. See Appendix § 10 for a description of how compression is computed.
We first compare several predictive relationships within and across layers under a matched memory budget and find that reconstructing current-layer values from current-layer keys best preserves downstream performance. Based on this finding, we introduce value cache amortisation: during prefill, we fit a lightweight per-head network to map cached keys to their associated values. We then rank tokens by reconstruction error and store residual corrections only for the least accurately predicted values. The resulting value cache consists of the network parameters and a sparse set of residuals. During decoding, values are recovered by applying the network to the keys and adding the stored corrections, as illustrated in Figure 2.
Figure 2. At inference, a per-head MLP learns to reconstruct values from keys, then is cached for decoding alongside residuals for the worst-reconstructed values.
Amortising the values still requires keys to be retained, both for attention and as inputs to the learned map. A complete compression scheme must therefore reduce the key cache as well. Our source-target analysis suggests that applying the same predictive strategy to the keys leads to less downstream performance than for values. Instead, for keys, prior work points to low-rank decomposition: their energy is concentrated in a relatively small number of feature directions, producing a short spectral tail and correspondingly small approximation error at reduced rank (Saxena et al., 2024). These findings motivate an asymmetric design: values are reconstructed from keys, while the keys themselves are compressed according to their low-rank structure. LowRAM realises this design by pairing value cache amortisation with xKV ([8]), which shares a low-rank basis across layers within each group to exploit alignment between their dominant key-cache subspaces.
Contributions
We identify within-head key-to-value predictability as a previously underexplored source of KV-cache redundancy. Building on this observation, we introduce value cache amortisation, which replaces most stored values with a context-adapted predictor and retains residuals only for the hardest-to-predict tokens.
We propose LowRAM, an asymmetric pipeline that combines amortised values with cross-layer low-rank key compression. LowRAM remains close to lossless at moderate compression across three model families and composes with quantisation and eviction, reaching 26.9× compression while retaining 94.5% of baseline performance.
We reduce the computational overhead of LowRAM using a meta-learned initialisation and selective reconstruction. The former dramatically reduces prefill overhead introduced by amortisation, while the latter achieves up to higher peak decode throughput.
Related Works
Low-Rank Decomposition of KVs. Training-free SVD-based KV compression largely targets per-layer redundancy, via activation-aware or grouped SVD on the KV projection weights [7] (Yuan et al. 2025; Yu et al. 2024; Zhang et al. 2024; Yan et al. 2025), projecting attention into an offline-fitted eigen-subspace (Saxena et al., 2024), or direct channel pruning (Xu et al., 2025). Complementary learned-projection methods [5] (Lin et al. 2025; Shi et al. 2025; Hao et al. 2026; Wang et al. 2026; Lesens et al. 2025; Zhu et al. 2025; Shaikh et al. 2026) reach similar ratios with trained or online-adapted projectors. xKV [8] instead exploits cross-layer redundancy, sharing a single SVD basis across contiguous layer groups since adjacent layers share dominant singular vectors.
Eviction. Eviction reduces KV cache memory by discarding unimportant tokens. Early methods keep attention sinks and a recent window (Xiao et al., 2024), or a mix of recent and high-attention tokens (Zhang et al., 2023). Later work selects tokens using an observation window (Li et al., 2024), or allocates budgets per layer [6] or per head [14, 13] (Tang et al. 2025). Query-agnostic methods instead score tokens by how well they support reconstructing the context (Kim et al., 2025; Jegou & Jeblick, 2026).
Quantisation. Quantisation compresses the KV cache by lowering the precision of stored keys and values. However, these exhibit channel-wise outliers [11] (Xiao et al. 2023) that dominate a uniform quantisation grid, so naive quantisation degrades sharply below 4 bits (Liu et al., 2024; Hooper et al., 2024). Sub-4-bit methods therefore target these outliers by quantising keys channel-wise and values token-wise (Liu et al., 2024), allocating bit-widths per token [12] (Yang et al. 2024a; He et al. 2024) or per layer (Li et al., 2025), inverting positional embeddings before quantisation (Hooper et al., 2024), or removing outliers with random rotations [2] (Han et al. 2026; Zandieh et al. 2026). AQuA-KV (Shutova et al., 2025) instead quantises a prediction residual, using linear adapters that exploit both inter-layer and intra-layer dependencies: keys are predicted from previous-layer keys, while values are predicted jointly from previous-layer values and current-layer keys. Our amortised value cache uses context-adapted prediction of values from keys within each attention head, storing residuals only for tokens that are hard to reconstruct.
Methods
We propose LowRAM, an asymmetric compression pipeline that treats the key and value caches as structurally distinct objects. For keys, we adopt xKV [8], which exploits the low-rank cross-layer redundancy of key states. For values, we introduce amortisation, a learned associative memory that predicts each token’s value vector directly from its key, fit once per context, and retains residuals only for the values the predictor reconstructs poorly. Prior to both decomposition and prediction, we invert RoPE, following evidence that pre-RoPE representations are more amenable to compression (Hooper et al., 2024; Su et al., 2025; Sun et al., 2025a), with the rotation reapplied at attention time. Both predictor fitting and value reconstruction use the pre-RoPE keys reconstructed by xKV. During decoding, generated key-values are stored exactly in the cache.
Identifying the Predictive Structure of the KV Cache
Prior work has explored predictive compression of the KV cache using inter-layer and intra-layer dependencies (Shutova et al., 2025) or across tokens (Hao et al., 2026). Here, we focus on learning a mapping from keys to values within individual attention heads. To determine which representation is best suited to predictive compression within our framework, we perform a source-target analysis over the KV cache. For each layer, we train the same predictor to reconstruct either the current-layer keys or values while varying the available source representation. Specifically, we consider previous-layer keys (), previous-layer values (), their concatenation (), current-layer values (), and current-layer keys (). To match our proposed compression scheme, each predictor is paired with a residual store, with the residual budget fixed across all configurations. All configurations are evaluated under the same compression budget using both reconstruction error and recovered LongBench performance.
Figure 3 shows that reconstructing values from current-layer keys, , achieves the highest recovered downstream performance among the configurations evaluated. This source-target pair therefore forms the basis of our proposed method. Our comparison revisits the probing configurations examined by AQUA-KV (Shutova et al., 2025) (Figure 2), evaluating them within our predictor and residual-storage framework using both reconstruction error and recovered LongBench performance. Under this setting, lower reconstruction error does not necessarily imply better preserved downstream performance.
Two further observations are worth noting. First, value reconstruction generally recovers more downstream performance than key reconstruction. Second, reconstruction error correlates only weakly with downstream performance: source-target pairs with similar reconstruction errors can exhibit significantly different recovered performance, and the differing scales of keys and values make their absolute error difficult to compare directly. We therefore base our design decisions primarily on recovered downstream performance rather than reconstruction error.
Value Cache Amortisation
Having established as the preferred prediction direction, we model this relationship with a lightweight parametric function. Let and denote the key and value matrices for a single attention head over a sequence of length . We learn a mapping applied row-wise to the key cache, independently per layer and KV head . Concretely, is a two-layer MLP with a learned linear residual branch,
where is a ReLU nonlinearity and is initialised from the Slim Attention prior introduced below; Appendix B gives the full architecture and layer-wise prefill-fitting procedure. During prefill, the parameters are adapted to the current context by minimising the value reconstruction loss
At decoding time, the value cache is reconstructed from the retained keys as . This prediction captures much of the value cache, but not all tokens are reconstructed equally well. Therefore, we store residual corrections only for the tokens with the largest prediction errors: for each token , the reconstruction error is .
Let denote the fraction of residuals retained. We select the top- fraction of tokens according to and define a binary gate
where larger rank corresponds to larger reconstruction error. Writing for the per-token residual, the reconstructed value cache is, in matrix form,
which replaces the original value cache in standard attention, . Algorithm 1 in Appendix B summarises this procedure end to end.
The parameter controls the memory-fidelity trade-off. Smaller stores fewer residuals and hence increases compression, while larger retains more of the original value cache. The total memory footprint consists of two terms: a sequence-independent term from the per-head MLP parameters and the linear residual , and a sequence-dependent term from the retained residuals and their indices. Figure 8 shows that the fixed parameter cost is amortised after the first few thousand tokens, after which memory grows at a fraction of the uncompressed value cache slope.
Figure 8. Analytical memory footprint of the amortised value cache as a function of sequence length. Left: total value cache memory for the uncompressed baseline against value cache amortisation at compression. Right: decomposition of our footprint on a logarithmic scale. The MLP parameters and the linear residual are constant in sequence length, while only the stored residuals and their indices scale with . The fixed parameter cost dominates only at short contexts (below 2–3k tokens), and is quickly amortised as the context grows.
Initialisation from Slim Attention. Following Slim Attention [15], we initialise the residual branch from the key and value projection matrices. When the key projection is square and invertible, values can be reconstructed exactly from keys via ; in GQA models is rectangular, so we instead use the Moore–Penrose pseudoinverse, . This gives the best linear approximation to the key-to-value mapping the model weights can offer, and a strong starting point for training, on top of which the MLP learns a non-linear correction that captures value components the Slim map cannot represent in GQA models.
Warm-starting the Amortised Value Cache MLP. Adapting from scratch on each context wastes gradient steps relearning shared structure. We instead split the work in two: a one-off offline stage that learns the general key-to-value mapping (one pass over 32 batches of 4096-token FineWeb-Edu sequences, 25 gradient steps, under five minutes on one GPU, run once per model), and an online stage that overfits this mapping to the current context. At inference, these weights initialise each layer’s MLP before per-context adaptation, reducing the steps needed to reach a useful approximation.
Efficient Adaptation and Selective Reconstruction
Meta-learned initialisation. Even with warm-starting, fitting the value predictor requires many per-context gradient updates, making it costly when a context is queried only once. We therefore meta-learn an initialisation using First-Order MAML Nichol et al., 2018. Each sequence defines a meta-learning task consisting of its observed KV pairs, and the inner loop adapts the per-head predictor for gradient updates. At inference, the predictor is initialised from the learned parameters and adapted to the new context using the same -update budget. Appendix E gives the complete training procedure.
Selective reconstruction. Reconstructing the complete cache at every decoding step can offset the computational benefit of compression. Following landmark-guided retrieval (Sun et al., 2025a), we partition the prefix into chunks represented by mean-key landmarks, retain outlier chunks and a local window exactly, and reconstruct only the top- chunks selected by the current query. A persistent buffer reuses chunks selected across consecutive decoding steps. We additionally replace the exact key cache SVD with a randomised Cholesky-QR decomposition, giving a speedup with no loss in downstream quality [8]. Appendix F describes the retrieval configuration, kernels, and memory accounting.
Results
Experimental Setup
Benchmarks. We evaluate on RULER (Hsieh et al., 2024) and LongBench [3], and additionally report LongBench filtered to sequences of at least 10K tokens. For this subset we exclude tasks with fewer than 20 samples meeting the length requirement. Appendix G gives per-task statistics. All results are reported as recovered performance relative to the uncompressed baseline. We evaluate across three model families: Mistral-7B-Instruct-v0.3 (Jiang et al., 2023), Llama-3.1-8B-Instruct [16], and Qwen3.5-4B (Qwen Team, 2026). Qwen3.5-4B is a hybrid architecture that interleaves full attention with linear attention layers; only the full attention layers maintain a conventional KV cache, so LowRAM is applied to that subset of layers alone, and the compression ratios we report for Qwen3.5-4B are relative to that subset’s cache rather than the whole model’s.
Eviction. When combined with LowRAM, we apply eviction before fitting the map, discarding tokens unlikely to be attended to so the MLP need only reconstruct and store residuals for the tokens it retains. Importance is estimated with a small CNN predictor adapted from Yang et al. (2025b): during prefill, a pre-hook on each attention layer scores every token by predicted future attention (averaged across heads), and we retain the top- positions by this score, with attention sinks (first 8 tokens) and a local window (last 64 tokens) kept unconditionally.
Quantisation. We quantise with TurboQuant (Zandieh et al., 2026), omitting the 1-bit QJL (Zandieh et al., 2025) error-correction step. Although QJL debiases the inner product estimations, we found that its variance is amplified by Softmax, thus degrading both attention quality and downstream accuracy. For the key cache, XKV factorises the layer-grouped cache as , where holds the shared low-rank basis and is layer-specific; since only scales with context length , we quantise only , which captures most of the memory reduction (Appendix B gives the crossover analysis for when quantising too becomes worthwhile). We use asymmetric precision, 4-bit for the key factor and 2-bit for value residuals.
End-to-End Compression
LowRAM retains near-baseline performance at moderate compression across all three model families (Table 1). Mistral-7B and Llama-3.1-8B retain 97.9–99.4% of baseline performance at compression. At , recovery remains 96.0–99.6% except for Llama-3.1-8B on RULER, where it falls to 93.9%. We attribute the performance drop at this compression ratio to the reduced residual budget. As for Qwen3.5-4B, its RULER performance matches the uncompressed baseline, while LongBench recovery varies between 94.4% and 97.8%.
Efficiency
The results in Table 1 were obtained by fitting the value-cache MLP with 150 gradient updates per context (300 in Table 2, where performance has largely saturated, giving the strongest full-adaptat
| Model | RULER 4K (%) 2x | RULER 4K (%) 3x | RULER 4K (%) 4x | LongBench (%) 2x | LongBench (%) 3x | LongBench (%) 4x |
| Mistral-7B | 99.4 | 98.8 | 96.0 | 100.7 | 99.4 | 97.9 |
| Llama3.1-8B | 99.6 | 97.9 | 93.9 | 98.7 | 98.5 | 99.6 |
| Qwen3.5-4B | 100.0 | 100.0 | 100.1 | 97.8 | 95.0 | 94.4 |
caption: tbl:1: Recovered average performance of LowRAM on RULER 4K and LongBench. Scores are averaged across each benchmark and reported as a percentage of the corresponding uncompressed baseline. The compression ratio reported for Qwen3.5-4B concerns only the full attention layers.
:::
| SR | Meta-init | RULER (%) 16K | RULER (%) 32K | RULER (%) 64K | LongBench (%) >10K |
| 96.9 | 100.0 | 95.8 | 95.3 | ||
| 97.7 | 99.0 | 96.8 | 95.3 | ||
| 96.3 | 96.8 | 92.4 | 92.9 |
caption: Table 2: Recovered LongBench (>10K) and RULER 16-64K performance of LowRAM on Llama3.1-8B-Instruct at compression, with and without selective reconstruction (SR) and a meta-learned initialisation (Meta-init). Rows without Meta-init use 300 gradient updates per context (vs. 150 in Table 1); Meta-init uses 2.
:::
reference) and reconstructing the full cache at every decoding step, significantly increasing prefill and decoding cost, respectively. § 4.3 introduced one mechanism targeting each; here we measure their effect on latency, throughput and accuracy.
Meta-learning the MLP initialisation reduces per-context adaptation to 2 gradient updates, lowering the time-to-first-token (TTFT) overhead of the amortised value cache to that of the uncompressed baseline, down from with 150 updates and with 300 (Figure 6). Figure 4 compares the decode throughput of the baseline with that of LowRAM with selective reconstruction (LowRAM-SR) at compression. The baseline is faster at small batch sizes, but it quickly becomes memory-bound and plateaus, while LowRAM-SR continues to scale. The crossover occurs earlier at longer contexts, and beyond it the gap widens until the baseline runs out of memory. LowRAM-SR supports a batch size of 28 at all three context lengths and, at 262K tokens, reaches roughly the peak throughput of the baseline.
Figure 4. Decode throughput for Llama-3.1-8B at different batch sizes and sequence lengths at compression, comparing exact KV attention against LowRAM-SR (LowRAM with selective reconstruction). Appendix F reports the full selective-reconstruction configuration.
Figure 6. TTFT overhead of value cache amortisation alone versus the number of gradient updates used to fit the MLP, at a 4K-token sequence length with batch size 4. Keys are left uncompressed, so this isolates the cost of fitting the value predictor. Bars show raw TTFT; annotations give the overhead relative to the uncompressed baseline. See Figure 7 for the full LowRAM pipeline, including key factorisation.
Figure 7. TTFT of the uncompressed baseline (exact KV) and the full LowRAM pipeline at compression across context lengths and batch sizes. LowRAM timings include both value cache amortisation (2 gradient updates) and low-rank factorisation of the keys via randomised Cholesky-QR, so overheads exceed those of the value cache alone (Figure 6). Annotations give the overhead relative to the baseline; OOM marks configurations where the baseline runs out of memory.
tbl:2 reports the accuracy cost of each mechanism at compression. Selective reconstruction matches full reconstruction on LongBench (>10K) and stays within 1 point of it, or above it, at every RULER context length. The remaining difference comes from meta-initialisation, which costs 1.4-4.4 points on RULER and 2.4 on LongBench (>10K) relative to 300-update adaptation, in exchange for the TTFT reduction above. With both mechanisms enabled, LowRAM recovers over 92.4-96.8% of baseline performance on RULER and 92.9% on LongBench (>10K). Meta-initialisation is therefore the natural choice when a context is queried once. When the same context serves many queries, the cost of full adaptation is paid only once, and its extra accuracy comes at little amortised cost.
Stacking with Eviction and Quantisation
Table 3 evaluates LowRAM in combination with eviction and quantisation, and Table 10 reports a full sweep in which both are applied on top of LowRAM at , and compression. Quantisation combines particularly well with our method, increasing both the overall compression ratio and recovered performance. For Mistral on LongBench, for example, adding quantisation to the LowRAM cache raises compression to and recovery from 97.9% to 99.4%. The benefit is largest where LowRAM alone degrades most, e.g. raising recovery for Llama on RULER at from 93.9% to 99.4%. We attribute this to two factors. First, storing residuals at lower precision reduces the memory each one requires, so LowRAM can retain more residuals within the same budget. Second, in most configurations quantisation alone introduces very little error, so the gain from additional residuals outweighs the error it adds. The exception is Llama on LongBench, where quantisation alone recovers only 96.3%. In this case the additional error is not fully offset, and adding quantisation lowers recovery at every compression ratio.
| Model | Method | RULER (4K) CR | RULER (4K) Score (%) | LongBench CR | LongBench Score (%) |
| Mistral-7B | Eviction | 99.9 | 99.4 | ||
| Mistral-7B | Quantisation | 99.5 | 98.9 | ||
| Mistral-7B | LowRAM | 96.0 | 97.9 | ||
| Mistral-7B | LowRAM-E | 91.8 | 96.6 | ||
| Mistral-7B | LowRAM-Q | 99.6 | 99.4 | ||
| Mistral-7B | LowRAM-EQ | 93.6 | 96.9 | ||
| Llama-3.1-8B | Eviction | 99.6 | 98.8 | ||
| Llama-3.1-8B | Quantisation | 99.8 | 96.3 | ||
| Llama-3.1-8B | LowRAM | 93.9 | 99.6 | ||
| Llama-3.1-8B | LowRAM-E | 79.9 | 90.6 | ||
| Llama-3.1-8B | LowRAM-Q | 99.4 | 95.8 | ||
| Llama-3.1-8B | LowRAM-EQ | 96.4 | 91.3 |
Table 3. Recovered performance of LowRAM and LowRAM combined with eviction (LowRAM-E), quantisation (LowRAM-Q), and both (LowRAM-EQ). The achieved compression ratio varies across benchmarks because it depends on sequence length. CR refers to compression ratio.
| Model | Method | RULER (4K) CR | RULER (4K) % | LongBench CR | LongBench % |
| Mistral-7B | Eviction | 99.9 | 99.4 | ||
| Mistral-7B | Eviction | 99.3 | 97.6 | ||
| Mistral-7B | Eviction | 98.0 | 96.3 | ||
| Mistral-7B | Quantisation | 99.5 | 98.9 | ||
| Mistral-7B | LowRAM | 99.4 | 100.7 | ||
| Mistral-7B | LowRAM | 98.8 | 99.4 | ||
| Mistral-7B | LowRAM | 96.0 | 97.9 | ||
| Mistral-7B | LowRAM-E | 99.2 | 97.8 | ||
| Mistral-7B | LowRAM-E | 97.3 | 97.5 | ||
| Mistral-7B | LowRAM-E | 91.8 | 96.6 | ||
| Mistral-7B | LowRAM-Q | 99.8 | 99.9 | ||
| Mistral-7B | LowRAM-Q | 99.6 | 99.5 | ||
| Mistral-7B | LowRAM-Q | 99.6 | 99.4 | ||
| Mistral-7B | LowRAM-EQ | 99.1 | 97.7 | ||
| Mistral-7B | LowRAM-EQ | 93.6 | 97.4 | ||
| Mistral-7B | LowRAM-EQ | 89.1 | 96.9 | ||
| Llama-3.1-8B | Eviction | 99.6 | 98.8 | ||
| Llama-3.1-8B | Eviction | 98.8 | 96.6 | ||
| Llama-3.1-8B | Eviction | 98.1 | 95.4 | ||
| Llama-3.1-8B | Quantisation | 99.8 | 96.3 | ||
| Llama-3.1-8B | LowRAM | 99.6 | 98.7 | ||
| Llama-3.1-8B | LowRAM | 97.9 | 98.5 | ||
| Llama-3.1-8B | LowRAM | 93.9 | 99.6 | ||
| Llama-3.1-8B | LowRAM-E | 97.3 | 94.3 | ||
| Llama-3.1-8B | LowRAM-E | 90.0 | 92.4 | ||
| Llama-3.1-8B | LowRAM-E | 79.9 | 90.6 | ||
| Llama-3.1-8B | LowRAM-Q | 99.9 | 96.5 | ||
| Llama-3.1-8B | LowRAM-Q | 99.7 | 95.8 | ||
| Llama-3.1-8B | LowRAM-Q | 99.4 | 95.8 | ||
| Llama-3.1-8B | LowRAM-EQ | 98.2 | 93.5 | ||
| Llama-3.1-8B | LowRAM-EQ | 96.4 | 91.3 | ||
| Llama-3.1-8B | LowRAM-EQ | 82.7 | 89.4 |
Table 10. Recovered performance of LowRAM, used alone or combined with eviction and quantisation, across a range of compression settings for Mistral-7B and Llama-3.1-8B. Results are reported as percentages of the corresponding uncompressed baseline on RULER 4K and LongBench. Grey rows show eviction and quantisation applied independently, blue rows show LowRAM alone, and unshaded rows show LowRAM combined with eviction (LowRAM-E), quantisation (LowRAM-Q), or both (LowRAM-EQ). CR denotes the total compression ratio achieved on each benchmark; for combined methods, it accounts for the composition of the individual compression components and may differ between benchmarks because of sequence-length variation and across models due to model-specific effective context lengths induced by their different tokenizers.
Eviction introduces a clearer trade-off, since it shortens the effective sequence without reducing the fixed cost of the reconstruction network, leaving less memory for residuals. At moderate compression, stacking remains beneficial for Mistral, where LowRAM-E at outperforms eviction alone on both RULER (99.2% vs. 98.0%) and LongBench (97.8% vs. 96.3%), though not for Llama. As compression increases, however, the fixed cost of the network becomes limiting: Figure 8 shows that at amortisation it dominates below roughly 2-3K tokens, a regime that RULER’s 4K-token sequences enter after moderate eviction. Accordingly, raising the LowRAM ratio from to while keeping eviction fixed at reduces RULER recovery by 7.4 percentage points for Mistral and 17.4 for Llama (Table 10). LongBench is less affected, with reductions of 1.2 and 3.7 points, because although its shortest tasks (around 1K tokens) have little residual budget to begin with, its longer tasks (up to 18K tokens) remain well above this threshold even after eviction.
Therefore, to evaluate extreme compression without conflating compression error with the fixed overhead of the reconstruction network, we evaluate on LongBench samples longer than 10K tokens. Table 4 reports the maximum compression each method can achieve while retaining at least 90% of baseline performance (see Table 11 for a sweep over compression ratios). This subset provides a more representative setting for pushing compression further, as the network parameters are sufficiently
| Method | CR | Score (%) |
| Eviction | 3.0× | 98.1 |
| Eviction | 4.0× | 97.8 |
| Eviction | 8.0× | 93.9 |
| Eviction | 10.0× | 91.6 |
| Quantisation | 3.9× | 100.0 |
| Quantisation | 5.1× | 99.3 |
| Quantisation | 7.5× | 92.3 |
| Quantisation | 14.2× | 33.4 |
| LowRAM | 3.0× | 99.6 |
| LowRAM | 4.0× | 97.2 |
| LowRAM | 6.0× | 95.2 |
| LowRAM | 7.0× | 93.4 |
| LowRAM | 8.0× | 91.9 |
| LowRAM | 9.0× | 90.7 |
| LowRAM | 10.0× | 89.0 |
| LowRAM-EQ | 9.2× | 99.2 |
| LowRAM-EQ | 17.8× | 98.0 |
| LowRAM-EQ | 22.1× | 96.2 |
| LowRAM-EQ | 26.9× | 94.5 |
Table 11. Compression ratio and recovered performance for Mistral-7B using different KV cache compression methods, evaluated on the subset of LongBench examples with sequence lengths exceeding 10k tokens. Scores are reported as percentages of the uncompressed baseline. CR refers to compression ratio and is calculated from the byte-level key and value footprints described in Appendix § 10
| Eviction | Quantisation | LowRAM | LowRAM-EQ | |
| CR | ||||
| Score (%) | 91.6 | 92.3 | 90.7 | 94.5 |
caption: Maximum evaluated compression retaining at least 90% of Mistral-7B performance on LongBench contexts longer than 10K tokens. CR uses the byte-level accounting in Appendix C.
:::
amortised to leave a meaningful residual budget. The full stack achieves compression while recovering 94.5% of baseline performance, matching or exceeding the standalone methods despite substantially higher compression ratios.
Value Cache Amortisation in Isolation
To isolate value compression from key-compression error, we keep the keys exact and compare amortisation against xKV applied to the value cache (Table 5; Llama-3.1-8B, two-layer per-head MLP, 300 updates, no warm-starting or ). Value-cache amortisation remains close to lossless across the evaluated compression ratios, retaining 98.2% of RULER performance and 99.3% of LongBench performance at compression. xKV edges slightly ahead on RULER at higher compression ratios, but amortisation is consistently stronger on LongBench, where it stays ahead at every compression ratio. This is consistent with our motivation for treating keys and values asymmetrically. Appendix D (Table 8) reports the complete sweep over adaptation budgets and the effect of the Slim initialisation . Those results show that the Slim prior substantially improves performance under short adaptation budgets, but that its advantage narrows as the predictor converges.
| Method | RULER 4K (%) 2× | RULER 4K (%) 3× | RULER 4K (%) 4× | LongBench (%) 2× | LongBench (%) 3× | LongBench (%) 4× |
| Amortisation | 100.0 | 99.6 | 98.2 | 100.4 | 100.4 | 99.3 |
| xKV | 99.9 | 99.7 | 99.3 | 95.0 | 95.3 | 95.8 |
Table 5. Recovered performance of amortisation and xKV applied to the value cache in isolation on Llama-3.1-8B, reported as a percentage of the uncompressed-cache baseline. The displayed configuration uses a two-layer per-head MLP with 300 per-context gradient updates, without warm-starting or . Table 8 provides the complete optimisation-budget and initialisation sweep.
| Predictor | Updates | RULER 4K (%) 2× | RULER 4K (%) 3× | RULER 4K (%) 4× | LongBench (%) 2× | LongBench (%) 3× | LongBench (%) 4× |
| MLP | 10 | 90.8 | 78.0 | 61.5 | 76.2 | 69.0 | 59.0 |
| MLP + | 10 | 98.4 | 94.4 | 79.0 | 84.9 | 82.3 | 78.1 |
| MLP | 100 | 99.4 | 97.6 | 88.9 | 93.6 | 91.7 | 88.9 |
| MLP + | 100 | 99.9 | 99.1 | 95.1 | 94.4 | 93.8 | 91.5 |
| MLP | 200 | 99.9 | 99.2 | 97.1 | 98.1 | 98.7 | 97.9 |
| MLP + | 200 | 99.9 | 99.5 | 97.3 | 98.8 | 98.4 | 97.3 |
| MLP | 300 | 100.0 | 99.6 | 98.2 | 100.4 | 100.4 | 99.3 |
Table 8. Recovered performance of value cache amortisation on Llama-3.1-8B as a function of the per-context optimisation budget and Slim Attention initialisation. Keys are retained exactly, and results are reported relative to the uncompressed-cache baseline. All predictors are two-layer per-head MLPs trained without warm-starting. The 300-update configuration is the high-fidelity result summarised in Table 5.
Discussion
Our results suggest that keys and values should be compressed differently. Within each attention head, values are largely predictable from their keys. A small per-head predictor fitted to the context, with exact residuals kept for the tokens it reconstructs worst, stays close to lossless at value compression and holds up better on LongBench than low-rank compression of the same cache (Table 5). Pairing it with low-rank keys gives LowRAM, which remains close to baseline at moderate compression and combines with quantisation and eviction to reach on long contexts.
The ablations suggest why prediction suits values. The key-to-value map appears strongly head-specific (sharing one predictor across a layer’s heads costs almost 20 points, Table 7). Keys are harder to predict, and their errors are likely more harmful: a key error perturbs the attention logits and can shift attention onto the wrong tokens, whereas a value error enters only after the attention weights are fixed. This matches findings on asymmetric KV quantisation ((Tao et al., 2025)) and may explain why reconstruction error tracks downstream performance so poorly (Figure 3). Residual placement also matters: at a fixed budget, error-ranked selection recovers almost all accuracy, whereas random selection loses most of it (Figure 5).
Figure 3. Source-target analysis for predictive KV cache compression at a fixed compression ratio for Mistral-7B. The left and centre panels show recovered LongBench performance when reconstructing current-layer keys and values , respectively, from alternative current- or previous-layer representations. The dotted line indicates uncompressed baseline performance. The right panel plots recovered performance against reconstruction mean-squared error for each source-target pair, showing that lower reconstruction error does not necessarily correspond to better downstream performance.
| Method | LongBench (%) | |
| Amortisation | 99.0 | |
| w/o error-residual store | 64.9 | -34.1 |
| w/o slim initialisation | 97.2 | -1.8 |
| w/o warm start | 97.1 | -1.9 |
| w per-layer MLP | 79.5 | -19.5 |
| w linear predictor | 57.5 | -41.5 |
Table 7. Ablation of the main components of the amortised value cache method for Mistral-7B at value cache compression. LongBench scores are reported as percentages of the uncompressed KV cache baseline, and denotes the absolute change in recovered performance relative to the full amortisation method.
Figure 5. Residual selection strategies at equal residual budgets on RULER NIAH-Multikey-1 for Llama-3.1-8B-Instruct, using the full LowRAM pipeline at compression with the meta-learned initialisation (2 gradient updates). Left: mean squared error between attention outputs computed with the compressed and exact KV caches (). Right: NIAH accuracy (). + 8 sinks forces residuals for the first 8 tokens within the same budget. All strategies share the same compressed key cache and are evaluated on the same sequences.
LowRAM is best suited to long contexts. The predictor’s parameters are a fixed cost that is only amortised after a few thousand tokens (Figure 8). This is why LowRAM degrades more on 4K RULER than on LongBench, and why it combines less cleanly with eviction, which shortens the sequence but leaves this cost unchanged. The choice of initialisation adds a second trade-off. Meta-learning brings prefill close to baseline speed at a small cost in accuracy, while full per-context fitting is slower but more accurate. Since both components of LowRAM depend only on the context, the slower option becomes attractive whenever a context is reused, as in document QA or shared-prefix serving, because the fitting cost is paid only once.
Limitations and Future Work
We have not evaluated long generations, which would require periodically compressing the suffix of newly generated tokens (e.g. every tokens) rather than relying on a single prefill-time fit. It is also unclear how much of the learned parametrisation persists across contexts: gains from warm-starting and Slim Attention initialisation suggest part of the key-to-value relationship is shared, while gains from test-time adaptation indicate a remaining context-specific component. Understanding this split, and whether hard-to-predict tokens are stable across contexts, could enable more efficient allocation of memory and computation.
AI Use Statement. We used generative AI to edit text we had written ourselves and to write and edit code, all of which we reviewed and verified before use. We take responsibility for the final content of this work.
Reproducibility Statement. Reproduction settings are in the appendix: value-reconstruction network (B), eviction/quantisation (4.1), meta-learning (E), selective reconstruction (F), benchmarks (G).
Extended Related Works
Test-time Adaptation and Learned Associative Memory
A separate line of work reframes sequence modelling itself as online learning at inference. The motivating observation is that the distribution of sequences encountered at test time differs substantially from the training distribution, and that a model which can fit itself to the stream it is currently processing should enjoy better in-context performance.
Test-time training (Sun et al., 2020) gives this idea a concrete architectural form. Rather than maintaining a fixed memory state, Sun et al. (2025b) treat the state itself as the parameters of a small predictive model, updated during inference by a self-supervised reconstruction loss over the incoming tokens. This makes the memory a sequence-specific map learned online, rather than a hand-specified accumulation rule or a static summary of the past. Titans [4] similarly add a neural long-term memory trained at test time, using a surprise-based gradient update and adaptive forgetting mechanism, and combine it with short-window attention for longer-range storage.
This view has roots in Fast Weight Programmers (Schmidhuber, 1992; 1993), where a slow controller network writes key-value associations into fast weights. Schlag et al. (2021) later showed that linear attention (Katharopoulos et al., 2020) can be interpreted as such a fast-weight memory, with later variants improving the write rule using delta-style corrections and gates (Yang et al., 2024b;c; 2025c). We inherit the core abstraction of this line (a map from keys to values trained by a reconstruction loss) but not its deployment regime: the map is fit once, after prefill, to the completed KV cache of a pretrained Transformer, and neither the base model nor the attention computation is modified at inference.
Reconstruction Network and Optimisation Details
Architecture. For each Transformer layer and KV head , we fit an independent reconstruction network that maps keys to values, repeated here from Equation (1) in § 4 with the hidden-width factor made explicit. The predictor is a two-layer MLP with hidden width , where is the hidden-width factor, and uses ReLU as the activation function :
The linear branch is initialised from the Slim Attention prior, , while the MLP learns a non-linear correction to this mapping. Although each head has separate parameters, all heads within a layer are evaluated and optimised in parallel by storing their parameters with the head dimension as an additional batch dimension. This preserves head-specific key–value mappings without requiring a separate training loop for every head.
Algorithm 1 Value Cache Amortisation
Input KV cache , residual budget
Initialise from warm-started parameters
Train on using MSE loss
(per-token error)
Select top- fraction of tokens with largest errors
Store residuals only for selected tokens
for each decoding step do
Reconstruct values
Add stored residuals to selected tokens
Compute attention using
end for
Layer-wise Fitting during Prefill. We train the reconstruction networks sequentially as the model progresses through prefill. Once attention layer has produced its key and value states, we fit the reconstruction networks for that layer before continuing to layer . The base-model parameters remain frozen throughout.
| Hyperparameter | Setting |
| Number of MLP layers | 2 |
| Hidden-width factor | 1 |
| Activation | ReLU |
| Loss | MSE |
| Optimiser | Adam |
| Learning rate | 0.001 |
Table 6. Hyperparameters used to train the value-reconstruction networks used for value cache amortisation.
For a layer with KV heads, we minimise the head-averaged reconstruction objective
The sequence dimension is processed in a single batch, while the head dimension is always batched. After optimisation, we compute the reconstruction error of every token, retain residual corrections for the highest-error tokens under the specified residual budget, and discard the dense value targets and optimiser state before processing the next layer. Consequently, dense value targets and reconstruction-network optimiser states are required for only one layer at a time, substantially reducing peak memory relative to fitting all layers simultaneously.
Key-cache quantisation crossover. xKV factorises the layer-grouped key cache as , where is a layer-specific reconstruction matrix and holds the left singular vectors of the shared low-rank subspace. With layer-group size , this stores in memory and in , where is the context length, the decomposition rank, and the flattened key width of one layer. Since only scales with , quantising it captures most of the memory reduction at long context. The basis is independent of : quantising it would save only while injecting error into every reconstructed key, and with the crossover is roughly , beyond which quantising is not worth its cost – this is why we quantise only in the main experiments.
Compression Ratio Calculation
since uncompressed key bytes = uncompressed value bytes, we can write the compression ratio as the harmonic mean:
Keys are compressed across layers. Groups of adjacent attention layers are flattened and horizontally concatenated into one token-by-feature matrix, which is factorised as at a rank chosen to hit a target ratio. What is stored is and . is the bytes per element.
where
so
The rank is chosen to meet compression ratio by solving
Values are compressed within a layer. The value cache stores the MLP’s parameters plus the residuals and indices for the hard to predict values. For a single layer, with values, each of size ,
where , is the parameter count, is the cost of a residual ( is the bytes per residual index), and is the number of residuals.
The number of residuals is chosen based on the target compression ratio ,
Quantisation, when applied, reduces the storage cost of the factor for the decomposed keys and the residual for the amortised value cache. For , each of the rows becomes bytes of packed indices plus one norm,
and for the residual count,
where is packed codes, is norm, and is sparse index.
Eviction, when applied, index-selects the key and value states down to the retained positions before either cache sees them. For layer , the predictor’s value-importance scores select
positions, with the first 8 (attention sinks) and the last 64 (local window) forced in and the remainder filled by top importance. The total compression ratio becomes
Extended results
Ablation Studies
To better understand which factors drive the performance of the amortised value cache, we perform a series of ablation studies. We first quantify the contribution of each component of the proposed method, including residual storage, Slim initialisation, warm-starting, and per-head modelling. We then compare alternative Slim initialisation schemes to determine how best to exploit the pretrained key-value relationship.
Architecture Variants. To examine the contribution of each component in the amortised value cache to performance, we remove or vary each individually while keeping all others fixed. The results of this ablation can be seen in Table 7. The residual store and the per-head MLP are by far the most important. Removing residual corrections drops recovered performance from 99.0% to 64.9% (34.1 points), confirming that a small subset of tokens contributes disproportionately to reconstruction error and needs explicit storage. Using a linear predictor instead of an MLP leads to a recovered performance of 57.5%, a performance drop even greater than not having a residual store. The linear map is fitted by least squares, , where denotes the pseudoinverse of the unrotated key matrix. Values are reconstructed as , retaining the same residual-storage mechanism and target compression budget. Per-head reconstruction is the second most important choice: replacing per-head MLPs with a shared per-layer model costs 19.5 points, suggesting different heads learn distinct key-value geometries.
Residual Selection. To test whether the choice of which tokens receive residuals matters beyond the residual budget itself, we compare four selection strategies at equal budgets on RULER NIAH-Multikey-1 (4K tokens) with Llama-3.1-8B-Instruct. We apply the full LowRAM pipeline at compression with the meta-learned initialisation (2 gradient updates). This is the regime in which the predictor is least converged, and in which residual selection should therefore matter most. Random assigns residuals to uniformly sampled tokens. MSE is our default, which ranks tokens by reconstruction error. The + 8 sinks variants additionally force residuals for the first 8 tokens (attention sinks) within the same budget. All strategies share the same compressed key cache and are evaluated on the same sequences, so differences between them isolate the effect of residual selection. We report NIAH accuracy over 200 samples and, on 16 of them, the mean squared error between attention outputs computed with the compressed and the exact KV cache.
Figure 5 shows that error-ranked selection is necessary. Random selection scores less than 40%, with or without sinks, whereas MSE-ranked selection recovers 98%. Forcing residuals for attention sinks reduces attention-output error for both random and MSE-ranked selection but does not positively affect accuracy. This observation is consistent with § 4.1 in which we show that reconstruction error is an imperfect proxy for downstream performance for values.
Optimisation Budget and Slim Attention Initialisation. We examine how the fidelity of value cache amortisation changes with the per-context optimisation budget and with the Slim Attention initialisation. These experiments isolate value compression by retaining the key cache exactly. We use Llama-3.1-8B throughout. Warm-starting is disabled so that the effect of can be measured independently.
The plain MLP configuration learns the key-to-value map entirely from the current context. In the configuration, the predictor additionally contains a trainable linear residual branch initialised using the Slim Attention prior,
where denotes the Moore-Penrose pseudoinverse of the key projection. The residual branch is jointly optimised with the MLP during per-context adaptation.
Increasing the optimisation budget consistently improves the fidelity of the amortised value cache. At 4× compression, the plain MLP improves from 61.5% to 98.2% on RULER and from 59.0% to 99.3% on LongBench as the budget increases from 10 to 300 updates. Performance is largely saturated after 200-300 updates.
The Slim prior is most useful when the adaptation budget is small. With only 10 updates, improves 4× recovery by 17.5 percentage points on RULER and 19.1 points on LongBench. Its effect becomes much smaller after 200 updates, where the plain MLP has sufficient time to learn a comparable mapping directly from the context. This indicates that primarily accelerates adaptation rather than improving the converged solution.
Slim Attention Initialisation. We consider two parameterisations. In the joint variant, all KV heads are concatenated and a single mapping is estimated. This map may use key dimensions from any KV head to reconstruct the value dimensions of any other KV head. In the per-head variant, we instead estimate one independent map per KV head, which prohibits cross-head mixing,
For the MLP in the amortised value cache, the linear map is incorporated as a residual branch:
where is the learned MLP predictor. In the per-head setting, the residual is applied independently to each head: . In the joint setting, the MLP remains head-wise, but its residual branch can mix information across KV heads. The joint and per-head rows use the linear reconstruction alone, without the learned MLP correction.
Table 9 shows that the linear-only joint initialisation is substantially more accurate than its per-head counterpart, but requires considerably more storage. For the 8-KV-head, configuration used by both models, the joint map contains parameters per layer, whereas
| Model | Value Predictor | shape | CR | LongBench (%) |
| Mistral-7B | Joint | 39.7 | ||
| Mistral-7B | Per-head | 9.7 | ||
| Mistral-7B | joint residual | 9.5 | ||
| Mistral-7B | per-head residual | 84.7 | ||
| Llama-3.1-8B | Joint | 67.8 | ||
| Llama-3.1-8B | Per-head | 20.3 | ||
| Llama-3.1-8B | joint residual | 18.5 | ||
| Llama-3.1-8B | per-head residual | 82.3 |
caption: Ablation of joint and per-head initialisation for value cache amortisation on LongBench. Joint permits mixing across KV heads, whereas per-head applies an independent map to each KV head. LongBench scores are percentages of the uncompressed baseline. CR refers to compression ratio. Differences in achieved compression ratio between Mistral and Llama reflect model-specific effective context lengths induced by their different tokenizers.
:::
the per-head map contains . This accounts for the much higher compression of per-head relative to joint .
When is added as a residual branch and jointly optimised (10 gradient updates) with the MLP, the per-head residual consistently outperforms the joint residual. This difference may reflect both the more constrained head-wise parameterisation and the compression budget: at a fixed target compression ratio, the smaller per-head leaves more memory available for storing exact value residuals, which correct the largest reconstruction errors.
Sweep over Compression Ratios
Table 10 provides the full compression-performance sweep underlying Table 3. It reports results for LowRAM, eviction, and quantisation, both individually and in combination, across models and benchmarks. Table 11 similarly expands Table 4, showing how performance varies across compression ratios on the subset of LongBench examples exceeding 10K tokens.
Meta-learning
We reduce the cost of fitting the value cache MLP by meta-learning an initialisation from which the reconstruction network can rapidly memorise the key-to-value mapping of a previously unseen sequence. This differs from the generalisation-after-adaptation objective typically considered in meta-learning. During prefill, all key-value pairs are observed, and the adapted predictor is evaluated on its ability to reconstruct those same values. Our objective is therefore rapid fitting of the observed context rather than generalisation to a held-out query set.
We use First-Order Model-Agnostic Meta-Learning (FOMAML). Let
denote one meta-learning task, consisting of the key-value pairs from a single sequence. We omit layer and head indices for clarity; the same procedure is applied to each per-head reconstruction network. The reconstruction loss for is
Starting from the shared initialisation , the task-specific parameters are adapted for inner-loop updates:
where denotes one update of the inner optimiser with learning rate . The same sequence is used in the inner and outer loops because the desired behaviour is rapid memorisation of the observed key-value pairs.
The meta-loss is evaluated only after the final inner-loop update:
Following FOMAML, we ignore derivatives through the inner-loop optimisation trajectory. For a sequence , its first-order contribution to the meta-gradient is therefore
This avoids the second-order derivatives that would otherwise arise when differentiating the final adapted parameters with respect to .
During meta-training, we process one sequence at a time. For each sequence , the reconstruction network is independently initialised from the current shared parameters and adapted for inner-loop updates. We compute its final-step meta-gradient and accumulate this gradient without performing an outer update. After processing sequences, we average the accumulated gradients:
and use to perform one outer-optimiser update of . Because no outer update occurs during the accumulation window, all sequences begin their inner-loop adaptation from the same current initialisation. The effective meta-batch size is therefore sequences, although only one sequence and its adapted reconstruction network are processed at a time.
| Component | Hyperparameter | Setting |
| Meta-optimisation | Algorithm | FOMAML |
| Inner-loop updates, | 2 | |
| Inner optimiser | Adam | |
| Inner learning rate, | ||
| Outer optimiser | AdamW | |
| Outer learning rate, | ||
| Outer-loop gradient accumulation | 8 | |
| Inner-loop batch size | 4 | |
| Predictor | Number of layers | 2 |
| Hidden-width factor, | 1 | |
| Activation | ReLU | |
| Objective | MSE |
Table 12. Hyperparameters used to meta-learn the value-predictor initialisation. The inner and outer objectives use the same key-value pairs because the desired behaviour is rapid memorisation of the observed context.
At inference time, each reconstruction network is initialised from the learned parameters and adapted to the new sequence for updates using its complete set of key-value pairs. Meta-training is performed offline and therefore does not contribute to TTFT.
| RULER 4K (%) | 2× | 3× | 4× | LongBench (%) | 2× | 3× | 4× |
| 98.4 (−1.2) | 96.1 (−1.8) | 91.2 (−2.7) | 93.6 (−5.1) | 93.0 (−5.5) | 91.1 (−8.5) |
Table 13. Recovered performance of LowRAM when the amortised value cache is initialised from meta-learned weights and adapted with only 2 gradient steps for Llama-3.1-8B. Scores are reported as percentages of the uncompressed baseline; values in parentheses give the drop in percentage points relative to vanilla LowRAM (Table 1), which uses 150 gradient steps per context.
Selective Reconstruction
To reduce decode-time compute, we adopt landmark-guided selective reconstruction, following ShadowKV (Sun et al., 2025a): rather than reconstructing the full key/value cache at every decode step, we retrieve and reconstruct only a query-dependent subset of tokens, using inexpensive per-chunk landmarks as a cheap proxy for full attention when deciding which tokens to retrieve.
Prefill: landmark construction and exact retention. During prefill, keys are partitioned into fixed-size chunks of tokens, and a landmark is computed for each chunk as the mean of its keys. Two subsets of chunks are then set aside and kept exact (both keys and values, uncompressed) rather than represented by their landmark: the chunks whose keys are least similar to their own landmark (outlier chunks, as in ShadowKV), and the chunks overlapping the most recent tokens (a local window; unlike the landmark and outlier mechanism, this is not part of ShadowKV’s original design). The remaining chunks form the landmark pool used for query-time retrieval at decode time.
This exact retention is not free: the landmarks, their chunk indices, and the outlier/local exact keys and values are all additional persistent state on top of the compressed key cache itself. Rather than reporting a compression ratio that ignores this, we account for it directly in rank selection. Because xKV shares one low-rank factor across every layer in a group rather than factorising each layer independently, this overhead is measured per layer but spent per group: at the point a group’s decomposition is triggered, we sum the selective-reconstruction overhead of every layer in the group and subtract it from the byte budget implied by the group’s target compression ratio before choosing the shared rank, so the group’s realised ratio (low-rank factors plus the combined overhead of its member layers) matches the configured target.
We additionally accelerate the low-rank key decomposition using a custom randomised SVD kernel adapted from xKV [8 Appendix C.2], which executes power iterations in 16-bit precision and replaces Householder QR with a parallelisable shifted Cholesky-QR factorisation, in place of exact SVD when factorising the key matrices. This modification leads to a speedup over exact SVD without harming performance.
Decoding: landmark-guided retrieval. Landmarks are computed per key/value head, so under grouped-query attention we pool scores across each group of query heads sharing a head by taking the max, before selecting chunks. At each decode step, the incoming query is scored against every landmark in the pool, and the top- chunks (a chunk budget , equal to the token budget in Table 14 divided by chunk size ) are retrieved and expanded back into token positions. The retrieved positions are combined with the exact outlier and local positions; keys and values, though compressed independently, are reconstructed at this shared set of positions.
| Parameter | Value |
| Chunk size | 8 tokens |
| Token budget | 2048 tokens |
| Local window | 32 tokens |
| Outlier chunks | 48 chunks (384 tokens) |
| Max. tokens attended per step | 2464 tokens |
Table 14. Selective reconstruction configuration.
Persistent reconstruction cache. Following ShadowKV, we avoid redundant reconstruction work across decode steps by keeping a persistent buffer of the most recently reconstructed chunks (capacity tokens). At each step, the newly selected chunk set is diffed against what the buffer already holds: chunks selected again are reused as-is, and only chunks newly entering the selection are reconstructed and written into the buffer. This is implemented via fused CUDA kernels adapted from ShadowKV/xKV.
Figure 4 presents the results for decode throughput, comparing LowRAM-SR (LowRAM with landmark-guided selective retrieval) at compression against the uncompressed baseline at the same context length and batch size.
Benchmarks
| Dataset | Source | Avg len | Metric | Language | #data | #data (>10K) | |
| Mistral | Llama | ||||||
| Single-Document QA | |||||||
| NarrativeQA | Literature, Film | 18,409 | F1 | English | 200 | 181 | 181 |
| Qasper | Science | 3,619 | F1 | English | 200 | 11 (excl.) | 7 (excl.) |
| MultiFieldQA-en | Multi-field | 4,559 | F1 | English | 150 | 46 | 24 |
| Multi-Document QA | |||||||
| HotpotQA | Wikipedia | 9,151 | F1 | English | 200 | 171 | 158 |
| 2WikiMultihopQA | Wikipedia | 4,887 | F1 | English | 200 | 49 | 38 |
| MuSiQue | Wikipedia | 11,214 | F1 | English | 200 | 197 | 195 |
| Summarization | |||||||
| GovReport | Government report | 8,734 | Rouge-L | English | 200 | 99 | 88 |
| QMSum | Meeting | 10,614 | Rouge-L | English | 200 | 160 | 153 |
| MultiNews | News | 2,113 | Rouge-L | English | 200 | 5 (excl.) | 4 (excl.) |
| Few-shot Learning | |||||||
| TREC | Web question | 5,177 | Accuracy (CLS) | English | 200 | 59 | 36 |
| TriviaQA | Wikipedia, Web | 8,209 | F1 | English | 200 | 136 | 123 |
| SAMSum | Dialogue | 6,258 | Rouge-L | English | 200 | 111 | 90 |
| Synthetic Task | |||||||
| PassageRetrieval-en | Wikipedia | 9,289 | Accuracy (EM) | English | 200 | 200 | 200 |
| Code Completion | |||||||
| LCC | Github | 1,235 | Edit Sim | Python/C#/Java | 500 | 26 | 14 (excl.) |
| RepoBench-P | Github repository | 4,206 | Edit Sim | Python/Java | 500 | 307 | 229 |
| Total | 3,550 | 1,742 | 1,515 |
Table 15. LongBench datasets used for evaluation. #data (>10K) gives the number of samples in the long-context subset used in Tables 4 and 11, i.e. samples whose prompt exceeds 10K tokens under the Mistral-7B and Llama tokenizers, respectively. Qasper and MultiNews (and LCC for Llama) are excluded from this subset because less than 20 of their samples meet the length requirement. Avg len is reported in words, following the original LongBench statistics.
References
- [1]Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901, 2023.
- [2]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=dfqsW38v1X.arxiv.org/abs/2404.00456
- [3]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119–3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172/.DOI
- [4]Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=8GjSf9R h7Z.arxiv.org/abs/2501.00663
- [5]Luca Benfenati, Matteo Risso, Andrea Vannozzi, Ahmet Caner Yüzügüler, Lukas Cavigelli, Enrico Macii, Daniele Jahier Pagliari, and Alessio Burrello. Don’t be so Stief! learning KV cache low-rank approximation over the Stiefel manifold. arXiv preprint arXiv:2601.21686, 2026.arxiv.org/abs/2601.21686
- [6]Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling, 2025. URL https://openreview.net/forum?id=jZVNmDiU86.arxiv.org/abs/2406.02069
- [7]Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: Compressing KV-cache with low-rank projection. In International Conference on Learning Representations (ICLR), 2025. arXiv:2407.21118.arxiv.org/abs/2407.21118
- [8]Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Hung-Yueh Chiang, Yash Akhauri, Xilai Dai, Huiqiang Jiang, Yucheng Li, Luis Ceze, Kai-Chiang Wu, and Mohamed S. Abdelfattah. xKV: Cross-layer KV-cache compression via aligned singular vector extraction. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=XDb69IykA8.arxiv.org/abs/2503.18893
- [9]Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=ztn8FCR1td.arxiv.org/abs/2405.21060
- [10]DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zhang, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, Q. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Yuan Xu, Yuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. URL https://arxiv.org/abs/2405.04434.
- [11]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=dxiaXiGWqBoxaD.
- [12]Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643, 2024.arxiv.org/abs/2403.04643
- [13]Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=tcisuhGsQZ.arxiv.org/abs/2407.11550
- [14]Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=uNrfpDPMyo.arxiv.org/abs/2310.01801
- [15]Nils Graef and Andrew Wasielewski. Slim attention: cut your context memory in half without loss – k-cache is all you need for mha, 2025. URL https://arxiv.org/abs/2503.05840.
- [16]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
- [17]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.arxiv.org/abs/2312.00752







