betaFeedback

WorksComputer Science

Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression

Abstract

Private, personalised language models must run on local consumer devices, where large, frontier models cannot fit, and retaining long contexts is non-trivial. Sequential memory—the key-value (KV) cache—grows linearly with context length, quickly dwarfing the model itself: for Qwen3-1.7B-Base, 200,000 retained entries require 21 GiB, 6.6 times more than the model weights. Yet the KV-cache compression methods we measured fail beyond moderate compression ratios. We introduce Transformer-FSQ, addressing extreme KV-cache compression by replacing standard key and value projections with a finite scalar quantisation bottleneck. Our cache stores a packed vector of discrete integer codes per token, with no per-token quantisation metadata. Converting a pretrained transformer trains only these projections on 0.25 tokens per original parameter. We evaluate Transformer-FSQ against five baseline families on models from 0.6B to 8B parameters. On long-context retrieval (RULER), from 12×\times to 97.5×\times Transformer-FSQ is the most accurate method we evaluate, across scales. A fused Triton kernel decodes directly from the integer codes; with baselines given their authors' kernels where available, it decodes at 1.01×1.01\times (95% interval 0.980.98–1.051.05) the throughput of native SDPA attention on the uncompressed Qwen3-8B-Base at 32×32\times and 1.54×1.54\times at 97.5×97.5\times. Extreme KV-cache compression is therefore a practical way to extend how much context a fixed device can hold.

Introduction

A language model’s key-value (KV) cache grows with every token it processes, and on local hardware, the cache competes directly with model weights for a fixed memory budget. Unlike the model weights, whose size does not depend on context, the KV cache grows with every retained entry and soon outgrows the model itself. Local deployments matter: prompts stay on device within user control [46, 53, 71], and processing spreads beyond centralised intermediaries [3, 50]; releasing weights alone does not achieve this [34, 60]. For Qwen3-8B-Base, caching 144 KiB per token in half precision (36 layers, 8 key-value heads of width 128) [63] alongside 15.3 GiB of weights on a 16 GiB GPU leaves room for only 5 thousand uncompressed tokens before runtime assertions, reaching 1.39M retained entries only at 256× compression (Figure 1 highlights the difference in magnitudes). Large capacity is essential: in-context learning, multi-step reasoning, and concurrent rollouts all depend on holding many tokens in memory simultaneously [30].

KV-cache capacity on a 16 GiB GPU

Figure 1. KV-cache capacity on a 16 GiB GPU. With ≈0.74 GiB left for cache after Qwen3-8B model weights, extreme compression (256×) expands token capacity from 5.4k to 1.39M, enabling million-token contexts on consumer hardware.

KV-cache compression trades accuracy for context. Existing methods shrink the cache along four axes (§ 2). Token eviction keeps fewer tokens. Scalar quantisation stores each coordinate in fewer bits. Low-rank projection and latent attention store fewer coordinates per token. Vector quantisation replaces groups of coordinates with indices into a learned codebook. Four of the five baselines we compare report accuracy at compression ratios of at most about 14× Table 1, whereas holding several million tokens beside a model’s weights on a 16 GiB GPU needs ratios of tens for the smaller models studied here, and deeper compression for the 8B model Figure 1; Appendix § 12.§ 3, the extreme regime (above 16×) this paper targets.

MethodFamilyCache contents per tokenTrainedDeepestMax tokens (16 GiB, ≥0.30\ge0.30)
PyramidKVtoken evictionfull-precision KV of kept tokensno174×\times552k
KIVIscalar quantisation2-bit KV, full-precision windowno5.2×\times684k
VecInfervector quantisationcodebook indices, window, sinksno11.0×\times592k
Palulow-rank projectionquantised low-rank SVD latentyes10.2×\times1.42M
TransMLAlatent attentionlatent, fp16 rotary keyyes14.2×\times542k
Transformer-FSQ (ours)finite scalar quantisationbit-packed integer codeyes410×\times2.21M

Table 1. Baselines and their limits on a 16 GiB GPU. Deepest: the deepest ratio at which the original authors report downstream accuracy, priced as in § 4.4 (PyramidKV: needle retrieval); for Transformer-FSQ, the deepest we evaluate, where RULER falls to 0.003 on Qwen3-0.6B-Base. Max tokens: input tokens whose cache fits beside the converted model’s weights in 16 GiB, on Qwen3-0.6B-Base at RULER ≥0.30\ge0.30; for PyramidKV, tokens represented after eviction.

To reach extreme compression we use finite scalar quantisation (FSQ) [45]. FSQ projects a continuous representation into a low-dimensional space and rounds each coordinate to a fixed number of levels. Unlike learned-codebook vector quantisation [57, 12], FSQ requires no stored codebook: the discrete grid is defined implicitly by the bounding levels, eliminating codebook search, commitment losses and learned codebook collapse. Transformer-FSQ replaces the key and value projections of attention with an FSQ encoder and a linear decoder, storing bit-packed integer codes in the KV cache rather than floating-point vectors. Conversion inserts these projections into a pretrained transformer in closed form, and finetuning trains only the new parameters on 0.25 tokens per original parameter, leaving the backbone model frozen. The inserted projections are lightweight at extreme compression: at 97.5× on Qwen3-1.7B-Base they hold under 10M parameters in total, fewer than the key and value projections they replace, though at 2× on Qwen3-0.6B-Base the inserted bottlenecks hold 352M parameters and replace 59M original key and value projection parameters, a net increase of 294M (≈ 49% of the model), as the code approaches the residual dimension. At extreme compression, where the bottleneck is narrow, Transformer-FSQ reduces the KV cache by large factors with a small parameter overhead § 4.2.

Figure 2 evaluates Transformer-FSQ against five baselines, one from each family in Table 1, plus a training-free TurboQuant reference, on RULER [17] (13 official tasks, 50 items each) at 8,192 tokens across three Qwen3 models. Up to about 5×, training-free vector quantisation and low-rank projection preserve accuracy with less overhead. At these ratios Transformer-FSQ’s code approaches or exceeds the model’s hidden dimension, and beyond it the closed-form initialisation repeats principal directions, so part of the cache holds duplication information § 4.2. From 12× to 97.5×, Transformer-FSQ retains substantially higher retrieval accuracy than every baseline, outscoring token eviction by 0.12 and 0.14 at 12× on Qwen3-0.6B and 1.7B, and retaining the highest share of uncompressed accuracy through 97.5×. Because Transformer-FSQ compresses channel width while token eviction reduces sequence length, the two compose: on a 16 GiB GPU, Qwen3-0.6B-Base with both yields an aggregate projected capacity of 3.8M tokens across concurrent sequences at RULER ≥ 0.30, 1.7× the best baseline, our head-shared PyramidKV (95% interval 1.0–2.3). Because the cached codes are small, decoding from them is fast: a fused Triton kernel matches native SDPA decoding of the uncompressed model at 32× and exceeds it from 97.5× § 5.2.

Our contributions are: (1) Transformer-FSQ, a discrete latent compression method that stores packed integer codes in the KV cache (§ 4.1); (2) a closed-form conversion and finetuning operator requiring only 0.25 tokens per original parameter on ClimbMix with the backbone frozen (§ 4.2); (3) an empirical evaluation showing that beyond 12×12 \times, Transformer-FSQ achieves the highest RULER accuracy at 8,192 tokens of the methods we evaluate, stacking with eviction to expand aggregate projected capacity by 1.7×1.7 \times (95% interval 1.0–2.3) on a 16 GiB GPU (§ 5); and (4) a fused Triton decode kernel that reads the packed codes directly, timed against native SDPA under strict dispatch, with KIVI, VecInfer and Palu on their authors’ released kernels (§ 4.5).

Related work

Attention remains the dominant sequence mixer in large language models, despite alternatives (sparse, linear, recurrent, state-space) that alter sequence processing to reduce long-context costs. We study a complementary problem: retaining Transformer attention while reducing the state stored during inference. In autoregressive decoding, persistent key-value caches grow linearly with context length; FlashAttention [8] optimises working memory during execution, but leaves this persistent cache uncompressed.

KV-cache compression mitigates this bottleneck while preserving existing model weights and architectures. Prior methods compress along four axes: token count, coordinate precision, coordinate dimension, or discrete codebooks. We review these below (extended review: Appendix § 10; baseline details: Appendix § 11.§ 7).

Token eviction: fewer stored tokens. Token eviction discards less important key-value pairs along the sequence dimension, keeping full precision for surviving tokens. StreamingLLM [62] retains attention sinks and a local sliding window; PyramidKV [6], SnapKV [33], and H2_2O [70] allocate cache budgets by attention patterns. While eviction lowers both memory and attention arithmetic, discarded token context is permanently lost.

Scalar quantisation: fewer bits per coordinate. Scalar quantisation preserves key-value dimensionality while reducing numerical precision. KIVI [39] quantises keys per channel and values per token to 2 bits alongside a full-precision window; KVQuant [16] and GEAR [22] employ non-uniform grids or error-correcting residuals. TurboQuant [66] rotates activations before scalar quantisation; it appears in Figure 2 evaluated training-free at moderate compression, without recovery finetuning or extreme-budget adaptation (Appendix § 11.§ 7).

Figure 2. RULER at 8,192 tokens against achieved compression on Qwen3-0.6B, 1.7B and 8B. Solid: finetuned; dashed: training-free; grey horizontal line: uncompressed. Points average up to three seeds (one on 8B), with 95% bootstrap intervals over seeds and then items stratified by task. Beyond about 12×, Transformer-FSQ scores highest on all three models.

Latent projection: fewer coordinates per token. Latent projection reduces key-value dimensionality into narrower continuous vectors. Multi-Head Latent Attention (MLA) [37] replaces per-head storage with a shared continuous latent and decoupled rotary key; TransMLA [44] adapts models to MLA via finetuning, while Palu [7] factorises key-value projections via SVD. These latents remain floating-point vectors unless subsequently quantised.

Vector and discrete quantisation: discrete representations per token. Vector quantisation replaces continuous representations with discrete codes. VQ-VAE [57] maps inputs to their nearest codebook entries; product quantisation codes subspaces independently [19], and residual quantisation codes successive error stages [28, 67]. For KV caches, coupled quantisation [69] and CommVQ [32] learn codebooks; VecInfer [65] suppresses outliers before VQ; and PQCache [68], Kumar (2024), VQKV [59], and GSRQ [24] use product or residual variants. While VQ enables discrete representations, codebook memory, nearest-neighbour search latency, and codebook collapse during training constrain its scalability under extreme compression.

Transformer-FSQ combines dimensionality reduction with a discrete representation: it maps the residual stream to a compact latent, rounding coordinates onto a finite grid. Unlike conventional VQ, this Cartesian grid is implicit and unlearned, eliminating stored codebooks and nearest-neighbour search.

Discrete latent geometry comparing vector quantisation and finite scalar quantisation

Figure 3. Discrete latent geometry: (a) VQ searches an explicit codebook, inducing irregular Voronoi cells [14]. (b) FSQ quantises coordinates independently onto a fixed Cartesian grid L1×L2\mathcal{L}_1 \times\mathcal{L}_2, an implicit codebook that is never materialised or searched.

Discrete latents for KV-cache compression

Our goal is extreme KV-cache compression, where reducing precision or width alone is not enough. We seek a representation reducing both coordinates and bits per coordinate without auxiliary tables: a wide, low-bit latent narrower than the full KV cache but wider than an equal-byte floating-point vector. We examine VQ and FSQ as two routes to this discrete bottleneck.

Vector quantisation. Discrete representation learning maps a continuous latent vector to a finite set of states. In vector quantisation (VQ), states are stored explicitly as codebook C={c1,…,cK}\mathcal{C} = \{c_1, \ldots, c_K\}, with ck∈Rdc_k \in\mathbb{R}^d, mapping input z∈Rdz \in\mathbb{R}^d to its nearest codeword:

QVQ(z)=ck⋆,k⋆=arg⁡min⁡k∥z−ck∥22.(1)Q_{\mathrm{VQ}}(z) = c_{k^\star}, \quad k^\star= \arg\min_k \|z - c_k\|_2^2. \tag*{(1)}

Nearest-neighbour assignment partitions latent space into Voronoi cells [14]. However, learned codebooks require commitment losses, updates, and nearest-neighbour search (shown on Figure 3a), risking centroid collapse where entries become inactive [57, 45].

Finite scalar quantisation. FSQ replaces the learned vector codebook with fixed scalar levels in each latent coordinate [45]. Let Li\mathcal{L}_i denote the finite set of allowed scalar values for coordinate ii, and let Li=∣Li∣L_i = |\mathcal{L}_i| be the number of levels in that coordinate; for instance, in Figure 3b every coordinate may take one of the Li=3L_i = 3 values Li={−1,0,1}\mathcal{L}_i = \{-1, 0, 1\}. FSQ first bounds each encoder output and quantises coordinates independently: a coordinate uu is bounded and rounded as

u~=tanh⁡(u+atanh⁡(δ/h)) h−δ,u^=Q(u)=u~+sg⁡(⌊u~⌉−u~)∈Li,(2)\tilde{u} = \tanh(u + \operatorname{atanh}(\delta/h))\,h - \delta,\qquad\hat{u} = Q(u) = \tilde{u} + \operatorname{sg}(\lfloor\tilde{u} \rceil- \tilde{u}) \in\mathcal{L}_i, \tag*{(2)}

where ⌊⋅⌉\lfloor\cdot\rceil denotes round-to-nearest with round-half-to-even tie breaking, the stop-gradient sg⁡\operatorname{sg} implements the straight-through estimator [2], and hh and δ\delta place the LL levels on the integers: bounding scale ε=10−3\varepsilon= 10^{-3}, half-width h=(L−1)(1+ε)/2=7.5075h = (L - 1)(1 + \varepsilon)/2 = 7.5075 for L=16L = 16, and centre shift δ=0.5\delta= 0.5 for even LL (δ=0\delta= 0 for odd LL). Packing stores the unsigned integer shift δ=0.5\delta= 0.5 for even LL (δ=0\delta= 0 for odd LL), plus the rounded signed level (00 to L−1L - 1 for L=16L = 16, i.e. −8,…,7↦0,…,15-8,\ldots,7 \mapsto0,\ldots,15) in b=⌈log⁡2L⌉b = \lceil\log_2 L\rceil bits; the decoder subtracts the same offset before applying Wk/WvW_k/W_v. The forward pass returns the rounded value; the stop-gradient makes the backward pass treat rounding as the identity, so gradients reach the encoder as if the rounding were absent. Rounding is applied in every training step, so the encoder learns representations that survive it. The complete set of representable vectors is therefore the Cartesian product

CFSQ=L1×⋯×Ld,∣CFSQ∣=∏i=1dLi.(3)\mathcal{C}_{\mathrm{FSQ}} = \mathcal{L}_1 \times\cdots\times\mathcal{L}_d,\qquad|\mathcal{C}_{\mathrm{FSQ}}| = \prod_{i=1}^{d} L_i. \tag*{(3)}

When all coordinates use Li=LL_i = L levels, the implicit code space contains LdL^d vectors (32=93^2 = 9 grid points in Figure 3b); in practice the LiL_i may differ across coordinates to meet a target code-space size [45]. This space is combinatorially large but is never materialised: encoding requires only coordinate-wise rounding, with no centroid table or nearest-neighbour search. Because codewords are fixed to a regular Cartesian grid rather than learned vectors, FSQ avoids commitment losses, codebook updates, and learned codebook collapse [45], while code utilisation across individual coordinates is empirically evaluated in Appendix § 13.§ 6.

FSQ latents for KV-cache compression. These properties make FSQ well suited to transformer KV-cache compression: (i) Simplicity: stateless coordinate-wise rounding requires no learned codebook or commitment objectives [45]; (ii) Implicit codebook: the code space (Ld)(L^d) is never learned, so it cannot collapse; (iii) Zero metadata: fixed integer coordinates cost ⌈log⁡2Li⌉\lceil\log_2 L_i\rceil bits each, with no per-token or per-channel scale overhead; and (iv) Cheap decoding: dequantisation is an integer offset with no table lookup, so a kernel can read the packed codes directly. § 4 integrates this bottleneck into the attention pathway.

Transformer-FSQ

Now we retrofit a pretrained transformer with an extremely compact per-token state that can replace the conventional KV cache, without pretraining the model from scratch. This setting differs from both conventional discrete tokenisation and existing KV-cache quantisation. FSQ is typically used to map a continuous signal to a small number of quantised coordinates whose Cartesian product acts as a discrete token, while KV-cache quantisation usually operates after keys and values have already been formed. Transformer-FSQ instead makes two design choices tailored to the internal state of a pretrained language model.

Wide product codes. A cached language-model state must preserve enough information to support many future, query-dependent KV readouts. We therefore use FSQ in a different regime from conventional image and audio tokenisers [31]: as a wide product representation of many low-precision coordinates rather than a single categorical token. Under a fixed byte budget, lowering the precision of each coordinate buys proportionally more latent dimensions (RQ3 in §5.2 tests this trade-off), while FSQ can scale to this width without materialising or searching the exponentially large Cartesian code space. Thus LGL^G should be understood as the number of representable product codes, not as a categorical vocabulary.

Compress once, before K/V expansion. Standard attention first maps the residual state xtx_t into separate keys and values, possibly across several KV heads, and caches these derived representations. Post-hoc KV quantisation therefore compresses multiple projections of the same underlying state. Transformer-FSQ instead places the bottleneck on the shared residual representation,

xt⟶zt⟶{k^t(h),v^t(h)}h=1nkvx_t \longrightarrow z_t \longrightarrow\left\{\hat{k}_t^{(h)},\hat{v}_t^{(h)}\right\}_{h=1}^{n_{kv}}

where xtx_t is the residual state of token tt, ztz_t is its cached integer code, and k^t(h)\hat{k}_t^{(h)} and v^t(h)\hat{v}_t^{(h)} are the key and value reconstructed from ztz_t for each of the nkvn_{kv} KV heads hh. Each token is therefore compressed once, and the same cached code serves all keys, values and KV heads. This is the discrete analogue of caching a shared continuous latent as in MLA. Figure 4 illustrates the resulting architecture.

Standard attention and Transformer-FSQ decoding steps

Figure 4. One decoding step. Left: standard attention caches keys and values. Right: Transformer-FSQ caches one bit-packed code ztz_t per token, shared by all KV heads. Keys are reconstructed and rotated at every cached position (×T\times T); values are decoded once, after attention pools the codes, zˉ=∑tatzt\bar{z} = \sum_t a_t z_t. Orange: added by Transformer-FSQ; grey: frozen backbone. Details: Appendix § 14.

The FSQ bottleneck

Let xt∈RCx_t \in\mathbb{R}^C denote the input to the original key and value projections at token tt: the residual stream after the layer’s pre-attention normalisation. A learned linear encoder E∈RG×CE \in\mathbb{R}^{G \times C} maps it to a GG-dimensional latent, which is bounded and rounded coordinate-wise to LL fixed levels by the FSQ quantiser QFSQQ_{\mathrm{FSQ}}, which applies the map QQ of (2) to each coordinate:

zt=QFSQ(Ext)∈L1×⋯×LG.(4)z_t = Q_{\mathrm{FSQ}}(E x_t) \in\mathcal{L}_1 \times\cdots\times\mathcal{L}_G . \tag*{(4)}

We use the standard FSQ bounding parametrisation of Mentzer et al. [45], with the exact form and integer packing convention of Eq. 2.

With b=⌈log⁡2L⌉b = \lceil\log_2 L\rceil bits per coordinate, the cache occupies ⌈Gb8⌉\lceil\frac{Gb}{8}\rceil bytes per token per layer and stores nothing beside the packed code. All main-text experiments use L=16L = 16, i.e. four bits per coordinate. At equal storage, this permits four times as many coordinates as an FP16 latent.

A single decoder Wkv∈R2nkvdh×GW_{kv} \in\mathbb{R}^{2n_{kv}d_h \times G}, split into WkW_k and WvW_v, reconstructs all KV heads:

k^t=RoPE⁡t(Nk(Wkzt)),v^t=Wvzt(5)\hat{k}_t = \operatorname{RoPE}_t\left(N_k(W_k z_t)\right), \qquad\hat{v}_t = W_v z_t \tag*{(5)}

where NkN_k denotes the model’s native key normalisation when present. The value projection need not be applied separately to every cached token. For query head hh with attention weights at(h)a_t^{(h)}, writing WvW_v for the rows of the value decoder that belong to the key-value head g(h)g(h) shared by query head hh,

∑tat(h)Wvzt=Wv(∑tat(h)zt).(6)\sum_t a_t^{(h)} W_v z_t = W_v \left(\sum_t a_t^{(h)} z_t\right). \tag*{(6)}

Values are therefore decoded only after attention has pooled the cached codes. Keys, in contrast, must be reconstructed before their dot products with the query because positional transformations act on each cached position separately. Transformer-FSQ thus trades cache traffic for additional key-side arithmetic; § 5.2 measures this runtime cost.

Retrofitting a pretrained transformer

Transformer-FSQ is applied as a post-training operator rather than requiring a new model to be pretrained from scratch. For each attention layer, the original write-time key and value projections are replaced by the encoder EE and shared decoder WkvW_{kv}. These inserted tensors are initialised in closed form from a small calibration set using the pretrained model’s own activations, and only the inserted parameters are subsequently finetuned under the language-model objective while the backbone remains frozen. Conversion alone already produces a functional model, analogous to the post-training conversion procedures of Palu (Chang et al., [7] and TransMLA (Meng et al., [44].

The closed-form initialisation uses principal directions of the calibration residual states for the encoder and a ridge-regularised least-squares fit for the decoder. Let X∈Rm×CX \in\mathbb{R}^{m \times C} hold the residual states of a calibration sample of m=32,768m = 32{,}768 ClimbMix tokens, and let uju_j and σj\sigma_j (j=0,…,C−1j = 0, \ldots, C - 1, in decreasing order of variance) be the principal directions of the centred XX and the standard deviation along each. The encoder’s rows are

Ei,:=ui mod C⊤3σi mod C,i=0,…,G−1,(7)E_{i,:} = \frac{u_{i \bmod C}^{\top}}{3\sigma_{i \bmod C}}, \qquad i = 0, \ldots, G - 1, \tag*{(7)}

so that one standard deviation falls at a third of the tanh bounding range. The encoder has no bias, and every layer is calibrated at once on the original model’s activations. When the latent coordinate width GG exceeds the hidden dimension CC (G>CG > C, as occurs at moderate compression ratios), the encoder basis cycles through the CC orthonormal principal directions i mod Ci \bmod C, so that every direction is repeated, and the leading directions once more when GG is not a multiple of CC; repeated rows start identical and nothing breaks their symmetry, so at moderate ratios some coordinates carry duplicate information. The decoder is fitted by ridge regression to reconstruct the residual states from their quantised latents Z=QFSQ(XE⊤)∈Rm×GZ = Q_{\mathrm{FSQ}}(XE^{\top}) \in\mathbb{R}^{m \times G}, and is then composed with the original key and value projections:

D=(Z⊤Z+λIG)−1Z⊤X∈RG×C,Wkv=[WK;WV]D⊤∈R2nkvdh×G,(8)D = (Z^{\top}Z + \lambda I_G)^{-1}Z^{\top}X \in\mathbb{R}^{G \times C}, \qquad W_{kv} = [W^{K}; W^{V}]D^{\top} \in\mathbb{R}^{2n_{kv}d_h \times G}, \tag*{(8)}

with penalty λ=1\lambda= 1. The converted bottleneck replaces 2Cnkvdh2Cn_{kv}d_h parameters with G(C+2nkvdh)G(C + 2n_{kv}d_h): at 97.5× on Qwen3-1.7B-Base (G=84G = 84) the bottlenecks hold under 10M parameters in total and replace about 117M. The bottleneck therefore has fewer parameters than the projections it replaces whenever G<2Cnkvdh/(C+2nkvdh)G < 2Cn_{kv}d_h/(C + 2n_{kv}d_h): below G=683G = 683 on Qwen3-0.6B, 1,024 on Qwen3-1.7B and 1,365 on Qwen3-8B, which at 4 bits per coordinate are the compression ratios 12×, 8× and 6×. The initialised bottleneck already approximates the pretrained key and value computation.

Interaction with positional encoding

The converted model retains its native positional encoding. In Qwen-style RoPE attention (Su et al., [54]), the cached code itself carries no explicit position; position is introduced only after reconstructing each cached key, k^t=RoPE⁡t(Nk(Wkzt))\hat{k}_t = \operatorname{RoPE}_t(N_k(W_k z_t)). Because the position-dependent rotation lies between WkW_k and the query-key dot product, the key decoder cannot in general be folded into the query. The score of a query qq against cached token tt is q⊤RoPE⁡t(Nk(Wkzt))q^{\top}\operatorname{RoPE}_t(N_k(W_k z_t)); without the rotation and the normalisation it would equal (Wk⊤q)⊤zt(W_k^{\top}q)^{\top}z_t, and WkW_k could be applied to the query once per step, as Eq. (6) applies WvW_v once. Because RoPE⁡t\operatorname{RoPE}_t is a different rotation at every position tt, no single folded query exists. Consequently, our implementation reconstructs keys at every decoding step, which is the main arithmetic overhead of Transformer-FSQ.

This overhead is specific to the interaction with RoPE rather than to the discrete bottleneck itself. Without position-dependent key rotation, the linear key decoder can be absorbed into the query when key normalisation is absent or handled separately. A pre-RoPE variant, FSQ-pre, is described in Appendices E and F; the exact interaction with query-key normalisation is also detailed there.

Sizing every cache from one byte budget

We compare all methods using the same realised cache-byte budget. Every quantity stored per token is charged, including quantisation metadata, full-precision windows, positional state, and auxiliary cached representations; Appendix C.4 lists what each method is charged. Ratios are priced at a context of 8,192 tokens. Reported compression ratios therefore divide the uncompressed FP16 KV-cache size,

cfull=2nkvdh⋅sizeof⁡(float16)bytes per token per layer,(9)c_{\mathrm{full}} = 2n_{kv}d_h \cdot\operatorname{sizeof}(\mathrm{float16}) \qquad\text{bytes per token per layer}, \tag*{(9)}

by the actual bytes retained by each configuration; cfullc_{\mathrm{full}} is 4096 B for the three Qwen models.

The decode kernel

A fused Triton kernel (Tillet et al., [55]) computes decode-time attention directly from the packed codes: it reconstructs keys on chip, accumulates attention-weighted codes and applies WvW_v once per step (Eq. (6)), so no reconstructed key or value is written to GPU memory. Appendix D.4 gives its data flow and Appendix D.5 the timing protocol.

Experiments

Setup

Models. We convert three pretrained models, Qwen3-0.6B-Base, Qwen3-1.7B-Base and Qwen3-8B-Base (Yang et al., [63]) (Appendix C). Against Transformer-FSQ we compare representative methods from five distinct families (Table 1): token eviction (PyramidKV (Cai et al., [6])), scalar quantisation (KIVI (Liu et al., [39])), vector quantisation (VecInfer (Yao et al., [65])), low-rank projection (Palu (Chang et al., [7])), and latent attention (TransMLA)

[44]. The first three require no training, while Palu and TransMLA are finetuned with the same data, optimiser and token budget as Transformer-FSQ. Each follows its authors’ specification except where noted (Appendix § 11), is sized by § 4.4, and runs as far as that sizing allows. A converted model trains only inserted tensors, with backbone frozen, on ClimbMix [11] with AdamW [40], at 0.25 tokens per parameter (and 0.025, 0.075 on Qwen3-0.6B-Base to test finetuning budget). Conversion and finetuning took 1,867 GPU-hours on NVIDIA GH200 GPUs (Appendix § 11.§ 6). Decode throughput was benchmarked on 4×4 \times NVIDIA L40S GPUs with the fused Triton decoding kernel (§ 4.5) and, where available, each baseline’s released kernels (Appendix § 12.§ 5).

Benchmarks. RULER [17] is our primary synthetic benchmark because its items provide exact ground truth across controlled sequence lengths without confounding from pretraining knowledge. We prefer RULER to common alternatives: single-needle retrieval [21] does not test associations across multiple tokens, sliding-window perplexity is dominated by local context and can miss long-range context loss, and synthetic reasoning suites like BABILong [27] fail on uncompressed base models, confounding cache fidelity with reasoning limits. LongBench [1] complements RULER with real-world multi-page tasks (summarisation, question answering and classification), testing whether results on synthetic retrieval carry over to natural documents.

Metrics. Long-context retrieval is evaluated at 8,192 tokens across RULER’s 13 standard tasks (50 items each). LongBench complements this by evaluating natural documents up to 32,768 tokens (truncated to 32,736) across three tasks. Generation is greedy, up to 32 (narrativeqa), 200 (qov_report) and 16 (trec) new tokens; a prompt longer than the window means that budget is truncated in the middle, keeping its start and end, and every method in a panel is scored on the same documents. All evaluations are scored with official task metrics. RULER runs in bfloat16, except KIVI, which runs in float16, the only precision its released kernel accepts; on LongBench, Transformer-FSQ and the uncompressed model run in bfloat16 and the other methods in float32; Appendix § 11.§ 5 compares the two precisions on LongBench for the uncompressed model and Transformer-FSQ (equivalent in 5 of 6 (model, method) groups, different in 0, inconclusive in 1). Reported RULER points are averaged up to three checkpoint seeds covering all items (one on Qwen3-8B-Base), with 95% bootstrap confidence intervals that resample seeds and then stratified by task; pairwise leads resample identical seeds and items. This protocol isolates compression degradation from prompt sampling noise under a unified evaluation harness.

Decode throughput. Decode throughput is the tokens generated per second by a full-model decode step (every layer, MLP and the output head; no sampling) divided by that of native SPDA decoding (PyTorch’s scaled_dot_product_attention) on the uncompressed model, timed in the same process with the same batch and context. We time every combination of batch size (1, 4, 16 and 64) and 8 contexts from 2,048 to 150,000 tokens and report the median ratio over these configurations with a 95% bootstrap interval. Each of the 4 GPUs of a host runs one process pinned to its own CPU core, and a calibration stage before every session admits concurrent timing only if native SDPA alone and beside three loaded GPUs agree within [0.97,1.03][0.97, 1.03] (95% interval). KIVI, VecInfer and Palu use their authors’ released kernels (PyramidKV and TransMLA have none; Appendix § 12.§ 5): KIVI’s matrix–vector kernels for keys and values, VecInfer’s fused kernel and Palu’s reconstruction kernel, with Palu’s key dequantisation, which has no released fused kernel, timed as a separate stage; KIVI and VecInfer run in float16 sessions against a float16 native-SDPA reference, so no ratio divides across precisions.

Results

Our experiments address five central questions, explored in detail below:

  • RQ1: How does Transformer-FSQ fare against baselines across compression regimes?

  • RQ2: Do synthetic retrieval gains transfer to natural multi-page document summarisation?

  • RQ3: Does low-bit latent geometry matter under extreme compression?

  • RQ4: What is the runtime cost in decode throughput?

  • RQ5: Does Transformer-FSQ work on a weight-quantised backbone?

RQ1: Accuracy across compression regimes. Across compression regimes, training-free and low-rank methods excel at moderate compression (≤5×\le5\times), while Transformer-FSQ is the most accurate from 12×12\times to 97.5×97.5\times, well into extreme compression (>16×> 16\times). At ≤5×\le5\times, VecInfer retains at least 93% of uncompressed RULER scores across all models (93%, 97%, 96%; Figure 2). Beyond 12×12\times, continuous latents and eviction collapse: Palu drops to 15% at 15.1×15.1\times, while Transformer-FSQ outclasses token eviction by 0.12 at 12×12\times and 0.10 at 32×32\times on Qwen3-0.6B-Base, reaching 0.49 on Qwen3-8B-Base (a lead of 0.25; Table 11). Discrete FSQ latents thus establish the retrieval-accuracy–cache-size Pareto frontier from 12×12\times to 97.5×97.5\times.

Methodseedsniah-single-1niah-single-2niah-single-3niah-multikey-1niah-multikey-2niah-multikey-3niah-multivalueniah-multiqueryvtcwefweqa-1qa-2Mean
Uncompressed11.000.981.001.000.961.000.990.990.900.920.900.900.440.92
Transformer-FSQ (8.0×)11.000.960.960.820.240.040.840.780.600.700.850.620.220.66
Transformer-FSQ (12.0×)11.001.001.000.660.140.020.790.770.580.580.730.520.200.61
Transformer-FSQ (32.0×)11.001.000.900.620.040.000.560.520.400.460.380.360.180.49
Transformer-FSQ (97.5×)10.960.740.300.200.000.000.170.140.000.070.150.140.140.23
Transformer-FSQ (409.6×)10.000.000.000.000.000.000.000.000.000.000.000.060.000.00
FSQ + window (28.4×)11.001.000.900.600.040.000.590.560.580.520.470.400.220.53
FSQ + window (87.1×)10.960.780.240.280.000.000.200.150.000.070.130.240.100.24
FSQ + window (372.4×)10.000.000.000.000.000.000.000.000.000.000.000.060.000.00
Palu (10.2×)10.980.620.140.300.000.000.160.170.290.400.430.240.100.30
Palu (15.1×)10.000.000.000.000.000.000.000.000.000.000.000.060.000.00
Palu (36.6×)10.000.000.000.000.000.000.000.000.000.000.000.080.000.01
Palu (102.4×)10.000.000.000.000.000.000.000.000.000.000.000.040.000.00
Palu (512.0×)10.000.000.000.000.000.000.000.000.000.000.000.020.000.00
TransMLA (8.0×)10.060.000.000.000.000.000.000.000.000.110.110.040.000.02
TransMLA (12.0×)10.000.000.000.000.000.000.000.000.000.000.030.040.000.01
TransMLA (32.0×)10.000.000.000.000.000.000.000.000.000.000.000.000.000.00
TransMLA (97.5×)10.000.000.000.000.000.000.000.000.000.000.000.020.000.00
TransMLA (409.6×)10.000.000.000.000.000.000.000.000.000.000.000.000.000.00
KIVI (4.9×)11.000.980.980.940.520.720.990.930.810.920.870.740.300.82
PyramidKV (4.0×)11.001.000.380.980.840.000.990.970.830.500.740.880.440.74
PyramidKV (12.0×)11.000.960.001.000.420.000.560.600.870.170.610.840.420.57
PyramidKV (32.0×)10.960.420.000.300.080.000.020.010.210.010.050.700.400.24
VecInfer (4.3×)11.001.001.000.960.880.980.970.980.830.920.910.800.320.89
VecInfer (12.7×)10.020.040.040.120.020.000.010.040.010.040.420.260.120.09
VecInfer (35.9×)10.020.040.040.120.020.000.010.040.010.100.550.220.080.10

Table 11. Per-task RULER on Qwen3-8B-Base (13 tasks of 50 items, 8,192 tokens, bfloat16). Finetuned methods use 0.25 tokens per parameter; KIVI, PyramidKV and VecInfer are training-free. Rows give achieved compression ratios; scores average complete seeds. Shading encodes the task score.

RQ2: Natural document summarisation. Synthetic retrieval resilience carries over to natural multi-page document summarisation. On LongBench gov_report (Figure 5), baselines track uncompressed scores (16.9 ROUGE-L on Qwen3-8B-Base) at moderate compression but collapse at higher ratios. In contrast, Transformer-FSQ scores highest from 32×32\times to 97.5×97.5\times on all three models; on Qwen3-8B-Base it scores 12.5 at 32×32\times and 11.4 at 97.5×97.5\times, where Palu at 102.4×102.4\times scores 7.0. Palu’s rise in ROUGE-L at extreme compression is likely the metric rewarding degraded text that repeats stopwords. Across all three scales (Table 12–Table 14), Transformer-FSQ, finetuned on 1,024-token sequences, keeps the highest ROUGE-L under extreme compression on far longer documents.

Figure 5. LongBench gov_report ROUGE-L against achieved compression on Qwen3-0.6B, 1.7B and 8B. Solid: finetuned; dashed: training-free; grey horizontal line: uncompressed; bands: one standard deviation over seeds.

MethodB/tokRatioseedsnnskipgov-reportnarrative-qatrec
Uncompressed––160018.213.010.0
Compression 1.4×–2.8×
Transformer-FSQ20482.03180017.1±1.35.3±1.225.0±5.0
TransMLA20482.03180012.7±0.02.7±0.616.7±5.8
VecInfer1601§2.6160018.18.010.0
Compression 2.8×–5.7×
Transformer-FSQ10244.03180017.4±0.96.0±1.121.7±5.8
Palu10403.93180017.5±0.410.7±2.516.7±2.9
Palu7845.23180016.0±1.510.5±0.520.0±5.0
TransMLA10244.03180012.9±0.12.7±0.28.3±5.8
KIVI832§4.9160015.17.835.0
Compression 5.7×–11.3×
Transformer-FSQ5128.03180016.7±1.15.7±1.123.3±7.6
FSQ-small (2048 B)6826.0160014.46.630.0
FSQ-small (1024 B)6826.03180014.9±0.56.7±0.430.0±5.0
FSQ-small5128.03180014.0±1.14.1±0.635.0±8.7
Palu40010.23180014.5±0.36.1±0.55.0±5.0
TransMLA5128.0318009.3±1.01.6±0.410.0±8.7
PyramidKV509§8.0160012.710.410.0
VecInfer705§5.8160013.12.915.0
Compression 11.3×–22.6×
Transformer-FSQ25616.03180014.2±0.43.9±1.031.7±7.6
FSQ-small25616.03180013.7±0.54.5±0.531.7±5.8
Palu20819.73180012.1±0.42.2±0.515.0±5.0
TransMLA25616.0318008.5±0.71.5±0.66.7±7.6
PyramidKV238§17.2160011.39.415.0
PyramidKV@8k226§18.1160011.214.315.0
VecInfer322§12.716006.12.95.0
VecInfer@8k225§18.216006.10.70.0
Compression 22.6×–45.3×
Transformer-FSQ12832.03180011.7±0.75.3±1.516.7±7.6
FSQ-small12832.03180011.3±1.14.5±1.318.3±5.8
TransMLA12832.0318003.6±0.31.1±1.10.0
PyramidKV103§39.616008.28.420.0
PyramidKV@8k96§42.516008.48.820.0
VecInfer@8k113§36.116003.90.415.0
Compression 90.5×–181.0×
Transformer-FSQ4297.5318006.8±0.63.9±0.53.3±5.8
FSQ-small4297.5318007.9±1.04.0±1.05.0±5.0
Palu40102.4318000.3±0.51.6±0.70.0
TransMLA4297.5318002.3±1.20.7±0.60.0
Compression 362.0×–724.1×
Transformer-FSQ10409.6318001.4±0.22.2±1.40.0
FSQ-small10409.6318001.4±0.52.3±0.40.0
Palu8512.0318007.3±1.51.5±0.60.0
TransMLA10409.6318000.9±0.82.5±2.00.0

Table 12. Per-task LongBench on Qwen3-0.6B-Base, each task in its own metric (over-length prompts truncated), over the 20 documents per task that every configuration scored. Finetuned methods use 0.25 tokens per parameter; KIVI, PyramidKV and VecInfer are training-free. Transformer-FSQ and uncompressed rows run in bfloat16, all other rows in float32.

Table 14: Per-task LongBench on Qwen3-8B-Base, each task in its own metric (over-length prompts truncated), over the 180 gov_report, 70 narrativeqa and 200 trec documents that every configuration scored. Finetuned methods use 0.25 tokens per parameter; KIVI, PyramidKV and VecInfer are training-free. Transformer-FSQ and uncompressed rows run in bfloat16, all other rows in float32.

MethodB/tokRatioseedsnnskipgov_reportnarrativeqatrec
Uncompressed––146213816.917.463.0
Compression 2.8×–5.7×
KIVI847{sup}§`4.8145015015.423.558.0
Compression 5.7×–11.3×
Transformer-FSQ5128.01600015.010.843.5
VecInfer721{sup}§`5.7145015015.616.359.0
Compression 11.3×–22.6×
Transformer-FSQ34112.01600014.510.146.0
PyramidKV287{sup}§`14.3147812210.518.860.5
Compression 22.6×–45.3×
Transformer-FSQ12832.01600012.58.538.0
Palu11236.614621388.03.02.0
VecInfer145{sup}§`28.214621389.27.420.0
Compression 45.3×–90.5×
PyramidKV56{sup}§`73.114801205.411.527.5
Compression 90.5×–181.0×
Transformer-FSQ4297.51600011.46.53.5
Palu40102.414621387.02.22.0
Compression 362.0×–724.1×
Transformer-FSQ10409.6160008.84.73.5
Palu8512.014621383.62.20.0

Table 14. Notes: ±\pm: one sample standard deviation over seeds. B/tok: cache bytes per token per layer; ratio: uncompressed size divided over B/tok; n/skip: examples the row’s own runs scored/did not score (prefill out of memory), over seeds and tasks; task columns use only the caption’s documents. Task columns: raw scores (0–100), averaged over documents, then seeds; never averaged across tasks. §: B/tok, ratio depend on tokens held (KIVI, PyramidKV, VecInfer); averaged over the row’s scored examples.

RQ3: Impact of latent geometry. Learning to fit a low-bit grid provides the strongest retrieval–memory operating point under extreme compression (Table 2). Table 2 compares bottlenecks at fixed budgets, including quantisation metadata; we discuss four of them at 80 B per token per layer (51× compression). All keep the pretrained Qwen backbone frozen and finetune only the inserted bottleneck parameters. Transformer-FSQ applies its grid throughout quantisation-aware finetuning (QAFT); post-hoc FSQ instead trains a continuous latent and applies the grid only at inference.

MethodTrainingCacheWidthB/tokRULER
Transformer-FSQ (ours)Discrete (STE, L=16L=16)4-bit implicit grid2881440.232 ± 0.022
Continuous latentContinuous (bf16)16-bit float721440.009 ± 0.008
Latent + post-hoc FSQContinuous (bf16)4-bit post-hoc FSQ2881440.175 ± 0.030
Latent + post-hoc KIVIContinuous (bf16)4-bit blocks + scales2881800.206 ± 0.021
PaluLow-rank SVD4-bit SVD + scales1441440.000 ± 0.000
TransMLALatent attention4-bit + 16-bit RoPE641440.001 ± 0.001
Transformer-FSQ (ours)Discrete (STE, L=16L=16)4-bit implicit grid160800.177 ± 0.017
Continuous latentContinuous (bf16)16-bit float40800.001 ± 0.001
Latent + post-hoc FSQContinuous (bf16)4-bit post-hoc FSQ160800.136 ± 0.014
Latent + post-hoc KIVIContinuous (bf16)4-bit blocks + scales1601000.168 ± 0.014
PaluLow-rank SVD4-bit SVD + scales80800.000 ± 0.000
TransMLALatent attention4-bit + 16-bit RoPE32760.000 ± 0.000

Table 2. Discreteness at matched byte budgets: RULER at 8,192 tokens on Qwen3-0.6B-Base (three seeds, 0.25 tokens per parameter) at nominal 144 and 80 B per token per layer; mean ± 95% bootstrap half-width. The uncompressed model (4,096 B) scores 0.718 ± 0.023. Continuous latents are 4× narrower to fit the budget; post-hoc variants quantise a wide continuous latent at evaluation. KIVI’s per-block scales add 36 and 20 B; TransMLA spends 32 B on rotary keys. Training through the 4-bit grid (STE, L=16L=16) scores highest at both budgets.

At the same width of 160 and 4-bit precision (scores: mean ± 95% bootstrap half-width), QAFT-FSQ achieves 0.177 ± 0.017 RULER, compared with 0.136 ± 0.014 for post-hoc FSQ. This comparison directly supports learning representations that accommodate the inference-time grid. Conventional min–max 4-bit QAFT achieves 0.151 ± 0.021 with width 128 (Table 7): its scale metadata leaves fewer latent coordinates within the same budget. Pure continuous rank compression, stored at 16-bit precision, permits only width 40 and achieves 0.001 ± 0.001.

Compression / quantisationWidthBytesRULER
Nominal budget: 144 B/token/layer (28.4×)
Continuous latent721440.009 ± 0.008
Latent + post-hoc FSQ2881440.175 ± 0.030
Transformer-FSQ (ours)2881440.232 ± 0.022
Latent + trained min–max 4-bit2241400.187 ± 0.020
Latent + post-hoc KIVI2881800.206 ± 0.021
Palu, 4-bit1441440.000 ± 0.000
TransMLA, 4-bit641440.001 ± 0.001
Nominal budget: 80 B/token/layer (51.2×)
Continuous latent40800.001 ± 0.001
Latent + post-hoc FSQ160800.136 ± 0.014
Transformer-FSQ (ours)160800.177 ± 0.017
Latent + trained min–max 4-bit128800.151 ± 0.021
Latent + post-hoc KIVI1601000.168 ± 0.014
Palu, 4-bit80800.000 ± 0.000
TransMLA, 4-bit32760.000 ± 0.000

Table 7. Cache width, precision and when quantisation is applied. RULER at 8,192 tokens on Qwen3-0.6B-Base (three seeds, 0.25 tokens per parameter); the uncompressed cache (4,096 B per token per layer) scores 0.718 ± 0.023. Scores: mean ± 95% bootstrap half-width; bold: best. Bytes: actual storage per token per layer, metadata included; Width: bottleneck dimension. Trained FSQ rounds during training (straight-through estimator); post-hoc FSQ and KIVI quantise a continuously trained latent only at evaluation. KIVI adds fp16 scales and offsets (25% over budget); Palu stores 4-bit SVD codes with scales; TransMLA stores 4-bit content codes plus 16-bit rotary keys.

Together, these results favour a wider low-bit latent trained with its grid over either inference-only quantisation or a narrow continuous bottleneck. The min–max comparison confounds geometry with width, so it only suggests that FSQ’s metadata-free grid is an effective budget-aware design. Appendix § 13.§ 3 further examines the trade-off between latent width and per-coordinate precision at a fixed cache budget.

RQ4: Decode throughput across compression regimes. Transformer-FSQ matches native SDPA decoding of the uncompressed model at 32× and exceeds it at extreme compression. Benchmarking on NVIDIA L40S GPUs (Figure 6) shows median throughput relative to native SDPA of 1.38×, 1.34× and 1.01× at 32× on Qwen3-0.6B, 1.7B and 8B, where the saved cache reads and the per-step key reconstruction approximately offset each other. At extreme compression, memory bandwidth savings dominate arithmetic: 1.98×, 2.20× and 1.63× at 409.6×, and up to 5.86× on Qwen3-0.6B when the batch holds at least 100,000 tokens. At matched cache bytes, Transformer-FSQ decodes faster than Palu in 88–100% of batch-size and context configurations, but slower than KIVI’s released kernels in most: it is faster in only 15% of configurations on Qwen3-0.6B and Qwen3-1.7B. At batch 16, native SDPA runs out of memory by 24,555 tokens on Qwen3-0.6B and Qwen3-1.7B, while Transformer-FSQ decodes to 150,000, at least 6.1× further. Per-step key reconstruction is thus outweighed once bandwidth limits dominate.

Figure 6. Decode throughput against context length on NVIDIA L40S in bfloat16; rows: model, columns: batch size, both axes logarithmic. Grey dashed: the uncompressed model with native SDPA, timed in the same process. Orange: Transformer-FSQ at 32×32 \times, 97.5×97.5 \times and 409.6×409.6 \times, darker for deeper compression; a line that ends early ran out of memory. Protocol: Appendix § 12.§ 5.

RQ5: Weight-quantised backbone. Transformer-FSQ remains effective when paired with a weight-quantised backbone. To evaluate compatibility under compound compression, we quantise all 252 decoder linear layers of Qwen3-8B-Base to 4-bit NormalFloat (NF4) (Dettmers et al., [10]) in groups of 64, keeping embeddings and the output head in 16-bit precision, and evaluate long-context retrieval on RULER at 8,192 tokens. The uncompressed model with 4-bit NF4 weights achieves a RULER score of 0.913 (within 0.010 of its 16-bit baseline of 0.922). For 32× Transformer-FSQ (128 B per token per layer), applying NF4 after finetuning on 16-bit weights drops the RULER score from 0.493 to 0.382 (a loss of 0.111). However, when finetuned on the frozen NF4 backbone, Transformer-FSQ recovers to 0.464 (95% bootstrap CI [0.444, 0.485]), recovering 74% of the loss from applying NF4 after finetuning and trailing the same model on the 16-bit backbone by only 0.029. Directly training the discrete KV bottleneck on top of frozen 4-bit weights confirms that Transformer-FSQ adapts to weight-quantised backbones, recovering long-context retrieval to within 0.029 of the same Transformer-FSQ model on the 16-bit backbone under joint 4-bit weight and 32×32 \times KV-cache compression.

Ablations and stacking with eviction

Appendix E evaluates architectural variants and training dynamics: nonlinear encoders and per-head codes degrade validation loss, MLA-FSQ (Transformer-FSQ codes plus the separate half-precision rotary key of multi-head latent attention) imposes compression ceilings, recent-token windows do not shift Pareto frontiers, and lossless entropy coding saves at most about 13% of cache bytes at a large decode-latency cost (§E.6). Because Transformer-FSQ shrinks per-token cache bytes while PyramidKV retains fewer tokens, the two compose at inference (Appendix § 12.§ 3). At a RULER threshold of 0.30, stacking them on Qwen3-0.6B-Base yields an aggregate projected cache capacity of 3.8M tokens across concurrent sequences on a 16 GiB GPU, 1.7× the furthest baseline, our head-shared PyramidKV (95% interval 1.0–2.3; capped by peak prefill memory). These projections extrapolate 8,192-token scores, extending memory capacity without establishing million-token retrieval accuracy.

Limitations

High-dimensional product grids operate beyond initial low-dimensional validations. While Mentzer et al. [45] recommend Li≥5L_i \ge5 levels for dimensions d≤6d \le6 (implicit codebooks ≤216\le2^{16}), we deploy L=16L = 16 across latent widths from 16 to 4,096. At wide settings, rounding thousands of coordinates independently acts as product scalar quantisation on a learned basis rather than joint vector quantisation, forgoing multidimensional cell shaping. Every configuration except FSQ-small (Transformer-FSQ with a capped code width and 8-bit levels, used only in the LongBench tables; Appendix § 11.§ 7) uses these 16 levels, compared against 4 and 256 only at 28× and 51× (Appendix § 13.§ 3); at moderate compression, where existing methods remain stronger, fewer coordinates with finer levels are untested.

Cache memory is verified analytically at conversion. Because HuggingFace cache objects maintain padded allocations to preserve attention masks, we verify physical footprint by counting stored bytes per token at conversion, rejecting deviations from target budgets. Measured GPU memory directly confirms this analytical model (Appendix § 12.§ 6): on Qwen3-0.6B-Base, after a 7,855-token prompt, the physical buffer equals the analytic size, tokens × 28 layers × 128 B (28,152,320 bytes).

Our long-context evaluation measures retrieval, tracking and question answering on RULER at one context length, 8,192 tokens, and three LongBench tasks on documents of up to 32,768 tokens; it does not test arithmetic reasoning, code generation or many-shot in-context learning under compression, where prior work reports sharper degradation than on retrieval-style tasks. A method’s score here does not show that it preserves multi-step reasoning under compression, which we leave to future measurement.

Conclusion and future work

Transformer-FSQ caches a quantised code of each token’s normalised residual state in place of its keys and values. A learned finite scalar quantisation bottleneck replaces separate KV vectors with a shared, packed integer code, requiring no per-token quantisation metadata. Closed-form conversion and finetuning adapt pretrained models while leaving the backbone frozen. Across Qwen3 models from 0.6B to 8B parameters, Transformer-FSQ achieves the highest measured RULER accuracy from 12× to 97.5× compression and the highest gov_report ROUGE-L from 32× to 97.5×. In our experiments on Qwen3-0.6B-Base, many learned low-bit coordinates outperform a few high-precision ones at the same tight memory budget.

Three directions follow from our findings.

Allocation. Every main configuration in our work gives every layer the same code of 16-level coordinates. Varying code width and levels by layer, as PyramidKV varies kept tokens (Cai et al., [6]), sharing projections across layers, and using fewer, finer coordinates at moderate (2×–4×) ratios, where existing methods remain stronger (Appendix § 13.§ 3), could each recover accuracy at the same cache size.

Training and evaluation. Distilling each block from the uncompressed model rather than training on the language-modelling loss alone, and low-rank adapters (LoRA) on the frozen backbone, may improve recovery. Evaluation beyond retrieval at 8,192 tokens, on reasoning, code generation, many-shot learning and longer contexts, would test whether the gains extend to tasks where retaining isolated facts is not enough.

Dynamic and hierarchical compression. A cache could compress lightly while the sequence is short and more as it grows. A stored code can be coarsened in place by rounding it to fewer levels, whereas a coarse code cannot be refined, since rounding discards the detail; whether a richer stored representation or recomputation from the activation could support refinement is untested.

Attention does not need full-precision keys and values in its cache: a learned discrete state per token keeps working at compression ratios where the methods we tested fail.

Statements

AI Use Statement

In this work, we used generative AI tools, large-language-model coding and writing assistants, for two tasks that require disclosure: designing the experiment that stacks Transformer-FSQ with PyramidKV token eviction (Appendix § 12.§ 3), and implementing the fused decode kernels, part of the method itself. We have not used generative AI tools to generate synthetic datasets, develop theoretical models, formulate or prove mathematical claims, propose hypotheses, design the other experiments, implement the other parts of the methods, translate, clean or reformat datasets, analyse qualitative data or interpret results; the synthetic RULER items come from RULER’s own generator. Additionally, we used generative AI tools for tasks with recommended disclosure: creating and editing the evaluation harness, benchmarking and plotting code (distinct from the compression methods’ own implementation, disclosed above), creating the scientific figures and the scripts that draw them, editing the paper for readability, and formatting references.

We have reviewed all AI-assisted work. Code produced with AI assistance is covered by an automated test suite, and every reference was checked against its publisher’s or the preprint server’s record for title, authors, venue and year. We take responsibility for the final content of this work, including text, claims or artefacts produced with the aid of generative AI.

Reproducibility Statement

The reported tables and figures were generated by scripts from the result files rather than typed by hand. Appendix § 11.§ 1 lists the three released models this paper evaluates, how each is loaded and at what precision; Appendix § 11.§ 3 gives the closed-form conversion that inserts Transformer-FSQ’s encoder and decoder, including the calibration sample and every constant in Eq. (2) (Appendix § 14.§ 1); Appendix § 11.§ 2 gives the optimiser, learning rate, schedule, data, sequence length, batch size and precision behind every finetuned configuration. Appendix § 11.§ 7 gives each baseline’s deviation from its published specification, at the bit width, window, sink-token count, rank allocation and codebook grouping we run it at. Every cache size in every table and figure comes from the one byte-budget procedure of § 4.4 and Table 1, so methods are compared under a uniform byte-pricing model accounting for all cache tensors and metadata.

Appendix § 11.§ 5 gives the RULER and LongBench protocols: task lists, item counts, the fixed item seed, per-task generation lengths, scoring rules and evaluation precision; Appendix § 15 tabulates every per-task score behind the figures, and Appendix § 12 gives the Qwen3-8B-Base campaign’s narrower coverage. Appendix § 11.§ 6 gives the GPU-hours by model and the software versions every run used.

Code for Transformer-FSQ, covering conversion, finetuning, the fused decode kernel, and RULER and LongBench evaluation, is available at https://github.com/ASzecsenyi/transformer-fsq.

Acknowledgments

This work was supported by PhD studentships within the Centre for Doctoral Training (CDT) in Machine Learning Systems, School of Informatics, University of Edinburgh, funded by the UK Engineering and Physical Sciences Research Council (EPSRC) under grant EP/Y03516X/1. The project also received funding from the European Union’s Horizon Europe Research and Innovation programme under grant agreement No. 101120726 and from UK Research and Innovation (UKRI) through the UK government’s Horizon Europe funding Guarantee (10085198). The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) ([43]) under award 0261-4485-3326-1. Isambard-AI is operated by the University of Bristol and funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation and the Science and Technology Facilities Council [ST/AIRR/I-A-1/1023].

References

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024.

Future directions

Interpretable and recurrent latents. Discrete codes could be inspected as features [5], edited to steer generation [56], or aggregated recurrently in place of attention [23], giving a state that does not grow with context.

Extended background and related work

Attention memory and sequence-length scaling

Under standard multi-head attention [58], computing the output for each query token requires inner products against every preceding key, followed by a weighted aggregation over every preceding value. Consequently, decoding step tt must access t−1t - 1 cached key-value pairs per layer. For long sequences and modern LLM serving, this per-token cache storage quickly surpasses the static memory required for model weights.

FlashAttention and compute optimisations. FlashAttention [8] restructures attention computation using tiling and online softmax rescaling to minimise memory bandwidth between high-bandwidth memory (HBM) and on-chip SRAM. While FlashAttention speeds up prefill and generation by not materialising the intermediate T×TT \times T attention score matrix, it leaves the persistent key-value cache unchanged: each key and value vector must still reside in HBM for subsequent autoregressive decoding steps.

Linear attention and recurrent state-space models. A parallel line of research replaces the growing per-token KV cache with a fixed-size recurrent hidden state. Linear attention methods [23, 36] use kernel feature maps to enable efficient associative computation, while linear RNNs and state-space models (SSMs), including Mamba [15], Gated DeltaNet [64], and Bayesian filtering-based models [51], compress sequence history into fixed-size states. Their effectiveness depends on the expressivity of their update rules and underlying online learning objectives [38]. Recent architectures such as Echo [52] combine recurrent mechanisms to improve associative recall, while hybrid models such as Nemotron-H [4] and Kimi K3 [25] interleave recurrent mixers with full-attention layers.

However, fixed-size recurrent states do not preserve individually addressable representations of arbitrary past tokens, while hybrids with full attention still require a growing KV cache. These approaches also typically require training alternative architectures rather than directly compressing the cache of pretrained Transformers. We therefore focus exclusively on techniques that retain a compressed per-token cache in pretrained standard Transformers.

Extended KV-cache taxonomy

Methods that compress the per-token KV cache operate along different dimensions: along the sequence length (token eviction), along coordinate precision (scalar quantisation), along latent subspace projections (low-rank factorisation), and along coordinate clusters (vector quantisation).

Token eviction. Token eviction methods discard positions judged uninformative. During generation, StreamingLLM [62] observes that preserving initial “attention sinks” alongside a local sliding window prevents catastrophic perplexity spikes in autoregressive generation. H2O (Heavy Hitter Oracle) [70] dynamically tracks tokens with the highest accumulated attention weights, continually evicting positions falling outside the heavy-hitter set and local window. TOVA [47] maintains a fixed token quota by evicting the token with the lowest attention score at each decode step. For prompt compression, SnapKV [33] uses an observation window at the end of the prompt to identify consistently attended prompt tokens. ■ PyramidKV [6] extends this insight by allocating non-uniform token capacities across layers: early layers retain broader dispersed contexts while deeper layers enforce sharper pruning. However, because eviction reduces token count rather than per-token width, the resulting cache memory remains tied to prompt and generation dynamics.

Low-rank projection. Low-rank methods exploit cross-head redundancy by projecting key and value vectors into narrower continuous subspaces. Multi-head Latent Attention (MLA) [37] replaces separate per-head keys and values with a shared low-rank latent vector while using a dedicated rotary key for positional encoding. ■ TransMLA [44], MHA2MLA [20], and X-EcoMLA [29] demonstrate that standard multi-head attention (MHA) and grouped-query attention (GQA) models can be converted into MLA post-training via rank-reduction and finetuning. Palu [7] computes low-rank SVD approximations of existing projection weights, caching intermediate latent vectors that are projected back to key and value dimensions at attention time. While continuous low-rank methods reduce cache width, their representations remain unquantised floating-point vectors unless paired with subsequent quantisation stages.

Scalar quantisation. Scalar quantisation maps individual coordinates onto low-bit representations. KIVI [39] quantises keys per channel and values per token to 2 bits without requiring finetuning, while preserving a small full-precision sliding window. KVQuant [16] applies non-uniform scalar quantisation pre-RoPE with dense outlier handling. GEAR [22] compresses the cache to 2–4 bits and approximates quantisation residual error with low-rank and sparse matrices. QServe [35] integrates 4-bit KV caching alongside 4-bit weights and 8-bit activations. These methods retain original activation dimensions and rely on per-channel or per-token scaling factors and auxiliary buffers.

Vector quantisation of the KV cache. Vector quantisation (VQ) maps activation blocks to discrete centroids in a codebook. VQ-VAE [57] and Product Quantisation [19] laid the groundwork for discrete representations in continuous spaces. VQ-VAE trains through a straight-through gradient estimator [2] because the codebook lookup, a nearest-neighbour argmin, is not differentiable; the Gumbel-softmax relaxation [18, 42] is an alternative route around the same non-differentiability, and Scannell et al. [49] use a discrete codebook as the latent state of a world model for continuous control. In LLM caching, Coupled Quantisation [69] groups 2 to 8 channels into learned codebooks without finetuning, though storing 67.1M centroid parameters for LLaMA-7B beside the cache. CommVQ [32] trains additive multi-stage codebooks using 2×22 \times2 block structures that commute with RoPE. RVQ-KV [26] adapts residual vector quantisation with sequential codebook stages. VecInfer [65] combines per-channel smoothing and Walsh–Hadamard transforms with offline-fitted codebooks at 1.25 to 4 bits per coordinate. [24] decouples magnitude from direction in residual updates, achieving sub-1-bit perplexity. PQCache [68] utilises product quantisation to retrieve top-kk attention states while storing the complete cache on host memory. NOVA-KV [13] and VQKV [59] also explore codebook vector quantisation for keys and values. TurboQuant [66] applies random orthogonal rotations to concentrate coordinate distributions, followed by precomputed scalar quantisation grids.

Weight compression

Weight compression addresses the static portion of the GPU memory budget. For post-training weight quantisation, empirical scaling studies demonstrate that a larger model at lower precision consistently outperforms a smaller model at higher precision under identical memory budgets, with 4-bit weights achieving near-optimal performance across diverse model architectures (Dettmers & Zettlemoyer, 2023). Quantisation-aware pretraining pushes this frontier further, achieving viable ternary and 1.58-bit models [41]. However, while weight footprint remains constant regardless of sequence length, the KV cache scales linearly with generation context, so it dominates memory at long contexts.

Rate–distortion bounds

Theoretical bounds. Rate–distortion theory characterises the fundamental trade-off between bit-rate and reconstruction error for arbitrary quantisers. Zandieh et al. [66] prove that for any quantiser operating at bb bits per coordinate, there exists a unit vector whose expected squared reconstruction error is lower-bounded by 4−b4^{-b}. Their construction approaches within a factor of 2.7 of this lower bound. However, this theoretical bound addresses the fidelity of reconstructing a given, arbitrary input vector. Transformer-FSQ does not attempt to reproduce the continuous key and value activations of a frozen model. Instead, its linear encoder and decoder are trained end-to-end such that the representations surviving the discrete integer bottleneck optimise downstream attention task performance. This task-aware training changes what is minimised, a downstream loss rather than reconstruction error, but it does not escape the rate–distortion bound itself: at a fixed bit rate, some distortion is unavoidable under any objective, and training only chooses which directions absorb it.

Table 3: Models and their memory on a 16 GiB GPU. cfullc_{\mathrm{full}}: the uncompressed half-precision cache per token per layer (Eq. (9)), the reference for every compression ratio. The uncompressed context limits are the baseline points of Figure 11. ‡^{\ddagger}Qwen3-8B-Base runs one seed throughout (§ 5.1).

Qwen3-0.6B-BaseQwen3-1.7B-BaseQwen3-8B-Base
parameters596,049,9201,720,574,9768,190,735,360
layers282836
model width CC102420484096
query heads161632
key-value heads nkvn_{kv}888
head width dhd_h128128128
vocabulary151,936151,936151,936
sequence length (pretraining)4096, then 32,7684096, then 32,7684096, then 32,768
sequence length (finetuning)102410241024
uncompressed cache cfullc_{\mathrm{full}}4096 B4096 B4096 B
weights in half precision1.1 GiB3.2 GiB15.3 GiB
cache per token, all layers112 KiB112 KiB144 KiB
cache for 200,000 tokens21 GiB21 GiB27 GiB
as a multiple of the weights19×7×2×
tokens that fit on a 16 GiB GPU139 k120 k5 k
seeds per configuration331‡^{\ddagger}

Table 3.

Experimental details

Models

Table 3 summarises architectural specifications and uncompressed cache sizes (Eq. (9)) for the three pretrained transformers evaluated on long-context benchmarks: Qwen3-0.6B-Base, Qwen3-1.7B-Base, and Qwen3-8B-Base ([63]), loaded through the Transformers library ([61]). Models are loaded from their public Hugging Face repositories.

Finetuning

Table 4 gives the finetuning hyperparameters. All released models are finetuned using AdamW with learning rate 10−510^{-5} and constant weight decay 0.28 on ClimbMix in bfloat16, the default of our finetuning script, without method-specific tuning. Checkpoints are evaluated in bfloat16 (Appendix § 11.§ 5). A shared training recipe controls the data and optimisation budget across methods, although it may not be optimal for every method.

Each run trains on τN\tau N tokens, where τ∈{0.025,0.075,0.25}\tau\in\{0.025, 0.075, 0.25\} tokens per parameter (NN being the base model parameter count). Qwen3-0.6B-Base and Qwen3-1.7B-Base train on one GPU with micro-batches of 4 sequences of 1,024 tokens and gradient accumulation 4 (16,384 tokens per step). Qwen3-8B-Base runs exclusively at τ=0.25\tau= 0.25 across 4 GH200 GPUs under data parallelism with micro-batch 1 sequence (1,024 tokens) and gradient accumulation 8, yielding an effective batch of 8,192 tokens per GPU and a global effective batch of 32,768 tokens (32 sequences), so the total training tokens are unchanged.

Conversion

Conversion fits the inserted tensors in closed form (§ 4.2) from activations captured on a calibration sample of ClimbMix: 32,768 tokens for the released models, Qwen3-8B-Base included. The sample is read from the training shards at an offset set by the seed, without finetuning a seed changes only the calibration text for methods fitted in closed form; the RULER items are drawn once from a fixed item seed shared by every configuration, and generation is greedy. We report conversion without finetuning for Palu at its mildest ratio on Qwen3-0.6B-Base; every TransMLA configuration is finetuned.

Cache-byte accounting

KIVI is charged its 2-bit keys and values, a scale and zero point per group, and its full-precision window of the 128 most recent tokens; VecInfer its codebook indices plus the same window and 4 sink tokens (its codebook is fitted once and shared, so it is not charged per token); PyramidKV the full-precision keys and values of the kept prompt tokens and of every generated token, since it evicts only at prefill, plus a 4-byte position per kept token; Palu its quantised low-rank latent with its scales; TransMLA its latent and its separate rotary key; and Transformer-FSQ its packed code, plus the window and sink tokens in the variant that keeps them. The rotary table is not charged: it is computed from the model’s own frequencies and does not grow with the number of cached tokens; methods that cache their own rotary key, such as TransMLA, are charged for that key.

Qwen3-0.6B / 1.7B-BaseQwen3-8B-Base
finetuning budgets τ\tau0.025*, 0.075*, 0.250.25
optimiserAdamWAdamW
peak learning rate10−510^{-5}10−510^{-5}
schedulelinear warm-up over 5% of steps, cosine decay to 5% of peaklinear warm-up over 5% of steps, cosine decay to 5% of peak
weight decay0.28, constant0.28, constant
dataClimbMixClimbMix
sequence length10241024
tokens per step16,384 (4 × 1024, accumulation 4)32,768 (4 GPUs, 1 × 1024, accumulation 8)
trainable tensorsencoder, decoderencoder, decoder
backbonefrozenfrozen

Table 4. Finetuning hyperparameters. τ\tau: finetuning tokens per parameter; τ=0\tau= 0 is conversion without finetuning. All methods at a model scale share data, optimiser and budget. The released models’ results (§ 5.2) use τ=0.25\tau= 0.25; Figure 7 shows the effect of τ\tau on Qwen3-0.6B-Base. *Qwen3-0.6B-Base only.

Figure 7. RULER against finetuning tokens on Qwen3-0.6B-Base (0.025, 0.075 and 0.25 per parameter) at nominal 512 and 1,024 B per token per layer. Labels give achieved ratios (Palu stores 400 and 784B). Transformer-FSQ plateaus with fewer finetuning tokens than the baselines.

Evaluation

RULER (Hsieh et al., [17]), the suite behind every RULER number this paper reports (§ 5.1), evaluates all 13 official tasks at 8,192 tokens with 50 items per task across all three Qwen sizes, drawn once from a fixed item seed and shared by every configuration. Scoring follows RULER’s published protocol, by substring containment of the reference answers: the fraction of references the output contains on multi-answer tasks and whether it contains any reference on QA tasks, with outputs capped at official generation lengths. As detailed in § 5.1, evaluation forward passes run in bfloat16, and KIVI’s in float16.

LongBench (Bai et al., [1]) runs narrativeqa, gov_report and trec at 20 examples per task, scored by F1, ROUGE-L and classification accuracy (0 to 100). On Qwen3-8B-Base the LongBench draw is 200 documents per task, and each task is scored on the documents that every configuration completed; some methods run out of memory in prefill on the longest documents, leaving 180 gov_report, 70 narrativeqa and all 200 trec documents (Appendix § 12). Prompts exceeding 32,768 tokens are middle-truncated to 32,736 tokens. Long-context benchmarks run three seeds at 0.25 tokens per parameter, except Qwen3-8B-Base, which runs one seed throughout (§ 5.1).

Precision tests. For RULER, a configuration is one bfloat16 file paired item by item with the float32 file of the same configuration. A group is one (model, method) pair, pooling its configurations weighted by item count. The 95% bootstrap resamples configurations, then items within each. The precisions are equivalent if the interval lies inside ±0.02\pm0.02, different if it lies wholly outside that band, and inconclusive otherwise; separately, no configuration’s mean difference may exceed 0.05. LongBench applies this rule on a 0–100 scale, with margin 2.0 and per-configuration bound 8, to the uncompressed model and Transformer-FSQ per model. KIVI runs in float16, the only precision its kernel accepts, and is reported separately. NF4 rounds every decoder linear layer of Qwen3-8B-Base, inserted ones included but not the embeddings or output head, in groups of 64 at evaluation after finetuning; the hypothesis holds if the upper bound of the interval of ΔFSQ−Δdense\Delta_{\mathrm{FSQ}}-\Delta_{\mathrm{dense}}, with Δ\Delta the bfloat16 minus NF4 score, lies below 0.05. Finetuning directly on the NF4-quantised backbone recovers the RULER score to 0.464 (recovering 74% of the post-hoc gap), consistent with Transformer-FSQ adapting to 4-bit backbone weights during recovery training.

Benchmark selection. We select RULER and LongBench to probe both structured associative retrieval and realistic document comprehension. Vanilla single-needle retrieval is insufficiently discriminative, as coarse scalar quantisation or naive token retention can retrieve isolated keys against uniform text without preserving relational context. In contrast, RULER tests 13 synthetic patterns—including multi-key, multi-value, and variable tracking—directly exposing when aggressive compression destroys token relationships. LongBench complements synthetic retrieval by evaluating multi-page real documents on summarisation, narrative QA, and classification.

Unless stated otherwise, a spread is a sample standard deviation across seeds, and an interval is a 95% bootstrap interval that resamples seeds and then items stratified by task (items only for the one-seed Qwen3-8B-Base).

Compute

The reported configurations’ conversion and finetuning took 1,867 GPU-hours on NVIDIA GH200 GPUs (1 GPU per run for ≤1.7\le1.7B, 4 GPUs for 8B); Table 5 gives the average time of one run by model and finetuning budget. Over the whole project, Slurm accounting of every job on Isambard-AI, including jobs that failed or reached the time limit, records about 5,620 GH200 GPU-hours: 4,180 for all conversion and finetuning, including unreported baseline configurations, ablations and Qwen3-8B-Base runs, 1,030 for RULER and LongBench evaluation, and 410 for decode-throughput benchmarking. The final evaluation campaign used a further 235 GPU-hours on NVIDIA RTX 6000 Ada GPUs, 110 of them in failed jobs. Reported decode throughput was measured on NVIDIA L40S GPUs.

Modelτ=0.025\tau= 0.025τ=0.075\tau= 0.075τ=0.25\tau= 0.25
Qwen3-0.6B-Base0.19 n=30n=30; 0.15–0.250.56 n=27n=27; 0.44–0.782.59 n=54n=54; 1.47–5.14
Qwen3-1.7B-Base––7.32 n=63n=63; 5.57–9.19
Qwen3-8B-Base––138.34 (34.6 h) n=9n=9; 108.88–158.72

Table 5. Conversion and finetuning GPU-hours per run on NVIDIA GH200 (one GPU per run; four for Qwen3-8B). Each cell: the mean over nn finished runs of the finetuned methods (Transformer-FSQ, Palu, TransMLA; all budgets and seeds), then the min–max range; –: not run; for Qwen3-8B, mean wall-clock hours in parentheses. τ\tau: finetuning tokens per parameter.

Implementation

All models are implemented in PyTorch 2.9.1 ([48]) against CUDA 12.8 wheels, and the fused decode kernels are written in Triton 3.5.1 ([55]). The Qwen conversion and finetuning path loads models through Transformers ([61]) (4.57.6). Baselines follow published specifications: KIVI preserves its 128-token full-precision window, while VecInfer keeps 4 sink tokens following StreamLLM ([62]). PyramidKV selects tokens per head, as published, following SnapKV.

VecInfer uses shared kk-means codebooks across heads and symmetric bit allocations; its released fused kernel is used for decode timing. Transformer-FSQ keeps no full-precision window or sink tokens except in the ablation of Appendix E.

Choice of baselines. We take one published method per family of KV-cache compression (Table 1). For token eviction, PyramidKV [6] rather than SnapKV, StreamingLLM or H2O, since it also varies the budget across layers. For scalar quantisation, KIVI [39] rather than round-to-nearest, since its per-channel keys, per-token values and full-precision window handle activation outliers. For vector quantisation, VecInfer [65] rather than Coupled Quantisation, which stores tens of millions of codebook parameters; VecInfer spreads outliers with a Walsh–Hadamard transform instead. For low-rank projection, Palu [7], and for latent attention, TransMLA [44], both of which convert a pretrained model by finetuning rather than pretraining an MLA architecture [37]. TurboQuant [66] rotates every cached key and value by a normalised Walsh–Hadamard matrix and quantises it per vector on a min–max grid, with an optional one-bit QJL residual correction; we run its PyTorch reference path training-free at nominal budgets of 1,024 and 2,048 B per token per layer (one seed), and it appears only in Figure 2. FSQ-small, which appears only in the LongBench tables, is Transformer-FSQ with its code width capped where the bottleneck stops adding parameters over the key and value projections it replaces, using 8-bit levels where they fit the budget. State-space and linear-attention models keep no per-token cache and are not compared (Appendix § 10).

Additional results

Qwen3-8B-Base

Qwen3-8B-Base retrieval and question classification. Figure 2 reports RULER and Figure 5 reports gov_report (50 RULER examples per task, 180 gov_report documents, seed 42 throughout). At extreme compression, Transformer-FSQ at 409.6×409.6\times and Palu at 512×512\times reach floor retrieval scores, as on the smaller models. On trec (Figure 10; 200 documents, uncompressed score 63.0), KIVI and VecInfer stay close to uncompressed at moderate compression (58.0 at 4.7×4.7\times, 59.0 at 5.5×5.5\times) before dropping. For methods whose cache depends on the tokens held (KIVI, VecInfer, PyramidKV), ratios in this subsection are those of the named task’s documents; Table 14 averages them over a row’s tasks. Transformer-FSQ maintains 38.0 at 32×32\times and 3.5 at 97.5×97.5\times, outperforming Palu (2.0 and 2.0). TransMLA ran out of memory during long-document prefill and is omitted.

Figure 10. LongBench trec accuracy (0–100) against achieved compression. Line styles as in Figure 9.

Document-level narrative comprehension on 8B parameters. On LongBench narrativeqa (Figure 9; the 70 documents every configuration completed, the others exceeding prefill memory for at least one method), the uncompressed model scores 17.4 F1. Token eviction performs strongly on this narrative task: PyramidKV scores 18.8 at 26.6×26.6\times and 11.5 at 197.8×197.8\times. Transformer-FSQ scores 8.5 (32×32\times) and 6.5 (97.5×97.5\times), above Palu’s 3.0 and 2.2.

Figure 9. LongBench narrativeqa F1 (0–100) against achieved compression. Solid: finetuned; dashed: training-free; grey horizontal line: uncompressed. PyramidKV scores highest on 0.6B and 1.7B; on 8B, Transformer-FSQ scores above Palu.

Long context

This subsection evaluates long-context retrieval and comprehension across all three Qwen models under compression. Figure 8 presents RULER scores relative to uncompressed models, while Figure 9 and Figure 10 report LongBench narrativeqa and trec performance across models. Complete per-task scores appear in Table 9–Table 11 and Table 12–Table 14.

Figure 8. RULER relative to the uncompressed model: score at 8,192 tokens divided by the same model’s uncompressed score; bands as in Figure 2. Below about 5×5 \times, VecInfer and Palu keep the most; from 32×32 \times to 97.5×97.5 \times, Transformer-FSQ does.

Methodseedsniah-single 1niah-single 2niah-single 3niah-multikey 1niah-multikey 2niah-multikey 3niah-multivalueniah-multiqueryvtcwefweqa-1qa-2Mean
Uncompressed11.000.960.980.900.820.420.830.880.800.300.510.620.300.72
Transformer-FSQ (2.0×)31.000.920.970.660.410.030.670.530.710.340.420.470.230.57
Transformer-FSQ (4.0×)31.000.890.970.690.430.030.680.540.610.310.420.530.250.57
Transformer-FSQ (8.0×)31.000.870.970.680.330.040.690.520.590.310.450.530.270.56
Transformer-FSQ (12.0×)30.990.850.660.400.110.010.330.410.180.310.240.430.170.39
Transformer-FSQ (16.0×)30.990.690.610.260.030.000.290.270.180.220.210.310.150.32
Transformer-FSQ (32.0×)30.940.480.330.230.000.000.220.230.000.130.010.190.120.22
Transformer-FSQ (97.5×)30.250.090.000.070.000.000.030.030.000.000.000.050.020.04
Transformer-FSQ (409.6×)30.000.000.000.000.000.000.000.000.000.000.000.020.000.00
FSQ + window (1.8×)11.000.881.000.680.420.000.650.510.490.340.450.500.280.55
FSQ + window (3.5×)11.000.881.000.700.340.000.610.660.490.310.450.500.260.55
FSQ + window (7.1×)11.000.961.000.720.320.040.690.610.450.300.400.580.260.56
FSQ + window (10.6×)10.980.820.760.380.120.000.260.350.060.320.220.340.160.37
FSQ + window (14.2×)11.000.520.560.280.020.000.230.230.010.250.110.300.140.28
FSQ + window (28.4×)10.940.400.380.220.000.000.220.190.010.150.020.220.120.22
FSQ + window (87.1×)10.480.180.000.120.000.000.050.050.000.000.000.020.000.07
FSQ + window (372.4×)10.000.000.000.000.000.000.000.000.000.000.000.020.000.00
Palu (3.9×)31.001.000.990.930.820.250.800.780.790.340.480.690.340.71
Palu (5.2×)31.001.000.990.910.750.110.810.780.710.360.460.570.310.67
Palu (10.2×)30.990.870.850.570.050.000.390.340.000.050.170.320.130.36
Palu (15.1×)30.490.190.010.170.000.000.110.090.000.010.040.150.130.11
Palu (19.7×)30.000.000.000.010.000.000.000.000.000.010.000.060.040.01
Palu (102.4×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
Palu (512.0×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
TransMLA (2.0×)30.960.860.790.340.000.000.380.320.340.150.280.330.170.38
TransMLA (4.0×)31.000.710.530.310.000.000.350.260.290.140.200.280.150.32
TransMLA (8.0×)30.930.370.250.150.000.000.180.120.000.080.010.150.140.18
TransMLA (12.0×)30.910.140.040.090.000.000.090.020.000.020.050.110.080.12
TransMLA (16.0×)30.610.070.000.070.000.000.060.010.000.000.020.090.030.07
TransMLA (32.0×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
TransMLA (97.5×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
TransMLA (409.6×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
KIVI (4.9×)10.800.740.120.360.040.000.240.340.380.220.320.400.180.32
PyramidKV (4.0×)11.000.740.020.600.080.000.330.290.780.280.150.680.260.40
PyramidKV (12.0×)11.000.300.000.140.000.000.040.020.760.150.230.640.160.26
PyramidKV (32.0×)10.440.040.000.040.000.000.000.000.240.020.010.500.140.11
VecInfer (4.3×)11.000.900.960.800.740.200.750.810.770.350.530.600.260.67
VecInfer (12.7×)10.020.040.040.120.020.000.010.040.000.000.050.080.100.04
VecInfer (35.9×)10.000.040.000.040.000.000.000.010.000.000.000.040.080.02

Table 9. Per-task RULER on Qwen3-0.6B-Base (13 tasks of 50 items, 8,192 tokens, bfloat16). Finetuned methods use 0.25 tokens per parameter; KIVI, PyramidKV and VecInfer are training-free. Rows give achieved compression ratios; scores average complete seeds. Shading encodes the task score.

On Qwen3-0.6B-Base (Figure 8), Transformer-FSQ keeps 79% of the uncompressed RULER score at 4×4\times and 78% at 8×8\times (Table 9), ahead of TransMLA’s 26% at 8×8\times and Palu’s 15% at 15.1×15.1\times. Single-key retrieval (niah_single_1) survives longest at extreme compression, while multi-key retrieval with distractors and variable tracking collapse past 16×16\times. Training-free baselines drop earlier: VecInfer retains 93% at 4.3×4.3\times but falls to 6% at 12.7×12.7\times, and PyramidKV holds 56% at 4.0×4.0\times.

On Qwen3-1.7B-Base (Figure 8), the same hierarchy holds at higher retention: Transformer-FSQ keeps 67% of uncompressed RULER at 8×8\times and 34% at 32×32\times (Table 10), exceeding TransMLA (21% at 8×8\times) and Palu (94% at 5.2×5.2\times, falling to 5% at 19.7×19.7\times). VecInfer drops to 9% at 12.7×12.7\times, while PyramidKV holds 62% at 4.0×4.0\times. As on 0.6B, single-key retrieval survives longest; the ordering of methods is unchanged and every score is higher.

Methodseedsniah-single-1niah-single-2niah-single-3niah-multikey-1niah-multikey-2niah-multikey-3niah-multivalueniah-multiqueryvtcwefweqa-1qa-2Mean
Uncompressed11.001.001.000.980.900.620.970.980.620.360.680.600.320.77
Transformer-FSQ (2.0×)21.001.001.000.850.290.040.840.860.720.330.510.630.320.64
Transformer-FSQ (4.0×)31.000.990.990.860.430.040.850.860.790.330.470.620.330.66
Transformer-FSQ (8.0×)30.990.840.970.640.090.010.720.630.490.400.330.430.210.52
Transformer-FSQ (12.0×)30.990.630.930.500.050.000.560.430.480.340.180.320.190.43
Transformer-FSQ (16.0×)31.000.620.870.380.040.000.430.360.250.270.170.270.180.37
Transformer-FSQ (32.0×)30.990.540.600.260.000.000.200.220.000.150.060.200.140.26
Transformer-FSQ (97.5×)30.590.140.000.110.000.000.070.050.000.000.000.080.020.08
Transformer-FSQ (409.6×)30.000.000.000.000.000.000.000.000.000.000.000.010.000.00
FSQ + window (1.8×)31.000.991.000.840.490.010.850.880.680.320.320.610.310.64
FSQ + window (3.5×)31.000.990.990.820.470.010.840.830.650.350.350.610.320.63
FSQ + window (7.1×)31.000.841.000.630.060.000.710.620.480.350.230.450.220.51
FSQ + window (10.6×)31.000.690.980.550.070.000.570.520.480.340.160.350.200.45
FSQ + window (14.2×)31.000.650.890.370.020.000.420.360.390.260.250.270.170.39
FSQ + window (28.4×)30.990.650.580.280.000.000.220.240.000.160.050.190.130.27
FSQ + window (87.1×)30.770.150.000.110.000.000.090.060.000.000.000.090.030.10
FSQ + window (372.4×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
Palu (3.9×)31.001.001.000.940.880.200.950.950.840.370.680.670.350.76
Palu (5.2×)31.000.991.000.900.840.030.940.920.790.400.690.630.290.72
Palu (10.2×)30.990.450.300.330.010.000.380.210.570.330.310.430.160.34
Palu (19.7×)30.010.190.000.060.000.000.030.030.000.000.020.110.100.04
Palu (102.4×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
Palu (512.0×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
TransMLA (2.0×)30.960.750.720.360.000.000.340.290.020.190.010.330.190.32
TransMLA (4.0×)30.970.710.490.170.000.000.190.220.330.250.220.270.140.30
TransMLA (8.0×)30.830.410.140.090.000.000.170.070.000.110.060.150.070.16
TransMLA (16.0×)30.050.050.000.030.000.000.030.020.000.000.080.060.000.02
TransMLA (32.0×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
TransMLA (97.5×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
TransMLA (409.6×)30.000.000.000.000.000.000.000.000.000.000.000.000.000.00
KIVI (4.0×)10.960.940.700.540.140.000.600.540.440.480.610.520.260.52
PyramidKV (4.0×)11.000.940.020.700.260.000.580.510.580.330.430.600.300.48
PyramidKV (12.0×)11.000.420.000.240.000.000.090.010.520.230.410.560.280.29
PyramidKV (32.0×)10.520.120.000.020.000.000.000.000.380.030.050.440.200.14
VecInfer (4.3×)11.000.981.001.000.720.560.940.950.540.410.690.600.320.75
VecInfer (12.7×)10.020.040.040.120.020.000.010.040.000.010.310.180.080.07
VecInfer (35.9×)10.020.020.000.040.000.000.000.030.000.020.010.020.040.01

Table 10. Per-task RULER on Qwen3-1.7B-Base (13 tasks of 50 items, 8,192 tokens, bfloat16). Finetuned methods use 0.25 tokens per parameter; KIVI, PyramidKV and VecInfer are training-free. Rows give achieved compression ratios; scores average complete seeds. Shading encodes the task score. Notes: seeds states how many complete finetuning seeds the score above averages; where seeds >1> 1, the score is the mean of that many complete seeds’ own per-task means. ±\pm is omitted for compactness. Ratio: uncompressed bytes per token per layer over the cache actually held (§4.4), not the nominal budget. FSQ + window: Transformer-FSQ that also keeps the newest 128 tokens as an unrounded latent (Appendix E), whose bytes the ratio includes. Omitted: PyramidKV and VecInfer at 42 and 10B, where no configuration of the method fits the budget.

Document comprehension and question classification reveal complementary strengths between uniform compression and token eviction (Figure 9 and Figure 10). On LongBench narrativeqa, token eviction (PyramidKV) stays within a few points of the uncompressed model across budgets (8.4–14.3 F1 on 0.6B, 12.1–18.5 on 1.7B), outperforming uniform compression on long narrative tracking where broad context can be pruned. Conversely, on trec classification and gov_report summarisation (Figure 10 and Figure 5), Transformer-FSQ scores above Palu and TransMLA from 32×32\times to 97.5×97.5\times, where both collapse.

On Qwen3-8B-Base (Table 11), Transformer-FSQ keeps 54% of RULER at 32×32 \times and 25% at 97.5×97.5 \times, with single-key retrieval again surviving longest.

Combining Transformer-FSQ with eviction

Stacking FSQ with token eviction on Qwen3-0.6B-Base. We applied PyramidKV at inference to finetuned Transformer-FSQ checkpoints without retraining, using our own variant of PyramidKV rather than the published method: it keeps one set of prompt tokens per layer, shared by all its heads, where published PyramidKV selects tokens per head, so that the kept tokens line up with Transformer-FSQ’s codes, which every head shares. This variant scores 0.13–0.24 higher on RULER than the published per-head method; it is the PyramidKV of this subsection, of Figure 11 and of Table 6, and every other PyramidKV result in this paper is the published method. We evaluated RULER (8,192 tokens) across combinations where FSQ carries at least half the log-compression (Table 6). Stacked ratios charge PyramidKV’s 4-byte position per kept prompt token and the generated tokens at the full code width, so they are not the product of the two standalone ratios. Stacking FSQ 8×8\times with PyramidKV 2×2\times achieves 16×16\times total compression (256 B/token/layer) and scores 0.39, outperforming PyramidKV alone at the matched budget (0.30 at 255.75 B). Stacking FSQ 16×16\times with PyramidKV 2×2\times achieves 32×32\times total compression (128 B/token/layer) and scores 0.22, outperforming matched PyramidKV alone (0.10 at 127.75 B). At 64 B/token/layer (64×64\times, FSQ 16×16\times with PyramidKV 4×4\times), the stack maintains 0.22, whereas PyramidKV alone cannot reach 64 B at 8,192 tokens, below its minimum window.

Figure 11. Tokens held on a 16 GiB GPU against RULER at 8,192 tokens. Crosses: out of memory. The stacked line includes only stacks in which Transformer-FSQ supplies at least half the compression in log terms. At matched RULER thresholds, Transformer-FSQ and its eviction stacks hold up to 1.7× more tokens than the baselines.

Table 6: Stack-grid results at 8,192 tokens on Qwen3-0.6B-Base (RULER, 13 tasks of 50 items; Appendix § 12.§ 3). Every stack that passes the rule fixed before the runs is listed, against Transformer-FSQ alone, the uncompressed model, and PyramidKV alone at the same achieved total budget (own block below). Ratios and B/token/layer are achieved, not nominal. Score: mean over complete seeds ± item-bootstrap half-width (Appendix § 11.§ 5); seeds: complete finetuning seeds.

FSQPyramidKVTotal ratioB/tok/layerScore ± half-CISeeds
––1.0×40960.72 ± 0.021
8×–8.0×5120.56 ± 0.033
16×–16.0×2560.32 ± 0.033
32×–32.0×1280.22 ± 0.013
97.5×–97.5×420.04 ± 0.023
8×2×16.0×2560.39 ± 0.021
8×4×32.0×1280.32 ± 0.021
8×8×64.0×640.28 ± 0.021
16×2×32.0×1280.22 ± 0.021
16×4×64.0×640.22 ± 0.021
16×8×128.1×320.20 ± 0.021
16×16×256.3×160.16 ± 0.021
32×2×64.0×640.17 ± 0.021
32×4×128.0×320.15 ± 0.021
32×8×256.1×160.13 ± 0.021
32×16×512.5×8.00.08 ± 0.021
32×32×1026.0×4.00.01 ± 0.011
97.5×2×195.1×210.06 ± 0.021
97.5×4×390.2×100.04 ± 0.011
97.5×8×780.6×5.20.03 ± 0.011
97.5×16×1562.1×2.60.01 ± 0.011
97.5×32×3129.1×1.30.00 ± 0.011
PyramidKV (shared, our variant; Appendix C.7) alone, at a stack cell’s own achieved total budget
–matched32.1×127.750.101
–matched16.0×255.750.301

Table 6.

Diminishing returns at extreme stacking ratios. Returns diminish as Transformer-FSQ’s own compression grows (FSQ 32×32\times stacked with PyramidKV 32×32\times drops to 0.01). On Qwen3-1.7B-Base (three seeds), the 8×+2×8\times+2\times stack scores 0.39±0.020.39\pm0.02 against 0.32 for matched PyramidKV alone, and single-seed stacks on Qwen3-8B-Base follow the same trend. Figure 11 summarises the resulting capability frontier on a 16 GiB GPU.

The decode kernel

A compressed cache saves time only if attention reads it without first expanding it in memory. We implement the decoding step of Figure 4 as a fused Triton kernel (Tillet et al. [55]), fsq_rope_decode_mma. Figure 13 compares its data flow with that of native SDPA decoding on the uncompressed cache. For each key-value head, the kernel splits the cached positions into blocks and, for each block,

One decoding step of one layer: native SDPA and Transformer-FSQ fused decode

Figure 13. One decoding step of one layer: native SDPA (left) and the Transformer-FSQ kernel (right). Top band: GPU memory (HBM); everything below runs on chip. Native SDPA streams 4096 B of 16-bit keys and values per cached token. At 32×, the kernel streams 128 B of packed codes per token, reconstructs keys on the tensor cores and accumulates attention-weighted codes; WvW_v is applied once, after the blocks are merged.

  1. reads the packed codes from GPU memory and unpacks them into registers;

  2. reconstructs the keys WkztW_k z_t with a matrix multiplication on the tensor cores;

  3. applies the key normalisation and the rotary embedding in registers, using a rotary table in the activation precision;

  4. computes the scores of the query heads that share the key-value head, and updates a running softmax (elementwise arithmetic, not tensor cores);

  5. adds the attention-weighted codes atzta_t z_t to an accumulator, as Equation (6) allows.

Blocks of positions run in parallel, each producing a partial accumulator and softmax statistics. A second, small kernel merges the partials and applies WvW_v once; it rounds to the activation precision only at its output. Neither kernel writes a reconstructed key or value to GPU memory. Per cached token, the cache therefore stores cFSQc_{\mathrm{FSQ}} bytes whereas the uncompressed cache stores cfullc_{\mathrm{full}}, in exchange for the key reconstruction in step 2. Every key-value head’s program streams the same shared codes, so the bytes issued per token exceed cFSQc_{\mathrm{FSQ}} unless repeated reads hit in the L2 cache; the dataflow figure shows the stored representation, not measured traffic.

Figure 12. Mean RULER at 8,192 tokens for each Transformer-FSQ ×\times PyramidKV stack on Qwen3-0.6B-Base. Grey cells fail the rule, fixed before the runs, that Transformer-FSQ supplies at least half the compression in log terms; values in Table 6. Retrieval degrades gradually up to 32× total and falls off at extreme stacked ratios.

Kernel checks. Every timed or scored run declares the attention kernel its method should use, and an observation inside each attention call records the kernel that executed. Throughput runs abort on any mismatch, for example when a kernel rejects its inputs and a slower reference path runs instead; such rows are discarded. KIVI keeps its most recent tokens in full precision; on steps where its quantised part is still empty, no kernel is expected, and the log records this. Quality runs record the kernel without aborting; Palu’s quality scores come from its reference implementation, which computes the same arithmetic as its kernel. Baselines use their authors’ kernels where available; PyramidKV, which has none, runs in PyTorch, and TransMLA runs on our latent-attention kernel.

Decode throughput protocol

The throughput campaign’s grid, statistics and exclusion rules were fixed before it ran. Each host has 4 NVIDIA L40S GPUs; one process runs on each, pinned to its own CPU core on the GPU’s socket, and the host’s one-minute load average is recorded before every timed row. A session times one group of (model, precision) pairs, so every ratio divides by a native-SDPA row of the same model, precision and session. Each cell is one (method, cache size, batch, context) combination, with batch sizes 1, 4, 16 and 64 and 8 log-spaced contexts from 2,048 to 150,000 tokens; every cache is priced at 8,192 tokens. A cell discards 2 warm-up repetitions and times 5 repetitions of 32 decode steps, each bounded by device synchronisation, and reports their median. The native-SDPA row is timed before and after the method in the same process. A row starts only when the host load is below 5.0.

Calibration. Running four processes at once could slow each through shared host resources. Before each session, a calibration stage times native SDPA on one GPU alone and beside three GPUs running the heaviest cells of the grid, in at least 6 paired runs. The session proceeds only if the 95% bootstrap interval of the ratio of the two lies inside [0.97, 1.03]; otherwise it runs one GPU at a time.

Statistics. A method’s ratio at a cache size is the median over its cells of its throughput divided by the native-SDPA throughput of the same cell, with a 95% percentile bootstrap over cells (10,000 resamples). We report the median over every cell both sides completed, and over the subset of cells with at least 100,000 tokens across the batch, where reading the cache dominates the step. A cell in which native SDPA runs out of memory has no ratio; such cells give the context reach of RQ4 instead.

Per-cell results. Kernels are checked as described in Appendix § 12.§ 4. Figure 14–Figure 16 give the measured throughput of every baseline in each (batch, context) cell, against the compression ratio it achieves.

Figure 14. Decode throughput against compression ratio, Qwen3-0.6B-Base: median tokens/s on NVIDIA L40S; rows: sequence length, columns: batch size, both axes logarithmic. Grey: the uncompressed model with native SDPA, timed in the same process; orange circles: Transformer-FSQ at every ratio timed; teal upward triangles: PyramidKV; green diamonds: Palu; blue downward triangles: KIVI; purple plus signs: VecInfer. KIVI and VecInfer run in their native float16, the rest in bfloat16. Missing markers: out of memory or outside the method’s grid.

Figure 16. Decode throughput against compression ratio, Qwen3-8B-Base; layout as in Figure 14.

Empirical memory footprint

To check the byte accounting against what is allocated, we profile resident GPU memory on Qwen3-0.6B-Base in bfloat16 at an 8,192-token context window (7,855 prompt tokens in practice; Figure 17). Across configurations, the physical buffer allocated for the KV cache equals the analytic formula, tokens × layers × bytes per token per layer, exactly. At 128 B per token per layer (32× compression), the measured cache buffer requires exactly 28,152,320 bytes (28.15 MB, or 1.005 MB per layer for 28 layers; 29.36 MB at full 8,192 capacity). For FSQ at 512 B per token per layer (8× compression), measured cache occupancy is 112,609,280 bytes (112.61 MB; 117.44 MB at full window). The uncompressed cache holds 900,874,240 bytes (900.87 MB; 939.52 MB at full window).

Figure 17. Measured against analytical KV-cache memory on Qwen3-0.6B-Base in bfloat16 at 8,192 context (7,855 prompt tokens). Grey: the analytical byte count (§ 4.4) at 8,192 tokens; coloured: the buffer allocated after the prompt, which equals the analytical bytes per token times the prompt’s tokens exactly.

Ablations

The ablations below test our architectural choices by validation bits per byte (BPB), long-context retrieval and decode throughput. Spreads report sample standard deviations over evaluated seeds.

MLA-FSQ

MLA-FSQ combines discrete FSQ codes for key-value content with Multi-Head Latent Attention’s separate rotary key krk^r, cached in half precision and shared across heads (Figure 18). On Qwen3-0.6B-Base, shared query-key normalisation causes validation BPB to spike from 0.75 to 5.5, as large rotary norms (121.8 RMS vs 19.6 content) suppress discrete representations. The half-precision rotary key and the per-head key norms take a fixed share of every token’s bytes, and the deepest configuration we trained with query-key normalisation is 32×. We therefore keep the rotary embedding on reconstructed keys.

Alternative decoding topologies: FSQ-pre (left) and MLA-FSQ (right)

Figure 18. Alternative decoding topologies: FSQ-pre (left) and MLA-FSQ (right). FSQ-pre applies RoPE before the encoder, so codes carry position and keys need no per-step reconstruction. MLA-FSQ caches a half-precision rotary key krk^r beside the content codes and adds its query dot product to the logits. Shaded plates (×T\times T) scale with context length TT. Appendix E gives why we keep post-projection Transformer-FSQ.

FSQ-pre and positional encodings

FSQ-pre evaluates applying RoPE to the residual stream before the encoder (Figure 18), rotating activations as C/dhC/d_h blocks of width dhd_h so attention reads cached codes without reconstructing keys (§ 4.3). In this topology, query-key normalisation breaks relative-position preservation, inducing severe logit errors in a 12-layer, 768-wide model trained from scratch (0.60±0.080.60 \pm0.08 relative error vs 3×10−73 \times10^{-7} without normalisation). We therefore keep the bottleneck after the projection, as in § 4.2.

Figure 15. Decode throughput against compression ratio, Qwen3-1.7B-Base; layout as in Figure 14.

Discreteness and coordinate allocations

Table 7 compares, at two equal cache budgets on Qwen3-0.6B-Base, Transformer-FSQ (16 levels per coordinate) against a trained half-precision latent at the same byte budget, a wider continuously trained latent quantised to 4 bits post hoc with FSQ’s grid or with KIVI’s block scheme, a latent trained through a min–max 4-bit grid, and Palu and TransMLA with 4-bit latents, on RULER at 8,192 tokens. Transformer-FSQ scores above every alternative at both sizes. Table 8 isolates levels per coordinate: at fixed cache memory, 16 levels per coordinate score higher than 4 or 256.

CacheB/token/layerRatioRULER (8,192)
FSQ, 4 levels14428×28\times0.178±0.0130.178 \pm0.013
FSQ, 16 levels14428×28\times0.232±0.0220.232 \pm0.022
FSQ, 256 levels14428×28\times0.161±0.0110.161 \pm0.011
FSQ, 4 levels8051×51\times0.104±0.0130.104 \pm0.013
FSQ, 16 levels8051×51\times0.177±0.0170.177 \pm0.017
FSQ, 256 levels8051×51\times0.035±0.0140.035 \pm0.014

Table 8. Levels per coordinate: RULER at 8,192 tokens on Qwen3-0.6B-Base (0.25 finetuning tokens per parameter) for FSQ with 4, 16 and 256 levels at 144 and 80 B per token per layer; mean ±\pm 95% bootstrap half-width over three seeds and items. 16 levels score highest at both sizes.

Straight-through estimation vs continuous relaxations

When optimising discrete bottlenecks, alternatives to the straight-through estimator (STE) include continuous relaxations such as Gumbel-softmax [18, 42] or codebook exponential moving average (EMA) updates [57, 49]. In our architecture, FSQ projects activations onto a fixed, implicit Cartesian grid by rounding, with the straight-through estimator taking ∂round⁡(z)/∂z=1\partial\operatorname{round}(z)/\partial z = 1 on the backward pass. Because the grid is fixed, there is no codebook to collapse and no commitment loss. Relaxations would add a temperature schedule across hundreds of coordinates, while STE trained stably with no extra hyperparameters; we did not evaluate the relaxations.

Joint end-to-end training vs layerwise conversion

An alternative to joint end-to-end training of all discrete bottlenecks is sequential layerwise conversion, progressively freezing earlier layers and training each bottleneck in isolation. While layerwise training avoids backpropagating through multiple STE rounding steps simultaneously, joint end-to-end training allows subsequent layers to adapt directly to the discrete quantisation errors of earlier representations. We use joint training throughout; layerwise conversion was not evaluated.

Code utilisation and entropy coding

We measure the empirical bit-rate of the codes across two million held-out tokens per configuration: coordinates carry 79–83% of available 4-bit capacity on Qwen3-0.6B-Base, with fewer than 0.3% of channels inactive even at 4,096 dimensions. Entropy coding is lossless, so it would change the cache size, not accuracy.

An ideal order-0 entropy coder would shrink the cache by 17–21% on Qwen3-0.6B-Base. We implemented a per-channel static Huffman coder over blocks of 128–512 tokens; net of per-block offset tables and byte alignment (≈ 3.8% overhead), savings reached 12.9% at extreme compression on Qwen3-0.6B-Base. However, variable-length decoding ran 16.6×–21.2× slower than our fused kernel (measured on a 12-layer, 768-wide model trained from scratch) and makes the cache size input-dependent, so we store codes packed.

Nonlinear encoders

We evaluated replacing the linear encoder with a rank-128 residual MLP (z=Wdownx+W2SiLU⁡(W1x)z = W_{\mathrm{down}}x + W_2 \operatorname{SiLU}(W_1x), with W2W_2 initialised to zero) on Qwen3-0.6B-Base (three seeds). The nonlinear encoder degraded validation loss across all cache budgets, increasing validation bits per byte from 1.000±0.0011.000 \pm0.001 to 1.015±0.0021.015 \pm0.002 at 128 B per token per layer (32×32\times). In a one-seed probe on a small model, a residual squared-ReLU encoder likewise raised validation BPB after a short finetune from conversion (3.466 vs 3.337). We therefore retain the linear encoder, which also eliminates non-linear projection overhead during prefill.

Per-head codes

Grouping codes by key-value head allows each head to decode its own slice independently. We evaluated block-diagonal key and value decoders where each of Qwen3-0.6B-Base’s 8 heads reads 1/81/8 of the code (64, 32 and 10 coordinates per head, so 256, 128 and 40 B, against the shared code’s 256, 128 and 42 B; the smallest per-head code has 80 coordinates in total, the shared one 84). Per-head codes are worse at every cache size, increasing validation bits per byte from 0.797±0.0010.797 \pm0.001 to 0.911±0.0010.911 \pm0.001 at 16×16\times, 0.879±0.0020.879 \pm0.002 to 1.109±0.0031.109 \pm0.003 at 32×32\times, and 1.173±0.0031.173 \pm0.003 to 1.718±0.0081.718 \pm0.008 at 97.5×97.5\times. A shared code pools what the heads have in common, so we keep it.

Recent-token window on Transformer-FSQ

Keeping the newest 128 tokens and the first four (sink) tokens as unrounded latents beside Transformer-FSQ’s codes shifts achieved memory: the nominal 8×8\times target (512 B per token per layer) expands to 578 B (7.1×7.1\times achieved) on Qwen3-0.6B-Base, and 32×32\times expands to 28.4× on 8B. At nearly matched achieved ratios on RULER (Table 9 and Table 11), windowed FSQ scores 0.56 at 7.1×7.1\times vs plain FSQ’s 0.56 at 8×8\times on 0.6B, and 0.53 vs 0.49 on 8B. Across ratios from 1.8×1.8\times to 372.4×372.4\times, windowed variants lie on the same score-against-compression curve as plain Transformer-FSQ once the window’s bytes are counted (Table 9–Table 11), so at 8,192 tokens the window neither helps nor hurts.

Architecture diagrams

The FSQ bottleneck

The quantiser and its packing are defined in Eq. (2) (§3).

Figure 4 contrasts one decoding step of an unmodified transformer with one step of Transformer-FSQ, and Figure 18 shows the two variants of Appendix E that change how the cache is read: FSQ-pre and MLA-FSQ. The shaded region of each diagram marks the work that grows with context length.

Per-task results in full

The tables below give the complete per-task RULER and LongBench results for every model.

Evaluation benchmarks and notation

Benchmarks. We evaluate long-context retrieval and comprehension on two suites:

  • RULER (Hsieh et al. [17]): 13 synthetic retrieval tasks at 8,192 tokens (50 items each): single-key retrieval (niah_single_1, 2, 3), multi-key retrieval (niah_multikey_1, 2, 3), multi-value retrieval (niah_multivalue), multi-query retrieval (niah_multiquery), variable tracking (vt), common/frequent words extraction (cwe, fwe), and QA (qa-1, qa-2).

  • LongBench (Bai et al. [1]): Real-world multi-page document comprehension: narrativeqa (narrative question answering, F1), gov_report (government report summarisation, ROUGE-L), and trec (few-shot in-context question classification, accuracy).

Table notation. In RULER tables (Table 9–Table 11), rows report the achieved compression ratio computed from the analytical byte budget of § 4.4 and the mean over seeds; standard deviations are omitted for compactness. Finetuned methods (Transformer-FSQ, Palu, TransMLA) use 0.25 finetuning tokens per parameter; KIVI, PyramidKV, and VecInfer are training-free. In LongBench tables (Table 12–Table 14), columns report achieved bytes per token per layer (B/tok), achieved compression ratio (Ratio), evaluated seeds, scored examples (nn), and task scores. {sup}§` marks dynamic-cache methods (KIVI, PyramidKV, VecInfer) whose cache footprint depends on context length.

Per-task long-context retrieval: RULER

Tables 9–11 give the per-task RULER breakdown behind Figure 2, Figure 8, and Appendix § 12.§ 2; every RULER number in the main text uses this protocol (Appendix § 11.§ 5).

Notes: seeds states how many complete finetuning seeds the score above averages; where seeds >> 1, the score is the mean of that many complete seeds’ own per-task means. ± is omitted for compactness. Ratio: uncompressed bytes per token per layer over the cache actually held (§ 4.4), not the nominal budget. FSQ + window: Transformer-FSQ that also keeps the newest 128 tokens and the first 4 tokens as unrounded latents (Appendix § 13), whose bytes the ratio includes. Omitted: PyramidKV and VecInfer at 42 and 10 B, where no configuration of the method fits this budget.

Notes: seeds states how many complete finetuning seeds the score above averages; ± is omitted for compactness. Ratio: uncompressed bytes per token per layer over the cache actually held (§4.4), not the nominal budget. FSQ + window: Transformer-FSQ that also keeps the newest 128 tokens and the first 4 tokens as unrounded latents (Appendix E), whose bytes the ratio includes. Omitted: PyramidKV and VecInfer at 42 and 10 B, where no configuration of the method fits the budget.

Per-task long-context results: LongBench

Tables 12 and 13 give LongBench on Qwen3-0.6B-Base and Qwen3-1.7B-Base, each task in its own metric (narrativeqa F1, gov_report ROUGE-L, trec accuracy, all from 0 to 100) and never averaged across tasks; each task is averaged only over the documents that every configuration compared in its table scored.

Notes: ±: one sample standard deviation over seeds. B/tok: cache bytes per token per layer; ratio: uncompressed size over B/tok; n/skip: examples the row’s own runs scored/did not score (prefill out of memory), over seeds and tasks; task columns use only the caption’s documents. Task columns: raw scores (0–100), averaged over documents, then seeds; never averaged across tasks. § suffix (e.g. PyramidKV@8k): cache sized at that context, not 2048 tokens; its own row. FSQ-small: Transformer-FSQ with its code capped at two thirds of the key-value width, where the bottleneck stops adding parameters over the K and V projections it replaces, with 8-bit levels where they fit more bytes; a parenthesised size is the budget the run asked for, which the cap holds to B/tok. §: B/tok ratio depend on tokens held (KIVI, PyramidKV, VecInfer); averaged over the row’s scored examples.

MethodB/tokRatioseedsnnskipgov-reportnarrative-qatrec
Uncompressed––160017.912.040.0
Compression 1.4×\times–2.8×\times
Transformer-FSQ20482.03180015.9±0.615.9\pm0.68.9±0.98.9\pm0.930.0±13.230.0\pm13.2
TransMLA20482.03180014.6±0.114.6\pm0.14.1±1.54.1\pm1.520.0±5.020.0\pm5.0
VecInfer1601§2.63180017.1±0.517.1\pm0.510.1±1.610.1\pm1.638.3±2.938.3\pm2.9
Compression 2.8×\times–5.7×\times
Transformer-FSQ10244.03180016.4±0.316.4\pm0.39.0±0.69.0\pm0.630.0±8.730.0\pm8.7
Palu10403.93180017.6±0.317.6\pm0.311.5±1.111.5\pm1.145.0±5.045.0\pm5.0
Palu7845.23180016.4±0.716.4\pm0.711.4±1.211.4\pm1.236.7±2.936.7\pm2.9
TransMLA10244.03180013.2±0.413.2\pm0.43.8±0.43.8\pm0.415.0±8.715.0\pm8.7
KIVI832§4.93180016.514.150.0
Compression 5.7×\times–11.3×\times
Transformer-FSQ5128.03180014.6±0.714.6\pm0.75.2±0.95.2\pm0.928.3±5.828.3\pm5.8
Palu40010.23180013.1±0.213.1\pm0.27.0±1.47.0\pm1.456.7±7.656.7\pm7.6
TransMLA5128.0318008.9±0.58.9\pm0.52.3±1.42.3\pm1.46.7±5.86.7\pm5.8
PyramidKV504§8.13180011.712.135.0
VecInfer705§5.83180013.9±0.613.9\pm0.65.0±2.35.0\pm2.335.0±5.035.0\pm5.0
Compression 11.3×\times–22.6×\times
Transformer-FSQ34112.03180013.4±1.313.4\pm1.34.3±1.54.3\pm1.525.0±5.025.0\pm5.0
Transformer-FSQ25616.03180013.1±0.813.1\pm0.86.8±0.56.8\pm0.531.7±2.931.7\pm2.9
Palu20819.73180010.6±0.710.6\pm0.73.8±1.13.8\pm1.111.7±7.611.7\pm7.6
TransMLA25616.0318005.8±0.85.8\pm0.81.7±0.91.7\pm0.95.0±5.05.0\pm5.0
PyramidKV229§17.9318009.914.630.0
PyramidKV@8k216§18.916009.112.830.0
VecInfer322§12.7318006.0±0.46.0\pm0.43.8±1.53.8\pm1.525.0±5.025.0\pm5.0
VecInfer@8k226§18.216006.72.430.0
Compression 22.6×\times–45.3×\times
Transformer-FSQ12832.03180011.7±1.211.7\pm1.26.1±1.46.1\pm1.435.0±5.035.0\pm5.0
TransMLA12832.0318004.4±1.14.4\pm1.11.5±0.31.5\pm0.30.0
PyramidKV93§44.0318008.212.325.0
VecInfer129§31.6318006.0±0.96.0\pm0.93.1±1.13.1\pm1.118.3±5.818.3\pm5.8
VecInfer@8k113§36.116006.81.325.0
Compression 45.3×\times–90.5×\times
PyramidKV47§87.4318007.114.820.0
PyramidKV@8k86§47.816008.518.525.0
Compression 90.5×\times–181.0×\times
Transformer-FSQ4297.5318009.6±0.49.6\pm0.42.0±0.32.0\pm0.320.0
Palu40102.4318000.1±0.10.1\pm0.10.5±0.40.5\pm0.40.0
TransMLA4297.5318003.3±0.53.3\pm0.50.3±0.60.3\pm0.60.0
Compression 362.0×\times–724.1×\times
Transformer-FSQ10409.6318001.7±1.81.7\pm1.83.5±1.33.5\pm1.30.0
Palu8512.0318001.9±1.51.9\pm1.50.6±0.50.6\pm0.50.0
TransMLA10409.6318000.3±0.40.3\pm0.40.8±0.80.8\pm0.80.0

Table 13. Per-task LongBench on Qwen3-1.7B-Base, each task in its own metric (over-length prompts truncated), over the 20 documents per task that every configuration scored. Finetuned methods use 0.25 tokens per parameter; KIVI, PyramidKV and VecInfer are training-free. Transformer-FSQ and uncompressed rows run in bfloat16, all other rows in float32.

Notes. ±\pm: one sample standard deviation over seeds. B/tok: cache bytes per token per layer; ratio: uncompressed size over B/tok; nn/skip: examples the row’s own runs scored/did not score (prefill out of memory), over seeds and tasks; task columns use only the caption’s documents. Task columns: raw scores (0–100), averaged over documents, then seeds; never averaged across tasks. §: suffix (e.g. PyramidKV@8k): cache sized at that context, not 2048 tokens; its own row. §: B/tok, ratio depend on tokens held (KIVI, PyramidKV, VecInfer); averaged over the row’s scored examples.

Table 14 gives the same breakdown for Qwen3-8B-Base, from the same 200-document-per-task campaign as Figure 5.

References

  1. [1]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024.arxiv.org/abs/2308.14508
  2. [2]Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.arxiv.org/abs/1308.3432
  3. [3]Yochai Benkler. The Wealth of Networks: How Social Production Transforms Markets and Freedom. Yale University Press, 2006.DOI
  4. [4]Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, et al. Nemotron-H: A family of accurate and efficient hybrid Mamba-Transformer models. arXiv preprint arXiv:2504.03624, 2025.arxiv.org/abs/2504.03624
  5. [5]Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. https://transformer-circuits.pub/2023/monosemantic-features/index.html, 2023.
  6. [6]Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. In Conference on Language Modeling (COLM), 2025.arxiv.org/abs/2406.02069
  7. [7]Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: KV-cache compression with low-rank projection. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2407.21118
  8. [8]Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2205.14135
  9. [10]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.14314
  10. [11]Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan (Celine) Lin, Jan Kautz, and Pavlo Molchanov. Nemotron-CLIMB: Clustering-based iterative data mixture bootstrapping for language model pre-training. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2504.13161
  11. [12]Patrick Esser, Robin Rombach, and Björn Ommer. Taming Transformers for high-resolution image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.arxiv.org/abs/2012.09841
  12. [13]Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez, Antonio Ortega, and Salman Avestimehr. Spend bits where queries look: KV cache vector quantization with attention-preserving transforms. arXiv preprint arXiv:2608.04074, 2026.
  13. [14]Robert M. Gray and David L. Neuhoff. Quantization. IEEE Transactions on Information Theory, 44(6):2325–2383, 1998.
  14. [15]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2312.00752
  15. [16]Coleman Hooper, Sehoon Kim, Hiva Mohammadazadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2401.18079
  16. [17]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2404.06654
  17. [18]Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-Softmax. In International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1611.01144
  18. [19]Hervé Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128, 2011.DOI
  19. [20]Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui. Towards economical inference: Enabling DeepSeek’s multi-head latent attention in any Transformer-based LLMs. In Annual Meeting of the Association for Computational Linguistics (ACL), 2025.arxiv.org/abs/2502.14837
  20. [21]Gregory Kamradt. Needle in a haystack – pressure testing LLMs. https://github.com/gkamradt/LLMTest_NeedleInAHaystack, 2023.
  21. [22]Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: An efficient error reduction framework for KV cache compression in LLM inference. In NeurIPS Efficient Natural Language and Speech Processing Workshop (ENLSP), 2024.
  22. [23]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. Transformers are RNNs: Fast autoregressive Transformers with linear attention. In International Conference on Machine Learning (ICML), 2020.arxiv.org/abs/2006.16236
  23. [24]Soosung Kim, Minjae Park, Eui-Young Chung, and Jaeyong Chung. GSRQ: Gain-shape residual quantization for sub-1-bit KV cache. In International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2607.01065
  24. [25]Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.arxiv.org/abs/2607.24653
  25. [26]Ankur Kumar. Residual vector quantization for KV cache compression in large language model. In NeurIPS Efficient Natural Language and Speech Processing Workshop (ENLSP), 2024.arxiv.org/abs/2410.15704
  26. [27]Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. In Advances in Neural Information Processing Systems (NeurIPS), 2024. doi: 10.52202/079017-3381.DOI
  27. [28]Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2203.01941
  28. [29]Guihong Li, Mehdi Rezagholizadeh, Mingyu Yang, Vikram Appia, and Emad Barsoum. X-EcoMLA: Upcycling pre-trained attention into MLA for efficient and extreme KV compression. In Conference on Language Modeling (COLM), 2025a.arxiv.org/abs/2503.11132
  29. [30]Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhangchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey on large language model acceleration based on KV cache management. Transactions on Machine Learning Research (TMLR), 2025b.arxiv.org/abs/2412.19442
  30. [31]Jindong Li, Yali Fu, Jiahong Liu, Linxiao Cao, Wei Ji, Menglin Yang, Irwin King, and Ming-Hsuan Yang. Discrete tokenization for multimodal LLMs: A comprehensive survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(8):9093–9113, 2026. doi: 10.1109/TPAMI.2026.3676982.DOI
  31. [32]Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, Pengsheng Guo, Foroozan Karimzadeh, Colorado Reed, Chong Wang, and Chuang Gan. CommVQ: Commutative vector quantization for KV cache compression. In International Conference on Machine Learning (ICML), 2025c.arxiv.org/abs/2506.18879
  32. [33]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyc Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2404.14469
  33. [34]Andreas Liesenfeld and Mark Dingemanse. Rethinking open source generative AI: Open-washing and the EU AI act. In ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2024.DOI
  34. [35]Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. In Proceedings of Machine Learning and Systems (MLSys), 2025.
  35. [36]Lucas D. Lingle. Transformer-VQ: Linear-time Transformers via vector quantization. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2309.16354
  36. [37]Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, et al. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a.arxiv.org/abs/2405.04434
  37. [38]Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2407.14207
  38. [39]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2-bit quantization for KV cache. In International Conference on Machine Learning (ICML), 2024b.
  39. [40]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
  40. [41]Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit LLMs: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 2024.arxiv.org/abs/2402.17764
  41. [42]Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1611.00712
  42. [43]Simon McIntosh-Smith, Sadaf R. Alam, and Christopher Woods. Isambard-AI: A leadership-class supercomputer optimised specifically for artificial intelligence. In Proceedings of the Cray User Group (CUG), pp. 44–54, 2024. doi: 10.1145/3725789.3725794.
  43. [44]Fanxu Meng, Pingzhi Tang, Xiaojun Tang, Zengwei Yao, Xing Sun, and Muhan Zhang. TransMLA: Migrating GQA models to MLA with full DeepSeek compatibility and speedup. In Advances in Neural Information Processing Systems (NeurIPS), 2025.DOI
  44. [45]Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE made simple. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2309.15505
  45. [46]Helen Nissenbaum. Privacy in Context: Technology, Policy, and the Integrity of Social Life. Stanford University Press, 2010.
  46. [47]Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state RNNs. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
  47. [48]Adam Paszke, Sam Gross, Francisco Massa, Adam Lehrer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köp f, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1912.01703
  48. [49]Aidan Scannell, Mohammadreza Nakhaeinezhadfard, Kalle Kujanpää, Yi Zhao, Kevin Sebastian Luck, Arno Solin, and Joni Pajari­nen. Discrete codebook world models for continuous control. In International Conference on Learning Representations (ICLR), 2025.
  49. [50]Elizabeth Seger, Ayiv Ovadya, Divya Siddarth, Ben Garfinkel, and Allan Dafoe. Democratising AI: Multiple meanings, goals, and methods. In AAAI/ACM Conference on AI, Ethics, and Society (AIES), 2023.arxiv.org/abs/2303.12642
  50. [51]Vaisakh Shaj, Cameron Barker, Aidan Scannell, Andras Szecsenyi, Elliot J. Crowley, and Amos Storkey. Kalman linear attention: Parallel Bayesian filtering for efficient language modelling and state tracking. In International Conference on Machine Learning (ICML), 2026.
  51. [52]Anupama Sridhar and Alexander Johansen. Echo: KV-cache-free associative recall with spectral Koopman operators. In ACM Conference on AI and Agentic Systems (CAIS), 2026.
  52. [53]Richard M. Stallman. Free Software, Free Society: Selected Essays of Richard M. Stallman. GNU Press, 2015.
  53. [54]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced Transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
  54. [55]Philippe Tillet, H. T. Kung, and David Cox. Triton: An intermediate language and compiler for tiled neural network computations. In ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL), 2019.DOI
  55. [56]Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2023.arxiv.org/abs/2308.10248
  56. [57]Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.arxiv.org/abs/1711.00937
  57. [58]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.arxiv.org/abs/1706.03762
  58. [59]Yixuan Wang, Qingyu Shi, Jiayu Zhou, Dianbo Liu, Ziwei He, and Zhouhan Lin. VQKV: High-fidelity and high-ratio cache compression via vector-quantization. arXiv preprint arXiv:2603.16435, 2026.arxiv.org/abs/2603.16435
  59. [60]David Gray Widder, Meredith Whittaker, and Sarah Myers West. Why “open” AI systems are actually closed, and why this matters. Nature, 635:827–833, 2024.DOI
  60. [61]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.DOI
  61. [62]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2309.17453
  62. [63]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenggen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuanchen Ren, Yang Fan, Yang Su, Yicheng Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.arxiv.org/abs/2505.09388
  63. [64]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule. In International Conference on Learning Representations (ICLR), 2025b.arxiv.org/abs/2412.06464
  64. [65]Dingyu Yao, Chenxu Yang, Zhengyang Tong, Zheng Lin, Wei Liu, Jian Luan, and Weiping Wang. VecInfer: Efficient LLM inference with low-bit KV cache via outlier-suppressed vector quantization. In Annual Meeting of the Association for Computational Linguistics (ACL), 2026.
  65. [66]Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. TurboQuant: Online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2504.19874
  66. [67]Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound-Stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2022.
  67. [68]Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. PQCache: Product quantization-based KVCache for long context LLM inference. Proceedings of the ACM on Management of Data (SIGMOD), 2025.arxiv.org/abs/2407.12820
  68. [69]Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. KV cache is 1 bit per channel: Efficient large language model inference with coupled quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2405.03917
  69. [70]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.14048
  70. [71]Jonathan Zittrain. The Future of the Internet – And How to Stop It. Yale University Press, 2008.

Paper details

vlnc:W_54e6jqd8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents