LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers

[!GitHub] Code: https://github.com/mscheong01/LoopVQ

Introduction

Looped transformers [10, 15, 30, 14] (Zhu et al. 2025) apply the same layers over multiple loop steps, effectively reusing the same parameters across multiple iterations. Built on this parameter-efficient architecture, Ouro-1.4B performs on par with non-looped models several times its size ((Zhu et al., 2025)).

This parameter efficiency, however, does not extend to the key-value (KV) cache. Although the loop steps share their weights, each loop step stores its own keys and values, so the KV cache grows linearly with the number of loop steps. As Figure 1a shows, Ouro-1.4B requires six times the per-token KV cache of Llama-3.1-8B ([16]), and the model’s KV cache exceeds its weights in memory even at moderate context lengths. This limits the batch size and context length at which the looped models can be served, making KV cache compression a promising way to improve their efficiency.

Among various KV cache compression methods, quantization compresses the KV cache by storing each key and value at lower precision, without discarding any tokens. It has been widely studied for standard LLMs, with both scalar ([26, 18, 1]) and vector quantization ([6, 31] (Zhang et al. 2024)). These methods can also be applied to looped models, but they do not exploit the redundancy of the cache across loop steps, and below one bit per element they lose much of their accuracy (Figure 1b).

We find that the keys and values of a layer are strongly correlated across loop steps. Grouping channels of several loop steps into one sub-vector exploits this correlation and substantially reduces the reconstruction error at the same bitwidth, which makes the loop dimension an additional axis for vector quantization, complementary to the channel axis used by existing methods. We also find that quantization errors at different loop steps affect the model output to very different degrees.

Based on these observations, we propose LoopVQ, a KV cache quantization method for looped models with two components. Loop-axis vector quantization codes channels of several loop steps jointly, so that one index is shared by multiple loop steps with no additional cost in bits. Loop-aware codebook fine-tuning refines the codebooks to match the output of the full-precision model, accounting for the unequal sensitivity of the loop steps.

We evaluate LoopVQ on general benchmarks with Ouro-1.4B, Ouro-2.6B ((Zhu et al., 2025)), and Huginn-3.5B ([14]), and on long chain-of-thought reasoning and needle-in-a-haystack (NIAH) tasks with the two Ouro models. Combined with TaSQ ([6]), which exploits structure along the channel axis, LoopVQ comes close to the BF16 baseline at 0.5 bit. We further implement LoopVQ in SGLang ((Zheng et al., 2023)), where the loop steps share the same indices and each reads them through its own slice of the codebook.

Our contributions are as follows:

• We show that the KV cache of looped models is highly redundant along the loop axis and that the loop steps differ substantially in their sensitivity to quantization.

• We propose LoopVQ, which combines loop-axis vector quantization with loop-aware codebook fine-tuning and is orthogonal to channel-axis methods such as TaSQ. At 0.5 and 0.25 bits per element, LoopVQ outperforms vector quantization baselines designed for non-looped LLMs on general, reasoning, and long-context benchmarks.

• We implement LoopVQ in SGLang, where the quantized cache stores one index row for several loop steps. It nearly halves the decode latency at long contexts and increases the peak throughput by up to 4.3× and 6.1× on the 1.4B and 2.6B Ouro models compared to the BF16 baseline.

Preliminaries

Looped language models. A looped transformer ([14] (Zhu et al. 2025)) applies the same $L_b$ decoder layers $T$ times, where the first layer of each loop step takes the previous loop step’s output as its input. Starting from the embedded sequence $h^{(0)}_{1:N}$, loop step $t \in\{1,\ldots,T\}$ computes

where $F_{\theta}$ denotes the $L_b$ layers, shared across loop steps. The output distribution $p_j$ is read from the final hidden state $h_j^{(T)}$. Each layer stores a separate KV cache at every loop step. Token $j$ produces a key $k_{j,h}^{(t,\ell)} \in\mathbb{R}^{D}$ and a value $v_{j,h}^{(t,\ell)} \in\mathbb{R}^{D}$ for each loop step $t$, layer $\ell$ and KV head $h \le H$. The cache is $T$ times larger than that of a non-looped model with the same weights.

Vector quantization of the KV cache. Prior work compresses the KV cache with vector quantization ((Zhang et al., 2024)). Every cached vector $x \in\mathbb{R}^{D}$ ($x = k$ or $v$ of one loop step, layer, and head) is split into $D/G$ sub-vectors $z \in\mathbb{R}^{G}$ of $G$ contiguous channels. Each sub-vector is stored as the index of its nearest entry in a codebook $\mathcal{B} = \{e_1,\ldots,e_K\} \subset\mathbb{R}^{G}$ of $K = 2^b$ entries,

at a cost of $b/G$ bits per element. Keys are quantized before the rotary embedding (RoPE), so attention reconstructs each key from its indices and applies RoPE to it, reading $D/G$ indices per cached vector. When these methods are applied to a looped model, codebooks are learned per loop step and layer $(t,\ell)$, and the $T$ loop steps of a layer are quantized as unrelated caches.

Codebook calibration. Codebooks are learned offline from sub-vectors $\{z_n\}$ collected on a small calibration set, typically by $k$-means on their squared reconstruction error. This objective treats all sub-vectors alike, although the error in some may affect the model output more than others. To address this, prior work weights each sub-vector by its sensitivity $\omega_n = \lVert\partial\mathcal{L}/\partial z_n\rVert_2^2$ to the next-token loss $\mathcal{L}$, which equals the sum of the diagonal empirical Fisher information over its coordinates:

This gradient-based weighting has been reported to improve over unweighted $k$-means in KV cache quantization [18] (Zhang et al. 2024). It only shapes the codebooks; encoding remains the nearest-entry rule (2).

Motivation

Unlike the caches of different layers in a non-looped model, the $T$ caches of a layer in a looped model are produced by applying the same parameters. We investigate whether this property can be used to compress the cache beyond what is achievable for a non-looped model.

Redundancy along the loop axis. The keys and values of a layer are highly similar across loop steps, as they are computed by the same projection weights from hidden states of the same token (1). In Figure 2a, the keys of one head show nearly the same channel pattern at all four loop steps. Averaged over all layers and heads of Ouro-1.4B, the correlation between adjacent loop steps increases from 0.69 between the first two loop steps to 0.94 between the last two.

This redundancy runs along a different axis from the one that existing vector quantization methods exploit. Figure 2b shows the correlation between every pair of (loop step, channel) coordinates within a 16-channel sub-vector. A key channel correlates with the same channel at the adjacent loop step at $|r| = 0.85$ on average, but with the other channels of its own sub-vector at only 0.16, and the gap is even wider for values. Figure 2c measures how much this matters for quantization. At the same bitwidth and with the same number of codebooks, we only change which coordinates share a code: $G$ channels of one loop step, or $G/4$ channels at each of four consecutive loop steps. With plain $k$-means, the second grouping roughly halves the reconstruction error of Ouro-1.4B at 0.5 bit, and reduces it substantially for both models at every bitwidth.

Unequal quantization sensitivity across loop steps. Grouping along the loop axis places coordinates from different loop steps in one sub-vector, which is then encoded by a single index. However, the loop steps do not affect the output equally. An error in the cache of an early loop step perturbs the hidden state passed to all later loop steps (1), whereas an error at the last loop step affects only that loop step.

Table 1 confirms this. We quantize the cache of a single loop step with loop-axis codebooks calibrated by (3), keep all other loop steps in full precision, and measure the KL divergence of the output distribution from that of the full-precision model. The sensitivity varies by roughly a factor of five across loop steps: quantizing the first loop step is by far the most harmful, and the third loop step the least.

Bitwidth	1	2	3	4
0.5 bit	0.069	0.018	0.013	0.030
0.25 bit	0.123	0.029	0.026	0.065

The calibration objective (3) cannot express this difference within a coupled sub-vector. Its weight $\omega_n$ is a single scalar per sub-vector, and its distance is the unweighted squared error, so an error is penalized equally regardless of the loop step on which it falls. The codebooks allocate the same precision to every loop step of a sub-vector, although errors at some loop steps affect the output far more than errors at others.

Existing KV cache quantization methods do not account for two properties of the looped structure: the cache is highly redundant along the loop axis, and the loop steps differ substantially in how strongly their quantization errors affect the output. § 4 builds on both observations.

Method

LoopVQ builds on the two properties identified in § 3: the redundancy of the cache along the loop axis (§ 4.1) and the unequal sensitivity of the loop steps (§ 4.2). Figure 3 gives an overview.

Loop-axis vector quantization

LoopVQ partitions the $T$ loop steps into blocks of $T_c$ consecutive loop steps, where $T_c$ divides both $T$ and $G$; block $\beta$ contains the loop steps $\beta T_c + m + 1$ with $m < T_c$. A code then covers $c = G/T_c$ channels at each of the $T_c$ loop steps of a block. For $x \in\{k,v\}$, layer $\ell$, head $h$, and channel group $g < D/c$, the coupled vector of token $j$ is

As in existing KV cache vector quantization methods, LoopVQ quantizes the keys before RoPE is applied, and attention applies RoPE to the reconstructed keys (§ 2). The coupled vector is encoded by its nearest entry (2) in a codebook $\mathcal{B}_{\beta,\ell,h,g} \subseteq\mathbb{R}^{G}$, and loop step $\beta T_c+m+1$ is reconstructed from its share of that entry:

One index serves $T_c$ loop steps, and the bitwidth remains $b/G$ bits per element. The codebooks are initialized with Fisher-weighted $k$-means (3), where $\omega_n = \lVert\partial\mathcal{L}/\partial z_n \rVert_2^2$ sums the squared gradients over all $T_c$ loop steps covered by $z_n$.

Loop-aware codebook fine-tuning

We refine the $k$-means codebooks by matching the output of the quantized model to that of the original model, as in knowledge distillation (Hinton et al., [17]) and the end-to-end fine-tuning of vector-quantized LLM weights (Egiazarian et al., [11]; Malinovskii et al., [27]). For a calibration sequence $s$, let $C$ be the cache of the unquantized forward pass and $p_j$ its output distribution at position $j$. Let $\hat{C}(\mathcal{B})$ be the cache reconstructed from the codes of $C$ by (2) and (5), where $\mathcal{B}$ collects all key and value codebooks, and let $q_j(\mathcal{B})$ be the output when all $T$ loop steps attend to $\hat{C}(\mathcal{B})$ instead of $C$. With the model weights frozen, we minimize

where the sum runs over the $N$ positions after the bf16 sink tokens.

Fisher-weighted $k$-means (3), like any weighted $k$-means objective, penalizes the error of each coded vector independently. It cannot account for the interaction between the errors of different loop steps, which propagate to later loop steps and combine (Appendix § 10.§ 1); its weights also treat all $T_c$ loop steps of a coupled vector equally, although they differ in sensitivity (§ 3). Minimizing (6) retains both effects. On Ouro-1.4B, fine-tuning roughly halves the KL divergence from the full-precision output (Figure 4).

System implementation

We implement LoopVQ in SGLang (Zheng et al., 2023) (Figure 5). Coupling loop steps affects how the codes are stored, when a token can be encoded, and how attention at one loop step reads codes shared by several loop steps.

Cache layout. The first $n_s$ tokens (sink) and the most recent $n_r$ tokens of a request are kept in bf16, and all other tokens are stored as codebook indices. The cache is organized in pages of 8 tokens, and $n_s$ and $n_r$ are multiples of 8, so every page is either bf16 or quantized as a whole. Since one index is shared across $T_c$ loop steps (5), a quantized page stores one index row per block of $T_c$ loop steps, and LoopVQ occupies the same memory per token as coding each loop step separately at the same $G$.

Encoding. A coupled vector (4) can be encoded only once all $T_c$ loop steps of its block have been computed. During prefill, the keys and values of each loop step are held until the last loop step of their block, and the block is then encoded by computing the distances to all codebook entries as a matrix product on tensor cores. During decoding, the key and value of each new token are written to the bf16 recent window, where all its loop steps are available by the end of the forward pass. After every $\delta$ generated tokens, the oldest page of the window is encoded across all layers and loop steps into one quantized page, so the window holds $n_r$ to $n_r+7$ tokens.

Attention. At loop step $t$, one kernel attends to the bf16 pages and another to the quantized pages (Algorithm 1). The

quantized kernel looks up each index in $B_t$, the slice of every entry that holds the coordinates of loop step $t = \beta T_c + m + 1$, i.e., $[e]_{iT_c+m}$ for $i < c$ (5), cut once when the model is loaded. Each loop step uses an ordinary vector quantizer with groups of $c$ channels, and the baselines, which code each loop step separately, run on the same kernel. Keys are cached before the rotary embedding, which the kernel applies to the reconstructed keys. At each loop step, the kernel reads $D/c$ indices per head and token. As in Flash-Decoding Dao et al. [9], the quantized tokens are divided into splits processed by separate thread blocks, and the partial outputs of all splits are combined using the logsumexp of their attention scores Dao [8], which recovers the exact softmax over all tokens. The bf16 pages are attended by SGLang’s Triton decoding kernel, which uses the same scheme, and its splits are combined with those of the quantized kernel. For batched decoding, the quantized kernel is optimized to process multiple requests per thread block, sharing the loads of the rotary coefficients and codebooks Appendix § 12.

Experiments

Experimental setup

Models. We evaluate Ouro-1.4B and Ouro-2.6B Zhu et al. (2025) ($L_b = 24$ and $48$, $T = 4$, $H = 16$, $D = 128$), using their Thinking variants for reasoning tasks, and Huginn-3.5B Geiping et al. [14], which iterates a four-layer core block $r = 32$ times between a two-layer prelude and coda (132 KV caches per token, 55 heads of dimension 96).

Methods. All methods quantize keys and values with 8-bit codes ($K = 256$) and $G \in\{16, 32\}$, i.e., 0.5 or 0.25 bits per element. The baselines are CQ Zhang et al. (2024), with a separate Fisher-weighted $k$-means codebook (3) per loop step and layer, and TaSQ Cheong et al. [6], which transforms the keys before coding each loop step. LoopVQ couples $T_c = 4$ consecutive loop steps, so a code spans $c = G/T_c$ channels ($c = 4$ at 0.5 bit, $c = 8$ at 0.25 bit); on Huginn, the 32 loop steps form eight blocks, and the prelude and coda, which have no loop axis, are coded per layer as in CQ. TaSQ+LoopVQ applies TaSQ’s key transforms before coupling Appendix § 11; its per-token key scale adds 0.008 bits per key element (0.003 on Huginn). Every method keeps the first and the most recent tokens in bf16: 8 and 64 on the general benchmarks, 64 and 256 on the reasoning and long-context benchmarks.

Calibration. All codebooks are fitted separately for each model, including the Thinking variants, on 16 sequences of 2,048 tokens from the GPQA main set excluding the Diamond subset, which does not overlap with the evaluation set. LoopVQ fine-tunes the $k$-means codebooks on 12 of these sequences with Adam (learning rate $10^{-3}$, one sequence per iteration) for four epochs and keeps the epoch with the lowest KL on the other four. On one RTX 6000 Ada GPU, the $k$-means calibration and the fine-tuning take 6.1 and 2.3 minutes for Ouro-1.4B and 14.3 and 5.9 minutes for Ouro-2.6B.

Benchmarks. We evaluate three settings, all with accuracy (%) as the metric. General benchmarks: the base models are evaluated with greedy decoding on GSM8K [7] (3-shot chain of thought, flexible extraction), MATH500 [22] (zero-shot), and HumanEval [4] and MBPP [2] (pass@1). Long chain-of-thought reasoning: the Thinking models are evaluated with thinking enabled on AIME’24, AIME’25 [28], GPQA-Diamond [29], HumanEval+, and MBPP+ [24], with the decoding settings of the model card (temperature 1.0, top-p 0.7, up to 32,768 generated tokens), averaged over three sampling seeds. Long-context retrieval: the base models are evaluated on the eight needle-in-a-haystack (NIAH) tasks of RULER [19] at context lengths from 4K to 64K tokens, the maximum context length of Ouro, with 200 examples per length, averaged over three seeds for the haystack and needle placement.

Implementation and hardware

All methods run in SGLang (Zheng et al., 2023) and store the quantized tokens as code indices. Decoding attends to the codes through a custom attention kernel; prefill attends to the exact keys and values of the current chunk and encodes each block of $T_c$ loop steps once its last loop step is computed. Efficiency is measured on an NVIDIA RTX 6000 Ada GPU (48 GB), and results on A100 (40 GB) are reported in Appendix § 13.§ 1.

Main results

General benchmarks. Table 2 reports the results on the base models. CQ and TaSQ lose much of the BF16 accuracy already at 0.5 bit, especially on MATH500 and HumanEval. LoopVQ substantially outperforms both baselines at both bitwidths, and TaSQ+LoopVQ further improves on LoopVQ, nearly matching BF16 at 0.5 bit. At 0.25 bit, the advantage of TaSQ+LoopVQ over LoopVQ becomes larger. LoopVQ and TaSQ+LoopVQ also outperform both baselines on Huginn-3.5B.

Model	Bitwidth	Method	Bits (K/V)	GSM8K	MATH500	HumanEval	MBPP	Avg.
Ouro-1.4B		BF16	16/16	78.77	77.20	73.17	73.54	75.67
Ouro-1.4B	0.5 bit	CQ	0.5/0.5	46.78	17.80	14.63	50.00	32.30
Ouro-1.4B	0.5 bit	TaSQ	0.508/0.5	57.32	38.80	24.39	53.97	43.62
Ouro-1.4B	0.5 bit	LoopVQ	0.5/0.5	75.82	69.40	63.41	64.55	68.29
Ouro-1.4B	0.5 bit	TaSQ+LoopVQ	0.508/0.5	77.56	75.20	70.12	71.43	73.58
Ouro-1.4B	0.25 bit	CQ	0.25/0.25	42.99	12.20	10.98	46.56	28.18
Ouro-1.4B	0.25 bit	TaSQ	0.258/0.25	44.88	12.80	12.20	48.41	29.57
Ouro-1.4B	0.25 bit	LoopVQ	0.25/0.25	58.91	41.20	25.61	54.50	45.05
Ouro-1.4B	0.25 bit	TaSQ+LoopVQ	0.258/0.25	70.96	63.40	55.49	59.26	62.28
Ouro-2.6B		BF16	16/16	86.13	77.40	81.10	81.22	81.46
Ouro-2.6B	0.5 bit	CQ	0.5/0.5	54.06	24.60	14.63	57.41	37.67
Ouro-2.6B	0.5 bit	TaSQ	0.508/0.5	69.07	49.20	31.10	61.38	52.69
Ouro-2.6B	0.5 bit	LoopVQ	0.5/0.5	85.22	74.60	75.00	75.93	77.69
Ouro-2.6B	0.5 bit	TaSQ+LoopVQ	0.508/0.5	85.37	78.60	78.66	78.57	80.30
Ouro-2.6B	0.25 bit	CQ	0.25/0.25	47.46	14.60	12.80	53.70	32.14
Ouro-2.6B	0.25 bit	TaSQ	0.258/0.25	52.99	17.60	14.02	56.08	35.18
Ouro-2.6B	0.25 bit	LoopVQ	0.25/0.25	70.20	52.00	28.66	60.05	52.73
Ouro-2.6B	0.25 bit	TaSQ+LoopVQ	0.258/0.25	81.80	68.60	62.80	69.58	70.70
Huginn-3.5B		BF16	16/16	34.04	12.80	17.07	35.98	24.97
Huginn-3.5B	0.5 bit	CQ	0.5/0.5	19.03	5.40	6.71	23.54	13.67
Huginn-3.5B	0.5 bit	TaSQ	0.503/0.5	19.11	5.80	10.37	23.81	14.77
Huginn-3.5B	0.5 bit	LoopVQ	0.5/0.5	32.37	9.40	12.80	34.92	22.37
Huginn-3.5B	0.5 bit	TaSQ+LoopVQ	0.503/0.5	33.13	12.20	12.20	36.51	23.51
Huginn-3.5B	0.25 bit	CQ	0.25/0.25	19.11	7.20	4.27	21.16	12.93
Huginn-3.5B	0.25 bit	TaSQ	0.253/0.25	19.18	6.60	4.88	21.16	12.96
Huginn-3.5B	0.25 bit	LoopVQ	0.25/0.25	25.02	8.20	13.41	27.78	18.60
Huginn-3.5B	0.25 bit	TaSQ+LoopVQ	0.253/0.25	28.13	9.80	13.41	32.54	20.97

Long chain-of-thought reasoning. Table 3 reports the results on the Thinking models. The baselines degrade more severely than on the general benchmarks and fall far below BF16 at both bitwidths. LoopVQ retains most of the BF16 accuracy at 0.5 bit, and combined with TaSQ it preserves much of the reasoning ability even at 0.25 bit, achieving the best result on every benchmark.

Model	Bitwidth	Method	Bits (K/V)	AIME’24	AIME’25	GPQA-D	HumanEval+	MBPP+	Avg.
Ouro-1.4B-Thinking		BF16	16/16	$47.78 \pm5.09$	$32.22 \pm5.09$	$45.12 \pm2.28$	$84.55 \pm0.70$	$72.49 \pm1.32$	$56.43 \pm1.54$
	0.5 bit	CQ	0.5/0.5	$1.11 \pm1.92$	$2.22 \pm1.92$	$11.78 \pm1.05$	$27.85 \pm0.70$	$35.27 \pm2.31$	$15.65 \pm0.76$
		TaSQ	0.508/0.5	$4.44 \pm1.92$	$5.56 \pm1.92$	$12.12 \pm2.31$	$35.16 \pm2.14$	$40.12 \pm2.72$	$19.48 \pm0.99$
		LoopVQ	0.5/0.5	$38.89 \pm5.09$	$31.11 \pm3.85$	$40.57 \pm1.27$	$75.81 \pm1.27$	$67.64 \pm1.19$	$50.80 \pm1.35$
		TaSQ+LoopVQ	0.508/0.5	$45.56 \pm8.39$	$34.44 \pm5.09$	$42.76 \pm3.04$	$81.50 \pm1.53$	$70.11 \pm0.70$	$54.87 \pm2.08$
	0.25 bit	CQ	0.25/0.25	$0.00 \pm0.00$	$1.11 \pm1.92$	$11.62 \pm3.07$	$25.61 \pm2.79$	$33.86 \pm0.00$	$14.44 \pm0.92$
		TaSQ	0.258/0.25	$2.22 \pm1.92$	$1.11 \pm1.92$	$11.45 \pm3.79$	$27.03 \pm2.82$	$32.01 \pm0.26$	$14.76 \pm1.09$
		LoopVQ	0.25/0.25	$7.78 \pm3.85$	$8.89 \pm3.85$	$14.48 \pm4.08$	$31.50 \pm1.27$	$37.13 \pm1.46$	$19.95 \pm1.41$
		TaSQ+LoopVQ	0.258/0.25	$36.67 \pm3.33$	$26.67 \pm5.77$	$36.53 \pm4.58$	$76.63 \pm2.31$	$63.84 \pm0.85$	$48.07 \pm1.69$
Ouro-2.6B-Thinking		BF16	16/16	$57.78 \pm11.71$	$51.11 \pm5.09$	$54.04 \pm3.31$	$86.18 \pm0.93$	$75.49 \pm0.81$	$64.92 \pm2.65$
	0.5 bit	CQ	0.5/0.5	$3.33 \pm0.00$	$5.56 \pm1.92$	$7.24 \pm3.04$	$30.08 \pm2.75$	$36.68 \pm1.10$	$16.58 \pm0.93$
		TaSQ	0.508/0.5	$10.00 \pm3.33$	$12.22 \pm1.92$	$22.73 \pm3.31$	$43.90 \pm2.79$	$52.73 \pm1.99$	$28.32 \pm1.23$
		LoopVQ	0.5/0.5	$47.78 \pm5.09$	$35.56 \pm1.92$	$47.81 \pm2.04$	$84.76 \pm1.61$	$74.34 \pm1.32$	$58.05 \pm1.24$
		TaSQ+LoopVQ	0.508/0.5	$61.11 \pm3.85$	$43.33 \pm5.77$	$51.01 \pm4.63$	$88.21 \pm0.93$	$75.49 \pm1.50$	$63.83 \pm1.71$
	0.25 bit	CQ	0.25/0.25	$2.22 \pm1.92$	$3.33 \pm3.33$	$8.25 \pm1.54$	$30.89 \pm1.96$	$38.80 \pm2.90$	$16.70 \pm1.09$
		TaSQ	0.258/0.25	$1.11 \pm1.92$	$5.56 \pm1.92$	$10.94 \pm0.77$	$33.33 \pm0.93$	$39.42 \pm0.26$	$18.07 \pm0.60$
		LoopVQ	0.25/0.25	$10.00 \pm3.33$	$5.56 \pm1.92$	$13.13 \pm3.31$	$43.50 \pm2.14$	$48.50 \pm1.25$	$24.14 \pm1.13$
		TaSQ+LoopVQ	0.258/0.25	$43.33 \pm3.33$	$34.44 \pm5.09$	$48.15 \pm1.27$	$82.72 \pm0.35$	$71.43 \pm1.15$	$56.02 \pm1.27$

Long-context retrieval. Figure 6 shows the NIAH results, which follow the same ordering. The baselines remain far below BF16 at every context length, while LoopVQ retrieves a large fraction of the needles at 0.5 bit. TaSQ+LoopVQ achieves the highest accuracy at every context length and bitwidth. Results on LongBench, which follow the same trend, are reported in Appendix § 13.§ 5.

System efficiency

We measure the serving efficiency of LoopVQ on Ouro-1.4B-Thinking with one RTX 6000 Ada GPU, keeping 8 sink and 64 recent tokens in bf16; results on an A100 GPU are reported in Appendix § 13.§ 1.

Prefill and decoding latency. Prefill with LoopVQ is slightly slower than with BF16, because of the encoding of each block of $T_c$ loop steps, and the overhead decreases with the context length (Figure 8a). During decoding, the kernel reads $D/c$ indices per head and token at each loop step, 8× and 16× fewer bytes than the bf16 keys and values. As decoding at long contexts is bound by reading the cache, the reduction in decode latency grows with the context length, reaching about half of the BF16 latency at 32K tokens (Figure 8b). Appendix § 13.§ 3 breaks down the GPU time by kernel type.

Batched throughput. For long generation, the batch size is limited by the KV cache (Figure 7). With BF16, only a few requests fit in memory, while LoopVQ can serve tens to over a hundred concurrent requests. At the same batch size LoopVQ is also faster, for the same reason as its lower decode latency. As a result, the peak throughput of LoopVQ exceeds that of BF16 by 3.2× to 4.3×, with the gap widening as the output grows longer and the BF16 batch shrinks. On Ouro-2.6B-Thinking, the BF16 cache holds even fewer requests, and the gain reaches 5.0× to 6.1× (Appendix § 13.§ 2).

Discussion

Ablation study

We ablate LoopVQ on Ouro-1.4B with GSM8K and HumanEval; all other settings follow § 5.1.

Components. To isolate the contribution of each component, we apply loop-axis coupling and loop-aware fine-tuning separately to CQ and TaSQ (Table 4). Coupling accounts for most of the gain at both bitwidths. Fine-tuning has little effect on GSM8K, whereas on HumanEval it tends to improve accuracy, most clearly with TaSQ. Combining both components gives the best or tied-best result in every setting. At 0.25 bit, coupling benefits TaSQ more than CQ.

	Base	+ Coupling	+ FT	GSM8K 0.5	GSM8K 0.25	HumanEval 0.5	HumanEval 0.25
CQ				46.8	43.0	14.6	11.0
CQ		✓		75.4	57.5	61.6	23.8
CQ			✓	48.7	42.5	14.6	11.0
CQ		✓	✓	75.8	58.9	63.4	25.6
TaSQ				57.3	44.9	24.4	12.2
TaSQ		✓		77.5	68.7	70.1	50.6
TaSQ			✓	59.7	47.1	31.7	14.0
TaSQ		✓	✓	77.6	71.0	70.1	55.5

Calibration corpus. We next test whether LoopVQ depends on the choice of calibration data by recalibrating every method on WikiText-2, C4, or SlimPajama (Figure 9). The corpus noticeably affects the baselines, with CQ varying by up to 11 points on GSM8K. TaSQ+LoopVQ is less sensitive overall, varying by at most 4 points on GSM8K and 1 point on HumanEval, and remains the best method on every corpus.

Calibration-set size. We then vary the number of 2,048-token GPQA calibration sequences from 8 to 32 (Table 5). Accuracy improves consistently as the set grows, most markedly on HumanEval, and even the smallest set keeps both variants well ahead of the baselines. As calibration takes only minutes (§ 5.1), the set can be enlarged at little cost.

	GSM8K			HumanEval
Sequences	8	16	32	8	16	32
LoopVQ	57.5	58.9	60.8	22.6	25.6	28.0
TaSQ+LoopVQ	69.3	71.0	71.9	49.4	55.5	59.8

Reasoning stability

When evaluating quantization methods, efficiency is usually measured as throughput or latency per generated token ignoring how many tokens are needed to reach an answer. Compression errors can lengthen reasoning or keep it from terminating, significantly wasting computation and decoding time ([25, 5]). On GPQA-Diamond, CQ and TaSQ exhaust the 32K-token budget on most problems and generate about twice as many tokens as BF16, whereas LoopVQ and TaSQ+LoopVQ follow the length distribution of BF16 (Figure 10). As a result, the throughput gains of LoopVQ (§ 5.4) carry over to the time needed to answer. Appendix § 13.§ 4 extends this analysis, including early termination, to both models and bitwidths.

Related work

Quantization of LLMs

Weight quantization methods for LLMs include scalar quantization with error compensation or activation-aware scaling ([13, 23]), joint quantization of weights and activations ([1] (Xiao et al. 2023)), and vector quantization with codebooks ([32, 11]). For the KV cache, scalar methods quantize keys and values per channel or per token ([26, 18, 1]), and vector quantization methods code groups of channels jointly ([6, 31] (Zhang et al. 2024)). These methods target non-looped models, where each layer writes one cache, and quantize the caches of different layers independently.

Quantization of looped transformers

The recurrence of looped transformers has motivated quantization methods tailored to it. LoopQ ([12]), LoopQuant (Zhao et al., 2026), and Loopy (Li et al., 2026) quantize the weights and activations of the shared block, accounting for the distribution shift and error accumulation across loop steps. For the KV cache, which dominates the memory of a looped model (Figure 1a), FlashLoop (Yang & Liu, 2026) and ResidualQuant ([20]) apply scalar quantization at 2 to 4 bits per element. LoopVQ instead applies vector quantization along the loop axis to a pretrained model, so that one index codes several loop steps, and compresses the KV cache below one bit per element.

Conclusion

We presented LoopVQ, a KV cache quantization method for looped transformers. LoopVQ groups channels of multiple loop steps into a sub-vector that shares one index, and fine-tunes the codebooks to match the output of the full-precision model, accounting for the different quantization sensitivities of the loop steps. We showed that, at 0.5 and 0.25 bits per element, LoopVQ outperforms existing vector quantization methods on general, reasoning, and long-context benchmarks, and that our SGLang implementation reduces the decode latency and raises the serving throughput over the BF16 baseline. The appendix provides further analysis of the fine-tuning objective, implementation and kernel optimization details, and additional experiment results.

Limitations

Our efficiency evaluation compares LoopVQ with BF16 serving in SGLang and does not include the baseline quantization methods. The serving efficiency of each method depends on kernels optimized for its own cache format, and it is difficult to determine a comparable level of optimization for every method. To avoid a comparison biased toward the method we optimized most, we compare against BF16 serving, which SGLang already supports with highly optimized kernels.

We also do not compare accuracy with FlashLoop (Yang & Liu, 2026) and ResidualQuant [20], the existing KV cache quantization methods for looped transformers. Both apply scalar quantization, which stores at least one bit per element and cannot reach the sub-1-bit regime that LoopVQ targets. They are designed for 2 to 4 bits per element and do not specify a configuration for lower bitwidths, so they cannot be compared with LoopVQ at matched bitwidths.

Analysis of Loop-Aware Fine-Tuning

This appendix supports two statements of § 4.2. First, the shortcomings of Fisher-weighted $k$-means cannot be removed by changing its weights, which is why we optimize the output objective (6) directly (Appendix § 10.§ 1). Second, although code assignment is discrete, the codebook entries receive an exact gradient and can be trained by ordinary backpropagation (Appendix § 10.§ 2).

Throughout, $s$ is a calibration sequence, $C$ the cache of its unquantized forward pass, and $\delta= \hat{C}(\mathcal{B}) - C$ the quantization error, viewed as one vector over all tokens, loop steps, layers, heads, and channels.

Limitation of weighted $k$-means

A natural alternative to fine-tuning is to improve the weighted $k$-means calibration itself (3). We show that no choice of weights can represent the fine-tuning objective (6).

The objective couples coded vectors. For small $\delta$, the KL divergence is quadratic in the error,

where $F_j$ is the Fisher information of $q_j$ with respect to the cache. Splitting $\delta$ into the errors $\delta_n = e_a(z_n) - z_n$ of the individual coded vectors (4),

where $F_{nn'}$ is the block of $F$ that pairs vectors $n$ and $n'$. The cross terms $F_{nn'}$ connect, among others, the codes of a token at different loop steps and layers: by the recurrence (1), the error of an early loop step propagates to the later loop steps and interacts with their errors.

A weighted $k$-means objective cannot express the cross terms. Any weighted $k$-means objective, Fisher-weighted $k$-means included, is a sum of terms that each depend on a single coded vector, $\sum_n \delta_n^{\mathsf{T}}W_n\delta_n$; Fisher-weighted $k$-means sets $W_n = \omega_n I$. Since each term involves only one vector, $k$-means can assign every vector to its nearest entry independently of the others and update each entry from the vectors assigned to it. For the same reason, such an objective can at best match the diagonal blocks $F_{nn}$ of (8) and always drops the cross terms. A better $W_n$ could account for the unequal sensitivity of the loop steps within a coded vector (§ 3), but not for the interaction between coded vectors. Fine-tuning minimizes (6) itself, which keeps both and, unlike any quadratic surrogate, also holds for large errors.

Gradient with respect to the codebook entries

Vector quantization selects entries by an arg min, which has no useful derivative. We show that the codebook entries nevertheless receive an exact gradient, because the selection does not need to be differentiated.

The reconstruction $\hat{C}(\mathcal{B})$ depends on $\mathcal{B}$ through the assignments $a(z_j)$ and the decoding (5). The coupled vectors $z_j$ are formed from the unquantized cache $C$ and do not depend on $\mathcal{B}$. Each assignment $a(z_j)$ is constant as long as the nearest entry of $z_j$ does not change, i.e., everywhere except on the measure-zero set of codebooks for which $z_j$ is equidistant from two entries. Off that set, $\partial a(z_j)/\partial\mathcal{B}=0$, and (5), which copies coordinate $iT_c+m$ of the selected entry into the reconstruction of loop step $\beta T_c+m+1$, is linear in that entry. For an entry $e_r$ of $\mathcal{B}_{\beta,\ell,h,g}$, the chain rule gives

for $i<c$ and $m<T_c$, where the sum runs over the calibration tokens assigned to $e_r$. Each entry accumulates the gradient of every token assigned to it, at each of the $T_c$ loop steps it covers. Because the coupled vectors $z_j$ do not depend on $\mathcal{B}$, no straight-through estimator is needed.

In practice, the backward pass runs in two stages: through the network down to the reconstructed cache, and then from the cache to the entries via (9), in chunks of tokens. This yields the same gradient as a single backward pass through one graph while bounding the memory of the second stage. After each update, the assignments are recomputed with the current codebooks.

Orthogonal application with TaSQ

TaSQ ([6]) improves key quantization along the channel axis: it transforms the channels of each key so that those sharing a code are correlated and the reconstruction error reflects the attention logits. LoopVQ is orthogonal to it, as it shares codes along the loop axis. TaSQ+LoopVQ applies TaSQ’s transform before loop-axis coupling, so that each code groups correlated channels across correlated loop steps.

Formulation

TaSQ maps the pre-RoPE key of loop step $t$ in block $\beta$, layer $\ell$, and head $h$ to

where $w^{(t,\ell)}_h$ weights each channel by the second moment $M$ of the post-RoPE queries that read this cache, $w_i^2 \propto M_{ii}$, $\sigma^{(t,\ell)}_j$ is the root mean square of $w\odot k$ over all heads and channels of token $j$, and $P_{\beta,\ell,h}$ is a channel permutation. TaSQ+LoopVQ builds the coupled key vectors (4) from $u$ instead of $k$,

so that each code spans $c$ channels chosen by $P$ along the channel axis and $T_c$ loop steps along the loop axis.

Grouping under loop coupling

In TaSQ, $P$ groups channels with high covariance into the same code, keeping each RoPE pair $\{i,i+D/2\}$ together so that RoPE can be applied after decoding. Under (11), a code contains the same channels at all $T_c$ loop steps, so the grouping must be shared by the loop steps of a block. We compute $P_{\beta,\ell,h}$ from the covariance of the transformed keys of the whole block,

a $T_cD\times T_cD$ matrix over the stacked loop steps with $\tilde{k}=w\odot k/\sigma$, and group RoPE pairs taken at all $T_c$ loop steps ($2T_c$ coordinates) until a group holds $G=cT_c$ coordinates. This grouping accounts for the correlation between channels both within and across loop steps.

Calibration and bitwidth

The Fisher weights of (3) are computed with respect to $u$. During fine-tuning (§ 4.2), $w$ and $P$ are fixed and $\sigma$ is recomputed from the keys. TaSQ in fp16 costs $16/(HD)$ bits per key element, about 0.008 on Ouro and 0.003 on Hugin, which gives the key bitwidths of TaSQ and TaSQ+LoopVQ in Table 2.

Optimizing pre-RoPE vector quantization serving efficiency

LoopVQ, like CQ and TaSQ, quantizes keys before the rotary embedding (RoPE) (§ 2), since RoPE rotates the same key differently at every position and would spread the keys of one channel group over many more codebook entries. The decode attention kernel then has to apply RoPE to every reconstructed key (Algorithm 1), which requires the cosine and sine of $D/2$ frequencies at the position of each cached token. Evaluating them in the kernel adds transcendental operations for every token and frequency, so the kernel reads them from a table precomputed for all positions, one row of $D$ values per position.

With one request per thread block, every block reads the table row of each token it attends to, and the blocks of different requests read the same rows again. At large batch sizes these reads account for about 40% of the kernel time. Since a token’s RoPE position depends only on its index in the sequence, the rows needed by two requests coincide wherever both requests have a token. The kernel assigns two requests to each thread block and cuts both into splits at the same boundaries, sized by the longer request, so that one load of a table row serves the tokens of both requests at the same index. The two requests also share the codebook slices $B_t^k$ and $B_t^v$ staged in shared memory, which halves the codebook loads per request. The shared-memory buffers of each request are halved accordingly, so a block uses the same shared memory as with one request. Each lane further handles the two channel groups that share the same frequencies, so that it loads their cosines and sines with single vector loads. At batch size 1 the kernel processes one request per block.

On an RTX 6000 Ada GPU with Ouro-1.4B-Thinking, this kernel reduces the decode attention time at batch size 64 and a 6K-token context by 37% and 48% at 0.5 and 0.25 bit compared with one request per block, and raises the end-to-end throughput at batch size 64 with 2K-token prompts and 1K-token outputs by 46% and 63%.

Additional experiment results

Efficiency results on an A100 (40 GB) GPU

We repeat the measurements of § 5.4 on one NVIDIA A100 (40 GB) GPU with the same settings. The results follow the same trends as on the RTX 6000 Ada. The prefill overhead of LoopVQ decreases with the context length, and the reduction in decode latency grows with it, reaching about half of the BF16 latency at 32K tokens (Figure 11). As BF16 fits only a few long requests in memory, the peak throughput of LoopVQ exceeds that of BF16 by 2.8× to 3.7× (Figure 12). On the A100, the k-means calibration and the fine-tuning take 5.5 and 2.0 minutes for Ouro-1.4B and 11.2 and 4.6 minutes for Ouro-2.6B.

Efficiency results on Ouro-2.6B-Thinking

We repeat the measurements of § 5.4 on Ouro-2.6B-Thinking with the same GPU and settings. Prefill with LoopVQ is 5 to 9% slower than with BF16, and the decode latency at 16K tokens is about 35% lower (Figure 13). As Ouro-2.6B has twice as many layers as Ouro-1.4B, BF16 fits only one to three long requests in memory, and the peak throughput of LoopVQ exceeds that of BF16 by 5.0× to 6.1× (Figure 14).

Time profile breakdown

Figure 15 breaks down the GPU time by kernel type, measured with the PyTorch profiler under the settings of § 5.4. During decoding, the linear layers take the same time in all methods, and the difference comes from attention. With BF16, attention grows with the context length and accounts for most of the decode time at 32K tokens, whereas with LoopVQ it stays below the time of the linear layers. At batch size 16, attention takes 1.87 ms per generated token with BF16 and 0.63 and 0.54 ms with LoopVQ at 0.5 and 0.25 bit. Encoding takes under 1% of the decode time and 5 to 7% of the prefill time.

Reasoning stability across models and bitwidths

Figure 16 extends Figure 10 to both Thinking models and both bitwidths. CQ and TaSQ exhaust the generation budget on a large share of problems in every setting. TaSQ+LoopVQ follows the length distribution of BF16 in all four settings, and LoopVQ does so at 0.5 bit.

Quantization errors can also end the reasoning early. With BF16, 16% and 22% of the generations of Ouro-1.4B-Thinking and Ouro-2.6B-Thinking end within 2K tokens. At 0.25 bit, this share rises to 33% and 59% with CQ and to 30% and 47% with TaSQ, whereas it stays at or below 30% with LoopVQ and TaSQ+LoopVQ in every setting.

LongBench results

We also evaluate the base models on the 16 English tasks of LongBench ([3]), using the official prompts and metrics, greedy decoding, and prompts truncated in the middle to 64K tokens. As in the other long-context experiments, 64 sink and 256 recent tokens are kept in bf16. CQ and TaSQ fall 7 to 13 points below the BF16 average (Table 6). LoopVQ outperforms both baselines on average in every setting, and TaSQ+LoopVQ comes within 1.5 points of BF16 at 0.5 bit and achieves the highest average at both bitwidths.

Model	Bitwidth	Method	NrtvQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
Ouro-1.4B	BF16		7.81	23.56	34.75	33.17	19.75	13.15	26.94	20.52	23.08	64.50	88.57	36.18	1.50	32.50	19.99	28.72	29.67
	0.5 bit	CQ	4.93	10.38	22.10	20.18	16.35	8.08	16.88	17.44	15.72	44.00	79.12	33.07	1.50	4.50	19.68	27.95	21.37
		TaSQ	7.02	13.39	25.05	24.90	17.46	9.22	18.38	17.88	16.58	47.00	81.79	34.07	1.00	4.50	21.85	27.87	23.00
		LoopVQ	7.31	19.29	30.71	32.63	19.23	12.82	23.73	18.94	21.06	58.50	87.80	35.61	2.00	16.00	22.09	29.60	27.33
		TaSQ+LoopVQ	8.11	22.84	33.34	34.56	18.89	12.60	26.07	19.68	22.15	64.00	89.03	36.09	1.50	23.00	22.16	30.55	29.04
	0.25 bit	CQ	4.29	9.22	21.13	20.51	16.83	7.17	15.79	17.11	14.96	44.50	78.44	33.01	1.00	4.50	20.55	28.45	21.09
		TaSQ	5.69	10.46	21.65	20.73	16.56	6.76	16.50	17.22	15.69	45.00	79.20	31.98	1.00	4.50	20.54	27.69	21.32
		LoopVQ	5.81	13.18	22.98	24.18	17.05	9.30	19.16	17.79	17.00	45.00	82.50	34.21	1.50	4.50	21.69	28.97	22.80
		TaSQ+LoopVQ	6.40	19.61	30.48	32.57	18.71	12.67	22.88	18.36	19.77	56.50	87.09	34.39	1.00	6.50	22.05	29.69	26.17
Ouro-2.6B	BF16		9.90	31.54	47.80	42.38	39.16	28.00	28.08	23.53	25.73	72.00	88.76	13.50	0.00	70.00	36.95	37.46	37.18
	0.5 bit	CQ	7.49	15.86	31.73	26.29	31.49	13.42	17.58	19.18	16.70	48.50	81.28	12.04	0.00	7.00	38.91	33.60	25.07
		TaSQ	8.41	17.82	35.24	32.51	33.24	20.24	19.03	19.17	17.74	52.00	85.74	11.25	0.00	9.00	40.47	35.81	27.35
		LoopVQ	10.21	25.53	45.30	41.84	35.03	23.20	22.61	21.32	22.39	62.00	88.93	13.21	0.00	27.00	40.38	39.22	32.39
		TaSQ+LoopVQ	9.83	29.41	48.46	43.18	38.93	26.11	25.80	22.38	24.39	72.00	89.01	12.87	0.00	49.00	39.73	39.93	35.69
	0.25 bit	CQ	7.48	15.22	31.20	25.60	30.73	12.91	16.84	18.84	15.17	47.00	81.12	12.07	0.00	7.50	38.51	33.85	24.63
		TaSQ	6.93	15.19	31.54	26.27	31.36	14.86	17.13	18.66	15.96	47.50	82.37	11.04	0.00	7.50	39.37	34.48	25.01
		LoopVQ	8.93	18.13	35.29	30.69	32.35	16.59	19.27	19.49	18.64	50.50	86.14	12.56	0.00	8.00	38.81	36.40	26.99
		TaSQ+LoopVQ	8.90	23.42	43.60	41.73	34.72	25.40	21.62	20.29	20.93	66.50	88.51	12.24	0.00	28.50	40.64	38.56	32.22