betaFeedback

WorksComputer Science

BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

Abstract

Looped architectures scale quality while keeping parameter count fixed by repeatedly passing activations through the same block of layers. During inference, each recurrence generates independent KV-caches, causing memory to grow linearly. To address this challenge, prior work has proposed KV-sharing (i.e., sharing KV-caches across depth), finding that, in some cases, the KV-cache of the final recurrence can be shared with the earlier recurrences without quality loss, a property we denote as KV-shareability. However, the mechanisms underlying KV-shareability remain poorly understood, with even identically trained looped architectures not consistently exhibiting it. In this paper, we study how KV-shareability can be enabled in looped models. We propose a two-stage method: in the first stage, we use stochastic depth during pretraining, which induces contractive residual dynamics that drive representations toward a common limit across recurrences. In the second stage, we propose BeNice!, a post-training method that broadcasts the final recurrence's KV-cache into the first recurrence, training it to produce compatible residuals while preserving the original model's quality. As a result, BeNice! enables KV-shareability while matching standard finetuning's unshared validation perplexity. Using BeNice!, we show that KV-sharing is complementary to alternative KV-compression methods, composing with memory-efficient sequence mixers (e.g., multi-latent attention, Kimi delta attention) and low-precision KV-cache quantization (e.g., fp8, mxfp8, nvfp4). Finally, in simulated low-memory, long-context inference, KV-sharing improves cache hit rates by up to 31.4%, increases throughput by up to 23.6%, and reduces p90 time-to-first-token by up to 83.6%.

Introduction

As inference deployments move to the edge [43, 45] and take up a large portion of compute [67], looped architectures create a new, predictable avenue to scale quality while keeping parameter count fixed [20, 41, 50]. Unfortunately, while looped architectures keep the memory overhead of parameters capped, they still create KVKV-caches at each recurrence [19, 50, 80], causing memory overhead to still grow linearly with recurrence count at inference time. One potential direction to reduce inference-memory overhead is KVKV-sharing (i.e., sharing KVKV-caches across recurrences). Specifically, some recent looped models have been shown to support reuse of the final recurrence’s KVKV-cache for all prior recurrences with minimal impact to quality [20, 21], a property we term KV-shareability (Figure 1 [right]).

Figure 1. Stochastic Depth, BeNice!, and KV-Shareability. (Left) During pretraining, stochastic depth induces contractive residual dynamics. (Middle) BeNice! post-trains by injecting the final recurrence KV-caches into the first recurrence. Together, they enable KV-shareability in looped models, retaining only the final recurrence’s KV-cache without degrading quality (Right).

Unfortunately, the mechanisms underlying KVKV-shareability remain poorly understood, with more recent work reporting significant quality degradation with KVKV-cache reuse [19, 80]. Moreover, even models trained under identical setups can differ in whether they exhibit KV-shareability (see Table 1). In this paper, we hypothesize that contractive recurrent dynamics drive residual states toward fixed points, whose agreement under shared and unshared KV-caches enables KV-shareability.

ModelLambada BPB (↓)Core (↑)Core-Extended (↑)
UnsharedSharedΔUnsharedSharedΔUnsharedSharedΔ
Parcae 140M0.9620.9620.00014.0714.170.1010.1310.09-0.04
Parcae 370M0.7620.7640.00220.4719.85-0.6213.2313.240.01
Parcae 770M0.6451.4600.81425.0521.10-3.9415.6913.95-1.74
Parcae 1.3B0.6012.2811.68028.7621.37-7.4016.7313.48-3.25
Huginn 3.5B0.4290.4290.00040.9740.53-0.4429.0628.61-0.44
Ouro 1.4B0.3531.2800.92754.5725.1729.4053.1132.1320.98
Ouro 2.6B0.3010.7440.44359.7826.5833.2061.4237.5723.84

Table 1. Emergence of KV-shareability in Looped Models. Quality of looped architectures with and without a shared KV-cache. Observe that KV-shareability is not universal to all looped architectures (Bolded models have KV-shareability). See Section C for evaluation details.

To encourage contractive residual dynamics, we train with stochastic depth, randomly sampling depth throughout training (Figure 1 [left]). By requiring low-loss predictions across multiple depths, stochastic depth discourages residual drift and promotes convergence across recurrences. Contractive residual dynamics are what allow KV-shareability, which we formally analyze and empirically validate by showing that replacing intermediate caches with the final cache perturbs the attention output by a term bounded by the difference between their input residuals (see § 3.1).

Unfortunately, even though stochastic depth creates contractive residual dynamics, it does not guarantee that residual states converge to the same fixed point with and without KV-sharing. Instead, KV-sharing sometimes introduces token-independent residual errors in the first recurrence, shifting the model’s fixed point and harming downstream quality (see § 3.2).

To enable KV-shareability, we pair stochastic depth pretraining with a targeted correction of the residual errors introduced by KV-sharing. We propose a two-stage method: first, stochastic depth pretraining to induce contractive residual dynamics; second, BeNice! to correct first recurrence residual errors created from KV-sharing. Specifically, BeNice! jointly optimizes both the first recurrence with KV-cache reuse and the final recurrence without reuse (Figure 1 [middle]), balancing KV-shareability and base-model quality. To further study KV-sharing with other KV-compression techniques, we extended BeNice! to memory-efficient sequence mixers, including multi-latent attention (MLA) [13] and Kimi delta attention (KDA) [65], by reusing the final recurrence’s latent or state, respectively. Overall, BeNice! unlocks near-lossless KV-cache reuse across recurrences, facilitating predictable quality scaling using extended looping [50] while keeping parameter count and KV-cache memory constant as depth grows.

We evaluate KV-shareability and BeNice! on end-to-end quality, compatibility with other KV compression methods, and how it accelerates inference workloads:

  • End-to-End Quality. BeNice! improves shared quality by up to 8.35 and 4.56 points for Core and Core-extended [38], respectively. Compared with standard finetuning, BeNice! improves validation perplexity by 9.75 points when sharing, while matching quality in the unshared case.

  • Complementary to other KV Compression Methods. To evaluate KV-sharing with other KV-compression methods, we study its composition with memory-efficient sequence mixers and KV-quantization. We find that KV-sharing composes with both MLA and KDA sequence mixers and low-precision KV-quantization (e.g., fp8, mxfp4, nvfp4).

  • Accelerating Inference Workloads. Under simulated long-context inference traffic, we observe

that memory reductions in KV-sharing improve cache-hit rates by up to 31.4% (8.1 points), increasing throughput by up to 23.6%, and reducing p90 TTFT by up to 83.6%.

Background

We first provide a brief background on prior looped models (§ 2.1) and the inference overhead of language models (§ 2.2). Looped architectures have been widely explored in synthetic problem solving tasks [7, 29, 31, 57, 70]; however, our work focuses on looped language models [14, 20, 50, 80]. These models differ in where recurrence occur (pre-, mid-, post-, or full-looping) [2], how it is halted (i.e., explicit router [80] or implicitly within the residual [20, 50]), and how gradients are computed (explicit or implicit backpropagation [5]). We study middle-looped models using implicit halting and explicit differentiation, with an emphasis on how pretraining for fixed-point convergence enables memory-efficient inference. An extended literature review is found in Section B.

Prior Looped Models

Most prior looped language models [20, 50, 80] follow a middle-looped structure sequentially composed of an initial prelude block P\mathcal{P}, a middle recurrent block R\mathcal{R}, and a final coda block C\mathcal{C}. For an input sequence s∈Vns \in\mathcal{V}^n, computation proceeds sequentially as e=P(s)e = \mathcal{P}(s), ht+1=R(ht,e)h_{t+1} = \mathcal{R}(h_t,e), p=C(hT)p = \mathcal{C}(h_T), where a model is looped TT times and h0h_0 is initialized randomly (i.e., h0∼N(0,Iσ2)h_0 \sim\mathcal{N}(0,I\sigma^2); [20, 50]) or with ee (i.e., h0←eh_0 \leftarrow e; [80]). Intuitively, P\mathcal{P} maps a sequence ss into a latent embedding e∈Rn×dee \in\mathbb{R}^{n \times d_e}, R\mathcal{R} iteratively refines a hidden state ht∈Rn×dhh_t \in\mathbb{R}^{n \times d_h} conditioned on ee, and then C\mathcal{C} maps the final hTh_T to an output p∈Rn×∣V∣p \in\mathbb{R}^{n \times|\mathcal{V}|}. To condition R\mathcal{R} on ee, input injection at each recurrence is often used (i.e., ht+1=R(Aˉ⋅ht+Bˉe)h_{t+1} = \mathcal{R}(\bar{A} \cdot h_t + \bar{B}e), where Aˉ∈Rdh×dh\bar{A} \in\mathbb{R}^{d_h \times d_h} and Bˉ∈Rdh×de\bar{B} \in\mathbb{R}^{d_h \times d_e}; [20, 50, 73]), and following Prairie et al. [50] is constrains ρ(Aˉ)<1\rho(\bar{A}) < 1 where ρ(⋅)\rho(\cdot) is the spectral radius, to prevent state explosion.

During training, prior works either sample depth stochastically [20, 50] or use adaptive computation time (ACT) [23]. Specifically, Parcae [50] and Huginn [20] sample a depth T∼ΛT \sim\Lambda independently per sequence/batch, where Λ\Lambda is a distribution (e.g., Poisson) with mean μrec\mu_{\mathrm{rec}}, training R\mathcal{R} to produce meaningful representations across depths. Ouro [80] uses ACT: a learned router predicts a halting probability at each recurrence, with computation stopping once the cumulative probability exceeds a threshold [23]. We will show the advantages of training with stochastically sampled depth in this work. Finally, to bound training memory, some models use truncated backpropagation through time (TBPTT) [20, 50], propagating gradients through only the final μbwd<T\mu_{\mathrm{bwd}} < T recurrences.

Inference Overhead of Looped Language Models

We will study how memory overhead at inference scales with context length and model depth, particularly within the scope of looped architectures [20, 50]. Under uniform numerical precision, inference memory MM for a PP-parameter model, with LL context length, DD depth, and dKVd_{\mathrm{KV}} KV width is M=Mθ+MkvM = M_{\theta} + M_{\mathrm{kv}}, where model-weight memory is Mθ=O(P)M_{\theta} = \mathcal{O}(P) and KV-cache memory is Mkv=O(LDdkv)M_{\mathrm{kv}} = \mathcal{O}(LDd_{\mathrm{kv}}). Beyond lowering numerical precision, prior work reduces MkvM_{\mathrm{kv}} overhead primarily by shrinking per-layer footprint LdkvLd_{\mathrm{kv}} (e.g., MLA [13], GQA [1], KDA [65]), leaving its linear dependence on depth DD unchanged. For a looped model evaluated with TT recurrences, the effective depth is D=DP+T⋅DR+DCD = D_{\mathcal{P}} + T \cdot D_{\mathcal{R}} + D_{\mathcal{C}}. Although looping keeps MθM_{\theta} fixed as TT increases, conventional inference stores a distinct KV cache for each recurrence, causing MkvM_{\mathrm{kv}} to grow linearly with TT.

Understanding KV-shareability in Looped Models

In this section, we study KV-shareability in looped architectures. To evaluate KV-shareability, we compare shared evaluations, which reuse the final-recurrence cache across all recurrences, against unshared evaluations, which retain every recurrence’s cache. Details on reproducing accurate KV-sharing inference setups for our benchmarks are provided in Section C.

We motivate our investigation by observing that KV-shareability is not universal to all looped models (see Table 1). We establish that stochastic depth enables contractive recurrent dynamics, which we formally analyze and empirically validate allows KV-shareability when fixed points agree under shared and unshared KV-caches (§ 3.1). However, stochastic depth alone is insufficient to guarantee agreement, leading to models not exhibiting KV-shareability (e.g., Parcae 1.3B in Table 1). Analyzing failure cases, we find that KV-sharing creates a token-independent basis shift, causing convergence to different fixed points and degraded quality under KV-sharing (§ 3.2).

Stochastic Depth Enables Contractive Residual Dynamics

In this section, we characterize how stochastic-depth training induces fixed-point convergence, driving KV-caches near a common limit and correlating with the emergence of KV-shareability.

Setup. We train Parcae-style [50] models from scratch at four scales (140M, 370M, 770M, and 1.3B), varying only whether recurrence depth is stochastic (i.e., randomly sampled during training) or static (i.e., fixed throughout training). KV-shareability is evaluated on final models and never trained for explicitly. Training and evaluation details are in Section G and Section C, respectively.

Fixed-Point Convergence encourages Shareable KV-Caches. Our key insight is that KV-shareability often emerges as a downstream consequence of fixed-point convergence in the recurrent block R\mathcal{R}. Specifically, consider a looped architecture in which R\mathcal{R} is a contraction on the hidden state hth_t, such that ht→h^h_t \to\hat{h} where h^\hat{h} is a fixed point (e.g., Parcae [50]; Huginn [20]; see Section F for empirical evidence). Since the key and value projections are locally-Lipschitz in hh, their caches also converge: kt(l),vt(l)→k^(l),v^(l)k_t^{(l)}, v_t^{(l)} \to\hat{k}^{(l)}, \hat{v}^{(l)} for every layer ll in R\mathcal{R}. Moreover, ∥kt(l)−kT(l)∥\lVert k_t^{(l)}-k_T^{(l)}\rVert is bounded by the step-to-step residual ∥ht−ht−1∥\lVert h_t-h_{t-1}\rVert (see Section D), as illustrated in Figure 2.

Figure 2. Residuals and Attention Convergence. Relative step-to-step change of the residual stream (∥hi−hi−1∥/∥hi−1∥\lVert h_i-h_{i-1}\rVert/\lVert h_{i-1}\rVert, left) and attention pattern (∥Ai−Ai−1∥/∥Ai−1∥\lVert A_i-A_{i-1}\rVert/\lVert A_{i-1}\rVert, right) across recurrence depth TT. Stochastic-depth training exhibits fixed-point convergence, while static-depth does not.

KV-cache convergence is necessary for KV-shareability: replacing an intermediate cache (kt(l),vt(l))(k_t^{(l)},v_t^{(l)}) with a fixed-point cache (k^(l),v^(l))(\hat{k}^{(l)},\hat{v}^{(l)}) perturbs the attention output by a term bounded by the fixed-point residual ∥h^−ht∥\lVert\hat{h}-h_t\rVert. To test this, we train Parcae models from scratch using stochastic and static depth training. Stochastic depth training induces a contraction1 and coincides with the emergence of KV-shareability (Table 2). In contrast, static-depth lacks an incentive to keep intermediate latents near a common limit, as reflected in their attention patterns (Figure 2) and shared quality (Table 2).

SizeΔ\Delta Val PPL (↓\downarrow)Δ\DeltaCore (↓\downarrow)
StochasticStaticStochasticStatic
140M-0.021.310.080.62
370M0.022.700.130.78
770M0.013.100.131.06
1.3B0.012.840.252.00

Table 2. KV-Sharing Relies on Stochastic Depth. We report the quality delta between unshared and shared KV-caching. Extended results and training details are in Section G.

Consistent with this mechanism, the final-recurrence cache is empirically the optimal broadcast target (Figure 3), as (kT(l),vT(l))(k_T^{(l)},v_T^{(l)}) lie closest to a fixed point across all tt, minimizing substitution error. While KV-shareability is still surprising, it suggests that looped architectures should favor stochastic fixed-point training over ACT objectives. By inducing a contraction towards a common fixed point, stochastic-depth training can enable KV-shareability without explicitly optimizing for it.

Figure 3. Final-Recurrence KV is Optimal. We find the best quality results are from converged KV caches.

KV-Sharing Fails to Converge to a Common Fixed Point

We now investigate why contractive residual dynamics alone are insufficient (e.g., 770M and 1.3B Parcae use stochastic depth and are fixed-point [Section F], yet lack KV-shareability [Table 1]). Setup. We focus our analysis on the released Parcae models [50], focusing on 770M and 1.3B scales that both fail to preserve quality under KV-sharing, yet were trained with stochastic depth and an equivalent setup to 140M and 370M Parcae models. We evaluate them using the setup in Section C.

KV-Sharing Fails to Converge to Similar Fixed Points. Figure 4 visualizes the converged representation with and without KV-sharing. We quantify the fixed-point agreement by the mean relative difference between final latents, ∥hT(shared)−hT(unshared)∥/∥hT(shared)∥\lVert h_T^{(\mathrm{shared})} - h_T^{(\mathrm{unshared})} \rVert/\lVert h_T^{(\mathrm{shared})} \rVert. Models with KV-shareability (e.g., Parcae 140M and 370M in Table 1) converge to similar fixed points, with a mean relative difference of at most 1.69%. In contrast, models without KV-shareability (e.g., Parcae 770M and 1.3B in Table 1) exhibit mean relative differences reaching up to 177.5%.

Figure 4. KV-Sharing Failures Converge to Incorrect Fixed Points. We visualize fixed points of several tokens after T=8T = 8 test-time recurrence with unshared and shared KV-caches. Models with KV-shareability (e.g., 140m and 370m) converge to the same fixed point under both setups, while models without KV-shareability (e.g., 770M and 1.3B) do not. See Section H for extended details.

KV-Sharing Failures are Token-Independent Basis Shifts Analyzing the source of KV-sharing failure, we observe that caching only the first layer KV of the first recurrence, on top of all final recurrence caches, restores quality (i.e., Layer > 1, Recurrence > 1 in Table 3). This localizes the failure to just the first layer of the first recurrence. More surprisingly, this failure manifests as a token-independent basis shift, altering the fixed point under KV-sharing and degrading quality.

Cache CompressionVal. PPL. (↓\downarrow)
770M1.3B
Unshared11.8610.80
Layer > 1, Recurrence > 111.8610.80
Naive Patching11.8710.84
Shared34.241.71e4

Table 3. Pinpointing Parcae Failures. Preserving first-layer KVs from the first recurrence or applying naive patching matches the unshared baseline, whereas full KV-sharing fails.

Table 3 and Figure 5 confirm this empirically: by constructing a subspace BB from the error delta between the first layer output with and without KV-sharing (see Section E for derivation) and projecting each token into BB, we observe an average projected pairwise cosine similarity of 0.99.

Figure 5. KV-Sharing Error Subspace Similarity. We find that tokens are strongly aligned within the subspace, but orthogonal in its complement and a random 8-dimensional subspace.

This confirms that KV-sharing errors act as a token-independent residual basis shift. To test this, we add the mean projected error to the residual after the first layer during shared evaluations (see Section E for details). This correction eliminates the need to separately cache the first layer and recurrence when sharing (see Naive Patching in Table 3).

Learning to Share Across Recurrences

One way to induce KV-shareability is to train directly on KV-sharing rollouts (i.e., sequentially rolling out KV-sharing dynamics token-by-token). However, this would be computationally infeasible and slow during training. Alternatively, using our analysis in § 3, we propose a two-stage method to enable KV-shareability (as visualized in Figure 1). In the first stage, we follow our observations in § 3.1 and pretrain models with stochastic depth to induce contractive residual dynamics that drive KV-caches to a common limit across recurrences (§ 4.1). In the second stage, we correct the token-independent basis shift observed in § 3.2 with BeNice! by optimizing both the first recurrence with KV-cache reuse and the standard final recurrence without reuse (§ 4.2).

Stochastic-Depth Pretraining: Enabling Contractive Recurrent Dynamics

In the first stage of our method, we train with stochastic depth, driving KV-caches to a common shareable limit across recurrences and setting the groundwork for KV-shareability based on § 3.1. Specifically, following Prairie et al. [50], we utilize per-sequence stochastic depth training, assigning a different depth for each sequence in a batch. This sampling structure was empirically observed to reduce loss spikes in prior work. We further follow Prairie et al. [50], sampling per-sequence depth from T∼Λ(μrec)T \sim\Lambda(\mu_{\mathrm{rec}}), where Λ\Lambda is a Poisson distribution with mean μrec\mu_{\mathrm{rec}}. Lastly, to keep memory bounded during pretraining, we use truncated backpropagation through recurrences, with a backprop depth of μbwd=⌊μrec2⌋\mu_{\mathrm{bwd}} = \left\lfloor\frac{\mu_{\mathrm{rec}}}{2} \right\rfloor [50].

BeNice! Post-training: Correcting First Recurrence Residual Errors

In the second stage of our method, we correct token-independent issues that still emerge when KV-sharing, even with stochastic-depth training. Using our analysis from § 3.2, we know quality degradations are primarily introduced in the first recurrence (see Table 3). This suggests that KV-sharing compatible inference can be recovered by training the first recurrence pass to be compatible with future KV-caches, rather than directly creating KV-sharing rollouts. Specifically, consider the ordinary TT-recurrence predictor pθ(T)p_{\theta}^{(T)} and a hybrid predictor pθ(1←T)p_{\theta}^{(1\leftarrow T)} that uses first-recurrence queries and residuals with final-recurrence KV (see Figure 1 [middle]). We define the BeNice! loss as:

LBeNice!=CE⁡(y,pθ(T))+γCE⁡(y,pθ(1←T))(1)\mathcal{L}_{\mathrm{BeNice!}} = \operatorname{CE}\left(y, p_{\theta}^{(T)}\right) + \gamma\operatorname{CE}\left(y, p_{\theta}^{(1\leftarrow T)}\right) \tag*{(1)}

where CE⁡(⋅)\operatorname{CE}(\cdot) denotes cross-entropy loss and γ\gamma controls auxiliary loss strength. Intuitively, the auxiliary term trains the first recurrence to be compatible with future recurrences’ KV-caches, while the standard language-modeling term maintains base quality at later depths. Finally, during finetuning, we sample TT from the same recurrence-depth distribution Λ\Lambda used during pretraining.

Additionally, to study if KV-sharing is complementary to other KV compression methods, we further generalize BeNice! to multiple memory-efficient sequence mixers. For example, we can extend BeNice! to multi-latent attention [13] and Kimi delta attention [65] by swapping the final recurrence’s latent or hidden state, respectively, over the KV-cache (see Section K for details).

Results

We evaluate end-to-end quality of finetuning methods for enabling KV-shareability (§ 5.1), its composition with other KV compression techniques (§ 5.2), and its latency and throughput gains in memory-bound setups (§ 5.3). Overall, we find BeNice! enables KV-shareability, improving shared Core and Core-Extended [38] quality by up to 8.35 and 4.56 points, respectively. Furthermore, BeNice! is complementary to memory-efficient sequence mixers (e.g., MLA [13], KDA [65]) and KV-quantization. Lastly, in synthetic memory-constrained inference workloads, KV-shareability improves cache-hit rate by up to 31.4% (8.1 percentage points), increases throughput by up to 23.6%, and reduces p90 TTFT by up to 83.6%.

Enabling KV-Shareability while Maintaining End-to-End Quality

On newly trained Parcae [50], BeNice! enables KV-shareability and improves base quality. BeNice! achieves comparable results to standard finetuning, while enabling KV-shareability.

Setup. We follow Parcae [50], training 140M, 370M, 770M, and 1.3B model classes (see Section N for definitions) on over 100B climbix [16] tokens. Baselines [50] are tuned by sweeping hyperparameters, using our hyperparameter-transfer observations in Section M. We finetune BeNice! with three seeds, and then evaluate using a test-time recurrence of T=μrecT = \mu_{\mathrm{rec}}, following Prairie et al. [50]. Individual hyperparameter sweeps are done for standard finetuning and BeNice! (see Section O).

BeNice! Enables KV-Shareability and Maintains Base Model Quality. Table 4 shows that KV-sharing failures emerge at all scales, reducing Core and Core-Ext [38] by up to 5.58 and 3.31 points, respectively. Finetuning with BeNice! further improves base model quality; it also enables KV-shareability, improving shared Core and Core-Ext by up to 8.35 and 4.56 points, respectively.

Overall, BeNice! allows looped architectures to scale recurrences to improve quality without scaling memory overhead at test-time.

BeNice! Matches Standard Finetuning and Gains KV-Shareability. We further compare BeNice! to standard finetuning (i.e., without an auxiliary loss term). As seen in Table 5, BeNice! achieves comparable quality to standard finetuning when no KV-cache reuse is done (i.e., unshared). However, BeNice! enables KV-shareability, reducing validation perplexity by up to 45.2% (9.75 PPL points), when KV-caches are reused across recurrences.

ModelVal. PPL (↓\downarrow)Core (↑\uparrow)Core-Extended (↑\uparrow)
UnsharedSharedΔ\DeltaUnsharedSharedΔ\DeltaUnsharedSharedΔ\Delta
770M Standard FT11.8621.61+9.7524.7423.42-1.3215.3414.54-0.80
770M BeNice!11.8511.86+0.0125.1525.19+0.0415.3515.29-0.06
1.3B Standard FT10.7918.52+7.7329.0127.49-1.5117.2316.66-0.58
1.3B BeNice!10.7910.78-0.0128.6328.67+0.0417.3217.41+0.09

Table 5. Comparing BeNice! with Standard Finetuning. We finetune public checkpoints of Parcae [50] on three different seeds, evaluating on fineweb-edu [49] validation and Core and Core Extended [38]. BeNice! matches the quality of standard finetuning, while enabling KV-shareability.

KV-Shareability is Complementary to Other KV Compression Methods.

In this section, we highlight how KV-shareability is complementary to alternative KV compression methods, by composing it with memory-efficient sequence mixers (e.g., MLA [13] and KDA [65]) and KV-quantization.

Setup. For experiments on KV-compression, we use Parcae models from § 5.1. For experiments composing KV-sharing with memory-efficient sequence mixers, we train models with multi-head attention (MHA) [68], multi-latent attention (MLA) [13] and Kimi delta attention (KDA) [65] (see Section K for more details). For experiments composing KV-sharing with KV-quantization, we directly reduce the precision of keys and values for MHA [68] or the shared latent for MLA [13].

KV-Sharing Composes with Memory-Efficient Sequence Mixers. In Figure 6, base MLA and KDA2 variants of Parcae fail to have KV-shareability, suffering quality drops when sharing final recurrence KV across earlier recurrences. These KV-sharing failures mirror the same pattern as the released MHA Parcae models explored in § 3 (see Section K for more details). BeNice! achieves KV-shareability with all sequence mixers. Across all model scales and sequence mixers, KV-sharing decreases Core and Core-Extended [38] by 0.09 and 0.04 points, respectively, on average.

KV-shareability with memory-efficient sequence mixers

Figure 6. KV-Shareability with Memory-Efficient Sequence Mixers. We train Parcae [50] models with MHA [68], MLA [13], and KDA [65]. We find that BeNice! allows memory-efficient sequence mixers to have KV-shareability, reducing inference overhead.

KV-Sharing is Compatible with KV-Quantization. In Figure 7, across scales and precisions, KV-sharing changes Core and Core-Ext by only −0.02-0.02 and +0.06+0.06 points on average, respectively, while increasing ClimbMix validation perplexity [16] by just 0.67% on average. KV-sharing and quantization interact by at most 0.53% in climbmix perplexity [16], indicating that KV-sharing is an orthogonal compression axis. We expect stronger KV-quantization methods (e.g., KVQuant [27], KIVI [40], ScaleSearch [24]) will compose with KV-sharing, further improving quality.

Figure 7. Quality of KV-Sharing and -Quantization. We report changes in climbmix PPL [16], Core, and Core-Extended [38] after quantizing the KV-shared cache across MHA [68] model scales. KV-sharing maintains quality across cache precisions.

Saving Memory and Accelerating Inference Workloads with KV-Shareability

In this section, we simulate how memory savings from KV-shareability improve TTFT and throughput under long-context inference workloads.

Setup. For long-context memory experiments, we implement efficient KV-sharing for Parcae [50] in sglang [77]. We report total GPU memory usage, including model weights, KV-cache storage, CUDA graphs, and runtime buffers. For workload experiments, we replay production-derived Mooncake requests [51] using AIPerf [46], preserving arrival times, input/output lengths, and prefix-sharing relations. To study realistic model scales, we construct proxy models from standard open-weights (e.g., GLM 4.7-FP8 [64]) by repeatedly looping the middle third transformer blocks, with and without KV-sharing. We vary both the offered request rate and maximum concurrency to characterize performance across load- and concurrency-limited regimes, respectively.

KV-Sharing Reduces Memory Overhead. In Figure 8, we observe that memory savings scale with recurrence count (μrec\mu_{\mathrm{rec}}) and KV-cache size. For our 1.3B Parcae on Core-Extended [38] with batch size and context length of 64 and 2048, respectively, KV-shareability reduces measured global HBM usage by 66.2% from 63.82 GB to 21.60 GB. At μrec=8\mu_{\mathrm{rec}} = 8, KV-sharing increases the total token capacity to 3.33×3.33\times the unshared capacity under the same KV-memory budget.

Figure 8. Global Memory Reduction of KV-Sharing. We measure and fit the memory overhead of KV-sharing, varying test-time recurrences.

KV-Sharing Accelerates Inference Workloads. With reduced memory overhead, KV-shareability increases the total tokens retained under a fixed cache budget (e.g., KV-sharing increases total token budgets by 3.33×3.33\times and 6×6\times for μrec=8\mu_{\mathrm{rec}} = 8 and μrec=16\mu_{\mathrm{rec}} = 16, respectively). Larger token budgets reduce prefix evictions and prefix recomputation (i.e., directly increasing cache hit rate by 8.1 points at 2 req/s in Figure 9). Higher cache-hit rates further improve throughput: at low loads (e.g., 0.5 req/s), KV-sharing matches the unshared baseline in TTFT and throughput; however, at high loads (e.g., 2 req/s), KV-shareability improves throughput by 23.6% and reduces p90 TTFT from 139.2 s to 23.2 s (Figure 10 left). Along the concurrency frontier, KV-shareability provides up to 6.7% higher throughput at a fixed median decode rate and 34.8% higher median decode rate at a fixed throughput (Figure 10 right). Overall, KV-shareability improves throughput and latency when workloads approach saturation.

Figure 9. Cache-Hit Rate Under Sharing. We average the cache-hit rate with and without KV-sharing using Mooncake traces [51].

Figure 10. Throughput Frontier of KV-Sharing. (Left) Under increasing offered load, KV-Sharing improves throughput and reduces TTFT over unshared looping. (Right) Under high concurrency load, KV-sharing improves overall throughput at matched decode rates.

Conclusion

In this work, we study the emergence and failure points of KV-shareability, finding that contractive residual dynamics drive KV-caches towards a common shareable limit. To enable KV-shareability, we propose a two stage method: first, we use stochastic depth to induce contractive residual dynamics across recurrences; second, we provide BeNICE!, a post-training method that corrects residual errors in the first recurrence to ensure that models approach a common limit with and without KV-sharing. KV-shareability composes with both memory-efficient sequence mixers and KV-quantization, detaches inference memory overhead from test-time recurrences, and improves throughput and TTFT in memory-constrained settings.

Acknowledgments

We graciously acknowledge the support of Together AI and AMD through the FAR research grant. This research was supported in part by the NSF under grants 2146151 and 2422214.

Glossary

We include a brief glossary of notations present in this work in Table 6.

NotationDescription
dhd_hHidden dimension of the model
ttDiscrete temporal state axis of R\mathcal{R} on N\mathbb{N}
γ\gammaHyperparameter used to select the strength of the auxiliary loss in BeNice!
P\mathcal{P}Initial prelude block of a recurrent architecture
R\mathcal{R}Middle recurrent block of a recurrent architecture
C\mathcal{C}Final coda block of a recurrent architecture
ssInput sequence to a model
eeOutput embedding of the prelude block P\mathcal{P}
hhHidden embedding of the recurrent block R\mathcal{R}
μrec\mu_{\mathrm{rec}}Mean recurrent forward propagation steps during pre-training
μbwd\mu_{\mathrm{bwd}}Mean recurrent backward propagation steps during pre-training
TTFinal number of recurrent steps actually taken
Λ\LambdaDistribution that recurrences are sampled from during training
KV-sharingUsing final recurrence’s KV-cache in early recurrences without quality loss
UnsharedDecoding performed with a KV cache per recurrence
SharedDecoding performed using only the final recurrence KV cache (KV-sharing)
ρθ(1←T)\rho_{\theta}^{(1\leftarrow T)}Output with first recurrence attending to final recurrence KV-cache
ρθ(T)\rho_{\theta}^{(T)}Output with first recurrence attending to first recurrence KV-cache (Unshared)

Table 6. Glossary of notation and terminology. (Top) Frequently used dimensions for tensors. (Middle) Definition of Parcae blocks, tensors, and distributions used to express recurrent depth models. (Bottom) KV-cache compression definitions.

Extended Literature Review

Looped architectures have been and continue to be explored by a growing body of work. We focus on architectural differences. Here, we discuss the main categories of architectural differences, as well as other works on memory efficiency in looped models.

Middle-Looped Models. This framework of models utilizes three blocks: a prelude block (P\mathcal{P}) that performs representation extraction using transformer layers, a recurrent block (R\mathcal{R}) that recurrently updates activations, and a coda block (C\mathcal{C}) that turns activations into outputs. Bansal et al. [7], Geiping et al. [20], McLeish et al. [41], and Prairie et al. [50] all train middle-looped models with stochastically sampled recurrence depth during pretraining. Schwethelm et al. [58] trained middle-looped models with fixed recurrence depth. These architectures tend to be trained with implicit halting mechanisms to determine when to stop looping [7, 20, 41, 50, 58], using either convergence methods like KL divergence between loops [7, 41], low relative differences in successive hidden states [20], or use the mean training recurrence as the optimal inference time recurrence depth [50, 58].

However, Bae et al. [4] is trained with a middle-looped architecture, but uses an explicit halting mechanism of lightweight routers to decide recurrence depth.

Fully Looped Models. This regime of looped models loops the entire model every recurrence. While not strictly a requirement, this regime of architecture tends to pull from Graves [23] work on adaptive compute time (ACT), using explicit halting mechanism to determine when to stop looping. Dehghani et al. [14], Zhu et al. [80] (Ouro), Elbayad et al. [17], Jolicoeur-Martineau [31], and Fu et al. [18] all extend this concept to language models, varying the mechanisms used for explicit halting. These halting mechanisms include prediction heads for dynamically deciding halting probabilities [17, 80] and Q-learning as a means of determining convergence [14, 23, 31]. Most of these models are trained with fixed recurrence depth [14, 17, 18, 80].

KV-sharing and Memory Efficiency in Looped Models. Huginn [20, 21] was noted as having KV-sharing ability out-of-the-box. Bae et al. [4] proposed a KV-sharing variant for their model that shares first recurrence KV-cache, with some quality loss. Deng et al. [15] extends the Ouro[80] style looped model to use with memory-efficient sequence mixers. Vendrell et al. [69] proposed a KV-sharing variant of the Ouro style model that shares the first recurrence KV-cache with later recurrences in addition to maintaining a sliding window KV-cache per recurrence.

Evaluation Setup of KV-Shareability

In this section, we will discuss how we augment benchmarks to evaluate whether a looped architecture can losslessly utilize KV-sharing. We will break benchmarks into two types (i.e., perplexity- and question-answering-based evaluations) and discuss how each is evaluated below.

Perplexity-Based Evaluations. For a standard perplexity benchmark SS evaluated on a looped architecture with TT test-time recurrences, the average per-token perplexity is

PPL⁡(S,T)=exp⁡(−1∑s∈S∣s∣∑s∈S∑i=1∣s∣log⁡pθ(T)(si∣s<i))(2)\operatorname{PPL}(S,T)=\exp\left(-\frac{1}{\sum_{s\in S}|s|}\sum_{s\in S}\sum_{i=1}^{|s|}\log p_{\theta}^{(T)}(s_i\mid s_{<i})\right) \tag*{(2)}

where pθ(T)p_{\theta}^{(T)} denotes the model’s next-token distribution when run for TT recurrences. Because the full prefix s<is_{<i} can be teacher-forced in a single parallel forward pass, the per-sequence cost is O(T∣s∣)O(T|s|). To assess whether a looped architecture has KV-shareability, we augment this benchmark in two ways. First, we replace the per-recurrence KV cache with the projection

Πshared({Kℓ(t),Vℓ(t)}t=1T):={Kℓ(T),Vℓ(T)}t=1T,∀ℓ∈R,\Pi_{\mathrm{shared}}\left(\left\{K_{\ell}^{(t)},V_{\ell}^{(t)}\right\}_{t=1}^{T}\right):=\left\{K_{\ell}^{(T)},V_{\ell}^{(T)}\right\}_{t=1}^{T},\qquad\forall\ell\in\mathcal{R},

which broadcasts the final recurrence’s keys and values across all TT recurrences in the looped block. To recover the standard perplexity evaluation, we can use the operator Πunshared\Pi_{\mathrm{unshared}}, where

Πunshared({Kℓ(t),Vℓ(t)}t=1T):={Kℓ(t),Vℓ(t)}t=1T,∀ℓ∈R.\Pi_{\mathrm{unshared}}\left(\left\{K_{\ell}^{(t)},V_{\ell}^{(t)}\right\}_{t=1}^{T}\right):=\left\{K_{\ell}^{(t)},V_{\ell}^{(t)}\right\}_{t=1}^{T},\qquad\forall\ell\in\mathcal{R}.

Finally, as populating the shared cache ΠT\Pi_T for the prediction at position ii requires re-running the entire prefix token-by-token (each token must complete all TT recurrences so its final-recurrence Kℓ(T),Vℓ(T)K_{\ell}^{(T)}, V_{\ell}^{(T)} is written into the shared slot before the next token can read it back), the evaluation no longer admits a single parallel forward pass per sequence, giving

PPL⁡(S,T;Πshared)=exp⁡(−1∑s∈S∣s∣∑s∈S∑i=1∣s∣log⁡pθ(T)(si∣Πshared(s<i)))(3)\operatorname{PPL}(S,T;\Pi_{\mathrm{shared}})=\exp\left(-\frac{1}{\sum_{s\in S}|s|}\sum_{s\in S}\sum_{i=1}^{|s|}\log p_{\theta}^{(T)}\left(s_i\mid\Pi_{\mathrm{shared}}(s_{<i})\right)\right) \tag*{(3)}

where Πshared(s<i)\Pi_{\mathrm{shared}}(s_{<i}) denotes the prefix s<is_{<i} processed under the KV-cache reuse projection Πshared\Pi_{\mathrm{shared}} (i.e., with each token’s per-recurrence keys and values {Kℓ(t),Vℓ(t)}t=1T\{K_{\ell}^{(t)},V_{\ell}^{(t)}\}_{t=1}^{T} collapsed to the final-recurrence cache {Kℓ(T),Vℓ(T)}t=1T\{K_{\ell}^{(T)},V_{\ell}^{(T)}\}_{t=1}^{T} before the next token attends back to it). The per-sequence cost is O(T∣s∣2)O(T|s|^2) as one factor of ∣s∣|s| comes from the outer position sum, and another from the prefix that must be re-processed for each prediction.

Question-Answering Evaluations. For a question prompt qq with candidate answers A={a(1),…,a(m)}\mathcal{A}=\{a^{(1)},\ldots,a^{(m)}\}, we rank candidates by their length-normalized conditional negative log-likelihood:

LΠ(a(j)∣q)=−1∣a(j)∣∑k=1∣a(j)∣log⁡pθ(T)(ak(j)∣Π(q∘a<k(j)))(4)\mathcal{L}_{\Pi}\left(a^{(j)}\mid q\right)=-\frac{1}{|a^{(j)}|}\sum_{k=1}^{|a^{(j)}|}\log p_{\theta}^{(T)}\left(a_{k}^{(j)}\mid\Pi\left(q\circ a_{<k}^{(j)}\right)\right) \tag*{(4)}

and predict

y^=arg⁡min⁡j∈{1,…,m}LΠ(a(j)∣q).(5)\hat{y}=\arg\min_{j\in\{1,\ldots,m\}}\mathcal{L}_{\Pi}\left(a^{(j)}\mid q\right). \tag*{(5)}

For KV-cache reuse, we prefill the common prompt qq once under Πshared\Pi_{\mathrm{shared}}, replicate the resulting cache across candidates, and decode each candidate token-by-token so that every token writes its final-recurrence keys and values before the next token is scored. For the unshared baseline, all candidate sequences are evaluated with a standard parallel teacher-forced forward pass.

Categorization of Benchmarks. Table 7 summarizes the 46 downstream tasks used in our evaluation, grouped by domain, scoring protocol, and number of in-context examples. CORE comprises the marked 22-task subset, while CORE-Extended includes all 46 tasks.

CategoryTaskTypeShotsKV ProtocolCORE
UnderstandingHellaSwag (0-shot) [74]MC0QA✓
UnderstandingHellaSwag (10-shot) [74]MC10QA✓
UnderstandingLAMBADA [47]LM0QA/PPL✓
UnderstandingWinograd WSC [6]S0QA✓
UnderstandingWinoGrande [55]S0QA✓
UnderstandingBIG-Bench Language ID [60]MC10QA✓
UnderstandingBIG-Bench Conlang Translation [60]LM0QA
UnderstandingBIG-Bench Conceptual Combinations [60]MC10QA
World KnowledgeJeopardy [33]LM10QA✓
World KnowledgeBIG-Bench QA WikiData [60]LM10QA✓
World KnowledgeARC-Easy [12]MC10QA✓
World KnowledgeARC-Challenge [12]MC10QA✓
World KnowledgeMMLU (0-shot) [25]MC0QA
World KnowledgeMMLU (5-shot) [25]MC5QA
World KnowledgeBIG-Bench Misconceptions [60]MC10QA
CommonsenseCOPA [22]MC0QA✓
CommonsenseCommonsenseQA [63]MC10QA✓
CommonsensePIQA [8]MC10QA✓
CommonsenseOpenBookQA [42]MC0QA✓
CommonsenseSIQA [56]MC10QA
CommonsenseBIG-Bench Novel Concepts [60]MC10QA
CommonsenseBIG-Bench Strange Stories [60]MC10QA
CommonsenseBIG-Bench StrategyQA [60]MC10QA
Symbolic/MathBIG-Bench Dyck Languages [60]LM10QA✓
Symbolic/MathAGI Eval LSAT AR [78]MC3QA✓
Symbolic/MathBIG-Bench CS Algorithms [60]LM10QA✓
Symbolic/MathBIG-Bench Operators [60]LM10QA✓
Symbolic/MathBIG-Bench Repeat Copy Logic [60]LM10QA✓
Symbolic/MathBIG-Bench Elementary Math QA [60]MC10QA
Symbolic/MathBIG-Bench Logical Deduction [60]MC10QA
Symbolic/MathSimple Arithmetic (no spaces) [44]LM10QA
Symbolic/MathSimple Arithmetic (with spaces) [44]LM10QA
Symbolic/MathMathQA [3]MC10QA
Symbolic/MathLogiQA [39]MC10QA
Reading Comp.SQuAD [52]LM10QA✓
Reading Comp.CoQA [53]LM0QA✓
Reading Comp.BoolQ [11]MC10QA✓
Reading Comp.PubMedQA (labeled) [30]LM10QA
Reading Comp.AGI Eval LSAT RC [79]MC3QA
Reading Comp.AGI Eval LSAT LR [71]MC3QA
Reading Comp.AGI Eval SAT English [79]MC3QA
Reading Comp.BIG-Bench Understanding Fables [60]MC10QA
SafetyWinogender MC (Female) [54]MC10QA
SafetyWinogender MC (Male) [54]MC10QA
SafetyEnterprise PII ClassificationMC10QA
SafetyBBQ [48]MC3QA

Table 7. Downstream evaluation tasks. Tasks marked with ✓ belong to the 22-task CORE suite; all 46 tasks belong to CORE-Extended. Under the QA KV-sharing protocol, the context is prefilled using the final-recurrence cache and candidate continuations are scored against that cache. Under the PPL protocol, used for the separate LAMBADA BPB evaluation, the complete sequence is streamed token-by-token so that every prediction reads only final-recurrence KVs. Type denotes the task scoring rule: MC uses mean candidate NLL, S uses schema-based NLL, and LM requires an exact greedy continuation match.

Proof of KV Convergence

In this section, we make precise the claim in § 3 that fixed-point convergence of the hidden state induces convergence of the per-layer keys and values. Specifically, we show that if the recurrent block R\mathcal{R} is a contraction with a fixed point h^\hat{h}, then for every attention layer l∈Rl \in\mathcal{R}, then kt(l),vt(l)→k^(l),v^(l)k_t^{(l)}, v_t^{(l)} \to\hat{k}^{(l)}, \hat{v}^{(l)} at a rate controlled by the step-to-step residual ∥ht−ht−1∥\lVert h_t-h_{t-1}\rVert.

Furthermore, we comment on the scope of this result. First, there is no guarantee that looped architectures such as Parcae [50] and Huginn [20] are contractive. However, both prior works empirically observe convergence to fixed points, and we independently verify this behavior in Section F. Second, we are not claiming that self-attention itself is Lipschitz, which has been proved not to be globally Lipschitz on unbounded inputs [35]. Our proof only requires that the per-layer key and value projection, h↦WK,V(l)⋅RMSNorm⁡(h)h \mapsto W_{K,V}^{(l)} \cdot\operatorname{RMSNorm}(h), is Lipschitz. This follows from standard bounds on linear maps composed with the Lipschitz bound of RMSNorm [10, 75].

Theorem 1. Let R:Rd→Rd\mathcal{R} : \mathbb{R}^{d} \to\mathbb{R}^{d} be κ\kappa-Lipschitz with κ<1\kappa< 1, and for each layer l∈Rl \in\mathcal{R} let ϕK(l),ϕV(l):Rd→Rdh\phi_K^{(l)}, \phi_V^{(l)} : \mathbb{R}^{d} \to\mathbb{R}^{d_h} be L(l)L^{(l)}-Lipschitz. Let ht+1=R(ht)h_{t+1}=\mathcal{R}(h_t), kt(l)=ϕK(l)(ht)k_t^{(l)}=\phi_K^{(l)}(h_t), vt(l)=ϕV(l)(ht)v_t^{(l)}=\phi_V^{(l)}(h_t). Then R\mathcal{R} has a unique fixed point h^\hat{h}, and for every t≤Tt \le T,

∥kt(l)−kT(l)∥, ∥vt(l)−vT(l)∥≤L(l)1−κ∥ht+1−ht∥,\lVert k_t^{(l)}-k_T^{(l)}\rVert,\ \lVert v_t^{(l)}-v_T^{(l)}\rVert\le\frac{L^{(l)}}{1-\kappa}\lVert h_{t+1}-h_t\rVert,

with kt(l)→k^(l):=ϕK(l)(h^)k_t^{(l)} \to\hat{k}^{(l)} := \phi_K^{(l)}(\hat{h}) and vt(l)→v^(l):=ϕV(l)(h^)v_t^{(l)} \to\hat{v}^{(l)} := \phi_V^{(l)}(\hat{h}) as t→∞t \to\infty.

Proof. Lipschitz continuity of R\mathcal{R} gives ∥hs+1−hs∥≤κs−t∥ht+1−ht∥\lVert h_{s+1}-h_s\rVert\le\kappa^{s-t}\lVert h_{t+1}-h_t\rVert for s≥ts \ge t. Summing,

∥hT−ht∥≤∑s=tT−1∥hs+1−hs∥(6)\lVert h_T-h_t\rVert\le\sum_{s=t}^{T-1}\lVert h_{s+1}-h_s\rVert \tag*{(6)}
≤∥ht+1−ht∥∑j=0T−t−1κj(7)\le\lVert h_{t+1}-h_t\rVert\sum_{j=0}^{T-t-1}\kappa^j \tag*{(7)}

via Triangle Inequality

≤∥ht+1−ht∥∑j=0∞κj(8)\le\lVert h_{t+1}-h_t\rVert\sum_{j=0}^{\infty}\kappa^j \tag*{(8)}
≤∥ht+1−ht∥1−κ(9)\le\frac{\lVert h_{t+1}-h_t\rVert}{1-\kappa} \tag*{(9)}

via Geometric Sum Closed Form. Lipschitz continuity of ϕK(l)\phi_K^{(l)} (resp. ϕV(l)\phi_V^{(l)}) then gives ∥kt(l)−kT(l)∥≤L(l)∥ht−hT∥\lVert k_t^{(l)}-k_T^{(l)}\rVert\le L^{(l)}\lVert h_t-h_T\rVert, yielding the stated bound. Existence of h^\hat{h} and convergence ht→h^h_t \to\hat{h} follow from Banach’s fixed-point theorem, and continuity of ϕK,V(l)\phi_{K,V}^{(l)} transfers convergence to the keys and values. □\square

KV-Sharing Error Subspace

In this section, we formalize the construction of the KV Sharing Error Subspace. This construction is inspired by [37].

Looking only at the first layer in the recurrent block and only during the first recurrence, let ashared∈RN×dha_{\mathrm{shared}} \in\mathbb{R}^{N \times d_h} be attention output for NN tokens attending to only final recurrence KV cache, while

aunshared∈RN×dha_{\mathrm{unshared}} \in\mathbb{R}^{N \times d_h} is the attention output using the first recurrence KV for the same NN tokens (we use N=24KN = 24\mathrm{K} tokens).

Then, define δ=aunshared−ashared\delta= a_{\mathrm{unshared}} - a_{\mathrm{shared}} as the per-token differences between the attention outputs using first recurrence KV and shared KV.

We perform SVD on δ\delta. In Figure 11, we observe that variance within δ\delta is low-rank and well explained with 8 right singular vectors, defining our KV-sharing error subspace.

Figure 11. KV-sharing Error Subspace is Low Rank. The variance within KV-sharing error subspace is well explained with a few directions.

δ=UΣVT∈RN×dh\delta= U\Sigma V^{\mathsf{T}} \in\mathbb{R}^{N \times d_h}
B8=V:,1:8B_8 = V_{:,1:8}

Naive Patching Here we provide a more formal explanation of our Naive Patching work. We take the average across per token differences within the subspace and project it back into the hidden dimension.

aˉ=(1N1TδB8)B8T(10)\bar{a} = \left(\frac{1}{N}\mathbf{1}^{\mathsf{T}}\delta B_8\right)B_8^{\mathsf{T}} \tag*{(10)}

Then, during KV-sharing decode, we add the average projected subspace error to the first layer, first recurrence attention output per token.

a^sharedt=asharedt+aˉ(11)\hat{a}_{\mathrm{shared}}^{t} = a_{\mathrm{shared}}^{t} + \bar{a} \tag*{(11)}

All other layers receive no patch, and the first layer on all subsequent recurrences receives no patch. Naive Patching is nearly lossless in terms of KV-sharing quality (Table 3).

Parcae and Huginn Converge to Fixed-Points

In this section, we provide evidence that Parcae [50] and Huginn [20] contract to fixed-points.

Setup. We evaluate the publicly released Parcae 140M, 370M, 770M, and 1.3B checkpoints [50], together with Huginn 3.5B [20], without additional training. For each model, we process 1,000 sequences of length 2,048 from the climbmix validation set in BF16. We evaluate Parcae for 15 recurrences and Huginn for its standard 32 recurrences. At each recurrence TT, we measure the relative change in the recurrent residual state,

Δh(T)=Ex[∥hT−hT−1∥RMS∥hT−1∥RMS],(12)\Delta_h(T) = \mathbb{E}_x\left[\frac{\lVert h_T - h_{T-1}\rVert_{\mathrm{RMS}}}{\lVert h_{T-1}\rVert_{\mathrm{RMS}}}\right], \tag*{(12)}

and the relative change in attention patterns,

ΔA(T)=Ex,ℓ,j[∥AT(ℓ,j)−AT−1(ℓ,j)∥F∥AT−1(ℓ,j)∥F](13)\Delta_A(T) = \mathbb{E}_{x,\ell,j}\left[\frac{\left\lVert A_T^{(\ell,j)} - A_{T-1}^{(\ell,j)}\right\rVert_F}{\left\lVert A_{T-1}^{(\ell,j)}\right\rVert_F}\right] \tag*{(13)}

where ℓ\ell and jj index recurrent-block layers and attention heads, respectively.

Results. As shown in Figure 12, residual states and attention patterns stabilize with recurrence depth across all models. Parcae 140M–770M converge rapidly, while Parcae 1.3B converges more slowly; Huginn exhibits the same behavior over its 32-recurrence horizon. Fixed-point dynamics are already present in the original pretrained models and are not specific to our KV-sharing experiments. All of these models were trained with stochastic-depth.

Figure 12. Fixed-point convergence of pretrained looped models. Relative changes in the recurrent residual state and attention pattern decrease with recurrence depth for the released Parcae models and Huginn.

Expanded Results Comparing Stochastic and Static Sampling Depth Training

In this section, we investigate the emergence of KV-sharing by connecting it to fixed-point convergence in looped architectures. Specifically, we train size-matched Parcae models, using either stochastic or static recurrence depth, and compare their residual convergence, attention convergence, and quality under KV-sharing.

Setup. We follow the experimental setup of Prairie et al. [50], using Parcae [50] as the architectural backbone for our experiments (see Section N for model definitions), and trained with μrec=8\mu_{\mathrm{rec}} = 8 and μbwd=4\mu_{\mathrm{bwd}} = 4. For stochastic depth training, we use a Poisson distribution following [50], while for static depth training, we forward pass exactly μrec\mu_{\mathrm{rec}} and a truncated backpropagation of exactly μbwd\mu_{\mathrm{bwd}}. We train on climbmix [16], using approximately 1:7 param-to-token ratios, and using the hyperparameters defined in Section O. Hyperparameters were selected after extensive tuning following Section M. Evaluation is done following Section C.

Stochastic and Static Depth Training. We compare models trained with stochastic recurrence depth against models trained at a static depth of T=8T = 8. Both exhibit similar training-loss trajectories (Figure 13) and comparable uncompressed quality. However, KV-sharing is nearly lossless for stochastic-depth models at every scale, while static-depth models degrade substantially under compression (Table 8). Thus, the emergence of KV-sharing is associated with stochastic-depth training rather than ordinary language-model optimization alone.

Figure 13. Training Curves for Stochastic- and Static-Depth Models. Training loss as a function of training tokens across model scales. The curves closely overlap, showing that stochastic- and static-depth training exhibit similar optimization dynamics.

TrainingVal. PPL (↓\downarrow)CORE (↑\uparrow)
UnsharedSharedΔ\DeltaUnsharedSharedΔ\Delta
140M Stochastic25.6825.67-0.026.416.48+0.08
140M Static24.4825.79+1.319.178.55-0.62
370M Stochastic17.3417.36+0.0214.4914.36-0.13
370M Static17.3020.00+2.7015.0014.22-0.78
770M Stochastic14.0314.04+0.0118.9118.78-0.13
770M Static14.1017.20+3.1018.1717.11-1.06
1.3B Stochastic12.3312.34+0.0123.6323.34-0.29
1.3B Static12.3215.17+2.8422.5920.82-1.77

Table 8. Quality under stochastic- and static-depth training. We compare the standard per-recurrence cache (Unshared) with final-recurrence KV sharing (Shared). Δ\Delta is Shared minus Unshared.

Extended Empirical Analysis on KV-Sharing Fixed Point Convergence Failures

In this section, we provide extended empirical evidence that KV-sharing failures in models trained with stochastic depth are fixed-point convergence failures. Specifically, we provide expanded empirical evidence that while contractive residual dynamics are necessary to induce KV-shareability, models can still fail to converge to the correct fixed points.

Setup. We use prior Parcae looped models [50] to explore our observations about KV-sharing failures. We additionally use models trained in § 5.1, showing how BeNice! impacts fixed-point convergence, before and after finetuning. Following convergence observations made in Section F and the saturation of test-time compute made by Prairie et al. [50], we use T=8T = 8 test-time recurrences, as both the residual differences and downstream quality saturate at this point. We use climbmix [16] validation data for all evaluations and explorations.

KV-sharing Fails to Converge Correctly. As visualized by Figure 4, we see that models that do not have KV-shareability from Table 1 (e.g., Parcae 770M and 1.3B) seem to have disjoint sets between unshared and shared fixed points, while models with KV-shareability have approximately similar distributions. We capture this numerically in Table 9, where we find that models that have KV-shareability approach a relatively similar fixed point. For example, Parcae 140M and 370M have a relative difference of 0.82% and 1.69%, respectively, which means they approached the same relative fixed point. However, Parcae 770M and 1.3B have an average relative difference of 134.8% and 177.5%, respectively, which means that they converge to different points with and without KV-sharing. Thus, Parcae 770M and 1.3B fail to have KV-shareability as they do not converge to the same fixed points as the base model.

ModelE[∥h8(shared)−h(unshared)∥2]\mathbb{E}[\lVert h_{8}^{(\mathrm{shared})} - h^{(\mathrm{unshared})} \rVert_{2}]E[∥h8(unshared)∥2]\mathbb{E}[\lVert h_{8}^{(\mathrm{unshared})} \rVert_{2}]E[∥h8(shared)−h(unshared)∥2∥h8(unshared)∥2]\mathbb{E}\left[\frac{\lVert h_{8}^{(\mathrm{shared})} - h^{(\mathrm{unshared})} \rVert_{2}}{\lVert h_{8}^{(\mathrm{unshared})} \rVert_{2}}\right]
140M0.458856.30.82
370M2.2766135.21.69
770M186.35153.4134.8
1.3B342.65301.6177.5

Table 9. Mean difference between shared and unshared final latents, mean unshared latent norm, and mean relative difference.

We further verify these results empirically with models trained in § 5.1, where we find that all trained baseline Parcae models fail to converge to the same fixed point under KV-sharing (see Table 10), preventing them from having KV-shareability. However, after finetuning with BeNice!, we empirically observe that models converge to the correct fixed point, as seen by Table 10.

ModelBaseline: Mean Relative DifferenceBeNice!: Mean Relative Difference
140M0.05260.0059
370M0.13040.0059
770M0.09570.0070
1.3B0.12530.0049

Table 10. Mean Relative Difference of Fixed Points With and Without KV-sharing Before and After Post-training with BeNice! Mean relative difference at each model scale for the baseline and BeNice!.

Extended Finetuning Setup

In this section, we provide more details about the BeNice! finetuning process.

Setup. Following the training procedures of Parcae in Prairie et al. [50], all models are trained using batch sizes of 256 at context length of 2048. Recurrence depth is sampled stochastically from Λ∼Poisson⁡(μrec)\Lambda\sim\operatorname{Poisson}(\mu_{\mathrm{rec}}), where μrec=8\mu_{\mathrm{rec}} = 8, as in pretraining.

We use BeNice! on two generations of models: previously released Parcae models (770m and 1.3B) [50] and new Parcae style models trained from scratch (including the MLA and KDA models in § 5.1). The publicly released models are finetuned using BeNice! with the same hyperparameters as their pretraining setup (see Prairie et al. [50]) and use 140M fineweb-edu [49] tokens. The new Parcae models are pretrained using hyperparameters described in Section O on climbmix [16] tokens, and are further finetuned on 140M climbmix tokens using BeNice!. Per-model finetuning learning rates and γ\gamma values are listed in Section O.

All BeNice! results are averaged across three differently seeded finetuning runs evaluated on Core and Core-Ext using three different seeds as well.

LoRA Finetuning for KV-sharing

In this section, we show that KV-sharing can be achieved using a LoRA[28] finetuning version of BeNice!, which we denote BeNice!{sup}†`.

Following our findings in § 3 that the KV-sharing error subspace is low-rank (Figure 11), we finetune released Parcae 770M and 1.3B with low-rank adapters on the key and value weights of the first layer of the recurrent block. We use rank-8 adapters on both key and value weights and freeze the rest of the model weights, leading to 41K trainable parameters in the 770M model and 49K trainable parameters in the 1.3B model. We finetune using 65M fineweb-edu tokens with batch size 32 and sequence length 2048.

We use the same BeNice! loss function defined in § 4. Hyperparameter details are in Section O. In Table 11, both BeNice! and BeNice!{sup}†` enable KV-sharing while maintaining quality across benchmarks.

ModelVal. PPL (↓\downarrow)Core (↑\uparrow)Core-Extended (↑\uparrow)
UnsharedSharedΔ\DeltaUnsharedSharedΔ\DeltaUnsharedSharedΔ\Delta
770M
Parcae11.8734.20+22.3325.0521.10-3.9415.6913.95-1.74
+ Standard FT11.8621.61+9.7524.7423.42-1.3215.3414.54-0.80
+ BeNice!11.8511.86+0.0125.1525.19+0.0415.3515.29-0.06
+ BeNice!{sup}†`11.8911.89+0.0024.6924.71+0.0215.3215.19-0.13
1.3B
Parcae10.781.6e4+1.6e428.7821.33-7.4516.7313.48-3.25
+ Standard FT10.7918.52+7.7329.0127.49-1.5117.2316.66-0.58
+ BeNice!10.7910.78-0.0128.6328.67+0.0417.3217.41+0.09
+ BeNice!{sup}†`10.8110.810.0028.2828.35+0.0716.9416.96+0.02

Table 11. Quality of BeNice! Finetuning Extended. We provide the extended results of finetuning using BeNice! and BeNice!{sup}†`. Both enable KV-sharing while maintaining quality.

Extended Discussion of Experimental Setup for MLA and KDA Evaluation

In this section, we provide a more detailed description of the experimental setup for evaluating MLA [13] and KDA [65] versions of Parcae [50]. We will fully discuss how KV-sharing is done for both alternative sequence mixing methods. Hyperparameters and model configurations can be found in Section O and Section N, respectively. BeNice! finetuning is explained in § 16.

KV-sharing with Multi-Latent Attention Similarly to MHA [68], KV-sharing with MLA [13] is done by discarding the latent cache of indeterminate recurrences, and keeping only the final recurrences latent cache. More formally, for an input ht∈Rn×dh_t \in\mathbb{R}^{n \times d} at recurrence tt, the latent cache and the decoupled RoPE [61] caches are calculated as

ctKV=WKVhtktRoPE=RoPE⁡(WRoPEht),(14)c_t^{\mathrm{KV}} = W^{\mathrm{KV}}h_t \qquad k_t^{\mathrm{RoPE}} = \operatorname{RoPE}(W^{\mathrm{RoPE}}h_t), \tag*{(14)}

where RoPE⁡(⋅)\operatorname{RoPE}(\cdot) denotes the RoPE rotation [61]. Thus with KV-sharing MLA models, we discard all ctKV,ktRoPEc_t^{\mathrm{KV}}, k_t^{\mathrm{RoPE}} where t<Tt < T keeping only the cache at t=Tt = T.

KV-sharing with Kimi Delta Attention Again, like both MHA [68] and MLA [13], KV-sharing with KDA [65] is done by keeping only the final recurrences sequence-mixing hidden state. More formally, for an input hth_t at recurrence tt, we focus on the linear update rule used in KDA [65]

St,i=(I−βt,ikt,ikt,iT)Diag⁡(αt,i)St,i−1+βt,ikt,ivt,iT∈Rdk×dv(15)S_{t,i} = (I - \beta_{t,i}k_{t,i}k_{t,i}^{\mathsf{T}})\operatorname{Diag}(\alpha_{t,i})S_{t,i-1} + \beta_{t,i}k_{t,i}v_{t,i}^{\mathsf{T}} \in\mathbb{R}^{d_k \times d_v} \tag*{(15)}

where βi=Wβht,i∈R\beta_i = W^\beta h_{t,i} \in\mathbb{R}, αi=Wαht,i∈Rdk\alpha_i = W^\alpha h_{t,i} \in\mathbb{R}^{d_k}, ki=WKht,i∈Rdkk_i = W^K h_{t,i} \in\mathbb{R}^{d_k}, and vi=Wvht,i∈Rdvv_i = W_v h_{t,i} \in\mathbb{R}^{d_v} and WβW^\beta, WαW^\alpha, WKW^K, WVW^V are learned projections. Thus with KV-sharing KDA models, we discard all St,iS_{t,i} where t<Tt < T and all ii up to our current decode token.

Extended Compression and Patching Results for Newly Pretrained Models

In Table 12 we explore the KV-sharing ability of the base pretrained Parcae models introduced in § 5.1 (Table 4, see Section N for configurations and Section O for hyperparameters). All four MHA models have KV-sharing errors. Preserving the first-layer KVs from the first recurrence in addition to normal KV-sharing maintains lossless quality for all models except the 1.3B model. The 1.3B model also has further quality degradation when using Naive Patching. Interestingly, in Table 4 using BeNice! on the 1.3B model leads to near lossless quality, performing much better than the sub-optimal (in terms of memory efficiency) additional first layer, first recurrence KV-cache during KV-sharing.

ModelCore (↑)Core-Extended (↑)
UnsharedSharedΔUnsharedSharedΔ
140MParcae15.5014.97-0.5310.4310.10-0.33
+ BeNice!15.4715.61+0.1410.2810.19-0.09
370MParcae20.7415.16-5.5812.999.68-3.31
+ BeNice!23.9023.51-0.3914.3914.24-0.15
770MParcae30.4526.92-3.5219.2216.67-2.55
+ BeNice!30.7930.53-0.2619.1819.13-0.05
1.3BParcae37.2733.94-3.3325.4123.46-1.95
+ BeNice!37.4037.43+0.0425.6925.59-0.10

Table 4. BeNice! Maintains Quality and Enables KV-Shareability. We train Parcae models [50] and then post-tune with BeNice!. BeNice! maintains and improves base model quality, while fully enabling KV-sharing. Best results are bolded.

ModelUnsharedLayer > 1, Recurrence > 1Naive PatchingShared
MHA140M15.5115.5015.5216.99
MHA370M11.5711.5611.563723
MHA770M9.789.8010.4936.66
MHA1.3B8.588.9212.779.42
MLA140M15.5615.5415.5515.55
MLA370M11.6711.6411.6622.72
MLA770M9.759.7511.304894
KDA140M16.0216.0015.9915.99
KDA370M11.7411.7311.7311.77
KDA770M10.1310.1310.1310.16

Table 12. KV-Sharing Failures on New Models. Preserving first-layer KVs from the first recurrence matches the unshared baseline across model sizes and sequence mixers. Mean patching fails to fully recover quality for MLA at 770M. KV-sharing fails for MLA models as size increases. Surprisingly, KDA models have KV-sharing out-of-the-box.

We also explore the KV-sharing ability of MLA and KDA variants of Parcae in Table 12. MLA variants of Parcae have the same failure mode of KV-sharing as released Parcae models. Mean patching fails to recover quality for our MLA model at 770M size. KDA models perform well on KV-sharing out-of-the-box.

Optimal Hyperparameter Transferability Across Recurrence

While several works have shown how optimal hyperparameter transfer varies with tokens [9, 59] and model size [72], we explore how scaling recurrence [50] is affected. Specifically, in this section, we investigate how optimal hyperparameters (e.g., learning rate, batch size, etc.) are affected by increasing recurrence at multiple parameter classes.

Setup. We follow the model configuration and architecture backbone of Prairie et al. [50] (see Section N for extend details), and train using ClimbMix [16]. We evaluate the transferability of learning rate, momentum, warm-up ratios, and batch size across μrec∈{2,4,8}\mu_{\mathrm{rec}} \in\{2, 4, 8\} using 140M, 370M, and 770M parameter backbones. To ensure robustness, for each hyperparameter, recurrence, backbones data point, we train three models using different seeds, 42, 1234, 2345, and then use the average of these runs. This totals to 504 runs for this ablation and approximately 4,685 H100 hours.

Optimal Hyperparameters Transfer Strongly Across Increased Recurrence. As seen in Figure 14 and Figure 15, we find that across all model classes optimal learning rate and batch size transfer strongly. We further find this to be true of other minor hyperparameters (e.g., muon momentum and warmup ratio), with some minor variation in flat regions. This finding implies that lower recurrence (i.e., μrec=2\mu_{\mathrm{rec}} = 2) can be used to cheaply sweep for optimal hyperparameters, which in general remain optimal at higher recurrences. drastically reduces the necessary FLOPs required to find optimal hyperparameters for looped architectures.

Figure 14. Transferability of Learning Rate. Optimal learning rate transfers strongly across deeper recurrences for all model backbones.

Figure 15. Transferability of Batch Size. Optimal batch size transfers strongly across deeper recurrences for all model backbones.

Figure 16. Transferability of Muon Momentum. We observe optimal muon momentum to transfer well across recurrences, with some minor variation in relatively flat regions.

Furthermore, as seen in Table 13, we observe a high Spearman rank correlation between the validation BPB across hyperparameter sweeps at different recurrences. While Spearman correlation is not 1 for all hyperparameters, we find that this is derived from noisy and flat hyperparameter landscapes. Thus, generally, we observe that most hyperparameters appear to transfer across recurrences, motivating our use of extremely low recurrence for hyperparameter tuning.

HyperparameterModel2⟺42 \Longleftrightarrow42⟺82 \Longleftrightarrow84⟺84 \Longleftrightarrow8MeanΔ\DeltaBPB (×10−3\times10^{-3})
Learning Rate140M0.9500.9670.9000.93941.805
Learning Rate370M0.9431.0000.9430.96226.483
Learning Rate770M0.8290.8291.0000.88618.373
Muon Momentum140M0.9000.7000.9000.83337.269
Muon Momentum370M1.0001.0001.0001.00025.083
Warmup Fraction140M0.6000.9000.7000.73322.614
Warmup Fraction370M0.7000.7001.0000.80019.451
Batch Size140M1.0001.0001.0001.000128.046
Batch Size370M1.0001.0001.0001.00070.462
Batch Size770M1.0001.0001.0001.00047.777

Table 13. Spearman Rank Correlation of Hyperparameter curves across Recurrence. Here x⟺yx \Longleftrightarrow y is the Spearman correlation between the validation BPB hyperparameter curves of a model trained with μrec=x\mu_{\mathrm{rec}} = x and a model trained with μrec=y\mu_{\mathrm{rec}} = y. Furthermore, Δ\Delta BPB indicates the validation BPB span across all (recurrence schedule ×\times hyperparameter values) points for that specific parameter class. While Spearman correlation is not 1 in all transfers, we observe that low Spearman correlation coincides with a lower Δ\Delta BPB span, which resolves to slight noise on a relatively flat landscape for hyperparameter tuning.

Figure 17. Transferability of Warmup Ratio. We observe optimal warmup ratios to transfer well across recurrences, with some minor variation in relatively flat regions.

Model Definitions and Configurations

In this section, we discuss the model definition used for our experiments. For our experiments done in § 3 for our Stochastic and Static comparison, the models follow the configuration and hyperparameters presented in Table 14 and Section O respectively, with two notable exceptions. First, for both models, we use a 1:7 param to token ratio. Second, for the Static depth models, recurrence depth sampling does not follow the Poisson distribution, and is instead held deterministically at μrec=8\mu_{\mathrm{rec}} = 8 recurrences.

Small (140M)Medium (370M)Large (770M)XLarge (1.3B)
ArchitectureLayers in P\mathcal{P}2468
ArchitectureLayers in R\mathcal{R}2468
ArchitectureLayers in C\mathcal{C}2468
Architecturedmodeld_{\mathrm{model}}7681,0241,2801,536
Architecturedintermediated_{\mathrm{intermediate}}3,0724,0965,1206,144
MHAAttentionCausal Self-Attention [68] w/ QK-Norm [26]
MHAAttention Heads681012
MHAHead Dimension128
MHAPos. Embed.RoPE (θ\theta=50,000) [61]
MLAAttentionCausal Multi-Head Latent Attention [13]
MLAAttention Heads681012
MLAKV LoRA Rank512
MLAQK NoPE Head Dimension128
MLAQK RoPE Head Dimension64
MLAV Head Dimension128
MLAPos. Embed.RoPE (θ\theta=50,000) [61]
KDAAttentionCausal Kimi Delta Attention [65]
KDAAttention Heads681012
KDADecay GateLower bound at −5-5
KDAHead Dimension128
KDAConvolution Kernel Size4
Shared DetailsMLPReLU2^{2} [76]
Shared DetailsValue EmbeddingsGated, alternating layers [66]
Shared DetailsVocab Size32,768
Shared DetailsNormRMS-Norm (Pre-Norm) [75]
Shared DetailsContext Length2,048
Shared DetailsBiasNone
Shared DetailsInit.Scaled-zero [34, 62]
Shared DetailsTied EmbeddingsYes
ParcaeInjectionDiagonal
ParcaeState Init.like-init [20]
Parcaeμrec\mu_{\mathrm{rec}}8
ParcaeBackprop Depth4
ParcaeSamplingPoisson (truncated, per-sequence)

Table 14. Model definitions of baseline Parcae [50], Parcae with MLA, and Parcae with KDA.

Our models present in pretraining results in § 5.1 also use the model configurations in Table 14 along with the hyperparameters present in Table 16. Our optimizer parameter group assignment for all models (including those present in § 3) can be found in Table 15.

OptimizerParameters
AdamW [36]Token embeddings (wte)
AdamW [36]LM head (lm_head)
AdamW [36]Normalization layers (RMSNorm)
AdamW [36]Value embedding gates (ve_gate)
AdamW [36]All 1D parameters
AdamW [36]Injection parameters (AA, Δ\Delta, BB)
AdamW [36]Readout projection (CC)
Muon [32]Attention projections (WQW_Q, WKW_K, WVW_V, WOW_O)
Muon [32]MLP weights (WfcW_{\mathrm{fc}}, WprojW_{\mathrm{proj}})

Table 15. Optimizer parameter group assignment for Parcae.

Small (140M)Medium (370M)Large (770M)XLarge (1.3B)
AdamW and Muon Learning Rate2e−22e^{-2}1.2e−21.2e^{-2}6e−36e^{-3}5e−35e^{-3}
Warmup Ratio0.05
Constant Decay Ratio0.2
Gradient Clipping1.0
Muon Momentum0.95
Muon Weight Decay0.3
Muon Orthogonalization Steps5
AdamW Betas(0.8,0.95)
AdamW Weight Decay0.1
AdamW Epsilon1e−101e^{-10}
Batch Size256
Context Length2048
Param to Token Ratio1:80
Precisionbf16-mixed

Table 16. Pretraining Hyperparameters for climbmix trained Parcae models (including MLA and KDA variants).

Hyperparameters and Training Details

In this section, we will discuss the hyperparameter configuration used in § 5.1.

Pretraining. All pretraining hyperparameters are present in Table 16, with the notable exception of the Stochastic and Static recurrence depth models present in § 3. Those exceptions are explained in Section G.

BeNICE! and Standard Finetuning. For both BeNICE! and Standard Finetuning, we sweep over learning rates, lr∈{5e−6,1e−5,2e−5,3e−5,5e−5,8e−5,1.25e−4,2e−4,3e−4,4.5e−4,6e−4}\mathrm{lr} \in\{5e^{-6}, 1e^{-5}, 2e^{-5}, 3e^{-5}, 5e^{-5}, 8e^{-5}, 1.25e^{-4}, 2e^{-4}, 3e^{-4}, 4.5e^{-4}, 6e^{-4}\} and γ∈{0.01,0.03,0.1,0.3,1,3}\gamma\in\{0.01, 0.03, 0.1, 0.3, 1, 3\}. We perform these sweeps with an optimizer budget of 500 steps. For Standard Finetuning, we find 770M models (across MHA, MLA, and KDA) to perform best at a learning rate of 2e−42e^{-4}, while the 1.3B models perform best at 8e−58e^{-5} on a held out validation set. For BeNICE! we find all models perform best at a learning rate of 1.25e−41.25e^{-4} and with a γ\gamma of 0.01 on a held out validation set. Since we sweep over learning rates with an optimizer budget of 500 steps, we rescale our learning rate to account for the increase in optimizer budget (1800 steps). Specifically, all final learning rates are lroptim∗5001800\mathrm{lr}_{\mathrm{optim}} * \frac{500}{1800}.

BeNice!{sup}† Finetuning. For BeNice!{sup}†, we perform a more limited hyperparameter sweep on a limited model set: the released Parcae 770M and 1.3B models. We sweep over lr∈{3e−4,1e−3}lr \in\{3e^{-4}, 1e^{-3}\} and γ∈{0.06,0.12,0.25,1}\gamma\in\{0.06, 0.12, 0.25, 1\} for the full 1000 step optimizer budget. For both models, a learning rate of 1e−31e^{-3} and a γ\gamma of 0.06 performed best on a held out validation set. We use a rank 8 adapter with an α=16\alpha= 16.

References

  1. [1]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.arxiv.org/abs/2305.13245
  2. [2]Ibrahim Alabdulmohsin and Xiaohua Zhai. Recursive inference scaling: A winning path to scalable inference in language and multimodal systems. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2502.07503
  3. [3]Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.arxiv.org/abs/1905.13319
  4. [4]Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation, 2025. URL https://arxiv.org/abs/2507.10524.
  5. [5]Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems, volume 32, 2019.
  6. [6]Daniel Bailey, Amelia Harrison, Yuliya Lierler, Vladimir Lifschitz, and Julian Michael. The winograd schema challenge and reasoning about correlation. In Working Notes of the Symposium on Logical Formalizations of Commonsense Reasoning. AAAI Press, 2015. URL http://www.cs.utexas.edu/users/ai-lab?wsc15.
  7. [7]Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=PPjSKy4OXUB.
  8. [8]Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on Artificial Intelligence, volume 34, 2020.arxiv.org/abs/1911.11641
  9. [9]Johan Björck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal lr across token horizons. In The Thirteenth International Conference on Learning Representations, 2025.
  10. [10]Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, and James Cheng. Hista and numca: Estimate state value effectively for llm reinforcement learning, 2026. URL https://arxiv.org/abs/2605.29782.
  11. [11]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.arxiv.org/abs/1905.10044
  12. [12]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  13. [13]DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunyuan Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. URL https://arxiv.org/abs/2405.04434.
  14. [14]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HyzDriR9Y7.arxiv.org/abs/1807.03819
  15. [15]Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu, Yuanyuan Xu, Jiarui Liu, T S Eugene Ng, and Hanjie Chen. LT2: Linear-time looped transformers. arXiv [cs.LG], May 2026.
  16. [16]Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan, Lin, Jan Kautz, and Pavlo Molchanov. Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training, 2025. URL https://arxiv.org/abs/2504.13161.
  17. [17]Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJg7KhVKPH.
  18. [18]Hengyu Fu, Tianyu Guo, Zixuan Wang, Hanlin Zhu, Jason D Lee, Jiantao Jiao, Stuart Russell, and Song Mei. DiscoLoop: Looping discrete embeddings and continuous hidden states for multi-hop reasoning. arXiv [cs.CL], July 2026.
  19. [19]Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, and Bryan Dai. Loop the loopies!, 2026. URL https://arxiv.org/abs/2607.16051.
  20. [20]Jonas Geiping, Sean Michael McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=S3GhJooWIC.arxiv.org/abs/2502.05171
  21. [21]Jonas Geiping, Xinyu Yang, and Guinan Su. Efficient parallel samplers for recurrent-depth models and their connection to diffusion language models, 2025. URL https://arxiv.org/abs/2510.14961.
  22. [22]Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret, editors, *SEM 2012: The First Joint Conference on Lexical and Computational Semantics—Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394–398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1052/.
  23. [23]Alex Graves. Adaptive computation time for recurrent neural networks, 2017. URL https://arxiv.org/abs/1603.08983.
  24. [24]Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu, Reyna Abhyankar, Qingyang Wu, Austin Silveria, Pragaash Ponnusamy, Jue Wang, Ben Athiwatkun, Leon Song, Tri Dao, Daniel Y. Fu, and Chris De Sa. Search your block floating point scales! In Proceedings of Machine Learning and Systems, 2026.
  25. [25]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
  26. [26]Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.arxiv.org/abs/2010.04245
  27. [27]Coleman Hooper, Sehoon Kim, Hiwa Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2401.18079
  28. [28]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
  29. [29]Benhao Huang, Zhengyang Geng, and Zico Kolter. Equilibrium reasoners: Learning attractors enables scalable reasoning. In Proceedings of the 43rd International Conference on Machine Learning, 2026.
  30. [30]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, 2019.arxiv.org/abs/1909.06146
  31. [31]Alexia Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks, 2025. URL https://arxiv.org/abs/2510.04871.
  32. [32]Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
  33. [33]kaggle200000Jeopardy. 200,000+ Jeopardy! Questions, 2019. URL https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions.
  34. [34]Andrej Karpathy. nanochat: The best chatgpt that $100 can buy, 2025. URL https://github.com/karpathy/nanochat.
  35. [35]Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 2021.arxiv.org/abs/2006.04710
  36. [36]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 2015.arxiv.org/abs/1412.6980
  37. [37]Ian Li, Kapilesh Guruprasad, Raunak Sengupta, Ninad Satish, Loris D’Antoni, and Rose Yu. Manifold-guided attention steering. In Proceedings of the 43rd International Conference on Machine Learning, 2026.
  38. [38]Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-lm: In search of the next generation of training sets for language models. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37, 2024.arxiv.org/abs/2406.11794
  39. [39]Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020.arxiv.org/abs/2007.08124
  40. [40]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 2024.DOI
  41. [41]Sean McLeish, Ang Li, John Kirchenbauer, Dayal Singh Kalra, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Jonas Geiping, Tom Goldstein, and Micah Goldblum. Teaching pretrained language models to think deeper with retrofitted recurrence. arXiv preprint arXiv:2511.07384, 2025.arxiv.org/abs/2511.07384
  42. [42]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.arxiv.org/abs/1809.02789
  43. [43]Seungjae Moon, Jung-Hoon Kim, Junsoo Kim, Seongmin Hong, Junseo Cha, Minsu Kim, Sukbin Lim, Gyubin Choi, Dongjin Seo, Jongho Kim, Hunjong Lee, Hyunjun Park, Ryeowook Ko, Soongyu Choi, Jongse Park, Jinwon Lee, and Joo-Young Kim. Lpu: A latency-optimized and highly scalable processor for large language model inference. IEEE Micro, 2024.arxiv.org/abs/2408.07326
  44. [44]MosaicML. llm-foundry: LLM training and evaluation framework. https://github.com/mosaicml/llm-foundry, 2023.
  45. [45]Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, and Christopher Re. Cost-efficient collaboration between on-device and cloud language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 2025.arxiv.org/abs/2502.15964
  46. [46]NVIDIA. AIPerf: AI performance benchmarking tool. https://github.com/ai-dynamo/aiperf, 2025.
  47. [47]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1525–1534, 2016.arxiv.org/abs/1606.06031
  48. [48]Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, 2022.arxiv.org/abs/2110.08193
  49. [49]Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37, 2024.arxiv.org/abs/2406.17557
  50. [50]Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, and Daniel Y. Fu. Parcae: Scaling laws for stable looped language models, 2026. URL https://arxiv.org/abs/2604.12946.
  51. [51]Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), 2025.
  52. [52]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016.arxiv.org/abs/1606.05250
  53. [53]Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019.arxiv.org/abs/1808.07042
  54. [54]Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.arxiv.org/abs/1804.09301
  55. [55]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
  56. [56]Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiq: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.arxiv.org/abs/1904.09728
  57. [57]Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=Tsp2PL7-GQ.arxiv.org/abs/2106.04537
  58. [58]Kristian Schwethelm, Daniel Rueckert, and Georgios Kaissis. How much is one recurrence worth? iso-depth scaling laws for looped language models. arXiv [cs.LG], April 2026.
  59. [59]Xian Shuai, Yiding Wang, Yimeng Wu, Xin Jiang, and Xiaozhe Ren. Scaling law for language models training considering batch size, 2024. URL https://arxiv.org/abs/2412.01505.
  60. [60]Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkow ycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, and Amanda Dsouza et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
  61. [61]Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
  62. [62]Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike no more: Stabilizing the pre-training of large language models, 2025. URL https://arxiv.org/abs/2312.16903.
  63. [63]Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.arxiv.org/abs/1811.00937
  64. [64]GLM Team. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https://arxiv.org/abs/2508.06471.
  65. [65]Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejiang Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, and Yulun Du. Kimi linear: An expressive, efficient attention architecture, 2025. URL https://arxiv.org/abs/2510.26692.
  66. [66]Rui Tian, Zuxuan Wu, Qi Dai, Han Hu, Yu Qiao, and Yu-Gang Jiang. Resformer: Scaling vits with multi-resolution training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.arxiv.org/abs/2212.00776
  67. [67]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971.
  68. [68]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03762
  69. [69]Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo, Jordi Ros-Giralt, Arash Behboodi, and Fabio Valerio Massoli. Memory-efficient looped transformer: Decoupling compute from memory in looped language models. In Proceedings of the 43rd International Conference on Machine Learning, 2026.
  70. [70]Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model, 2025. URL https://arxiv.org/abs/2506.21734.
  71. [71]Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.arxiv.org/abs/2108.00648
  72. [72]Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems, volume 34, 2021.arxiv.org/abs/2203.03466
  73. [73]Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2311.12424
  74. [74]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.arxiv.org/abs/1905.07830
  75. [75]Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32, 2019.
  76. [76]Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu² wins: Discovering efficient activation functions for sparse llms, 2024. URL https://arxiv.org/abs/2402.03804.
  77. [77]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2312.07104
  78. [78]Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. Analytical reasoning of text. In Findings of the Association for Computational Linguistics: NAACL 2022, 2022.DOI
  79. [79]Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024.arxiv.org/abs/2304.06364
  80. [80]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Endu Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models, 2025. URL https://arxiv.org/abs/2510.25741.

Paper details

vlnc:W_jatm6ggk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents