In this paper
AbstractIntroductionBackgroundPrior Looped ModelsInference Overhead of Looped Language ModelsUnderstanding KV-shareability in Looped ModelsStochastic Depth Enables Contractive Residual DynamicsKV-Sharing Fails to Converge to a Common Fixed PointLearning to Share Across RecurrencesStochastic-Depth Pretraining: Enabling Contractive Recurrent DynamicsBeNice! Post-training: Correcting First Recurrence Residual ErrorsResultsEnabling KV-Shareability while Maintaining End-to-End QualityKV-Shareability is Complementary to Other KV Compression Methods.Saving Memory and Accelerating Inference Workloads with KV-ShareabilityConclusionAcknowledgmentsGlossaryExtended Literature ReviewEvaluation Setup of KV-ShareabilityProof of KV ConvergenceKV-Sharing Error SubspaceParcae and Huginn Converge to Fixed-PointsExpanded Results Comparing Stochastic and Static Sampling Depth TrainingExtended Empirical Analysis on KV-Sharing Fixed Point Convergence FailuresExtended Finetuning SetupLoRA Finetuning for KV-sharingExtended Discussion of Experimental Setup for MLA and KDA EvaluationExtended Compression and Patching Results for Newly Pretrained ModelsOptimal Hyperparameter Transferability Across RecurrenceModel Definitions and ConfigurationsHyperparameters and Training DetailsReferencesWorksComputer Science
BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing
Abstract
Looped architectures scale quality while keeping parameter count fixed by repeatedly passing activations through the same block of layers. During inference, each recurrence generates independent KV-caches, causing memory to grow linearly. To address this challenge, prior work has proposed KV-sharing (i.e., sharing KV-caches across depth), finding that, in some cases, the KV-cache of the final recurrence can be shared with the earlier recurrences without quality loss, a property we denote as KV-shareability. However, the mechanisms underlying KV-shareability remain poorly understood, with even identically trained looped architectures not consistently exhibiting it. In this paper, we study how KV-shareability can be enabled in looped models. We propose a two-stage method: in the first stage, we use stochastic depth during pretraining, which induces contractive residual dynamics that drive representations toward a common limit across recurrences. In the second stage, we propose BeNice!, a post-training method that broadcasts the final recurrence's KV-cache into the first recurrence, training it to produce compatible residuals while preserving the original model's quality. As a result, BeNice! enables KV-shareability while matching standard finetuning's unshared validation perplexity. Using BeNice!, we show that KV-sharing is complementary to alternative KV-compression methods, composing with memory-efficient sequence mixers (e.g., multi-latent attention, Kimi delta attention) and low-precision KV-cache quantization (e.g., fp8, mxfp8, nvfp4). Finally, in simulated low-memory, long-context inference, KV-sharing improves cache hit rates by up to 31.4%, increases throughput by up to 23.6%, and reduces p90 time-to-first-token by up to 83.6%.
Introduction
As inference deployments move to the edge [43, 45] and take up a large portion of compute [67], looped architectures create a new, predictable avenue to scale quality while keeping parameter count fixed [20, 41, 50]. Unfortunately, while looped architectures keep the memory overhead of parameters capped, they still create -caches at each recurrence [19, 50, 80], causing memory overhead to still grow linearly with recurrence count at inference time. One potential direction to reduce inference-memory overhead is -sharing (i.e., sharing -caches across recurrences). Specifically, some recent looped models have been shown to support reuse of the final recurrence’s -cache for all prior recurrences with minimal impact to quality [20, 21], a property we term KV-shareability (Figure 1 [right]).
Figure 1. Stochastic Depth, BeNice!, and KV-Shareability. (Left) During pretraining, stochastic depth induces contractive residual dynamics. (Middle) BeNice! post-trains by injecting the final recurrence KV-caches into the first recurrence. Together, they enable KV-shareability in looped models, retaining only the final recurrence’s KV-cache without degrading quality (Right).
Unfortunately, the mechanisms underlying -shareability remain poorly understood, with more recent work reporting significant quality degradation with -cache reuse [19, 80]. Moreover, even models trained under identical setups can differ in whether they exhibit KV-shareability (see Table 1). In this paper, we hypothesize that contractive recurrent dynamics drive residual states toward fixed points, whose agreement under shared and unshared KV-caches enables KV-shareability.
| Model | Lambada BPB (↓) | Core (↑) | Core-Extended (↑) | ||||||
| Unshared | Shared | Δ | Unshared | Shared | Δ | Unshared | Shared | Δ | |
| Parcae 140M | 0.962 | 0.962 | 0.000 | 14.07 | 14.17 | 0.10 | 10.13 | 10.09 | -0.04 |
| Parcae 370M | 0.762 | 0.764 | 0.002 | 20.47 | 19.85 | -0.62 | 13.23 | 13.24 | 0.01 |
| Parcae 770M | 0.645 | 1.460 | 0.814 | 25.05 | 21.10 | -3.94 | 15.69 | 13.95 | -1.74 |
| Parcae 1.3B | 0.601 | 2.281 | 1.680 | 28.76 | 21.37 | -7.40 | 16.73 | 13.48 | -3.25 |
| Huginn 3.5B | 0.429 | 0.429 | 0.000 | 40.97 | 40.53 | -0.44 | 29.06 | 28.61 | -0.44 |
| Ouro 1.4B | 0.353 | 1.280 | 0.927 | 54.57 | 25.17 | 29.40 | 53.11 | 32.13 | 20.98 |
| Ouro 2.6B | 0.301 | 0.744 | 0.443 | 59.78 | 26.58 | 33.20 | 61.42 | 37.57 | 23.84 |
Table 1. Emergence of KV-shareability in Looped Models. Quality of looped architectures with and without a shared KV-cache. Observe that KV-shareability is not universal to all looped architectures (Bolded models have KV-shareability). See Section C for evaluation details.
To encourage contractive residual dynamics, we train with stochastic depth, randomly sampling depth throughout training (Figure 1 [left]). By requiring low-loss predictions across multiple depths, stochastic depth discourages residual drift and promotes convergence across recurrences. Contractive residual dynamics are what allow KV-shareability, which we formally analyze and empirically validate by showing that replacing intermediate caches with the final cache perturbs the attention output by a term bounded by the difference between their input residuals (see § 3.1).
Unfortunately, even though stochastic depth creates contractive residual dynamics, it does not guarantee that residual states converge to the same fixed point with and without KV-sharing. Instead, KV-sharing sometimes introduces token-independent residual errors in the first recurrence, shifting the model’s fixed point and harming downstream quality (see § 3.2).
To enable KV-shareability, we pair stochastic depth pretraining with a targeted correction of the residual errors introduced by KV-sharing. We propose a two-stage method: first, stochastic depth pretraining to induce contractive residual dynamics; second, BeNice! to correct first recurrence residual errors created from KV-sharing. Specifically, BeNice! jointly optimizes both the first recurrence with KV-cache reuse and the final recurrence without reuse (Figure 1 [middle]), balancing KV-shareability and base-model quality. To further study KV-sharing with other KV-compression techniques, we extended BeNice! to memory-efficient sequence mixers, including multi-latent attention (MLA) [13] and Kimi delta attention (KDA) [65], by reusing the final recurrence’s latent or state, respectively. Overall, BeNice! unlocks near-lossless KV-cache reuse across recurrences, facilitating predictable quality scaling using extended looping [50] while keeping parameter count and KV-cache memory constant as depth grows.
We evaluate KV-shareability and BeNice! on end-to-end quality, compatibility with other KV compression methods, and how it accelerates inference workloads:
End-to-End Quality. BeNice! improves shared quality by up to 8.35 and 4.56 points for Core and Core-extended [38], respectively. Compared with standard finetuning, BeNice! improves validation perplexity by 9.75 points when sharing, while matching quality in the unshared case.
Complementary to other KV Compression Methods. To evaluate KV-sharing with other KV-compression methods, we study its composition with memory-efficient sequence mixers and KV-quantization. We find that KV-sharing composes with both MLA and KDA sequence mixers and low-precision KV-quantization (e.g., fp8, mxfp4, nvfp4).
Accelerating Inference Workloads. Under simulated long-context inference traffic, we observe
that memory reductions in KV-sharing improve cache-hit rates by up to 31.4% (8.1 points), increasing throughput by up to 23.6%, and reducing p90 TTFT by up to 83.6%.
Background
We first provide a brief background on prior looped models (§ 2.1) and the inference overhead of language models (§ 2.2). Looped architectures have been widely explored in synthetic problem solving tasks [7, 29, 31, 57, 70]; however, our work focuses on looped language models [14, 20, 50, 80]. These models differ in where recurrence occur (pre-, mid-, post-, or full-looping) [2], how it is halted (i.e., explicit router [80] or implicitly within the residual [20, 50]), and how gradients are computed (explicit or implicit backpropagation [5]). We study middle-looped models using implicit halting and explicit differentiation, with an emphasis on how pretraining for fixed-point convergence enables memory-efficient inference. An extended literature review is found in Section B.
Prior Looped Models
Most prior looped language models [20, 50, 80] follow a middle-looped structure sequentially composed of an initial prelude block , a middle recurrent block , and a final coda block . For an input sequence , computation proceeds sequentially as , , , where a model is looped times and is initialized randomly (i.e., ; [20, 50]) or with (i.e., ; [80]). Intuitively, maps a sequence into a latent embedding , iteratively refines a hidden state conditioned on , and then maps the final to an output . To condition on , input injection at each recurrence is often used (i.e., , where and ; [20, 50, 73]), and following Prairie et al. [50] is constrains where is the spectral radius, to prevent state explosion.
During training, prior works either sample depth stochastically [20, 50] or use adaptive computation time (ACT) [23]. Specifically, Parcae [50] and Huginn [20] sample a depth independently per sequence/batch, where is a distribution (e.g., Poisson) with mean , training to produce meaningful representations across depths. Ouro [80] uses ACT: a learned router predicts a halting probability at each recurrence, with computation stopping once the cumulative probability exceeds a threshold [23]. We will show the advantages of training with stochastically sampled depth in this work. Finally, to bound training memory, some models use truncated backpropagation through time (TBPTT) [20, 50], propagating gradients through only the final recurrences.
Inference Overhead of Looped Language Models
We will study how memory overhead at inference scales with context length and model depth, particularly within the scope of looped architectures [20, 50]. Under uniform numerical precision, inference memory for a -parameter model, with context length, depth, and KV width is , where model-weight memory is and KV-cache memory is . Beyond lowering numerical precision, prior work reduces overhead primarily by shrinking per-layer footprint (e.g., MLA [13], GQA [1], KDA [65]), leaving its linear dependence on depth unchanged. For a looped model evaluated with recurrences, the effective depth is . Although looping keeps fixed as increases, conventional inference stores a distinct KV cache for each recurrence, causing to grow linearly with .
Understanding KV-shareability in Looped Models
In this section, we study KV-shareability in looped architectures. To evaluate KV-shareability, we compare shared evaluations, which reuse the final-recurrence cache across all recurrences, against unshared evaluations, which retain every recurrence’s cache. Details on reproducing accurate KV-sharing inference setups for our benchmarks are provided in Section C.
We motivate our investigation by observing that KV-shareability is not universal to all looped models (see Table 1). We establish that stochastic depth enables contractive recurrent dynamics, which we formally analyze and empirically validate allows KV-shareability when fixed points agree under shared and unshared KV-caches (§ 3.1). However, stochastic depth alone is insufficient to guarantee agreement, leading to models not exhibiting KV-shareability (e.g., Parcae 1.3B in Table 1). Analyzing failure cases, we find that KV-sharing creates a token-independent basis shift, causing convergence to different fixed points and degraded quality under KV-sharing (§ 3.2).
Stochastic Depth Enables Contractive Residual Dynamics
In this section, we characterize how stochastic-depth training induces fixed-point convergence, driving KV-caches near a common limit and correlating with the emergence of KV-shareability.
Setup. We train Parcae-style [50] models from scratch at four scales (140M, 370M, 770M, and 1.3B), varying only whether recurrence depth is stochastic (i.e., randomly sampled during training) or static (i.e., fixed throughout training). KV-shareability is evaluated on final models and never trained for explicitly. Training and evaluation details are in Section G and Section C, respectively.
Fixed-Point Convergence encourages Shareable KV-Caches. Our key insight is that KV-shareability often emerges as a downstream consequence of fixed-point convergence in the recurrent block . Specifically, consider a looped architecture in which is a contraction on the hidden state , such that where is a fixed point (e.g., Parcae [50]; Huginn [20]; see Section F for empirical evidence). Since the key and value projections are locally-Lipschitz in , their caches also converge: for every layer in . Moreover, is bounded by the step-to-step residual (see Section D), as illustrated in Figure 2.
Figure 2. Residuals and Attention Convergence. Relative step-to-step change of the residual stream (, left) and attention pattern (, right) across recurrence depth . Stochastic-depth training exhibits fixed-point convergence, while static-depth does not.
KV-cache convergence is necessary for KV-shareability: replacing an intermediate cache with a fixed-point cache perturbs the attention output by a term bounded by the fixed-point residual . To test this, we train Parcae models from scratch using stochastic and static depth training. Stochastic depth training induces a contraction1 and coincides with the emergence of KV-shareability (Table 2). In contrast, static-depth lacks an incentive to keep intermediate latents near a common limit, as reflected in their attention patterns (Figure 2) and shared quality (Table 2).
| Size | Val PPL () | Core () | ||
| Stochastic | Static | Stochastic | Static | |
| 140M | -0.02 | 1.31 | 0.08 | 0.62 |
| 370M | 0.02 | 2.70 | 0.13 | 0.78 |
| 770M | 0.01 | 3.10 | 0.13 | 1.06 |
| 1.3B | 0.01 | 2.84 | 0.25 | 2.00 |
Table 2. KV-Sharing Relies on Stochastic Depth. We report the quality delta between unshared and shared KV-caching. Extended results and training details are in Section G.
Consistent with this mechanism, the final-recurrence cache is empirically the optimal broadcast target (Figure 3), as lie closest to a fixed point across all , minimizing substitution error. While KV-shareability is still surprising, it suggests that looped architectures should favor stochastic fixed-point training over ACT objectives. By inducing a contraction towards a common fixed point, stochastic-depth training can enable KV-shareability without explicitly optimizing for it.
Figure 3. Final-Recurrence KV is Optimal. We find the best quality results are from converged KV caches.
KV-Sharing Fails to Converge to a Common Fixed Point
We now investigate why contractive residual dynamics alone are insufficient (e.g., 770M and 1.3B Parcae use stochastic depth and are fixed-point [Section F], yet lack KV-shareability [Table 1]). Setup. We focus our analysis on the released Parcae models [50], focusing on 770M and 1.3B scales that both fail to preserve quality under KV-sharing, yet were trained with stochastic depth and an equivalent setup to 140M and 370M Parcae models. We evaluate them using the setup in Section C.
KV-Sharing Fails to Converge to Similar Fixed Points. Figure 4 visualizes the converged representation with and without KV-sharing. We quantify the fixed-point agreement by the mean relative difference between final latents, . Models with KV-shareability (e.g., Parcae 140M and 370M in Table 1) converge to similar fixed points, with a mean relative difference of at most 1.69%. In contrast, models without KV-shareability (e.g., Parcae 770M and 1.3B in Table 1) exhibit mean relative differences reaching up to 177.5%.
Figure 4. KV-Sharing Failures Converge to Incorrect Fixed Points. We visualize fixed points of several tokens after test-time recurrence with unshared and shared KV-caches. Models with KV-shareability (e.g., 140m and 370m) converge to the same fixed point under both setups, while models without KV-shareability (e.g., 770M and 1.3B) do not. See Section H for extended details.
KV-Sharing Failures are Token-Independent Basis Shifts Analyzing the source of KV-sharing failure, we observe that caching only the first layer KV of the first recurrence, on top of all final recurrence caches, restores quality (i.e., Layer > 1, Recurrence > 1 in Table 3). This localizes the failure to just the first layer of the first recurrence. More surprisingly, this failure manifests as a token-independent basis shift, altering the fixed point under KV-sharing and degrading quality.
| Cache Compression | Val. PPL. () | |
| 770M | 1.3B | |
| Unshared | 11.86 | 10.80 |
| Layer > 1, Recurrence > 1 | 11.86 | 10.80 |
| Naive Patching | 11.87 | 10.84 |
| Shared | 34.24 | 1.71e4 |
Table 3. Pinpointing Parcae Failures. Preserving first-layer KVs from the first recurrence or applying naive patching matches the unshared baseline, whereas full KV-sharing fails.
Table 3 and Figure 5 confirm this empirically: by constructing a subspace from the error delta between the first layer output with and without KV-sharing (see Section E for derivation) and projecting each token into , we observe an average projected pairwise cosine similarity of 0.99.
Figure 5. KV-Sharing Error Subspace Similarity. We find that tokens are strongly aligned within the subspace, but orthogonal in its complement and a random 8-dimensional subspace.
This confirms that KV-sharing errors act as a token-independent residual basis shift. To test this, we add the mean projected error to the residual after the first layer during shared evaluations (see Section E for details). This correction eliminates the need to separately cache the first layer and recurrence when sharing (see Naive Patching in Table 3).
Learning to Share Across Recurrences
One way to induce KV-shareability is to train directly on KV-sharing rollouts (i.e., sequentially rolling out KV-sharing dynamics token-by-token). However, this would be computationally infeasible and slow during training. Alternatively, using our analysis in § 3, we propose a two-stage method to enable KV-shareability (as visualized in Figure 1). In the first stage, we follow our observations in § 3.1 and pretrain models with stochastic depth to induce contractive residual dynamics that drive KV-caches to a common limit across recurrences (§ 4.1). In the second stage, we correct the token-independent basis shift observed in § 3.2 with BeNice! by optimizing both the first recurrence with KV-cache reuse and the standard final recurrence without reuse (§ 4.2).
Stochastic-Depth Pretraining: Enabling Contractive Recurrent Dynamics
In the first stage of our method, we train with stochastic depth, driving KV-caches to a common shareable limit across recurrences and setting the groundwork for KV-shareability based on § 3.1. Specifically, following Prairie et al. [50], we utilize per-sequence stochastic depth training, assigning a different depth for each sequence in a batch. This sampling structure was empirically observed to reduce loss spikes in prior work. We further follow Prairie et al. [50], sampling per-sequence depth from , where is a Poisson distribution with mean . Lastly, to keep memory bounded during pretraining, we use truncated backpropagation through recurrences, with a backprop depth of [50].
BeNice! Post-training: Correcting First Recurrence Residual Errors
In the second stage of our method, we correct token-independent issues that still emerge when KV-sharing, even with stochastic-depth training. Using our analysis from § 3.2, we know quality degradations are primarily introduced in the first recurrence (see Table 3). This suggests that KV-sharing compatible inference can be recovered by training the first recurrence pass to be compatible with future KV-caches, rather than directly creating KV-sharing rollouts. Specifically, consider the ordinary -recurrence predictor and a hybrid predictor that uses first-recurrence queries and residuals with final-recurrence KV (see Figure 1 [middle]). We define the BeNice! loss as:
where denotes cross-entropy loss and controls auxiliary loss strength. Intuitively, the auxiliary term trains the first recurrence to be compatible with future recurrences’ KV-caches, while the standard language-modeling term maintains base quality at later depths. Finally, during finetuning, we sample from the same recurrence-depth distribution used during pretraining.
Additionally, to study if KV-sharing is complementary to other KV compression methods, we further generalize BeNice! to multiple memory-efficient sequence mixers. For example, we can extend BeNice! to multi-latent attention [13] and Kimi delta attention [65] by swapping the final recurrence’s latent or hidden state, respectively, over the KV-cache (see Section K for details).
Results
We evaluate end-to-end quality of finetuning methods for enabling KV-shareability (§ 5.1), its composition with other KV compression techniques (§ 5.2), and its latency and throughput gains in memory-bound setups (§ 5.3). Overall, we find BeNice! enables KV-shareability, improving shared Core and Core-Extended [38] quality by up to 8.35 and 4.56 points, respectively. Furthermore, BeNice! is complementary to memory-efficient sequence mixers (e.g., MLA [13], KDA [65]) and KV-quantization. Lastly, in synthetic memory-constrained inference workloads, KV-shareability improves cache-hit rate by up to 31.4% (8.1 percentage points), increases throughput by up to 23.6%, and reduces p90 TTFT by up to 83.6%.
Enabling KV-Shareability while Maintaining End-to-End Quality
On newly trained Parcae [50], BeNice! enables KV-shareability and improves base quality. BeNice! achieves comparable results to standard finetuning, while enabling KV-shareability.
Setup. We follow Parcae [50], training 140M, 370M, 770M, and 1.3B model classes (see Section N for definitions) on over 100B climbix [16] tokens. Baselines [50] are tuned by sweeping hyperparameters, using our hyperparameter-transfer observations in Section M. We finetune BeNice! with three seeds, and then evaluate using a test-time recurrence of , following Prairie et al. [50]. Individual hyperparameter sweeps are done for standard finetuning and BeNice! (see Section O).
BeNice! Enables KV-Shareability and Maintains Base Model Quality. Table 4 shows that KV-sharing failures emerge at all scales, reducing Core and Core-Ext [38] by up to 5.58 and 3.31 points, respectively. Finetuning with BeNice! further improves base model quality; it also enables KV-shareability, improving shared Core and Core-Ext by up to 8.35 and 4.56 points, respectively.
Overall, BeNice! allows looped architectures to scale recurrences to improve quality without scaling memory overhead at test-time.
BeNice! Matches Standard Finetuning and Gains KV-Shareability. We further compare BeNice! to standard finetuning (i.e., without an auxiliary loss term). As seen in Table 5, BeNice! achieves comparable quality to standard finetuning when no KV-cache reuse is done (i.e., unshared). However, BeNice! enables KV-shareability, reducing validation perplexity by up to 45.2% (9.75 PPL points), when KV-caches are reused across recurrences.
| Model | Val. PPL () | Core () | Core-Extended () | ||||||
| Unshared | Shared | Unshared | Shared | Unshared | Shared | ||||
| 770M Standard FT | 11.86 | 21.61 | +9.75 | 24.74 | 23.42 | -1.32 | 15.34 | 14.54 | -0.80 |
| 770M BeNice! | 11.85 | 11.86 | +0.01 | 25.15 | 25.19 | +0.04 | 15.35 | 15.29 | -0.06 |
| 1.3B Standard FT | 10.79 | 18.52 | +7.73 | 29.01 | 27.49 | -1.51 | 17.23 | 16.66 | -0.58 |
| 1.3B BeNice! | 10.79 | 10.78 | -0.01 | 28.63 | 28.67 | +0.04 | 17.32 | 17.41 | +0.09 |
Table 5. Comparing BeNice! with Standard Finetuning. We finetune public checkpoints of Parcae [50] on three different seeds, evaluating on fineweb-edu [49] validation and Core and Core Extended [38]. BeNice! matches the quality of standard finetuning, while enabling KV-shareability.
KV-Shareability is Complementary to Other KV Compression Methods.
In this section, we highlight how KV-shareability is complementary to alternative KV compression methods, by composing it with memory-efficient sequence mixers (e.g., MLA [13] and KDA [65]) and KV-quantization.
Setup. For experiments on KV-compression, we use Parcae models from § 5.1. For experiments composing KV-sharing with memory-efficient sequence mixers, we train models with multi-head attention (MHA) [68], multi-latent attention (MLA) [13] and Kimi delta attention (KDA) [65] (see Section K for more details). For experiments composing KV-sharing with KV-quantization, we directly reduce the precision of keys and values for MHA [68] or the shared latent for MLA [13].
KV-Sharing Composes with Memory-Efficient Sequence Mixers. In Figure 6, base MLA and KDA2 variants of Parcae fail to have KV-shareability, suffering quality drops when sharing final recurrence KV across earlier recurrences. These KV-sharing failures mirror the same pattern as the released MHA Parcae models explored in § 3 (see Section K for more details). BeNice! achieves KV-shareability with all sequence mixers. Across all model scales and sequence mixers, KV-sharing decreases Core and Core-Extended [38] by 0.09 and 0.04 points, respectively, on average.

Figure 6. KV-Shareability with Memory-Efficient Sequence Mixers. We train Parcae [50] models with MHA [68], MLA [13], and KDA [65]. We find that BeNice! allows memory-efficient sequence mixers to have KV-shareability, reducing inference overhead.
KV-Sharing is Compatible with KV-Quantization. In Figure 7, across scales and precisions, KV-sharing changes Core and Core-Ext by only and points on average, respectively, while increasing ClimbMix validation perplexity [16] by just 0.67% on average. KV-sharing and quantization interact by at most 0.53% in climbmix perplexity [16], indicating that KV-sharing is an orthogonal compression axis. We expect stronger KV-quantization methods (e.g., KVQuant [27], KIVI [40], ScaleSearch [24]) will compose with KV-sharing, further improving quality.
Figure 7. Quality of KV-Sharing and -Quantization. We report changes in climbmix PPL [16], Core, and Core-Extended [38] after quantizing the KV-shared cache across MHA [68] model scales. KV-sharing maintains quality across cache precisions.
Saving Memory and Accelerating Inference Workloads with KV-Shareability
In this section, we simulate how memory savings from KV-shareability improve TTFT and throughput under long-context inference workloads.
Setup. For long-context memory experiments, we implement efficient KV-sharing for Parcae [50] in sglang [77]. We report total GPU memory usage, including model weights, KV-cache storage, CUDA graphs, and runtime buffers. For workload experiments, we replay production-derived Mooncake requests [51] using AIPerf [46], preserving arrival times, input/output lengths, and prefix-sharing relations. To study realistic model scales, we construct proxy models from standard open-weights (e.g., GLM 4.7-FP8 [64]) by repeatedly looping the middle third transformer blocks, with and without KV-sharing. We vary both the offered request rate and maximum concurrency to characterize performance across load- and concurrency-limited regimes, respectively.
KV-Sharing Reduces Memory Overhead. In Figure 8, we observe that memory savings scale with recurrence count () and KV-cache size. For our 1.3B Parcae on Core-Extended [38] with batch size and context length of 64 and 2048, respectively, KV-shareability reduces measured global HBM usage by 66.2% from 63.82 GB to 21.60 GB. At , KV-sharing increases the total token capacity to the unshared capacity under the same KV-memory budget.
Figure 8. Global Memory Reduction of KV-Sharing. We measure and fit the memory overhead of KV-sharing, varying test-time recurrences.
KV-Sharing Accelerates Inference Workloads. With reduced memory overhead, KV-shareability increases the total tokens retained under a fixed cache budget (e.g., KV-sharing increases total token budgets by and for and , respectively). Larger token budgets reduce prefix evictions and prefix recomputation (i.e., directly increasing cache hit rate by 8.1 points at 2 req/s in Figure 9). Higher cache-hit rates further improve throughput: at low loads (e.g., 0.5 req/s), KV-sharing matches the unshared baseline in TTFT and throughput; however, at high loads (e.g., 2 req/s), KV-shareability improves throughput by 23.6% and reduces p90 TTFT from 139.2 s to 23.2 s (Figure 10 left). Along the concurrency frontier, KV-shareability provides up to 6.7% higher throughput at a fixed median decode rate and 34.8% higher median decode rate at a fixed throughput (Figure 10 right). Overall, KV-shareability improves throughput and latency when workloads approach saturation.
Figure 9. Cache-Hit Rate Under Sharing. We average the cache-hit rate with and without KV-sharing using Mooncake traces [51].
Figure 10. Throughput Frontier of KV-Sharing. (Left) Under increasing offered load, KV-Sharing improves throughput and reduces TTFT over unshared looping. (Right) Under high concurrency load, KV-sharing improves overall throughput at matched decode rates.
Conclusion
In this work, we study the emergence and failure points of KV-shareability, finding that contractive residual dynamics drive KV-caches towards a common shareable limit. To enable KV-shareability, we propose a two stage method: first, we use stochastic depth to induce contractive residual dynamics across recurrences; second, we provide BeNICE!, a post-training method that corrects residual errors in the first recurrence to ensure that models approach a common limit with and without KV-sharing. KV-shareability composes with both memory-efficient sequence mixers and KV-quantization, detaches inference memory overhead from test-time recurrences, and improves throughput and TTFT in memory-constrained settings.
Acknowledgments
We graciously acknowledge the support of Together AI and AMD through the FAR research grant. This research was supported in part by the NSF under grants 2146151 and 2422214.
Glossary
We include a brief glossary of notations present in this work in Table 6.
| Notation | Description |
| Hidden dimension of the model | |
| Discrete temporal state axis of on | |
| Hyperparameter used to select the strength of the auxiliary loss in BeNice! | |
| Initial prelude block of a recurrent architecture | |
| Middle recurrent block of a recurrent architecture | |
| Final coda block of a recurrent architecture | |
| Input sequence to a model | |
| Output embedding of the prelude block | |
| Hidden embedding of the recurrent block | |
| Mean recurrent forward propagation steps during pre-training | |
| Mean recurrent backward propagation steps during pre-training | |
| Final number of recurrent steps actually taken | |
| Distribution that recurrences are sampled from during training | |
| KV-sharing | Using final recurrence’s KV-cache in early recurrences without quality loss |
| Unshared | Decoding performed with a KV cache per recurrence |
| Shared | Decoding performed using only the final recurrence KV cache (KV-sharing) |
| Output with first recurrence attending to final recurrence KV-cache | |
| Output with first recurrence attending to first recurrence KV-cache (Unshared) |
Table 6. Glossary of notation and terminology. (Top) Frequently used dimensions for tensors. (Middle) Definition of Parcae blocks, tensors, and distributions used to express recurrent depth models. (Bottom) KV-cache compression definitions.
Extended Literature Review
Looped architectures have been and continue to be explored by a growing body of work. We focus on architectural differences. Here, we discuss the main categories of architectural differences, as well as other works on memory efficiency in looped models.
Middle-Looped Models. This framework of models utilizes three blocks: a prelude block () that performs representation extraction using transformer layers, a recurrent block () that recurrently updates activations, and a coda block () that turns activations into outputs. Bansal et al. [7], Geiping et al. [20], McLeish et al. [41], and Prairie et al. [50] all train middle-looped models with stochastically sampled recurrence depth during pretraining. Schwethelm et al. [58] trained middle-looped models with fixed recurrence depth. These architectures tend to be trained with implicit halting mechanisms to determine when to stop looping [7, 20, 41, 50, 58], using either convergence methods like KL divergence between loops [7, 41], low relative differences in successive hidden states [20], or use the mean training recurrence as the optimal inference time recurrence depth [50, 58].
However, Bae et al. [4] is trained with a middle-looped architecture, but uses an explicit halting mechanism of lightweight routers to decide recurrence depth.
Fully Looped Models. This regime of looped models loops the entire model every recurrence. While not strictly a requirement, this regime of architecture tends to pull from Graves [23] work on adaptive compute time (ACT), using explicit halting mechanism to determine when to stop looping. Dehghani et al. [14], Zhu et al. [80] (Ouro), Elbayad et al. [17], Jolicoeur-Martineau [31], and Fu et al. [18] all extend this concept to language models, varying the mechanisms used for explicit halting. These halting mechanisms include prediction heads for dynamically deciding halting probabilities [17, 80] and Q-learning as a means of determining convergence [14, 23, 31]. Most of these models are trained with fixed recurrence depth [14, 17, 18, 80].
KV-sharing and Memory Efficiency in Looped Models. Huginn [20, 21] was noted as having KV-sharing ability out-of-the-box. Bae et al. [4] proposed a KV-sharing variant for their model that shares first recurrence KV-cache, with some quality loss. Deng et al. [15] extends the Ouro[80] style looped model to use with memory-efficient sequence mixers. Vendrell et al. [69] proposed a KV-sharing variant of the Ouro style model that shares the first recurrence KV-cache with later recurrences in addition to maintaining a sliding window KV-cache per recurrence.
Evaluation Setup of KV-Shareability
In this section, we will discuss how we augment benchmarks to evaluate whether a looped architecture can losslessly utilize KV-sharing. We will break benchmarks into two types (i.e., perplexity- and question-answering-based evaluations) and discuss how each is evaluated below.
Perplexity-Based Evaluations. For a standard perplexity benchmark evaluated on a looped architecture with test-time recurrences, the average per-token perplexity is
where denotes the model’s next-token distribution when run for recurrences. Because the full prefix can be teacher-forced in a single parallel forward pass, the per-sequence cost is . To assess whether a looped architecture has KV-shareability, we augment this benchmark in two ways. First, we replace the per-recurrence KV cache with the projection
which broadcasts the final recurrence’s keys and values across all recurrences in the looped block. To recover the standard perplexity evaluation, we can use the operator , where
Finally, as populating the shared cache for the prediction at position requires re-running the entire prefix token-by-token (each token must complete all recurrences so its final-recurrence is written into the shared slot before the next token can read it back), the evaluation no longer admits a single parallel forward pass per sequence, giving
where denotes the prefix processed under the KV-cache reuse projection (i.e., with each token’s per-recurrence keys and values collapsed to the final-recurrence cache before the next token attends back to it). The per-sequence cost is as one factor of comes from the outer position sum, and another from the prefix that must be re-processed for each prediction.
Question-Answering Evaluations. For a question prompt with candidate answers , we rank candidates by their length-normalized conditional negative log-likelihood:
and predict
For KV-cache reuse, we prefill the common prompt once under , replicate the resulting cache across candidates, and decode each candidate token-by-token so that every token writes its final-recurrence keys and values before the next token is scored. For the unshared baseline, all candidate sequences are evaluated with a standard parallel teacher-forced forward pass.
Categorization of Benchmarks. Table 7 summarizes the 46 downstream tasks used in our evaluation, grouped by domain, scoring protocol, and number of in-context examples. CORE comprises the marked 22-task subset, while CORE-Extended includes all 46 tasks.
| Category | Task | Type | Shots | KV Protocol | CORE |
| Understanding | HellaSwag (0-shot) [74] | MC | 0 | QA | ✓ |
| Understanding | HellaSwag (10-shot) [74] | MC | 10 | QA | ✓ |
| Understanding | LAMBADA [47] | LM | 0 | QA/PPL | ✓ |
| Understanding | Winograd WSC [6] | S | 0 | QA | ✓ |
| Understanding | WinoGrande [55] | S | 0 | QA | ✓ |
| Understanding | BIG-Bench Language ID [60] | MC | 10 | QA | ✓ |
| Understanding | BIG-Bench Conlang Translation [60] | LM | 0 | QA | |
| Understanding | BIG-Bench Conceptual Combinations [60] | MC | 10 | QA | |
| World Knowledge | Jeopardy [33] | LM | 10 | QA | ✓ |
| World Knowledge | BIG-Bench QA WikiData [60] | LM | 10 | QA | ✓ |
| World Knowledge | ARC-Easy [12] | MC | 10 | QA | ✓ |
| World Knowledge | ARC-Challenge [12] | MC | 10 | QA | ✓ |
| World Knowledge | MMLU (0-shot) [25] | MC | 0 | QA | |
| World Knowledge | MMLU (5-shot) [25] | MC | 5 | QA | |
| World Knowledge | BIG-Bench Misconceptions [60] | MC | 10 | QA | |
| Commonsense | COPA [22] | MC | 0 | QA | ✓ |
| Commonsense | CommonsenseQA [63] | MC | 10 | QA | ✓ |
| Commonsense | PIQA [8] | MC | 10 | QA | ✓ |
| Commonsense | OpenBookQA [42] | MC | 0 | QA | ✓ |
| Commonsense | SIQA [56] | MC | 10 | QA | |
| Commonsense | BIG-Bench Novel Concepts [60] | MC | 10 | QA | |
| Commonsense | BIG-Bench Strange Stories [60] | MC | 10 | QA | |
| Commonsense | BIG-Bench StrategyQA [60] | MC | 10 | QA | |
| Symbolic/Math | BIG-Bench Dyck Languages [60] | LM | 10 | QA | ✓ |
| Symbolic/Math | AGI Eval LSAT AR [78] | MC | 3 | QA | ✓ |
| Symbolic/Math | BIG-Bench CS Algorithms [60] | LM | 10 | QA | ✓ |
| Symbolic/Math | BIG-Bench Operators [60] | LM | 10 | QA | ✓ |
| Symbolic/Math | BIG-Bench Repeat Copy Logic [60] | LM | 10 | QA | ✓ |
| Symbolic/Math | BIG-Bench Elementary Math QA [60] | MC | 10 | QA | |
| Symbolic/Math | BIG-Bench Logical Deduction [60] | MC | 10 | QA | |
| Symbolic/Math | Simple Arithmetic (no spaces) [44] | LM | 10 | QA | |
| Symbolic/Math | Simple Arithmetic (with spaces) [44] | LM | 10 | QA | |
| Symbolic/Math | MathQA [3] | MC | 10 | QA | |
| Symbolic/Math | LogiQA [39] | MC | 10 | QA | |
| Reading Comp. | SQuAD [52] | LM | 10 | QA | ✓ |
| Reading Comp. | CoQA [53] | LM | 0 | QA | ✓ |
| Reading Comp. | BoolQ [11] | MC | 10 | QA | ✓ |
| Reading Comp. | PubMedQA (labeled) [30] | LM | 10 | QA | |
| Reading Comp. | AGI Eval LSAT RC [79] | MC | 3 | QA | |
| Reading Comp. | AGI Eval LSAT LR [71] | MC | 3 | QA | |
| Reading Comp. | AGI Eval SAT English [79] | MC | 3 | QA | |
| Reading Comp. | BIG-Bench Understanding Fables [60] | MC | 10 | QA | |
| Safety | Winogender MC (Female) [54] | MC | 10 | QA | |
| Safety | Winogender MC (Male) [54] | MC | 10 | QA | |
| Safety | Enterprise PII Classification | MC | 10 | QA | |
| Safety | BBQ [48] | MC | 3 | QA |
Table 7. Downstream evaluation tasks. Tasks marked with ✓ belong to the 22-task CORE suite; all 46 tasks belong to CORE-Extended. Under the QA KV-sharing protocol, the context is prefilled using the final-recurrence cache and candidate continuations are scored against that cache. Under the PPL protocol, used for the separate LAMBADA BPB evaluation, the complete sequence is streamed token-by-token so that every prediction reads only final-recurrence KVs. Type denotes the task scoring rule: MC uses mean candidate NLL, S uses schema-based NLL, and LM requires an exact greedy continuation match.
Proof of KV Convergence
In this section, we make precise the claim in § 3 that fixed-point convergence of the hidden state induces convergence of the per-layer keys and values. Specifically, we show that if the recurrent block is a contraction with a fixed point , then for every attention layer , then at a rate controlled by the step-to-step residual .
Furthermore, we comment on the scope of this result. First, there is no guarantee that looped architectures such as Parcae [50] and Huginn [20] are contractive. However, both prior works empirically observe convergence to fixed points, and we independently verify this behavior in Section F. Second, we are not claiming that self-attention itself is Lipschitz, which has been proved not to be globally Lipschitz on unbounded inputs [35]. Our proof only requires that the per-layer key and value projection, , is Lipschitz. This follows from standard bounds on linear maps composed with the Lipschitz bound of RMSNorm [10, 75].
Theorem 1. Let be -Lipschitz with , and for each layer let be -Lipschitz. Let , , . Then has a unique fixed point , and for every ,
with and as .
Proof. Lipschitz continuity of gives for . Summing,
via Triangle Inequality
via Geometric Sum Closed Form. Lipschitz continuity of (resp. ) then gives , yielding the stated bound. Existence of and convergence follow from Banach’s fixed-point theorem, and continuity of transfers convergence to the keys and values.
KV-Sharing Error Subspace
In this section, we formalize the construction of the KV Sharing Error Subspace. This construction is inspired by [37].
Looking only at the first layer in the recurrent block and only during the first recurrence, let be attention output for tokens attending to only final recurrence KV cache, while
is the attention output using the first recurrence KV for the same tokens (we use tokens).
Then, define as the per-token differences between the attention outputs using first recurrence KV and shared KV.
We perform SVD on . In Figure 11, we observe that variance within is low-rank and well explained with 8 right singular vectors, defining our KV-sharing error subspace.
Figure 11. KV-sharing Error Subspace is Low Rank. The variance within KV-sharing error subspace is well explained with a few directions.
Naive Patching Here we provide a more formal explanation of our Naive Patching work. We take the average across per token differences within the subspace and project it back into the hidden dimension.
Then, during KV-sharing decode, we add the average projected subspace error to the first layer, first recurrence attention output per token.
All other layers receive no patch, and the first layer on all subsequent recurrences receives no patch. Naive Patching is nearly lossless in terms of KV-sharing quality (Table 3).
Parcae and Huginn Converge to Fixed-Points
In this section, we provide evidence that Parcae [50] and Huginn [20] contract to fixed-points.
Setup. We evaluate the publicly released Parcae 140M, 370M, 770M, and 1.3B checkpoints [50], together with Huginn 3.5B [20], without additional training. For each model, we process 1,000 sequences of length 2,048 from the climbmix validation set in BF16. We evaluate Parcae for 15 recurrences and Huginn for its standard 32 recurrences. At each recurrence , we measure the relative change in the recurrent residual state,
and the relative change in attention patterns,
where and index recurrent-block layers and attention heads, respectively.
Results. As shown in Figure 12, residual states and attention patterns stabilize with recurrence depth across all models. Parcae 140M–770M converge rapidly, while Parcae 1.3B converges more slowly; Huginn exhibits the same behavior over its 32-recurrence horizon. Fixed-point dynamics are already present in the original pretrained models and are not specific to our KV-sharing experiments. All of these models were trained with stochastic-depth.
Figure 12. Fixed-point convergence of pretrained looped models. Relative changes in the recurrent residual state and attention pattern decrease with recurrence depth for the released Parcae models and Huginn.
Expanded Results Comparing Stochastic and Static Sampling Depth Training
In this section, we investigate the emergence of KV-sharing by connecting it to fixed-point convergence in looped architectures. Specifically, we train size-matched Parcae models, using either stochastic or static recurrence depth, and compare their residual convergence, attention convergence, and quality under KV-sharing.
Setup. We follow the experimental setup of Prairie et al. [50], using Parcae [50] as the architectural backbone for our experiments (see Section N for model definitions), and trained with and . For stochastic depth training, we use a Poisson distribution following [50], while for static depth training, we forward pass exactly and a truncated backpropagation of exactly . We train on climbmix [16], using approximately 1:7 param-to-token ratios, and using the hyperparameters defined in Section O. Hyperparameters were selected after extensive tuning following Section M. Evaluation is done following Section C.
Stochastic and Static Depth Training. We compare models trained with stochastic recurrence depth against models trained at a static depth of . Both exhibit similar training-loss trajectories (Figure 13) and comparable uncompressed quality. However, KV-sharing is nearly lossless for stochastic-depth models at every scale, while static-depth models degrade substantially under compression (Table 8). Thus, the emergence of KV-sharing is associated with stochastic-depth training rather than ordinary language-model optimization alone.
Figure 13. Training Curves for Stochastic- and Static-Depth Models. Training loss as a function of training tokens across model scales. The curves closely overlap, showing that stochastic- and static-depth training exhibit similar optimization dynamics.
| Training | Val. PPL () | CORE () | ||||
| Unshared | Shared | Unshared | Shared | |||
| 140M Stochastic | 25.68 | 25.67 | -0.02 | 6.41 | 6.48 | +0.08 |
| 140M Static | 24.48 | 25.79 | +1.31 | 9.17 | 8.55 | -0.62 |
| 370M Stochastic | 17.34 | 17.36 | +0.02 | 14.49 | 14.36 | -0.13 |
| 370M Static | 17.30 | 20.00 | +2.70 | 15.00 | 14.22 | -0.78 |
| 770M Stochastic | 14.03 | 14.04 | +0.01 | 18.91 | 18.78 | -0.13 |
| 770M Static | 14.10 | 17.20 | +3.10 | 18.17 | 17.11 | -1.06 |
| 1.3B Stochastic | 12.33 | 12.34 | +0.01 | 23.63 | 23.34 | -0.29 |
| 1.3B Static | 12.32 | 15.17 | +2.84 | 22.59 | 20.82 | -1.77 |
Table 8. Quality under stochastic- and static-depth training. We compare the standard per-recurrence cache (Unshared) with final-recurrence KV sharing (Shared). is Shared minus Unshared.
Extended Empirical Analysis on KV-Sharing Fixed Point Convergence Failures
In this section, we provide extended empirical evidence that KV-sharing failures in models trained with stochastic depth are fixed-point convergence failures. Specifically, we provide expanded empirical evidence that while contractive residual dynamics are necessary to induce KV-shareability, models can still fail to converge to the correct fixed points.
Setup. We use prior Parcae looped models [50] to explore our observations about KV-sharing failures. We additionally use models trained in § 5.1, showing how BeNice! impacts fixed-point convergence, before and after finetuning. Following convergence observations made in Section F and the saturation of test-time compute made by Prairie et al. [50], we use test-time recurrences, as both the residual differences and downstream quality saturate at this point. We use climbmix [16] validation data for all evaluations and explorations.
KV-sharing Fails to Converge Correctly. As visualized by Figure 4, we see that models that do not have KV-shareability from Table 1 (e.g., Parcae 770M and 1.3B) seem to have disjoint sets between unshared and shared fixed points, while models with KV-shareability have approximately similar distributions. We capture this numerically in Table 9, where we find that models that have KV-shareability approach a relatively similar fixed point. For example, Parcae 140M and 370M have a relative difference of 0.82% and 1.69%, respectively, which means they approached the same relative fixed point. However, Parcae 770M and 1.3B have an average relative difference of 134.8% and 177.5%, respectively, which means that they converge to different points with and without KV-sharing. Thus, Parcae 770M and 1.3B fail to have KV-shareability as they do not converge to the same fixed points as the base model.
| Model | |||
| 140M | 0.4588 | 56.3 | 0.82 |
| 370M | 2.2766 | 135.2 | 1.69 |
| 770M | 186.35 | 153.4 | 134.8 |
| 1.3B | 342.65 | 301.6 | 177.5 |
Table 9. Mean difference between shared and unshared final latents, mean unshared latent norm, and mean relative difference.
We further verify these results empirically with models trained in § 5.1, where we find that all trained baseline Parcae models fail to converge to the same fixed point under KV-sharing (see Table 10), preventing them from having KV-shareability. However, after finetuning with BeNice!, we empirically observe that models converge to the correct fixed point, as seen by Table 10.
| Model | Baseline: Mean Relative Difference | BeNice!: Mean Relative Difference |
| 140M | 0.0526 | 0.0059 |
| 370M | 0.1304 | 0.0059 |
| 770M | 0.0957 | 0.0070 |
| 1.3B | 0.1253 | 0.0049 |
Table 10. Mean Relative Difference of Fixed Points With and Without KV-sharing Before and After Post-training with BeNice! Mean relative difference at each model scale for the baseline and BeNice!.
Extended Finetuning Setup
In this section, we provide more details about the BeNice! finetuning process.
Setup. Following the training procedures of Parcae in Prairie et al. [50], all models are trained using batch sizes of 256 at context length of 2048. Recurrence depth is sampled stochastically from , where , as in pretraining.
We use BeNice! on two generations of models: previously released Parcae models (770m and 1.3B) [50] and new Parcae style models trained from scratch (including the MLA and KDA models in § 5.1). The publicly released models are finetuned using BeNice! with the same hyperparameters as their pretraining setup (see Prairie et al. [50]) and use 140M fineweb-edu [49] tokens. The new Parcae models are pretrained using hyperparameters described in Section O on climbmix [16] tokens, and are further finetuned on 140M climbmix tokens using BeNice!. Per-model finetuning learning rates and values are listed in Section O.
All BeNice! results are averaged across three differently seeded finetuning runs evaluated on Core and Core-Ext using three different seeds as well.
LoRA Finetuning for KV-sharing
In this section, we show that KV-sharing can be achieved using a LoRA[28] finetuning version of BeNice!, which we denote BeNice!{sup}†`.
Following our findings in § 3 that the KV-sharing error subspace is low-rank (Figure 11), we finetune released Parcae 770M and 1.3B with low-rank adapters on the key and value weights of the first layer of the recurrent block. We use rank-8 adapters on both key and value weights and freeze the rest of the model weights, leading to 41K trainable parameters in the 770M model and 49K trainable parameters in the 1.3B model. We finetune using 65M fineweb-edu tokens with batch size 32 and sequence length 2048.
We use the same BeNice! loss function defined in § 4. Hyperparameter details are in Section O. In Table 11, both BeNice! and BeNice!{sup}†` enable KV-sharing while maintaining quality across benchmarks.
| Model | Val. PPL () | Core () | Core-Extended () | ||||||
| Unshared | Shared | Unshared | Shared | Unshared | Shared | ||||
| 770M | |||||||||
| Parcae | 11.87 | 34.20 | +22.33 | 25.05 | 21.10 | -3.94 | 15.69 | 13.95 | -1.74 |
| + Standard FT | 11.86 | 21.61 | +9.75 | 24.74 | 23.42 | -1.32 | 15.34 | 14.54 | -0.80 |
| + BeNice! | 11.85 | 11.86 | +0.01 | 25.15 | 25.19 | +0.04 | 15.35 | 15.29 | -0.06 |
| + BeNice!{sup}†` | 11.89 | 11.89 | +0.00 | 24.69 | 24.71 | +0.02 | 15.32 | 15.19 | -0.13 |
| 1.3B | |||||||||
| Parcae | 10.78 | 1.6e4 | +1.6e4 | 28.78 | 21.33 | -7.45 | 16.73 | 13.48 | -3.25 |
| + Standard FT | 10.79 | 18.52 | +7.73 | 29.01 | 27.49 | -1.51 | 17.23 | 16.66 | -0.58 |
| + BeNice! | 10.79 | 10.78 | -0.01 | 28.63 | 28.67 | +0.04 | 17.32 | 17.41 | +0.09 |
| + BeNice!{sup}†` | 10.81 | 10.81 | 0.00 | 28.28 | 28.35 | +0.07 | 16.94 | 16.96 | +0.02 |
Table 11. Quality of BeNice! Finetuning Extended. We provide the extended results of finetuning using BeNice! and BeNice!{sup}†`. Both enable KV-sharing while maintaining quality.
Extended Discussion of Experimental Setup for MLA and KDA Evaluation
In this section, we provide a more detailed description of the experimental setup for evaluating MLA [13] and KDA [65] versions of Parcae [50]. We will fully discuss how KV-sharing is done for both alternative sequence mixing methods. Hyperparameters and model configurations can be found in Section O and Section N, respectively. BeNice! finetuning is explained in § 16.
KV-sharing with Multi-Latent Attention Similarly to MHA [68], KV-sharing with MLA [13] is done by discarding the latent cache of indeterminate recurrences, and keeping only the final recurrences latent cache. More formally, for an input at recurrence , the latent cache and the decoupled RoPE [61] caches are calculated as
where denotes the RoPE rotation [61]. Thus with KV-sharing MLA models, we discard all where keeping only the cache at .
KV-sharing with Kimi Delta Attention Again, like both MHA [68] and MLA [13], KV-sharing with KDA [65] is done by keeping only the final recurrences sequence-mixing hidden state. More formally, for an input at recurrence , we focus on the linear update rule used in KDA [65]
where , , , and and , , , are learned projections. Thus with KV-sharing KDA models, we discard all where and all up to our current decode token.
Extended Compression and Patching Results for Newly Pretrained Models
In Table 12 we explore the KV-sharing ability of the base pretrained Parcae models introduced in § 5.1 (Table 4, see Section N for configurations and Section O for hyperparameters). All four MHA models have KV-sharing errors. Preserving the first-layer KVs from the first recurrence in addition to normal KV-sharing maintains lossless quality for all models except the 1.3B model. The 1.3B model also has further quality degradation when using Naive Patching. Interestingly, in Table 4 using BeNice! on the 1.3B model leads to near lossless quality, performing much better than the sub-optimal (in terms of memory efficiency) additional first layer, first recurrence KV-cache during KV-sharing.
| Model | Core (↑) | Core-Extended (↑) | |||||
| Unshared | Shared | Δ | Unshared | Shared | Δ | ||
| 140M | Parcae | 15.50 | 14.97 | -0.53 | 10.43 | 10.10 | -0.33 |
| + BeNice! | 15.47 | 15.61 | +0.14 | 10.28 | 10.19 | -0.09 | |
| 370M | Parcae | 20.74 | 15.16 | -5.58 | 12.99 | 9.68 | -3.31 |
| + BeNice! | 23.90 | 23.51 | -0.39 | 14.39 | 14.24 | -0.15 | |
| 770M | Parcae | 30.45 | 26.92 | -3.52 | 19.22 | 16.67 | -2.55 |
| + BeNice! | 30.79 | 30.53 | -0.26 | 19.18 | 19.13 | -0.05 | |
| 1.3B | Parcae | 37.27 | 33.94 | -3.33 | 25.41 | 23.46 | -1.95 |
| + BeNice! | 37.40 | 37.43 | +0.04 | 25.69 | 25.59 | -0.10 |
Table 4. BeNice! Maintains Quality and Enables KV-Shareability. We train Parcae models [50] and then post-tune with BeNice!. BeNice! maintains and improves base model quality, while fully enabling KV-sharing. Best results are bolded.
| Model | Unshared | Layer > 1, Recurrence > 1 | Naive Patching | Shared | |
| MHA | 140M | 15.51 | 15.50 | 15.52 | 16.99 |
| MHA | 370M | 11.57 | 11.56 | 11.56 | 3723 |
| MHA | 770M | 9.78 | 9.80 | 10.49 | 36.66 |
| MHA | 1.3B | 8.58 | 8.92 | 12.77 | 9.42 |
| MLA | 140M | 15.56 | 15.54 | 15.55 | 15.55 |
| MLA | 370M | 11.67 | 11.64 | 11.66 | 22.72 |
| MLA | 770M | 9.75 | 9.75 | 11.30 | 4894 |
| KDA | 140M | 16.02 | 16.00 | 15.99 | 15.99 |
| KDA | 370M | 11.74 | 11.73 | 11.73 | 11.77 |
| KDA | 770M | 10.13 | 10.13 | 10.13 | 10.16 |
Table 12. KV-Sharing Failures on New Models. Preserving first-layer KVs from the first recurrence matches the unshared baseline across model sizes and sequence mixers. Mean patching fails to fully recover quality for MLA at 770M. KV-sharing fails for MLA models as size increases. Surprisingly, KDA models have KV-sharing out-of-the-box.
We also explore the KV-sharing ability of MLA and KDA variants of Parcae in Table 12. MLA variants of Parcae have the same failure mode of KV-sharing as released Parcae models. Mean patching fails to recover quality for our MLA model at 770M size. KDA models perform well on KV-sharing out-of-the-box.
Optimal Hyperparameter Transferability Across Recurrence
While several works have shown how optimal hyperparameter transfer varies with tokens [9, 59] and model size [72], we explore how scaling recurrence [50] is affected. Specifically, in this section, we investigate how optimal hyperparameters (e.g., learning rate, batch size, etc.) are affected by increasing recurrence at multiple parameter classes.
Setup. We follow the model configuration and architecture backbone of Prairie et al. [50] (see Section N for extend details), and train using ClimbMix [16]. We evaluate the transferability of learning rate, momentum, warm-up ratios, and batch size across using 140M, 370M, and 770M parameter backbones. To ensure robustness, for each hyperparameter, recurrence, backbones data point, we train three models using different seeds, 42, 1234, 2345, and then use the average of these runs. This totals to 504 runs for this ablation and approximately 4,685 H100 hours.
Optimal Hyperparameters Transfer Strongly Across Increased Recurrence. As seen in Figure 14 and Figure 15, we find that across all model classes optimal learning rate and batch size transfer strongly. We further find this to be true of other minor hyperparameters (e.g., muon momentum and warmup ratio), with some minor variation in flat regions. This finding implies that lower recurrence (i.e., ) can be used to cheaply sweep for optimal hyperparameters, which in general remain optimal at higher recurrences. drastically reduces the necessary FLOPs required to find optimal hyperparameters for looped architectures.
Figure 14. Transferability of Learning Rate. Optimal learning rate transfers strongly across deeper recurrences for all model backbones.
Figure 15. Transferability of Batch Size. Optimal batch size transfers strongly across deeper recurrences for all model backbones.
Figure 16. Transferability of Muon Momentum. We observe optimal muon momentum to transfer well across recurrences, with some minor variation in relatively flat regions.
Furthermore, as seen in Table 13, we observe a high Spearman rank correlation between the validation BPB across hyperparameter sweeps at different recurrences. While Spearman correlation is not 1 for all hyperparameters, we find that this is derived from noisy and flat hyperparameter landscapes. Thus, generally, we observe that most hyperparameters appear to transfer across recurrences, motivating our use of extremely low recurrence for hyperparameter tuning.
| Hyperparameter | Model | Mean | BPB () | |||
| Learning Rate | 140M | 0.950 | 0.967 | 0.900 | 0.939 | 41.805 |
| Learning Rate | 370M | 0.943 | 1.000 | 0.943 | 0.962 | 26.483 |
| Learning Rate | 770M | 0.829 | 0.829 | 1.000 | 0.886 | 18.373 |
| Muon Momentum | 140M | 0.900 | 0.700 | 0.900 | 0.833 | 37.269 |
| Muon Momentum | 370M | 1.000 | 1.000 | 1.000 | 1.000 | 25.083 |
| Warmup Fraction | 140M | 0.600 | 0.900 | 0.700 | 0.733 | 22.614 |
| Warmup Fraction | 370M | 0.700 | 0.700 | 1.000 | 0.800 | 19.451 |
| Batch Size | 140M | 1.000 | 1.000 | 1.000 | 1.000 | 128.046 |
| Batch Size | 370M | 1.000 | 1.000 | 1.000 | 1.000 | 70.462 |
| Batch Size | 770M | 1.000 | 1.000 | 1.000 | 1.000 | 47.777 |
Table 13. Spearman Rank Correlation of Hyperparameter curves across Recurrence. Here is the Spearman correlation between the validation BPB hyperparameter curves of a model trained with and a model trained with . Furthermore, BPB indicates the validation BPB span across all (recurrence schedule hyperparameter values) points for that specific parameter class. While Spearman correlation is not 1 in all transfers, we observe that low Spearman correlation coincides with a lower BPB span, which resolves to slight noise on a relatively flat landscape for hyperparameter tuning.
Figure 17. Transferability of Warmup Ratio. We observe optimal warmup ratios to transfer well across recurrences, with some minor variation in relatively flat regions.
Model Definitions and Configurations
In this section, we discuss the model definition used for our experiments. For our experiments done in § 3 for our Stochastic and Static comparison, the models follow the configuration and hyperparameters presented in Table 14 and Section O respectively, with two notable exceptions. First, for both models, we use a 1:7 param to token ratio. Second, for the Static depth models, recurrence depth sampling does not follow the Poisson distribution, and is instead held deterministically at recurrences.
| Small (140M) | Medium (370M) | Large (770M) | XLarge (1.3B) | ||
| Architecture | Layers in | 2 | 4 | 6 | 8 |
| Architecture | Layers in | 2 | 4 | 6 | 8 |
| Architecture | Layers in | 2 | 4 | 6 | 8 |
| Architecture | 768 | 1,024 | 1,280 | 1,536 | |
| Architecture | 3,072 | 4,096 | 5,120 | 6,144 | |
| MHA | Attention | Causal Self-Attention [68] w/ QK-Norm [26] | |||
| MHA | Attention Heads | 6 | 8 | 10 | 12 |
| MHA | Head Dimension | 128 | |||
| MHA | Pos. Embed. | RoPE (=50,000) [61] | |||
| MLA | Attention | Causal Multi-Head Latent Attention [13] | |||
| MLA | Attention Heads | 6 | 8 | 10 | 12 |
| MLA | KV LoRA Rank | 512 | |||
| MLA | QK NoPE Head Dimension | 128 | |||
| MLA | QK RoPE Head Dimension | 64 | |||
| MLA | V Head Dimension | 128 | |||
| MLA | Pos. Embed. | RoPE (=50,000) [61] | |||
| KDA | Attention | Causal Kimi Delta Attention [65] | |||
| KDA | Attention Heads | 6 | 8 | 10 | 12 |
| KDA | Decay Gate | Lower bound at | |||
| KDA | Head Dimension | 128 | |||
| KDA | Convolution Kernel Size | 4 | |||
| Shared Details | MLP | ReLU [76] | |||
| Shared Details | Value Embeddings | Gated, alternating layers [66] | |||
| Shared Details | Vocab Size | 32,768 | |||
| Shared Details | Norm | RMS-Norm (Pre-Norm) [75] | |||
| Shared Details | Context Length | 2,048 | |||
| Shared Details | Bias | None | |||
| Shared Details | Init. | Scaled-zero [34, 62] | |||
| Shared Details | Tied Embeddings | Yes | |||
| Parcae | Injection | Diagonal | |||
| Parcae | State Init. | like-init [20] | |||
| Parcae | 8 | ||||
| Parcae | Backprop Depth | 4 | |||
| Parcae | Sampling | Poisson (truncated, per-sequence) |
Table 14. Model definitions of baseline Parcae [50], Parcae with MLA, and Parcae with KDA.
Our models present in pretraining results in § 5.1 also use the model configurations in Table 14 along with the hyperparameters present in Table 16. Our optimizer parameter group assignment for all models (including those present in § 3) can be found in Table 15.
| Optimizer | Parameters |
| AdamW [36] | Token embeddings (wte) |
| AdamW [36] | LM head (lm_head) |
| AdamW [36] | Normalization layers (RMSNorm) |
| AdamW [36] | Value embedding gates (ve_gate) |
| AdamW [36] | All 1D parameters |
| AdamW [36] | Injection parameters (, , ) |
| AdamW [36] | Readout projection () |
| Muon [32] | Attention projections (, , , ) |
| Muon [32] | MLP weights (, ) |
Table 15. Optimizer parameter group assignment for Parcae.
| Small (140M) | Medium (370M) | Large (770M) | XLarge (1.3B) | |
| AdamW and Muon Learning Rate | ||||
| Warmup Ratio | 0.05 | |||
| Constant Decay Ratio | 0.2 | |||
| Gradient Clipping | 1.0 | |||
| Muon Momentum | 0.95 | |||
| Muon Weight Decay | 0.3 | |||
| Muon Orthogonalization Steps | 5 | |||
| AdamW Betas | (0.8,0.95) | |||
| AdamW Weight Decay | 0.1 | |||
| AdamW Epsilon | ||||
| Batch Size | 256 | |||
| Context Length | 2048 | |||
| Param to Token Ratio | 1:80 | |||
| Precision | bf16-mixed |
Table 16. Pretraining Hyperparameters for climbmix trained Parcae models (including MLA and KDA variants).
Hyperparameters and Training Details
In this section, we will discuss the hyperparameter configuration used in § 5.1.
Pretraining. All pretraining hyperparameters are present in Table 16, with the notable exception of the Stochastic and Static recurrence depth models present in § 3. Those exceptions are explained in Section G.
BeNICE! and Standard Finetuning. For both BeNICE! and Standard Finetuning, we sweep over learning rates, and . We perform these sweeps with an optimizer budget of 500 steps. For Standard Finetuning, we find 770M models (across MHA, MLA, and KDA) to perform best at a learning rate of , while the 1.3B models perform best at on a held out validation set. For BeNICE! we find all models perform best at a learning rate of and with a of 0.01 on a held out validation set. Since we sweep over learning rates with an optimizer budget of 500 steps, we rescale our learning rate to account for the increase in optimizer budget (1800 steps). Specifically, all final learning rates are .
BeNice!{sup}† Finetuning. For BeNice!{sup}†, we perform a more limited hyperparameter sweep on a limited model set: the released Parcae 770M and 1.3B models. We sweep over and for the full 1000 step optimizer budget. For both models, a learning rate of and a of 0.06 performed best on a held out validation set. We use a rank 8 adapter with an .
References
- [1]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.arxiv.org/abs/2305.13245
- [2]Ibrahim Alabdulmohsin and Xiaohua Zhai. Recursive inference scaling: A winning path to scalable inference in language and multimodal systems. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2502.07503
- [3]Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.arxiv.org/abs/1905.13319
- [4]Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation, 2025. URL https://arxiv.org/abs/2507.10524.
- [5]Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems, volume 32, 2019.
- [6]Daniel Bailey, Amelia Harrison, Yuliya Lierler, Vladimir Lifschitz, and Julian Michael. The winograd schema challenge and reasoning about correlation. In Working Notes of the Symposium on Logical Formalizations of Commonsense Reasoning. AAAI Press, 2015. URL http://www.cs.utexas.edu/users/ai-lab?wsc15.
- [7]Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=PPjSKy4OXUB.
- [8]Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on Artificial Intelligence, volume 34, 2020.arxiv.org/abs/1911.11641
- [9]Johan Björck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal lr across token horizons. In The Thirteenth International Conference on Learning Representations, 2025.
- [10]Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, and James Cheng. Hista and numca: Estimate state value effectively for llm reinforcement learning, 2026. URL https://arxiv.org/abs/2605.29782.
- [11]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.arxiv.org/abs/1905.10044
- [12]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
- [13]DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunyuan Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. URL https://arxiv.org/abs/2405.04434.
- [14]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HyzDriR9Y7.arxiv.org/abs/1807.03819
- [15]Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu, Yuanyuan Xu, Jiarui Liu, T S Eugene Ng, and Hanjie Chen. LT2: Linear-time looped transformers. arXiv [cs.LG], May 2026.
- [16]Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan, Lin, Jan Kautz, and Pavlo Molchanov. Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training, 2025. URL https://arxiv.org/abs/2504.13161.
- [17]Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJg7KhVKPH.
- [18]Hengyu Fu, Tianyu Guo, Zixuan Wang, Hanlin Zhu, Jason D Lee, Jiantao Jiao, Stuart Russell, and Song Mei. DiscoLoop: Looping discrete embeddings and continuous hidden states for multi-hop reasoning. arXiv [cs.CL], July 2026.
- [19]Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, and Bryan Dai. Loop the loopies!, 2026. URL https://arxiv.org/abs/2607.16051.
- [20]Jonas Geiping, Sean Michael McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=S3GhJooWIC.arxiv.org/abs/2502.05171
- [21]Jonas Geiping, Xinyu Yang, and Guinan Su. Efficient parallel samplers for recurrent-depth models and their connection to diffusion language models, 2025. URL https://arxiv.org/abs/2510.14961.
- [22]Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret, editors, *SEM 2012: The First Joint Conference on Lexical and Computational Semantics—Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394–398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1052/.
- [23]Alex Graves. Adaptive computation time for recurrent neural networks, 2017. URL https://arxiv.org/abs/1603.08983.
- [24]Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu, Reyna Abhyankar, Qingyang Wu, Austin Silveria, Pragaash Ponnusamy, Jue Wang, Ben Athiwatkun, Leon Song, Tri Dao, Daniel Y. Fu, and Chris De Sa. Search your block floating point scales! In Proceedings of Machine Learning and Systems, 2026.
- [25]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
- [26]Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.arxiv.org/abs/2010.04245
- [27]Coleman Hooper, Sehoon Kim, Hiwa Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2401.18079
- [28]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
- [29]Benhao Huang, Zhengyang Geng, and Zico Kolter. Equilibrium reasoners: Learning attractors enables scalable reasoning. In Proceedings of the 43rd International Conference on Machine Learning, 2026.
- [30]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, 2019.arxiv.org/abs/1909.06146
- [31]Alexia Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks, 2025. URL https://arxiv.org/abs/2510.04871.
- [32]Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
- [33]kaggle200000Jeopardy. 200,000+ Jeopardy! Questions, 2019. URL https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions.
- [34]Andrej Karpathy. nanochat: The best chatgpt that $100 can buy, 2025. URL https://github.com/karpathy/nanochat.
- [35]Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 2021.arxiv.org/abs/2006.04710
- [36]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 2015.arxiv.org/abs/1412.6980
- [37]Ian Li, Kapilesh Guruprasad, Raunak Sengupta, Ninad Satish, Loris D’Antoni, and Rose Yu. Manifold-guided attention steering. In Proceedings of the 43rd International Conference on Machine Learning, 2026.
- [38]Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-lm: In search of the next generation of training sets for language models. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37, 2024.arxiv.org/abs/2406.11794
- [39]Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020.arxiv.org/abs/2007.08124
- [40]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 2024.DOI
- [41]Sean McLeish, Ang Li, John Kirchenbauer, Dayal Singh Kalra, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Jonas Geiping, Tom Goldstein, and Micah Goldblum. Teaching pretrained language models to think deeper with retrofitted recurrence. arXiv preprint arXiv:2511.07384, 2025.arxiv.org/abs/2511.07384
- [42]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.arxiv.org/abs/1809.02789
- [43]Seungjae Moon, Jung-Hoon Kim, Junsoo Kim, Seongmin Hong, Junseo Cha, Minsu Kim, Sukbin Lim, Gyubin Choi, Dongjin Seo, Jongho Kim, Hunjong Lee, Hyunjun Park, Ryeowook Ko, Soongyu Choi, Jongse Park, Jinwon Lee, and Joo-Young Kim. Lpu: A latency-optimized and highly scalable processor for large language model inference. IEEE Micro, 2024.arxiv.org/abs/2408.07326
- [44]MosaicML. llm-foundry: LLM training and evaluation framework. https://github.com/mosaicml/llm-foundry, 2023.
- [45]Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, and Christopher Re. Cost-efficient collaboration between on-device and cloud language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 2025.arxiv.org/abs/2502.15964
- [46]NVIDIA. AIPerf: AI performance benchmarking tool. https://github.com/ai-dynamo/aiperf, 2025.
- [47]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1525–1534, 2016.arxiv.org/abs/1606.06031
- [48]Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, 2022.arxiv.org/abs/2110.08193
- [49]Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 37, 2024.arxiv.org/abs/2406.17557
- [50]Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, and Daniel Y. Fu. Parcae: Scaling laws for stable looped language models, 2026. URL https://arxiv.org/abs/2604.12946.
- [51]Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), 2025.
- [52]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016.arxiv.org/abs/1606.05250
- [53]Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019.arxiv.org/abs/1808.07042
- [54]Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.arxiv.org/abs/1804.09301
- [55]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
- [56]Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiq: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.arxiv.org/abs/1904.09728
- [57]Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=Tsp2PL7-GQ.arxiv.org/abs/2106.04537
- [58]Kristian Schwethelm, Daniel Rueckert, and Georgios Kaissis. How much is one recurrence worth? iso-depth scaling laws for looped language models. arXiv [cs.LG], April 2026.
- [59]Xian Shuai, Yiding Wang, Yimeng Wu, Xin Jiang, and Xiaozhe Ren. Scaling law for language models training considering batch size, 2024. URL https://arxiv.org/abs/2412.01505.
- [60]Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkow ycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, and Amanda Dsouza et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
- [61]Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
- [62]Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike no more: Stabilizing the pre-training of large language models, 2025. URL https://arxiv.org/abs/2312.16903.
- [63]Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.arxiv.org/abs/1811.00937
- [64]GLM Team. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https://arxiv.org/abs/2508.06471.
- [65]Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejiang Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, and Yulun Du. Kimi linear: An expressive, efficient attention architecture, 2025. URL https://arxiv.org/abs/2510.26692.
- [66]Rui Tian, Zuxuan Wu, Qi Dai, Han Hu, Yu Qiao, and Yu-Gang Jiang. Resformer: Scaling vits with multi-resolution training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.arxiv.org/abs/2212.00776
- [67]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971.
- [68]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03762
- [69]Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo, Jordi Ros-Giralt, Arash Behboodi, and Fabio Valerio Massoli. Memory-efficient looped transformer: Decoupling compute from memory in looped language models. In Proceedings of the 43rd International Conference on Machine Learning, 2026.
- [70]Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model, 2025. URL https://arxiv.org/abs/2506.21734.
- [71]Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.arxiv.org/abs/2108.00648
- [72]Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems, volume 34, 2021.arxiv.org/abs/2203.03466
- [73]Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2311.12424
- [74]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.arxiv.org/abs/1905.07830
- [75]Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32, 2019.
- [76]Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu² wins: Discovering efficient activation functions for sparse llms, 2024. URL https://arxiv.org/abs/2402.03804.
- [77]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2312.07104
- [78]Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. Analytical reasoning of text. In Findings of the Association for Computational Linguistics: NAACL 2022, 2022.DOI
- [79]Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024.arxiv.org/abs/2304.06364
- [80]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Endu Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models, 2025. URL https://arxiv.org/abs/2510.25741.















