betaFeedback

WorksComputer Science

Architectures and KV Replay for Depth-Recurrent Language Models

Abstract

Depth recurrence increases effective model depth by repeatedly applying a fixed computation block. However, sharing attention parameters does not share the key-value (KV) states generated at different recurrence iterations, which causes memory growth with recurrence depth during autoregressive decoding. We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework. We first study a looped hybrid that repeats a stack of Transformer layers, with recurrent mLSTM or Gated DeltaNet blocks replacing most attention layers and periodic Transformer attention anchors retained. We then restrict recurrence to a dedicated recurrent workspace in a Parcae-style architecture and test its stabilising adapter as a wrapper for the looped hybrid. Finally, we train a looped Transformer with recurrent KV replay, where only selected recurrence iterations write KV states that later iterations reuse. This trains the model to operate with the same shared KV topology used during inference. We find that looped hybrids require sufficiently frequent attention anchors, with performance depending strongly on the recurrent mixer. For cross-iteration KV sharing, recurrent KV replay allows multiple recurrence iterations to reuse the same KV states while retaining performance close to the full looped Transformer with separate KV caches for each iteration.

Introduction

Depth-recurrent language models increase computational depth without allocating a distinct set of parameters to every effective layer. A stack of LL physical blocks applied for RR recurrence iterations implements an effective depth of LRLR while retaining substantially fewer parameters than an independently parameterized LRLR-layer Transformer. This construction is attractive when parameter storage or optimiser state is the principal constraint. It also permits inference-time compute scaling by changing the number of recurrence iterations without changing the stored model. However, autoregressive decoding stores keys and values whose size grows with sequence length, layer count, batch size, head dimension, and, in a looped architecture, the number of recurrence iterations whose attention states must remain distinct. Parameter sharing across depth does not imply memory sharing across depth, and a question arises:

Research question

How can inference memory be reduced in depth-recurrent language models while retaining the benefits of exact attention?

To study this question, we make three empirical contributions. First, we evaluate a looped hybrid architecture that replaces most repeated attention layers with recurrent sequence mixers while retaining periodic Transformer [10] attention anchors. Second, we study a PARCAE-style architecture [9], which confines recurrence to a dedicated latent workspace between Transformer encoder and decoder stages, and assess whether its stabilising adapter transfers to the looped-hybrid architecture. Third, we introduce cache-compression training, which trains a looped Transformer under the same KV-sharing policy that will later be used during inference. The first two contributions reduce inference memory through architectural changes. The third one leaves the architecture unchanged but reduces the number of persistent KV groups through training.

Related Work

Looped attention. Recent work studies how to make recurrence efficient by changing the token mixer itself. LT2 [5] compares full, linear, sparse, and hybrid attention operators inside recurrent loops. Li et al. (2025) show that pretrained Transformers can also adapt computation at inference time by dynamically skipping or repeating layers. PARCAE [9] separates computation into prelude, core, and coda stages. It confines recurrence to the latent core and applies a stabilising adapter that repeatedly injects the original representation into the recurrent state.

Layer-wise and cross-layer KV sharing. Cross-Layer Attention and related methods share key–value representations across different Transformer layers [3, 13, 15]. In contrast, we share KV states across recurrence iterations of the same weight-shared attention layer. Mixture-of-Recursions [1] routes tokens through different numbers of shared-layer recursions and includes a variant that reuses KV pairs from the first recursion. Parallel Loop Transformer [12] shares the first loop’s KV cache across subsequent loops and combines it with local information via gated sliding-window attention. MELT [11] shares one KV cache per layer across recursions and updates it via learned gate. Looped Latent Attention [8] compresses loop-specific keys and values into low-rank codes and reconstructs them when attention reads the cache. The training procedure proposed by Gelberg et al. (2026) masks context-token KV slots during continued pretraining, encouraging the model to learn representations that remain useful when the token-level cache is subsequently compressed by post-hoc methods. These studies address complementary aspects of recurrent computation. Our work studies a fixed write/reuse schedule across iterations of the same weight-shared attention layer.

Recurrent sequence mixers. Gated DeltaNet [14] and mLSTM [2] provide recurrent state updates of fixed size, which avoids the sequence-length-dependent KV storage of softmax attention. In the looped hybrid, these mixers do not replace exact attention; instead, they carry computation between Transformer anchors. This permits us to study how often token-level memory must be reconstructed and how much of the effective depth we can implement through recurrent state transitions.

Method

Looped Hybrid The looped hybrid architecture consists of exact-attention Transformer anchors separated by recurrent sequence mixer blocks. The complete logical stack is repeated, and Transformer parameters are shared across recurrence iterations, while recurrent mixer parameters are iteration-specific, and MLP parameters are shared across loops at the same vertical position. This architecture directly exposes the trade-off between exact token-level memory and compressed recurrent computation. Reducing the number of Transformer anchors lowers the number of attention caches, but also lengthens the interval over which recurrent mixers must propagate and transform information without reconstructing exact token-level interactions.

Let the logical stack contain LL vertical positions and execute for RR recurrence iterations. Denote the hidden state at recurrence iteration rr and vertical position ll by hr,lh_{r,l}. A set A⊆{0,…,L−1}\mathcal{A} \subseteq\{0,\ldots,L-1\} specifies the positions occupied by exact-attention Transformer anchors. Layers outside A\mathcal{A} are implemented using recurrent sequence mixers. For l∈Al \in\mathcal{A}, the update is

hr,l+1=Tl(hr,l;θlT).h_{r,l+1} = \mathcal{T}_l(h_{r,l}; \theta_l^T).

where the same parameters θlT\theta_l^T are reused for every rr. The attention parameters are fixed layer parameters. At recurrent positions l∉Al \notin\mathcal{A}, the update is

ur,l=Mr,l(hr,l;θr,lM),hr,l+1=ur,l+Fl(ur,l;θlF).\begin{aligned} u_{r,l} &= \mathcal{M}_{r,l}(h_{r,l}; \theta_{r,l}^M), \qquad h_{r,l+1} = u_{r,l} + \mathcal{F}_l(u_{r,l}; \theta_l^F). \end{aligned}

Figure 1. Looped hybrid model. Exact-attention Transformer anchors are interleaved with recurrent sequence mixers; the complete logical stack is applied across multiple recurrence iterations.

where Mr,l\mathcal{M}_{r,l} is an mLSTM ([2]) or GDN ([14]) mixer and Fl\mathcal{F}_l is the MLP assigned to the vertical position ll. We allocate separate recurrent mixer parameters θr,lM\theta^M_{r,l} for each recurrence iteration, whereas θlF\theta^F_l is shared across recurrence iterations. The final hidden state of one logical pass initialises the next recurrence iteration. A transformer-anchor stride sds_d places Transformer anchors every sds_d layers; the final layer can optionally also be made an anchor. Decreasing sds_d increases the frequency with which exact token-level interactions are reconstructed, but also increases attention computation and the number of KV-producing positions. Increasing sds_d delegates longer spans of the stack to fixed-size recurrent mixers.

PARCAE and PARCAE Wrapper Modes Unlike the looped hybrid, which repeats an entire logical stack, PARCAE confines recurrence to a dedicated latent workspace between fixed Transformer stages. Our implementation follows the overall PARCAE design ([9]), including the prelude/core/coda decomposition, recurrent workspace, input-injection adapter, stochastic recurrence schedule, and truncated-backpropagation training. Unlike the original architecture, we instantiate the core with mLSTM-based modules ([2]). This allows a direct comparison between the PARCAE organisation and the looped-hybrid architecture. Following PARCAE ([9]), we divide computation into a Transformer prelude, a recurrent core, and a Transformer coda. The prelude produces a representation ee, which is projected into a recurrent workspace. Each recurrence iteration first applies an input-injection adapter and then an mLSTM recurrent core ([2]):

z~t=I(zt,e),zt+1=C(z~t;θC).\widetilde{z}_t = \mathcal{I}(z_t,e), \qquad z_{t+1} = \mathcal{C}(\widetilde{z}_t;\theta^C).

After RR recurrence iterations, the recurrent state is projected back to the Transformer hidden dimension and processed once by the Transformer coda. For each training instance, PARCAE samples two depths: a prefix depth nn, used only to advance the recurrent state, and a suffix depth kk, retained in the autograd graph. The state is first advanced without constructing an autograd graph; optimisation then differentiates through only the final kk iterations. Evaluation uses deterministic recurrence depth.

The PARCAE adapter ([9]) is designed to prevent recurrent drift by repeatedly re-injecting the original input representation ee. Let xtx_t be the current recurrent state and ee be the representation that entered recurrence. The diagonal adapter computes

dt=softplus⁡(dtbias),A=exp⁡(Alog),decay=exp⁡(−dt⋅A),xt+1=xt⊙decay+dt(eBT).\mathrm{dt} = \operatorname{softplus}(\mathrm{dt}_{\mathrm{bias}}), \quad A = \exp(A_{\mathrm{log}}), \quad\mathrm{decay} = \exp(-\mathrm{dt} \cdot A), \quad x_{t+1} = x_t \odot\mathrm{decay} + \mathrm{dt}(eB^{\mathsf{T}}).

At each recurrence iteration, a learned decay retains part of the current state, while a learned projection of the original representation ee is injected to counteract recurrent drift.

The wrapper ablation applies the PARCAE adapter before each looped-hybrid recurrence pass:

h~r=I(hr,h0),hr+1=H(h~r).\widetilde{h}_r = \mathcal{I}(h_r,h_0), \qquad h_{r+1} = \mathcal{H}(\widetilde{h}_r).

Figure 2. In our implementation of the PARCAE architecture, Transformer layers encode the input, the representation is projected into a recurrent workspace, a shared mLSTM core is applied repeatedly with stabilising input injection, and the resulting state is projected back before the coda.

Here, h0h_0 is the representation entering depth recurrence, I\mathcal{I} is the input-injection adapter, and H\mathcal{H} denotes one pass through the looped-hybrid stack. Figure 6 in Appendix § 9.§ 1 illustrates this wrapper. The wrapper isolates input injection from the remainder of the PARCAE architecture. It does not introduce the prelude/core/coda decomposition, separate recurrent workspace, stochastic recurrence depth, nor truncated backpropagation.

Figure 6. PARCAE-style adapter applied as a wrapper around the Looped Hybrid recurrence. The adapter repeatedly injects the representation entering recurrence but does not introduce the PARCAE prelude/core/coda decomposition.

Cache Compression We distinguish two cache-compression regimes. Under inference-only compression, the model is trained with ordinary attention [10]:

Attn⁡r,l=softmax⁡(Qr,lKr^,l⊤dk)Vr^,l,r^=r for standard attention,r^≠r for KV replay.(1)\operatorname{Attn}_{r,l} = \operatorname{softmax}\left(\frac{Q_{r,l}K_{\hat{r},l}^{\top}}{\sqrt{d_k}}\right)V_{\hat{r},l}, \qquad\hat{r}=r\ \text{for standard attention},\qquad\hat{r}\ne r\ \text{for KV replay}. \tag*{(1)}

During standard training, keys and values are temporary activations and no persistent cache object is used. When decoding with standard attention, each iteration uses its own Kr,lK_{r,l}, Vr,lV_{r,l}. However, if during decoding a compressed cache policy is applied, the query is computed from the current hidden state but Kr^,lK_{\hat{r},l}, Vr^,lV_{\hat{r},l} are reused from an earlier writer iteration. Thus, inference-only compression exposes the model to a different KV provenance at evaluation time.

Cache-compression training incorporates the intended sharing policy into the training forward pass. Recurrence iterations are divided into writers, which compute and store fresh keys and values, and readers, which compute fresh queries but reuse the keys and values of the most recent writer. With cache stride scs_c, iterations divisible by scs_c are writers and use r^=r\hat{r}=r in (1); all other iterations are readers and use the corresponding earlier writer r^\hat{r}. The replay dictionary is not a generation cache. It exists only within one training forward pass and does not contain history from preceding autoregressive decode steps. Its purpose is to reproduce the cross-iteration provenance of keys and values that the model will encounter under compressed decoding.

Cache-Routing Policies Let r∈{0,…,R−1}r\in\{0,\ldots,R-1\} denote the recurrence iteration, let l∈Al\in\mathcal{A} denote an exact-attention layer, and let LL be the number of logical stack positions. Each policy maps (r,l)(r,l) to a cache group g(r,l)g(r,l) and therefore to the flat cache slot c(r,l)=g(r,l)L+lc(r,l)=g(r,l)L+l. The write rule w(r,l)∈{0,1}w(r,l)\in\{0,1\} specifies whether iteration rr creates fresh keys and values at layer ll or reuses the tensors already stored in the selected group. Figure 3 and Figure 4 illustrate two of the routing policies used in our experiments. Figures of other routing policies are provided in Appendix § 9.§ 2.

Figure 3. Stride-2 KV-cache routing. Orange connections indicate cache writes, dashed connections indicate cache reuse, and grey connections show hidden-state propagation.

Figure 4. Selective-loop KV-cache routing. Orange connections indicate cache writes, dashed connections indicate cache reuse, and grey connections show hidden-state propagation.

Stride-ss cache. Recurrence iterations are grouped into windows of size ss, with g(r,l)=⌊r/s⌋g(r,l)=\lfloor r/s\rfloor, c(r,l)=⌊r/s⌋L+lc(r,l)=\lfloor r/s\rfloor L+l, and w(r,l)=1[r mod s=0]w(r,l)=\mathbf{1}[r\bmod s=0]. Only the first recurrence iteration in each stride group writes new keys and values. The shared-cache and per-iteration-cache cases are the special cases s=Rs=R and s=1s=1, respectively. Figure 3 illustrates the routing pattern for s=2s=2.

Selective-loop cache. Let UU be the set of recurrence iterations which are selected to receive independent cache storage. Selected loops are assigned their own group using an offset of +1+1; all other loops use the shared fallback group:

g(r,l)={r+1,r∈U,0,r∉U,w(r,l)=1[r∈U]∨1[r=0].g(r,l)= \begin{cases} r+1, & r\in U,\\ 0, & r\notin U, \end{cases} \qquad w(r,l)=\mathbf{1}[r\in U]\lor\mathbf{1}[r=0].

The offset prevents a selected loop from colliding with the shared fallback group.

Selective-layer cache. Let SS be the set of transformer-anchor layers selected for independent per-loop KV storage. Selected layers receive a separate cache group for each recurrence iteration; other layers use the shared fallback group 0.

Experiments

We organise the experiments around three questions. First, how frequently must a looped-hybrid model reconstruct exact token-level interactions? Second, does restricting recurrence to a dedicated workspace improve over recurrence through the full hidden representation? Third, can a fully looped Transformer learn to operate under a compressed cross-iteration KV topology? The non-recurrent Transformer is included as a baseline in every quantitative comparison. Parameter counts differ across the looped-hybrid configurations because recurrent mixer parameters are iteration-specific and reported in Appendix C.2; the baseline and full looped Transformer contain 406.37M parameters. Higher is better for all length-normalised accuracy metrics; lower is better for perplexity and memory. Bold entries denote the best observed result in each table column. For CORE and CORE-Extended, we report the mean ±\pm standard deviation across evaluation seeds for the selected checkpoint. Additional training and evaluation details are provided in Appendix B.

Baseline. Our primary baseline is a 24-layer, non-recurrent Transformer trained with sequence length 4096, learning rate 3×10−33\times10^{-3}, and a token budget of 5.03B tokens. It provides a parameter- and training-budget

Results and Discussion

reference for the looped-hybrid and cache-compression experiments. The per-iteration control establishes the reference performance of a looped Transformer without cross-iteration KV compression.

Looped Hybrid. The main configuration contains 24 logical layers, executes 4 recurrence iterations, and is trained at sequence length 4096. The anchor stride sds_d controls the distance between exact-attention Transformer anchors. We also compare a matched 12-layer, 8-iteration configuration used for the PARCAE architecture and evaluate a wrapper ablation that applies the PARCAE adapter without the native PARCAE architecture.

PARCAE Implementation with mLSTM. We followed the original PARCAE training configuration [9] as faithfully as permitted by our infrastructure. In particular, the reference configuration uses 8 recurrence iterations, hidden and recurrent dimensions of 1024, backpropagation depth 4, gradient clipping at 1.0, no warm-up, and a 50% cool-down. Our principal difference is the training budget: the original 370M-parameter configuration specifies 29.6B tokens; models here are generally trained on 5.03B tokens. We also use our Nemotron-CC-HQ data pipeline, tokeniser, and AdamW optimiser rather than the original FineWeb-Edu, tokeniser, and Muon setup. The comparison should therefore be interpreted as an architectural study under our common budget, not as a reproduction of the published PARCAE results. Changing the recurrent workspace dimension drd_r also changes the number of model parameters and the computation performed by each recurrent-core application. The dr∈{768,1024,1536}d_r \in\{768, 1024, 1536\} configurations contain 241.10M, 259.76M, and 315.94M parameters, respectively; exact counts are reported in Appendix § 11.§ 2. We additionally train the 2/4/6, dr=1536d_r = 1536 configuration for the original PARCAE budget of 29.6B tokens.

KV Replay and Reasoning. We evaluate a fully looped Transformer with 24 physical layers and 6 recurrence iterations. Models trained with recurrent KV replay are compared with those trained using ordinary attention and compressed only during evaluation. Stride 2 retains three recurrence-dependent KV groups, stride 3 retains two. We evaluate both matched and mismatched train–inference topologies. Additionally, we evaluate GSM8K [4] using 8-shot chain-of-thought prompting with 8 generations per problem and report the mean Majority@8 and Pass@8 over all problems together with 95% percentile bootstrap confidence intervals from 10 000 resamples of the GSM8K evaluation set.

Results and Discussion

Looped Hybrid with mLSTM and Gated DeltaNet Mixers Figure 5 examines the effect of recurrent mixer choice in the looped-hybrid architecture and its sensitivity to the spacing of the remaining attention layers. Full results, including perplexity and the wrapper ablation, are provided in Appendix § 12, Table 6.

Looped-hybrid results on the main setup

Figure 5. Looped-hybrid results on the main setup: we report normalised accuracies on PIQA, WinoGrande, ARC-Easy, HellaSwag, and SciQ, together with CORE; sds_d denotes the Transformer-anchor stride.

Table 6: Complete evaluation results for the Looped-Hybrid experiments. “Main” denotes the 24-layer, four-iteration, 4096-token setup; “matched” denotes the 12-layer, 8-iteration, 2048-token setup. We report normalised accuracy on PIQA, WinoGrande, ARC-Easy, HellaSwag, ARC-Challenge, and SciQ; CORE metrics are reported in percentage points. The table includes WikiText perplexity omitted from the main-paper results table.

ModelRRLLsds_dPIQA-NWino.ARC-E-NHella.-NARC-C-NSciQ-NCORECORE Ext.WikiText PPL
Transformer baseline124–0.6640.5000.4970.3690.2760.71314.95 ± 0.2910.26 ± 0.3935.988
mLSTM, main42420.6330.5050.4480.3680.2570.64113.56 ± 0.259.87 ± 0.3557.423
mLSTM, main42430.6440.5220.4470.3720.2680.67413.08 ± 0.139.12 ± 0.1952.008
mLSTM, main42440.6520.5210.4730.3720.2660.67713.16 ± 0.159.35 ± 0.3848.746
mLSTM, main42460.6490.5110.4470.3530.2700.62611.76 ± 0.028.20 ± 0.0952.524
mLSTM, main42480.5870.5060.3910.2750.2290.4905.94 ± 0.054.31 ± 0.11105.098
mLSTM, main424120.5830.5110.3860.2710.2240.5004.70 ± 0.114.80 ± 0.14115.087
mLSTM, matched81230.6260.5000.4220.3080.2650.57310.83 ± 0.127.99 ± 0.3183.549
mLSTM, matched81240.6260.5240.4410.3070.2460.5759.22 ± 0.166.24 ± 0.2382.179
mLSTM, matched81260.6240.5070.4150.3050.2460.5876.89 ± 0.115.97 ± 0.2186.494
mLSTM + adapter, main42440.5340.4920.2940.2590.2240.304−1.36 ± 0.041.49 ± 0.091324.289
mLSTM + adapter, main42460.5230.4970.2810.2620.2430.273−1.67 ± 0.031.41 ± 0.122767.665
mLSTM + adapter, main42480.5350.4960.3000.2570.2060.265−1.28 ± 0.031.21 ± 0.041666.295
mLSTM + adapter, main424120.5360.5010.2930.2490.2300.278−1.88 ± 0.060.96 ± 0.06961.664
Gated DeltaNet, main42420.6690.5240.5300.4300.3050.77118.08 ± 0.1111.77 ± 0.1730.786
Gated DeltaNet, main42430.6760.5280.5370.4410.3010.78218.79 ± 0.0612.62 ± 0.1130.703
Gated DeltaNet, main42440.6960.5220.5630.4460.2920.78916.26 ± 0.0611.03 ± 0.2230.229
Gated DeltaNet, main42460.6800.5310.5430.4490.3150.76518.79 ± 0.1112.70 ± 0.0929.974

Table 6.

For mLSTM hybrids, models with denser Transformer anchors achieve higher downstream accuracy and lower perplexity than those with larger anchor strides, but the results do not establish a monotonic relationship among the strides. Applying the PARCAE adapter as an isolated wrapper degrades accuracy. The input-reinjection wrapper alone is therefore insufficient to improve the looped-hybrid architecture under the evaluated training setting. In our settings, the looped-hybrid performance depends strongly on the choice of recurrent mixer, and replacing the mLSTM mixers with Gated DeltaNet substantially improves the results.

PARCAE Implementation with mLSTM Table 1 compares two prelude/core/coda allocations and recurrent workspace dimensions dr∈{768,1536}d_r \in\{768, 1536\}; the complete sweep, including dr=1024d_r = 1024 and the 6/4/2 allocation, is provided in Appendix § 12.§ 2. Increasing the recurrent workspace dimension does not produce a uniform improvement across the two allocations. The 29.6B-token run improves every reported metric relative to the same 2/4/6, dr=1536d_r = 1536 configuration trained for 5.03B tokens.

Modeldrd_rArch.PIQA-NWino.ARC-E-NHella.-NARC-C-NSciQ-NCORECORE Ext.WikiText PPL
Transformer baseline––0.6640.5000.4970.3690.2760.71314.95 ± 0.2910.26 ± 0.3935.988
PARCAE with mLSTM7682/4/60.6340.5020.4400.3130.2560.64210.50 ± 0.177.57 ± 0.1348.741
PARCAE with mLSTM15362/4/60.6320.5100.4360.3230.2360.66812.47 ± 0.168.80 ± 0.3945.289
PARCAE with mLSTM7684/4/40.6320.5240.4390.3130.2660.63112.06 ± 0.178.90 ± 0.2348.700
PARCAE with mLSTM15364/4/40.6330.5150.4490.3260.2470.65012.09 ± 0.158.42 ± 0.1145.460
Extended training (original PARCAE budget)
PARCAE with mLSTM15362/4/60.6540.5130.4920.3690.2760.71915.10 ± 0.069.52 ± 0.2535.750

Table 1. Selected results of our PARCAE implementation. Architecture denotes the prelude/core/coda allocation, drd_r is the recurrent workspace dimension. The non-recurrent Transformer uses the main 24-layer setup. CORE metrics are reported in percentage points.

Cache-Compression Training Table 2 compares replay-trained models with models compressed only during evaluation. For both cache strides, matched replay training outperforms inference-only cache compression when training and evaluation use the same cache topology. A checkpoint can be evaluated with different replay strides, but our results show that this does not preserve performance equally across topologies.

TrainingEval.PIQA-NWino.ARC-E-NHella.-NARC-C-NSciQ-NCORECORE Ext.WikiText PPL
Transformer baseline–0.6640.5000.4970.3690.2760.71314.95 ± 0.2910.26 ± 0.3935.988
Full looped TransformerPer-iteration0.6780.5280.5420.4290.3030.78218.96 ± 0.2312.58 ± 0.0931.204
Replay s=2s = 2s=2s = 20.6600.5020.5080.3930.2830.73916.76 ± 0.0711.07 ± 0.2934.214
Replay s=2s = 2s=3s = 30.6060.5100.3740.3420.2790.54811.29 ± 0.028.59 ± 0.05162.992
Replay s=3s = 3s=2s = 20.6700.5100.5120.4240.3100.69811.34 ± 0.158.24 ± 0.0560.969
Replay s=3s = 3s=3s = 30.6780.5420.5320.4180.2960.78418.71 ± 0.1612.41 ± 0.1631.095
Standards=2s = 20.6510.4950.4600.3850.2820.61114.48 ± 0.1010.39 ± 0.1883.867
Standards=3s = 30.6150.4960.3880.3470.2850.48311.66 ± 0.098.86 ± 0.14180.065

Table 2. Cross-iteration KV compression in a 6-iteration looped Transformer. The evaluation column specifies the decoding cache stride. CORE metrics are reported in percentage points.

TrainingEval.Majority@8Pass@8
Replay s=2s = 2s=2s = 20.016 [0.010, 0.023]0.080 [0.066, 0.096]
Replay s=2s = 2s=3s = 30.005 [0.002, 0.010]0.039 [0.029, 0.049]
Replay s=3s = 3s=3s = 30.024 [0.017, 0.033]0.092 [0.077, 0.108]
Standards=2s = 20.017 [0.010, 0.024]0.072 [0.058, 0.086]
Standards=3s = 30.006 [0.002, 0.011]0.046 [0.034, 0.057]

Table 3. GSM8K reasoning with cross-iteration KV sharing. Values are point estimates with 95% percentile bootstrap confidence intervals.

Model and cache topologyPeak memory delta (MB)
Transformer baseline2205.1
Looped Transformer, stride 23241.5 ± 350.7
Looped Transformer, stride 32922.9 ± 348.3

Table 4. Peak evaluation-memory delta during 8-shot GSM8K decoding. Stride 2 retains three recurrence-dependent KV groups; stride 3 retains two.

The strongest observed results are obtained by replay training with stride 3, despite retaining fewer KV groups than stride 2. For the evaluated configurations, more aggressive sharing is therefore not inherently destructive when it is represented during optimisation.

Table 3 reports Majority@8 and Pass@8 together with 95% bootstrap confidence intervals computed from 10 000 resamples of the GSM8K evaluation set. Matched replay training improves reasoning under compressed decoding, with the strongest results obtained when both training and evaluation use stride 3.

Caution

However, the absolute GSM8K scores are low, and each configuration uses a single training seed. These results should therefore be treated as a comparison among cache policies and they do not establish that the observed ordering is robust across training runs.

Memory Analysis Table 4 reports the maximum GPU memory usage observed during eight-shot GSM8K decoding, measured relative to GPU memory usage immediately before launching the decoding process. Values for the looped Transformer are aggregated over evaluation seeds 42, 43, and 44. The full per-iteration-cache configuration with 6 iterations was computationally infeasible in our setup. Within the recurrent models, reducing the number of KV groups lowers the measured peak but the difference is smaller than either standard deviation. The measurement also includes parameters, temporary activations, allocator overhead, and other states unaffected by cache routing.

The analytical KV-cache sizes in bytes are computed as

2⏟K and V×24⏟layers×G⏟KV groups×16⏟KV heads×64⏟head dimension×2⏟bytes per BF16 element.\underbrace{2}_{K\ \text{and}\ V} \times \underbrace{24}_{\text{layers}} \times \underbrace{G}_{\text{KV groups}} \times \underbrace{16}_{\text{KV heads}} \times \underbrace{64}_{\text{head dimension}} \times \underbrace{2}_{\text{bytes per BF16 element}}.
Cache policyGGKV (KiB/token)
Per iteration6576
Stride 23288
Stride 32192

Table 8.

Conclusion

Depth recurrence reduces parameter growth but introduces additional inference memory through the KV cache. We studied three approaches to reducing this memory cost: replacing most attention layers with recurrent mixers, restricting recurrence to a dedicated latent workspace, and training a looped Transformer to reuse KV memory across recurrence iterations. In the looped-hybrid architecture, the effect of Transformer-anchor spacing depends on the recurrent mixer. For mLSTM, denser Transformer anchors are associated with higher downstream accuracy and lower language-modelling perplexity in the evaluated configurations, although the results do not establish a monotonic relationship across strides. Gated DeltaNet is comparatively robust across the evaluated spacings. Across the evaluated PARCAE configurations, no single architecture dominates all metrics, and the effect of recurrent workspace dimension depends on the allocation of Transformer layers around the recurrent core. The strongest observed result comes from recurrent KV replay. Replay-trained models outperform inference-only cache compression when training and evaluation use the same cache topology, with the strongest overall results obtained for the evaluated stride-3 configuration. Thus in our setting, cross-iteration KV sharing can be incorporated into training while retaining performance close to the full looped Transformer with separate KV caches.

Discussion and Future Work. The baseline and full looped Transformer contain 406.37M parameters, while the looped-hybrid configurations range from 965M to 1.47B parameters. Additionally, our experiments compare complete configurations rather than isolating every architectural factor. The results are therefore exploratory rather than resource-matched comparisons. The relative ordering of architectures and policies may change at larger scales, with different datasets, or under longer training. The replay results are evaluated only for fixed deterministic cache-sharing schedules and do not establish that the observed improvements extend to token-dependent routing or policies in which different layers use different training-time replay schedules. We report GSM8K performance only as a controlled comparison of cache policies; it should not be interpreted as evidence of strong mathematical reasoning. Reported memory measurements used to establish relative comparisons between routing policies are implementation-level peak values obtained within our evaluation pipeline and include runtime overhead beyond the KV cache itself.

Our work is foundational and has no specific deployment target. More memory-efficient depth-recurrent architectures could reduce the computational resources required for downstream language-model applications, and potentially make such applications easier to deploy at scale; their broader effects therefore depend on the downstream systems in which they are used.

Acknowledgements

This work was supported by European Union’s Horizon Europe research and innovation programme under grant agreement number 101214398 (ELLIOT). We acknowledge EuroHPC Joint Undertaking for awarding us access to Leonardo at CINECA, Italy, and Deucalion at MACC, Portugal.

Additional Figures

Parcae Wrapper

Figure 6 illustrates the wrapper ablation described in § 3.

Additional Cache-Routing Policies

Figures 7, 8, and 9 illustrate the cache-routing policies defined in § 3 that are omitted from the main paper. Orange connections denote cache writes, dashed connections denote cache reuse, and grey connections denote hidden-state propagation.

Figure 7. Shared KV-cache routing. Only the first recurrence iteration writes keys and values; all subsequent iterations reuse the same cache group.

Figure 8. Per-iteration KV-cache routing. Every recurrence iteration writes to and reads from an independent cache group.

Figure 9. Selective-layer KV-cache routing. Selected exact-attention layers retain independent per-iteration caches, while non-selected layers reuse the shared fallback group.

Experimental Setup

Training Data and Evaluation

All models are trained on the Nemotron-CC-HQ corpus using the m2rnn-400m tokenizer. Evaluation uses the Hugging Face model interface through lm-evaluation-harness. Unless stated otherwise, reported downstream metrics include PIQA, WinoGrande, ARC-Easy, HellaSwag, ARC-Challenge, SciQ, CORE, CORE Extended, and WikiText perplexity. Models are loaded from the exported Hugging Face checkpoints with batch size 1, BOS-token insertion enabled, and chat-template application disabled. The same selected checkpoint is used for all downstream evaluations.

Training Configurations

Main Looped Hybrid. Unless stated otherwise, looped-hybrid experiments use 24 logical layers, 4 recurrence iterations, and sequence length 4096. Because recurrent mixer parameters are iteration-specific, parameter count varies with the dense-anchor stride, ranging from 965.004M parameters at sd=2s_d = 2 to 1472.851M at sd=12s_d = 12. Exact counts are reported in Appendix § 11.§ 2.

The effective training budget is

4096×64×4×4800≈5.03×1094096 \times64 \times4 \times4800 \approx5.03 \times10^{9}

tokens.

Training uses TorchAdamW with learning rate 3×10−33 \times10^{-3}, weight decay 0.1, betas (0.9, 0.95), cosine learning-rate decay, and 480 warm-up steps. Models are trained for 4800 optimisation steps on four GPUs using BF16 precision, microbatch size 1, gradient accumulation 64, global batch size 256, and activation checkpointing.

Training and evaluation were performed on the EuroHPC Leonardo Booster supercomputer (4× NVIDIA A100 64GB GPUs per node) and the EuroHPC Deucalion supercomputer. Unless otherwise stated, the main experiments were trained on 4 GPUs using BF16 precision with activation checkpointing.

Parcae-Matched Configuration. The matched configuration follows the original PARCAE training setup as closely as practical while using the common training infrastructure. Models use 12 layers, 8 recurrence iterations, sequence length 2048, BF16 precision, learning rate 2×10−42 \times10^{-4}, gradient clipping with maximum norm 1.0, no warm-up, and linear cooldown over the final 50% of training. Unless otherwise stated, the target token budget is approximately 5.03×1095.03 \times10^{9} tokens.

Parcae with mLSTM. Unless otherwise stated, PARCAE uses a recurrent workspace dimension of 1024 with a 4/4/4 prelude/core/coda decomposition, zero recurrent-state initialisation, disabled prelude normalisation, 8 recurrence iterations, backpropagation depth 4, diagonal input injection, per-sequence recurrence scheduling, Poisson-truncated-full sampling, and lockstep sampling for both recurrence variables nn and kk.

Reproducibility. Unless otherwise stated, all architectures are trained with a single training seed (42). Checkpoints are selected solely according to the lowest validation loss among the predefined checkpoint saves. Perplexities are computed once from the selected checkpoint and are not averaged across training seeds.

Compute Resources

We conducted the experiments on the EuroHPC Leonardo Booster system at CINECA and the Deucalion GPU partition. On Leonardo Booster nodes we used four NVIDIA A100 GPUs with 64 GB memory each, a 32-core Intel Xeon 8358 CPU, and 512 GB of host memory per node. Deucalion GPU nodes contain four NVIDIA A100 GPUs, a 64-core AMD EPYC 7742 CPU, and 512 GB of host memory. We used approximately 2,400 Leonardo node-hours and 400 Deucalion node-hours, for approximately 2,800 node-hours in total. This included failed runs as well as the runs reported in the paper. Individual training runs took up to 4 days, while evaluation runs took up to 24 hours. Training used BF16 precision and activation checkpointing. The remaining training and evaluation settings are reported in Appendix § 10.

Implementation Details

Depth-Recurrence Modes

Our implementation exposes several depth-recurrence modes. Only none, looped_hybrid, parcae, and the full looped Transformer configuration are used for the reported experiments.

none. A standard non-recurrent Transformer executes the physical stack once.

looped_hybrid. Transformer attention anchors are interleaved with recurrent sequence mixers, and the complete logical stack is repeated. This is the architecture defined in § 3.

parcae. Fixed Transformer prelude and coda stages surround an iterated recurrent workspace. This is the architecture defined in § 3.

Full looped Transformer. The Transformer stack is repeatedly applied with shared parameters. Its cross-iteration KV behaviour is determined by the cache-routing and replay policies defined in § 3 and § 3.

Additional implementation modes. The codebase also supports dense, kv_shared, compressed, compressed_refresh, and parcae_cache. These exploratory modes are not included in the reported experiments.

ModelConfigurationParameters (M)
Transformer baseline–406.373
Full looped Transformerall cache policies406.373
Looped Hybrid mLSTM (main)sd=2s_d = 2965.004
Looped Hybrid mLSTM (main)sd=3s_d = 31168.143
Looped Hybrid mLSTM (main)sd=4s_d = 41269.712
Looped Hybrid mLSTM (main)sd=6s_d = 61371.282
Looped Hybrid mLSTM (main)sd=8s_d = 81422.066
Looped Hybrid mLSTM (main)sd=12s_d = 121472.851
Looped Hybrid mLSTM (matched)sd=3s_d = 3994.927
Looped Hybrid mLSTM (matched)sd=4s_d = 41100.692
Looped Hybrid mLSTM (matched)sd=6s_d = 61206.458
Looped Hybrid mLSTM + adaptermain, sd=4s_d = 41272.073
Looped Hybrid GDN (main)main, sd=3s_d = 31168.082
PARCAEdr=768d_r = 768241.103
PARCAEdr=1024d_r = 1024259.755
PARCAEdr=1536d_r = 1536315.935

Table 5 (PDF p. 17). Parameter counts for the reported model configurations.

Model Parameter Counts

Parameter counts are obtained directly from the exported model state dictionaries. The non-recurrent Transformer baseline and the full looped Transformer share the same 406.37M parameters. The looped-hybrid parameter count depends on the number of recurrent mixer blocks, while the PARCAE parameter count depends only on the recurrent workspace dimension drd_r.

Additional Results

This section provides the complete evaluation results omitted from the main paper, including the full looped-hybrid sweep and the complete PARCAE recurrent-workspace sweep.

Complete Looped-Hybrid Results

Table 6 reports the complete looped-hybrid evaluation results, including the baseline and the perplexity omitted from the main-paper figure.

Complete PARCAE Results

Table 7 reports the complete recurrent-workspace sweep for the matched 12-layer, 8-iteration PARCAE setup.

Table 7: Results of our PARCAE implementation under the 12-layer, 8-iteration matched setup. Architecture denotes the prelude/core/coda allocation, drd_r is the recurrent workspace dimension. The architectural sweep uses the common 5.03B-token budget; the final row reports the 2/4/6, drd_r = 1536 model trained for the original PARCAE budget of 29.6B tokens. The non-recurrent Transformer reference uses the main 24-layer setup. CORE metrics are reported in percentage points.

Modeldrd_rArch.PIQA-NWino.ARC-E-NHella.-NARC-C-NSciQ-NCORECORE Ext.WikiText PPL
Transformer baseline–24/0/00.6640.5000.4970.3690.2760.71314.95 ± 0.2910.26 ± 0.3935.988
PARCAE with mLSTM7682/4/60.6340.5020.4400.3130.2560.64210.50 ± 0.177.57 ± 0.1348.741
PARCAE with mLSTM10242/4/60.6320.4980.4450.3150.2530.65511.36 ± 0.068.33 ± 0.3547.243
PARCAE with mLSTM15362/4/60.6320.5100.4360.3230.2360.66812.47 ± 0.168.80 ± 0.3945.289
PARCAE with mLSTM7684/4/40.6320.5240.4390.3130.2660.63112.06 ± 0.178.90 ± 0.2348.700
PARCAE with mLSTM10244/4/40.6320.5070.4450.3240.2650.63111.40 ± 0.088.21 ± 0.1646.651
PARCAE with mLSTM15364/4/40.6330.5150.4490.3260.2470.65012.09 ± 0.158.42 ± 0.1145.460
PARCAE with mLSTM7686/4/20.6330.5070.4470.3120.2450.64911.27 ± 0.128.63 ± 0.1648.991
PARCAE with mLSTM10246/4/20.6380.5210.4470.3190.2530.65410.43 ± 0.067.95 ± 0.2546.993
PARCAE with mLSTM15366/4/20.6330.5130.4560.3310.2540.64711.26 ± 0.197.80 ± 0.1045.260
Extended training (original PARCAE budget)
PARCAE with mLSTM15362/4/60.6540.5130.4920.3690.2760.71915.10 ± 0.069.52 ± 0.2535.750

Table 7.

Existing Assets and Licenses

We use the Nemotron-CC-HQ training corpus, governed by the NVIDIA Data Agreement for Model Training. Our implementation is based on lm-engine at commit 661c45ce887afcb14b59f41ced3f55b70d435765 and xlstm-flash-linear-attention at commit 072988b1f23b6702e272069217a7c407a775ff7a. Evaluation uses lm-evaluation-harness at commit 64982776449fc5687b6d510e77f5fe6a4a5327ff, released under the MIT License. The existing methods and models used in the study are cited in the main paper.

References

  1. [1]Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation, 2025. URL https://arxiv.org/abs/2507.10524.
  2. [2]Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2405.04517.
  3. [3]William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing transformer key-value cache size with cross-layer attention. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://arxiv.org/abs/2405.12981.
  4. [4]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168.
  5. [5]Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu, Yuanyuan Xu, Jiarui Liu, T. S. Eugene Ng, and Hanjie Chen. Lt2: Linear-time looped transformers. arXiv preprint arXiv:2605.20670, 2026. URL https://arxiv.org/abs/2605.20670.
  6. [8]James O’Neill and Fergal Reid. Looped latent attention: Cross-loop kv compression for looped transformers, 2026. URL https://arxiv.org/abs/2607.15456.
  7. [9]Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, and Daniel Y. Fu. Parcae: Scaling laws for stable looped language models. arXiv preprint arXiv:2604.12946, 2026. URL https://arxiv.org/abs/2604.12946.
  8. [10]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03762
  9. [11]Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo, Jordi Ros-Giralt, Arash Behboodi, and Fabio Valerio Massoli. Memory-efficient looped transformer: Decoupling compute from memory in looped language models, 2026. URL https://arxiv.org/abs/2605.07721.
  10. [12]Bohong Wu, Mengzhao Chen, Xiang Luo, Shen Yan, Qifan Yu, Fan Xia, Tianqi Zhang, Hongrui Zhan, Zheng Zhong, Xun Zhou, Siyuan Qiao, and Xingyan Bin. Parallel loop transformer for efficient test-time computation scaling. arXiv preprint arXiv:2510.24824, 2025a. URL https://arxiv.org/abs/2510.24824.
  11. [13]You Wu, Haoyi Wu, and Kewei Tu. A systematic study of cross-layer kv sharing for efficient llm inference. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 396–403, 2025b. URL https://aclanthology.org/2025.naacl-short.34/.
  12. [14]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024a. URL https://arxiv.org/abs/2412.06464.
  13. [15]Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing. arXiv preprint arXiv:2410.18517, 2024b. URL https://arxiv.org/abs/2410.18517.

Paper details

vlnc:W_27f87kv8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents