betaFeedback

WorksComputer Science

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Abstract

Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later reads weight it. A full block output contains both its input and its new update, so weighting that output also reweights the history already carried by the input. Reading individual updates instead, with the shared stack's input kept on a separate path, gives separate controls over input strength, total update weight, and allocation across updates. The Loopback Transformer implements these controls: geometric decay sets the total update weight and supplies a recency prior, and attention learns how to allocate that weight. On ClimbMix, Loopback reaches lower validation loss than standard Transformer baselines with 40-75% of their Transformer parameters (excluding token embeddings and the output head) at reference scales from 191M to 3B, with training tokens and executed depth matched. It also has higher mean accuracy over thirteen downstream tasks at 1.4B and 3B. At 191M, 374M, and 778M, it has the lowest validation loss and the highest thirteen-task mean among the standard Transformer and seven looped residual designs.

*Equal contribution

Corresponding author

Correspondence to Xuan Ouyang, Zefan Cai, and Junjie Hu ([email protected], [email protected], [email protected]).

Introduction

Looped Transformers reduce parameter count by repeatedly applying shared blocks to an evolving hidden representation ([7, 16]). Sharing separates computational depth from the number of distinct blocks and supports latent reasoning and inference-time scaling ([22, 10]). Each visit applies the same transformation to a different hidden state and so produces a different update. Parameter sharing fixes which transformation is reused; it leaves open how the updates from successive visits are stored, carried forward, and used.

Residual connections turn depth into a form of memory: each layer writes new computation into a residual stream carried forward to later layers. Standard residual addition accumulates block updates with fixed unit coefficients ([13]), whereas Attention Residuals (AttnRes) learns how to weight these updates and the input ([27]). Hyper-Connections (HC) and Hyperloop introduce learned pathways across residual streams or loop visits ([38, 36]).

This depth-memory view ([27]) connects residual design to linear attention and related recurrent models. Linear attention expresses attention over token history as writes to and reads from a recurrent memory ([15]); RetNet and Mamba-2 additionally control how that memory is retained across steps ([25, 6]). We use this write–transport–read perspective to analyze repeated computation along depth. Here, each block produces a new write from a representation assembled from earlier writes, so the memory rule also influences subsequent computation. We therefore examine what is stored and how memory transitions and readout determine each update’s contribution.

Seen this way, common residual designs tie together quantities that could be controlled separately. First, a full block output contains both its input and its new update. Weighting that output therefore also weights the history carried by the input. Storing updates separately lets the reader assign a coefficient directly to each update. Second, a joint softmax over the original input and stored updates makes them compete for a fixed total weight. Ordinary addition instead fixes the input coefficient but assigns every update weight one, so total update weight grows with the number of block applications. We therefore want separate controls over input strength, total update weight, and allocation across updates.

The Loopback Transformer is built around these three controls. It stores individual block updates rather than full block outputs and keeps the shared stack’s input on a separate path with coefficient one. Geometric decay assigns a baseline weight to each update. A softmax reader redistributes their total weight according to update content, using the decay weights as a recency prior. This separates total update weight from allocation without changing the shared blocks or their token-axis attention.

We compare against the standard Transformer, which uses distinct parameters at every layer (labeled Standard in tables and figures), and against seven looped residual designs, always at the same budget of training tokens and the same executed depth. On ClimbMix ([8]), Loopback reaches lower validation loss and a higher thirteen-task mean than Standard at 191M, 374M, 778M, 1.4B, and 3B while storing 40–75% of its Transformer parameters (Table 2, Table 4 and Table 9 and Figure 1). At 191M, 374M, and 778M, it also has the lowest validation loss and the highest thirteen-task mean of all nine designs (Table 2).

Figure 1. Loopback versus the standard Transformer from 191M to 3B. Each pair uses the same token budget and executed depth at the 20-TPP setting (tokens per reference-model parameter). (a) Validation loss and (c) thirteen-task mean accuracy against the standard Transformer’s estimated training FLOPs; (b) the same losses against Transformer parameters, where percentages give Loopback’s share of the standard Transformer’s. Lines interpolate the points.

Table 2: Residual adapters at matched tokens and depth. Validation loss, zero-shot accuracy (%) and WikiText perplexity at 191M (20 layers), 374M (32 layers), and 778M (48 layers) (Table 12); every prompt starts with the beginning-of-sequence token, as in training (Section D.2). Arrows mark lower-is-better metrics; bold marks the best result per row. Percentages compare Transformer parameters with Standard, excluding embeddings and the vocabulary head and counting shared weights once.

BenchmarkStandardPlain LoopmHCHCiHCHyperloopMHARAttnResLoopback
191M, 20 layers, [2L → 4L (×4) → 2L], 40.0% of Standard, 3.82B tokens (20 TPP)
Transformer params (M)141.5956.6456.9356.6557.0456.7956.6456.6656.65
ClimbMix loss ↓2.62342.68022.62502.64492.65642.64892.65642.64042.6187
PIQA65.5164.9665.8964.7465.3465.0765.1364.4765.78
HellaSwag33.2632.1433.7933.1832.7433.0433.1233.0733.83
WinoGrande51.7051.3852.6451.2251.7049.8050.7552.7251.14
OpenBookQA32.6029.4031.4030.2032.8031.8031.8029.2032.40
CommonsenseQA21.5419.9819.9018.6719.9821.9518.9220.4821.13
BoolQ62.7246.8261.9959.6062.1760.5861.2260.7361.44
RACE28.0429.0927.7529.8629.6728.6130.1429.9528.33
ARC-Easy47.9046.9346.5147.1446.4245.6646.2144.2847.22
ARC-Challenge26.4525.9425.6825.9425.0926.1126.6226.1125.77
MMLU (57 subjects)23.5624.5724.0124.1323.6424.0223.2526.5723.68
ANLI-R130.4032.2027.3032.4031.3032.9033.0031.6030.70
RTE49.4650.9055.6053.4353.4353.7953.7953.0754.15
LAMBADA24.4722.0125.4625.3824.9224.7223.9124.1826.80
WikiText word ppl ↓46.3050.3045.8647.5248.7247.8748.3047.0145.74
Mean (13 tasks)38.2836.6438.3038.1538.4038.3138.3038.1938.64
374M, 32 layers, [2L → 14L (×2) → 2L], 56.3% of Standard, 7.47B tokens (20 TPP)
Transformer params (M)323.02181.70181.87181.71182.52181.79181.70181.75181.72
ClimbMix loss ↓2.43952.43712.41182.42732.41992.41472.43102.41662.4008
PIQA68.8268.2368.3968.8869.4869.2169.1068.7269.10
HellaSwag38.7239.4740.4439.8940.3640.4839.3340.4241.11
WinoGrande51.9352.9652.8852.1752.0152.4952.8851.1452.09
OpenBookQA30.8034.0033.6032.8031.2032.0033.6033.2034.60
CommonsenseQA19.9820.2319.9021.0520.3121.2120.6421.2119.74
BoolQ47.1360.8958.0458.6562.4256.7355.9958.2358.90
RACE29.2831.0030.8129.3830.1428.4230.9130.8129.67
ARC-Easy51.2652.9053.3751.4352.6554.6352.4451.7352.15
ARC-Challenge27.6528.5829.0128.4127.8228.4129.6928.1628.07
MMLU (57 subjects)25.2523.6425.3225.7624.8525.8126.2525.3726.24
ANLI-R133.3033.6032.0030.7032.6031.3033.4033.9032.40
RTE54.8753.7953.4351.2651.9950.1853.0746.5753.07
LAMBADA30.2931.6931.9431.8532.5432.7631.1933.0534.43
WikiText word ppl ↓34.7934.5733.3434.0533.6933.4534.2333.5932.61
Mean (13 tasks)39.1840.8440.7040.1740.6440.2840.6540.1940.89
778M, 48 layers, [3L → 21L (×2) → 3L], 56.3% of Standard, 15.60B tokens (20 TPP)
Transformer params (M)708.68398.63398.84398.65400.11398.74398.64398.73398.68
ClimbMix loss ↓2.26442.27372.24542.26422.25232.25392.26462.25292.2404
PIQA71.4471.5572.2071.7670.5771.1671.4971.7171.16
HellaSwag47.0747.0749.0347.7448.5648.0848.1048.7248.97
WinoGrande52.0949.8053.4353.0453.1253.2853.5155.6453.91
OpenBookQA34.8035.4035.2035.0035.6036.0034.0036.4036.20
CommonsenseQA19.1619.4118.1019.8218.7620.9720.1521.9520.80
BoolQ59.9758.9955.1760.2858.9360.3757.9857.9258.93
RACE31.4832.5431.9632.0632.3432.4433.4933.2133.30
ARC-Easy60.4457.5861.2060.5660.9858.6361.4158.4258.92
ARC-Challenge31.9131.4031.8332.2533.1931.0632.5931.7432.08
MMLU (57 subjects)25.7725.1025.7025.3724.5324.8425.4925.4425.30
ANLI-R131.3032.0033.7033.2030.7033.3033.2033.3035.20
RTE56.3257.4052.7151.2653.7953.7950.1853.0751.99
LAMBADA37.7637.8439.7138.8739.7838.0738.5437.9239.74
WikiText word ppl ↓26.3926.7825.6826.4925.8426.0126.3525.9525.33
Mean (13 tasks)43.0442.7843.0743.1743.1443.2343.0943.5043.58

Table 2.

(a) Decay and attention at two scales
191M (Standard: 2.6234)374M (Standard: 2.4198)
DecayFixedLearnedFixedLearned
Off (λ=1\lambda= 1)2.68022.66212.44152.4308
On (λ=1/4\lambda= 1/4)2.62202.6187 (−0.0047)2.39992.3971 (−0.0227)
(b) Weight initialization: 191M, 20 layers
ModelTransformer params (M)N(0,0.022)\mathcal{N}(0, 0.02^2)GPT-2PyTorch
Standard141.592.62342.64192.7145
Plain Loop56.642.68022.69772.6642
mHC56.932.62502.64082.6278
Loopback56.652.6187 (−0.0047)2.6178 (−0.0241)2.6138 (−0.1007)
(c) Weight initialization: 374M, 24 layers
ModelTransformer params (M)N(0,0.022)\mathcal{N}(0, 0.02^2)GPT-2PyTorch
Standard308.332.41982.42522.4296
Plain Loop179.862.44152.42722.4334
mHC180.062.40622.40912.4093
Loopback179.882.3971 (−0.0227)2.4117 (−0.0135)2.3938 (−0.0358)

Table 4. Decay, attention, and weight initialization. Validation loss (↓) at 20 layers (191M) and 24 layers (374M); shading marks Loopback and green gives its difference from Standard. (a) Fixed allocation uses zero queries. (b–c) Three weight-initialization schemes with learning rates held fixed; bold marks the lowest loss in each initialization column. Parameter counts exclude embeddings and the vocabulary head.

1.4B, 25 layers, 28.31B tokens3B, 24 layers, 59.77B tokens
StandardLoopbackStandardLoopback
Topology–[3L → 8L (×2) → 6L]–[6L → 6L (×2) → 6L]
Transformer params (M)1,284.61873.572,718.062,038.58
Relative to Standard100.0%68.0%100.0%75.0%
ClimbMix loss ↓2.12832.10732.00051.9849
PIQA73.8874.4877.0475.90
HellaSwag55.0256.4362.8463.72
WinoGrande56.5156.9957.2258.48
OpenBookQA37.8036.6040.4041.80
CommonsenseQA23.3418.6719.9021.87
BoolQ62.5163.2462.2364.95
RACE34.6433.4935.1236.17
ARC-Easy66.1667.0069.1971.13
ARC-Challenge36.3537.0340.5340.96
MMLU (57 subjects)24.7026.8126.2628.86
ANLI-R131.3034.8031.0032.90
RTE57.4055.2361.0156.68
LAMBADA42.6443.7248.5049.45
Thirteen-task mean (%) ↑46.3346.5048.5649.45
WikiText perplexity ↓21.0920.2917.2816.74

Table 9. Standard and Loopback at 1.4B and 3B. Results after 20 TPP, one run per arm. Task scores are zero-shot accuracies (%) with beginning-of-sequence prompts; bold marks the better result within each scale. Parameter shares compare Transformer parameters, excluding embeddings and the vocabulary head.

We make three contributions:

  • We connect residual designs with linear attention and related recurrent models by adapting their memory-update rules from token sequences to network depth. The resulting framework describes what each block writes and how its contribution is transported and read.

  • Within this framework, we design the Loopback Transformer, which adds a weighted sum of block updates to the original loop input. Geometric decay keeps the total update weight bounded as visits increase at fixed stack size. Attention adjusts each update’s weight according to its content while preserving the total.

  • On ClimbMix, Loopback reaches lower validation loss and higher zero-shot task means than matched-depth standard Transformers at five scales from 191M to 3B, with 40–75% of their Transformer parameters.

Preliminaries

Looped Transformer

A looped Transformer shares parameters across depth by repeatedly applying the same sequence of Transformer blocks. Each block contains causal token attention followed by an MLP, with a residual connection around each computation. We call this shared sequence a stack, consisting of KK distinct blocks with transformations f0,…,fK−1f_0,\ldots,f_{K-1}.

We adopt Huginn’s prefix–middle–coda layout (Geiping et al., [10]), with PP prefix blocks, a shared middle stack of KK blocks repeated for RR loop iterations, and CC coda blocks. Following Zeitoun et al. [36], we denote this topology by [PL→KL (×R)→CL][\mathrm{PL} \rightarrow\mathrm{KL}\ (\times R) \rightarrow\mathrm{CL}], where the suffix L denotes layers, counting each block application as one layer of executed depth. The model stores P+K+CP+K+C distinct blocks and executes P+RK+CP+RK+C block applications. Let h0h_0 be the prefix output entering the shared stack. Within this stack, r∈{0,…,R−1}r \in\{0,\ldots,R-1\} indexes loop iterations, and k∈{0,…,K−1}k \in\{0,\ldots,K-1\} identifies a block’s position within each iteration. The index s=rK+k∈{0,…,RK−1}s=rK+k \in\{0,\ldots,RK-1\} counts block applications across all iterations. At application ss, we apply the shared block fkf_k, where k=s mod Kk=s\bmod K.

To describe how residual connections carry computation across depth, we distinguish a block’s full output from the update it produces. Let hsh_s denote the aggregated hidden representation before application ss, and let xsx_s be the input passed to block fkf_k. A single-stream model passes one hidden representation between blocks, so xs=hsx_s=h_s; a multi-stream model may form xsx_s by mixing several parallel hidden states. Aggregating the block’s result with stored computation produces the next readout hs+1h_{s+1}. The full block output is fk(xs)f_k(x_s), and the block update is

δs=fk(xs)−xs.(1)\delta_s=f_k(x_s)-x_s. \tag*{(1)}

We suppress batch and token indices; each token’s block update is in Rd\mathbb{R}^d, where dd is the hidden width. The block update measures the change produced by the block itself, before residual aggregation.

Residual aggregation

We review ordinary residual addition, Attention Residuals, and Hyper-Connections to examine how block updates contribute to later computation. These designs accumulate updates directly, weight stored computation through attention, or maintain multiple residual streams.

Ordinary residual addition (He et al., [13]) takes xs=hsx_s=h_s and uses a fixed coefficient of one to accumulate each update:

hs+1=hs+δs=h0+∑i=0sδi.(2)h_{s+1}=h_s+\delta_s=h_0+\sum_{i=0}^{s}\delta_i. \tag*{(2)}

Attention Residuals (AttnRes) (Team et al., [27]) learns content-dependent weights over the input and stored computation. Before block application ss, its block-level formulation reads the sources y0=h0y_0=h_0 and yi+1=δiy_{i+1}=\delta_i for 0≤i<s0\leq i<s:

hs=∑i=0sπs,iyi,πs,i=softmax⁡i(qsTRMSNorm⁡(yi)).(3)h_s=\sum_{i=0}^{s}\pi_{s,i}y_i,\qquad\pi_{s,i}=\operatorname{softmax}_i(q_s^{\mathsf{T}}\operatorname{RMSNorm}(y_i)). \tag*{(3)}

The learned query qs∈Rdq_s\in\mathbb{R}^d scores normalized sources, while the read combines their original values.

Hyper-Connections (HC) (Zhu et al., [38]) carries nn parallel dd-dimensional hidden states per token through the network, forming nn residual streams. Before block application ss, these states are stored side by side in Hs∈Rd×nH_s\in\mathbb{R}^{d\times n}. At block level, HC combines these states into one input and distributes the block update back to the streams:

xs=Hsps,Hs+1=HsWs+δsbsT.(4)x_s=H_sp_s,\qquad H_{s+1}=H_sW_s+\delta_sb_s^{\mathsf{T}}. \tag*{(4)}

Here ps∈Rnp_s\in\mathbb{R}^n combines the states into the block input, Ws∈Rn×nW_s\in\mathbb{R}^{n\times n} mixes the existing states, and bs∈Rnb_s\in\mathbb{R}^n controls how much of the block update each stream receives. These weights are learned and may depend on HsH_s.

Linear attention and recurrent memory

Linear attention ([15]) summarizes token history in a recurrent memory. Each new token adds information to this memory, from which a query reads an output. A scalar-decay version of the unnormalized recurrence is

St=γtSt−1+vtkt⊤,ot=Stqt.(5)S_t = \gamma_t S_{t-1} + v_t k_t^\top,\qquad o_t = S_t q_t. \tag*{(5)}

Here tt indexes tokens, vt∈Rdv_t \in\mathbb{R}^d is the value, and kt,qt∈Rmk_t, q_t \in\mathbb{R}^m are the key and query, with feature dimension mm. The matrix St∈Rd×mS_t \in\mathbb{R}^{d \times m} stores the accumulated writes. The factor γt\gamma_t scales the previous memory, vtkt⊤v_t k_t^\top writes the current value, and qtq_t reads the unnormalized output oto_t. With feature maps absorbed into the keys and queries, additive linear attention uses γt=1\gamma_t = 1 and divides the read by the positive normalizer zt⊤qtz_t^\top q_t, where zt=∑i≤tkiz_t = \sum_{i \le t} k_i. Related recurrent models modify transport: RetNet uses fixed decay ([25]), while Gated Linear Attention (GLA) and Mamba-2 make it input dependent ([32, 6]). Along depth, each token position can maintain a memory of earlier block computations. After a block runs, its result for that token updates the memory, whose readout helps form the next block’s input. Attention inside each block still operates over tokens.

Residual connections as depth memory

Following the depth-memory view ([27]), we describe residual connections and recurrent memory by the same write, transport, and read operations. We first compare full block outputs with individual block updates as the stored value, then expand the recurrence to see how the original input and earlier block updates enter later hidden states. The expansion is what we use to compare how existing residual designs reuse earlier computation, and it leads to the design of Loopback.

A common memory interface

To trace how block results contribute to later computation, we describe residual aggregation through a common memory formulation. For each token, let Ms∈Rd×msM_s \in\mathbb{R}^{d \times m_s} hold the representations retained before block application ss, where msm_s counts the dd-dimensional vectors used to represent memory. The block input is xs=Mspsx_s = M_s p_s, with ps∈Rmsp_s \in\mathbb{R}^{m_s}. Its result supplies a value vs∈Rdv_s \in\mathbb{R}^d to write into memory, such as the full block output or the block update:

Ms+1=MsWs+vsbs⊤,hs+1=Ms+1rs.(6)\begin{aligned} M_{s+1} &= M_s W_s + v_s b_s^\top,\\ h_{s+1} &= M_{s+1} r_s. \tag*{(6)} \end{aligned}

The transport matrix Ws∈Rms×ms+1W_s \in\mathbb{R}^{m_s \times m_{s+1}} scales or mixes the existing representations. The write term vsbs⊤v_s b_s^\top adds the current value to memory, with bs∈Rms+1b_s \in\mathbb{R}^{m_{s+1}} specifying its weight in each memory component. The read combines the updated representations using rs∈Rms+1r_s \in\mathbb{R}^{m_{s+1}} to produce hs+1∈Rdh_{s+1} \in\mathbb{R}^d. Thus, psp_s forms the input before the block runs, while rsr_s reads the memory after it is updated; these weights may depend on content. Single-stream models pass hs+1h_{s+1} to the next block. Multi-stream models form the next input from Ms+1M_{s+1} using ps+1p_{s+1}. Unrolling this recurrence follows each written value through subsequent transport and read operations and gives its weight in a later readout, so different residual designs can be compared even when their memory representations differ. Table 1 summarizes their choices.

Table 1: Write, transport, and read operations for residual and attention memory, with Ms∈Rd×msM_s \in\mathbb{R}^{d \times m_s} applied per head where needed. Feature construction, output gates and normalization, feedthrough paths, and outer residual blends are not shown. The rows describe memory operations, not complete published architectures.

MethodTransport WsW_sWrite bsb_sRead rsr_s
Addition ([13]) (Standard, Plain Loop)111111
Explicit-history attention
History banks start from M0=h0M_0 = h_0 and store each new value separately. At block granularity, the value is δs\delta_s; enewe_{\mathrm{new}} selects its new storage location.
AttnRes ([27])[I,0][I, 0]enewe_{\mathrm{new}}πs+1\pi_{s+1}
MHAR ([18])[I,0][I, 0]enewe_{\mathrm{new}}headwise πs+1(h)\pi_{s+1}^{(h)}
Linear-attention family and related recurrent models
The value memory starts from M0=0M_0 = 0. Keys ksk_s and queries qsq_s include each rule’s feature mapping, when present.
Linear (normalized) ([15])IIksk_sqszs+1⊤qs\frac{q_s}{z_{s+1}^{\top}q_s}
RetNet ([25])γI\gamma Iksk_sqsq_s
GLA ([32])Diag⁡(gs)\operatorname{Diag}(g_s)ksk_sqsq_s
Mamba-2 ([6])γsI\gamma_s Iτsks\tau_s k_sqsq_s
HGRN-2 ([20])Diag⁡(gs)\operatorname{Diag}(g_s)1−gs1-g_sqsq_s
DeltaNet ([33])I−ηsksks⊤I-\eta_s k_s k_s^{\top}ηsks\eta_s k_sqsq_s
Gated DeltaNet ([34])γs(I−ηsksks⊤)\gamma_s(I-\eta_s k_s k_s^{\top})ηsks\eta_s k_sqsq_s
Kimi Delta Attention (KDA) ([26])Diag⁡(gs)(I−ηsksks⊤)\operatorname{Diag}(g_s)(I-\eta_s k_s k_s^{\top})ηsks\eta_s k_sqsq_s
RWKV-7 ([19])Diag⁡(gs)−κs(as⊙κs)⊤\operatorname{Diag}(g_s)-\kappa_s(a_s \odot\kappa_s)^{\top}ksk_sqsq_s
Mesa ([29])I−ksℓs⊤I-k_s\ell_s^{\top}ℓs\ell_sqsq_s
The regression gain ℓs\ell_s uses an additional inverse-covariance state.
LaCT-inspired ([37])Augmented prefix operators (Eq. (25))[qs;0][q_s; 0]
Hyper-connection family: M0=h01⊤M_0 = h_0\mathbf{1}^{\top}; read gives the next input xs+1x_{s+1}.
HC ([38])Learned WsW_sLearned bsb_sps+1p_{s+1}
mHC ([31])Sinkhorn⁡(Us)\operatorname{Sinkhorn}(U_s)bs≥0b_s \ge0ps+1≥0p_{s+1} \ge0
Hyperloop ([36])Diag⁡(gs)\operatorname{Diag}(g_s)bs≥0b_s \ge0ps+1≥0p_{s+1} \ge0
iHC¹IIbs≥0b_s \ge0ps+1≥0p_{s+1} \ge0

Table 1.

The memory can store past values separately or combine them in a fixed-width state. An explicit history bank grows as new values are added, while recurrent memory updates a state of fixed size. Along depth, each block application supplies a value viv_i. Let SiS_i denote the accumulated writes after application ii, starting from S−1=0S_{-1} = 0, and let qsq_s be the query used to read after application ss. With feature maps absorbed into the write vectors and queries, the unnormalized recurrence and its expansion are

Si=Si−1Wi+vibi⊤,S−1=0,(7)S_i = S_{i-1} W_i + v_i b_i^\top,\qquad S_{-1} = 0, \tag*{(7)}
os=Ssqs=∑i=0s(bi⊤Wi+1⋯Wsqs)vi.(8)o_s = S_s q_s = \sum_{i=0}^{s} \left(b_i^\top W_{i+1} \cdots W_s q_s\right)v_i. \tag*{(8)}

The second line follows by repeatedly substituting the first: each value is written by bib_i, transported through subsequent WjW_j, and read by qsq_s. Empty products are identity matrices. In the common interface, MiM_i denotes the memory before application ii. For the zero-initialized case M0=0M_0 = 0, the correspondence is Mi=Si−1M_i = S_{i-1}, Mi+1=SiM_{i+1} = S_i, and ri=qir_i = q_i. This initializes the value memory, not the network input; an input path can be kept separately or included in a nonzero M0M_0. For a general initial state, unrolling the common recurrence also contributes M0W0⋯WsrsM_0W_0 \cdots W_sr_s to the readout. The general block-update expansion below retains this initial-state contribution. When vi=δiv_i = \delta_i, these coefficients directly weight individual block updates. A full block output also contains its input, so its coefficient affects earlier computation as well.

Figure 2. Same shared blocks, three residual pathways. Repeated colors denote shared blocks emitting block updates. (a) Addition. (b) RsR_s aggregates the input and updates computed so far using explicit history or a fixed-size recurrent memory. (c) Multiple streams with input read, transport, write, and final readout.

Outputs versus updates

A full block output fk(xs)=xs+δsf_k(x_s) = x_s + \delta_s contains both the current block update and earlier computation carried by its input. A read weight on this output therefore applies to both. Storing δs\delta_s separately lets the read assign a coefficient directly to that block update.

Consider two successive full block outputs under ordinary residual addition: h1=h0+δ0h_1 = h_0 + \delta_0 and h2=h0+δ0+δ1h_2 = h_0 + \delta_0 + \delta_1. Let ρ∈[0,1]\rho\in[0, 1] be the read weight on h1h_1, with the remaining weight 1−ρ1 - \rho assigned to h2h_2. Their weighted read is

ρh1+(1−ρ)h2=h0+δ0+(1−ρ)δ1.\rho h_1 + (1 - \rho)h_2 = h_0 + \delta_0 + (1 - \rho)\delta_1.

Because δ0\delta_0 appears in both stored outputs, its coefficient is always one. The update δ1\delta_1 appears only in h2h_2, so its coefficient is 1−ρ1-\rho. More generally, a nonnegative normalized read over ordinary additive states assigns nonincreasing coefficients to successive updates (Section A.1). This ordering need not hold for reweighted trajectories. Reading updates separately removes the cumulative-sum restriction.

The example shows how a stored output carries updates already present in earlier states. In the common memory formulation, this occurs through the input term in fk(xs)=xs+δsf_k(x_s) = x_s + \delta_s. To make its effect explicit, let the binary indicator χ\chi specify the stored value: χ=0\chi= 0 selects the full block output, and χ=1\chi= 1 selects the block update. The definition in Eq. (1) then gives

vs(χ)=fk(xs)−χxs=δs+(1−χ)xs,χ∈{0,1}.(9)v_s^{(\chi)} = f_k(x_s) - \chi x_s = \delta_s + (1 - \chi)x_s,\qquad\chi\in\{0,1\}. \tag*{(9)}

Using xs=Mspsx_s = M_s p_s in Eq. (6) gives

Ms+1=Ms[Ws+(1−χ)psbsT]+δsbsT.(10)M_{s+1} = M_s\left[W_s + (1-\chi)p_s b_s^{\mathsf{T}}\right] + \delta_s b_s^{\mathsf{T}}. \tag*{(10)}

Here AsA_s is the effective transport when the recurrence is expressed in terms of block updates. With full-output writes, the additional term carries the input back into memory, since MspsbsT=xsbsTM_s p_s b_s^{\mathsf{T}} = x_s b_s^{\mathsf{T}}. With block-update writes, this term vanishes and As=WsA_s = W_s. Constraints on WsW_s alone therefore need not hold for the effective transport AsA_s. The block update δs\delta_s still depends on the input. An unrestricted transport operator can absorb the extra term; the distinction concerns parameterization and control, not a general expressivity advantage.

Input and update weights

Expanding Eq. (10) separates the original input from all subsequent block updates, including those contained in full block outputs. For M0=h0a0TM_0 = h_0a_0^{\mathsf{T}}, the vector a0∈Rm0a_0 \in\mathbb{R}^{m_0} specifies how the input is placed in memory (all ones, for example, copy it into every stream). The read becomes

hs+1=as+1h0+∑i=0scs,iδi,(11)h_{s+1} = a_{s+1}h_0 + \sum_{i=0}^{s} c_{s,i}\delta_i, \tag*{(11)}
cs,i=biTAi+1⋯Asrs,as+1=a0TA0⋯Asrs.(12)c_{s,i} = b_i^{\mathsf{T}}A_{i+1}\cdots A_s r_s,\qquad a_{s+1} = a_0^{\mathsf{T}}A_0\cdots A_s r_s. \tag*{(12)}

Each cs,ic_{s,i} combines write, subsequent transport, and final read; as+1a_{s+1} includes the input carried inside stored values. Empty products are identity matrices. All quantities are evaluated along the actual nonlinear trajectory: this is a value decomposition, not a linear model or Jacobian. It applies headwise; value projections require operator-valued coefficients (Section A).

For nonnegative coefficients, we can separate their total from their distribution. Suppressing the destination index and taking B=∑ici>0B = \sum_i c_i > 0, write

h=ah0+B∑iπiδi,πi=ci/B,∑iπi=1.(13)h = ah_0 + B\sum_i \pi_i\delta_i,\qquad\pi_i = c_i/B,\qquad\sum_i \pi_i = 1. \tag*{(13)}

Here aa is the input coefficient, BB is the total update weight, and πi\pi_i specifies the fraction assigned to update ii. The total BB sums coefficients; it is not a representation norm. Ordinary addition fixes a=1a = 1 and gives every update coefficient one, so B=s+1B = s + 1 grows with the number of block applications. A joint softmax over the input and block updates instead enforces a+B=1a + B = 1: increasing the total weight on updates reduces the input weight.

Residual and recurrent models

Table 1 contrasts explicit history, recurrent compression, and stream transport. Addition starts from M0=h0M_0 = h_0. AttnRes and Multi-Head Attention Residuals (MHAR) ([18]) use explicit history banks; the block-level formulations in the table append block updates. Our evaluated adapters use Block AttnRes and a loop-state MHAR reader with an outer residual blend; their sources and placements are specified in Section D.2. Manifold-constrained Hyper-Connections (mHC) ([31]) constrains transport by applying Sinkhorn normalization to a learned matrix UsU_s. Hyperloop’s transport is diagonal, and iHC1 uses identity transport.

For recurrent memory, γs\gamma_s scales the entire previous state, while Diag⁡(gs)\operatorname{Diag}(g_s) scales its components separately; Diag forms a diagonal matrix from a vector. RetNet uses a fixed γ\gamma. In normalized linear attention, zs+1=∑i=0skiz_{s+1} = \sum_{i=0}^{s} k_i accumulates keys and zs+1Tqs>0z_{s+1}^{\mathsf{T}}q_s > 0 is required. Our evaluated linear-attention adapters omit this kernel normalizer and use rs=qsr_s = q_s. GLA uses key-side gating, while Mamba-2 couples its scalar decay and write scale through γs=exp⁡(−ατs)\gamma_s = \exp(-\alpha\tau_s), α,τs>0\alpha,\tau_s > 0. The delta-rule families use ηs\eta_s to control a correction of the memory’s prediction of the new value; this correction is distinct from the block update δs\delta_s. RWKV-7 uses a removal direction κs\kappa_s with elementwise strength asa_s, where ⊙\odot denotes elementwise multiplication. Mesa additionally tracks inverse covariance; our LCT-inspired reader updates from the stored prefix. These extra states and the remaining implementation conventions are detailed in Section B.

Loopback Transformer

Storing block updates separately allows each read weight to act directly on one update. Loopback uses this representation to form each new hidden state from the original loop input h0h_0 and a weighted sum of past block updates (Figure 3). Geometric decay bounds their total weight as loop iterations increase, while attention adjusts their relative weights using update content. The input stays outside this allocation with coefficient one, preserving the usual residual identity path: if all updates vanish, the shared stack returns h0h_0. Only the residual aggregation changes; the shared blocks are unchanged.

Figure 3. Loopback separates input, decay, and allocation. (a) Shared blocks with causal history reads. (b) Iteration-end decay sets the prior and total weight. (c) Attention redistributes that total over block updates, keeping the input coefficient at one.

Geometric decay sets the prior and total weight

High representation similarity has been observed across depth in Pre-Norm Transformers and across iterations in looped Transformers ([38, 36]). If successive block updates share a common component, ordinary residual addition reinforces that component at every iteration. Decay limits this accumulation by shrinking the weights of earlier updates. The decay-only reference accumulates updates within each loop iteration and applies a fixed retention 0<λ≤10 < \lambda\le1 at its end.

Each new update enters with coefficient one; at an iteration boundary, all update coefficients are multiplied by λ\lambda, while the input coefficient stays at one. For K=2K = 2, the coefficients of the updates written so far, in application order, evolve as

[1]⟶[λ,λ]⟶[λ,λ,1]⟶[λ2,λ2,λ,λ].[1] \longrightarrow[\lambda,\lambda] \longrightarrow[\lambda,\lambda,1] \longrightarrow[\lambda^{2},\lambda^{2},\lambda,\lambda].

After two iterations, the first iteration’s updates have crossed two boundaries and the second iteration’s updates only one. Repeated scaling therefore produces geometric weights. In this decay-only reference, an iteration maps its incoming state hh to

h←h0+λ(F(h)−h0),F=fK−1∘⋯∘f0,0<λ≤1.(14)h \mathrel{\gets} h_0 + \lambda\left(F(h)-h_0\right), \qquad F = f_{K-1}\circ\cdots\circ f_0,\qquad0 < \lambda\le1. \tag*{(14)}

Decay follows every iteration, including the last before the coda. This reference defines Loopback’s prior and its trajectory at initialization; the learned reader below subsequently reweights updates after each block. No separate reference forward pass is evaluated.

This rule is a scalar instance of the recurrent memory in Eq. (7a). In the reference, write hs=h0+ush_s = h_0+u_s, with u0=0u_0 = 0. The accumulated update usu_s and the prior coefficient cs,i0c^0_{s,i} of update ii after application ss follow

us+1=γsus+αsδs,cs,i0=αi∏j=i+1sγj.(15)u_{s+1} = \gamma_s u_s + \alpha_s\delta_s,\qquad c^0_{s,i} = \alpha_i \prod_{j=i+1}^{s}\gamma_j. \tag*{(15)}

Here αs\alpha_s scales the new write and γs\gamma_s retains the previous accumulation. Both equal λ\lambda when (s+1) mod K=0(s+1)\bmod K = 0 and one otherwise. Each write is scaled on entry by αi\alpha_i and subsequently by the retention factors γj\gamma_j. For $0 \le i \le s < KR$, counting iteration boundaries from write ii through read ss gives

cs,i0=λ⌊(s+1)/K⌋−⌊i/K⌋,Bs=∑i=0scs,i0.(16)c^0_{s,i} = \lambda^{\lfloor(s+1)/K\rfloor-\lfloor i/K\rfloor},\qquad B_s = \sum_{i=0}^{s}c^0_{s,i}. \tag*{(16)}

Superscript 00 marks the prior, and BsB_s is its total weight at read ss. For λ<1\lambda< 1, the prior favors later iterations. After RR complete iterations, BKR−1=K∑j=1RλjB_{KR-1}=K\sum_{j=1}^{R}\lambda^j, which equals KRKR without decay and is bounded by Kλ/(1−λ)K\lambda/(1-\lambda) for fixed 0<λ<10 < \lambda< 1. Thus a fixed retention controls accumulation as the number of iterations grows.

Softmax attention reweights the updates

The geometric prior assigns weights by loop iteration. Earlier updates may remain useful for forming the next hidden state, and updates from the same iteration can play different roles despite receiving equal prior weights. We therefore use attention to adjust their weights using each token’s update content while preserving the total set by decay. Loopback is single-stream, so xs=hsx_s = h_s. After application ss, we append δs=fk(hs)−hs\delta_s = f_k(h_s) - h_s, with k=s mod Kk = s \bmod K, to its history. These updates come from the current Loopback trajectory; only the prior coefficients are fixed by the schedule above. For the read after this application, a learned vector qs∈Rd\mathbf{q}_s \in\mathbb{R}^d scores normalized updates ki\mathbf{k}_i for 0≤i≤s0 \le i \le s. Using ε>0\varepsilon> 0 for numerical stability, the scores zs,iz_{s,i}, weights cs,ic_{s,i}, and next state are

ki=δid−1∥δi∥2+ε,zs,i=qsTki/d,\mathbf{k}_i = \frac{\delta_i}{\sqrt{d^{-1}\lVert\delta_i\rVert^2 + \varepsilon}}, \qquad z_{s,i} = \mathbf{q}_s^{\mathsf{T}}\mathbf{k}_i/\sqrt{d},
cs,i=Bssoftmax⁡i(zs,i+log⁡cs,i0)=Bscs,i0ezs,i∑j=0scs,j0ezs,j,hs+1=h0+∑i=0scs,iδi(17)c_{s,i} = B_s\operatorname{softmax}_i(z_{s,i} + \log c^0_{s,i}) = B_s\frac{c^0_{s,i}e^{z_{s,i}}}{\sum_{j=0}^{s}c^0_{s,j}e^{z_{s,j}}}, \qquad h_{s+1} = h_0 + \sum_{i=0}^{s}c_{s,i}\delta_i \tag*{(17)}

Each baseline weight cs,i0c^0_{s,i} is multiplied by the content factor ezs,ie^{z_{s,i}}; normalization then restores the total BsB_s. Attention can therefore favor an older update while preserving ∑ics,i=Bs\sum_i c_{s,i} = B_s. This total depends on depth but is unchanged by attention at each read. The query is specific to application ss and shared across tokens; content enters through the keys. There are no query, key, or value projections. The language-model loss trains the queries to redistribute weight (Section C.2). The readout replaces the current mixture of updates and retains the input coefficient of one. It feeds the next block, or the coda after the final application; weighting does not modify the stored updates. We initialize the queries to zero, giving cs,i=cs,i0c_{s,i} = c^0_{s,i} and recovering the decay-only trajectory. With zero queries, λ=1\lambda= 1 also recovers ordinary addition. The first read has only one update and needs no learned query (Section C).

Loopback applies the write–transport–read view of linear attention and recurrent memory along depth: blocks supply the values, retention defines their prior weights, and attention reads them to form later block inputs. Its explicit history allows individual updates to be reweighted. The bank requires O(KRd)O(KRd) storage, and total read work is O((KR)2d)O((KR)^2d) per token beyond block computation.

Experiments

We evaluate parameter efficiency, residual designs, and readers on ClimbMix (Diao et al., [8]). Main comparisons match training tokens and executed depth within each scale. Scale names and the 20-tokens-per-parameter (TPP) budget use Standard’s total parameters; Transformer counts exclude embeddings and the vocabulary head. Training and evaluation details are in Section D.

Baselines. The standard Transformer uses independent blocks; Plain Loop repeats shared blocks with ordinary addition. HC learns full stream mixing, mHC uses Sinkhorn-normalized transport, Hyperloop uses diagonal transport, and iHC uses identity transport with learned read and write gates (Table 1). AttnRes and MHAR are our loop-level adaptations of Attention Residuals and multi-head attention residuals (Team et al., [27]; Luo et al., [18]). Adapters act only inside the loop. Looped arms share the topology and learning rate within each setting; adapter placement and memory sources differ.

Parameter-efficient language modeling

We first test whether Loopback can achieve lower validation loss than depth-matched standard Transformers with fewer Transformer parameters. Table 2 compares all nine rules at 191M with 20 layers, 374M with 32 layers, and 778M with 48 layers. Loopback stores 40.0%, 56.3%, and 56.3% of the standard baseline’s Transformer parameters, respectively, and attains the lowest validation loss at all three. Its losses are 0.005, 0.039, and 0.024 nats lower than those of the standard Transformer, respectively. MHC is the closest looped baseline at all three, trailing Loopback by 0.006, 0.011, and 0.005 nats. At 374M with 24 layers, Loopback stores 58.3% of the standard baseline’s Transformer parameters and again has the lowest loss among the nine rules (Table 12). AttnRes trails Loopback by 0.022 nats at 191M and 0.009 nats at 374M with 24 layers. Figure 4 shows the training-loss curves of Loopback and the standard Transformer across all five model scales.

Figure 4. Training loss of the standard Transformer and Loopback. 100-step moving averages against tokens seen; insets show the last quarter of training; Δ\Delta is the final training-loss difference, Loopback minus the standard Transformer, over the last 100 steps. The five panels use 20, 24, 24, 25, and 24 executed layers, respectively. Loopback stores 40–75% of the standard baseline’s Transformer parameters.

LayersWidthHeadsLooped topologyStandard total (M)StandardLoopbackShare
374M, 7.47B tokens at 20 TPP (7,120 steps of 1,048,576 tokens)
16121619[1L → 7L (×2) → 1L]366.75288.92162.5456.3%
24102416[2L → 10L (×2) → 2L]373.87308.33179.8858.3%
3289614[2L → 14L (×2) → 2L]380.36323.02181.7256.3%
4876812[3L → 21L (×2) → 3L]388.97339.81191.1856.3%
6464010[4L → 28L (×2) → 4L]366.10325.14182.9356.3%
778M, 15.60B tokens at 20 TPP (7,440 steps of 2,097,152 tokens)
16185629[1L → 7L (×2) → 1L]795.44676.65380.6456.3%
24153624[2L → 10L (×2) → 2L]777.86679.55396.4458.3%
32134421[2L → 14L (×2) → 2L]779.74693.72390.2556.3%
48108817[3L → 21L (×2) → 3L]778.31708.68398.6856.3%
6496015[4L → 28L (×2) → 4L]769.35707.91398.2556.3%
LayersStandardPlain LoopmHCHCiHCHyperloopMHARAttnResLoopback
374M: ClimbMix loss ↓, 20 TPP
162.42182.45362.42342.44062.43552.43272.44532.43122.4191
242.41982.44152.40622.42492.42172.41792.42232.40652.3971
322.43952.43712.41182.42732.41992.41472.43102.41662.4008
482.45442.44252.40892.43272.42972.41542.43602.41682.4005
642.49582.46952.43282.45372.46072.43462.45152.44152.4224
778M: ClimbMix loss ↓, 20 TPP
162.26042.28572.26262.27412.26132.26962.27392.27352.2583
242.24412.27132.24322.25792.24612.25512.25952.24952.2425
322.24712.27372.24712.26272.24982.25362.26122.24532.2415
482.26442.27372.24542.26422.25232.25392.26462.25292.2404
642.27882.28162.24952.26842.25802.26142.27382.25072.2512

Table 12. Depth-sweep configurations and validation loss at 20 TPP. Training tokens are fixed; Standard’s total parameters stay within 5% of its 24-layer reference. (a) Configurations: heads are 64-dimensional and looped models make two visits. Transformer counts exclude embeddings and the vocabulary head; shared weights are counted once. Other looped arms are within 0.7% of Loopback’s Transformer count. (b) ClimbMix validation loss: bold marks the lowest loss per row; shading marks Loopback.

Loopback also has the highest thirteen-task mean and the lowest WikiText perplexity in all three settings of Table 2, although individual-task results remain mixed. Across the tested training seeds, Loopback retains lower validation loss than the looped baselines retrained at 191M and than the standard Transformer and mHC at 374M with 24 layers (Section E.1).

Parameter efficiency at larger scale

At 1.4B and 3B, Loopback uses [3L→8L (×2)→6L][3\mathrm{L} \to8\mathrm{L}\ (\times2) \to6\mathrm{L}] and [6L→6L (×2)→6L][6\mathrm{L} \to6\mathrm{L}\ (\times2) \to6\mathrm{L}], respectively, matching the standard Transformer’s 25 and 24 executed layers with 68% and 75% of its Transformer parameters. We evaluate these 20-TPP checkpoints on general benchmarks and mathematical and code tasks. Tests of additional loop iterations also include 191M and 374M models for comparison with Plain Loop.

General benchmarks

Figure 5 summarizes Transformer parameter counts, validation loss, and the thirteen-task zero-shot mean. Loopback improves both aggregate metrics at each scale, and has lower WikiText perplexity: 20.29 versus 21.09 at 1.4B and 16.74 versus 17.28 at 3B. Individual tasks are mixed, with Loopback ahead on nine and eleven of the thirteen tasks, respectively; all task scores are in Table 9.

Figure 5. General benchmarks at 1.4B and 3B. Loopback uses 68% and 75% of Standard’s Transformer parameters while attaining lower validation loss and higher thirteen-task mean accuracy. Bars use the left parameter axis; connected points use the right metric axis. Training tokens and executed depth are matched within each scale, with one run per arm at 20 TPP.

Code and mathematical evaluation

We test whether shared computation improves the modeling of mathematical and code solutions with fewer Transformer parameters. At matched training tokens and executed depth, reference-solution loss on GSM8K, MATH-500, HumanEval, and MBPP measures how well the model predicts a worked solution given the problem. Five-shot numeric-answer accuracy on six word-problem sets separately tests final-answer correctness. Loopback has lower solution loss on all four datasets at both large scales (Table 3). At 3B, word-problem accuracy rises from 2.6% to 6.4%. The nine-design comparisons at 191M and 374M also give Loopback the lowest four-set mean (Table 13). Evaluation protocols are in Section G.

Reference-solution loss ↓\downarrow1.4B3B
StandardLoopbackΔ\DeltaStandardLoopbackΔ\Delta
GSM8K1.2751.227-0.0481.1431.086-0.057
MATH-5001.4211.391-0.0301.2891.259-0.030
HumanEval1.0651.035-0.0300.9060.868-0.038
MBPP1.2821.235-0.0471.1041.078-0.026
Four-set mean1.2611.222-0.0391.1101.073-0.037
Word problems (%) ↑\uparrow2.02.6+0.62.66.4+3.8

Table 3. Mathematical and code evaluation at 1.4B and 3B. Reference-solution loss is teacher-forced NLL (nats per solution token), not generation or code-execution accuracy. Word problems use mean five-shot numeric-answer accuracy over six sets. Δ\Delta is Loopback minus Standard (nats for loss; percentage points for accuracy). Training tokens and depth are matched, with one run per arm. Shading marks Loopback; bold scores are row-best, and green differences favor Loopback.

Reference solutions ↓StandardPlain LoopmHCHCiHCHyperloopMHARAttnResLoopback
Hyper-connection familymHCHCiHCHyperloop
Attention familyMHARAttnResLoopback
191M, 20 layers, [2L → 4L (×4) → 2L] (3.82B tokens)
GSM8K1.8221.8651.7951.8011.8261.8331.8261.7971.768
MATH-5001.9672.0281.9611.9992.0232.0241.9941.9621.972
HumanEval1.8271.9351.8611.9001.9241.9051.8971.8951.842
MBPP2.0212.1182.0282.0422.0812.0742.0762.0642.018
Mean1.9091.9871.9111.9351.9641.9591.9481.9291.900
374M, 32 layers, [2L → 14L (×2) → 2L] (7.47B tokens)
GSM8K1.6001.6121.5641.5961.5541.5521.5921.5631.529
MATH-5001.7551.7551.7401.7471.7281.7371.7461.7341.707
HumanEval1.5181.5201.4871.5141.4951.4931.5031.4811.447
MBPP1.6901.7071.6641.6841.6611.6801.6911.6751.643
Mean1.6411.6481.6141.6351.6101.6151.6331.6131.582

Table 13. Reference solutions at matched tokens. Teacher-forced loss (nats per token, lower is better) of the reference solutions of GSM8K, MATH-500, HumanEval and MBPP given the problem, for all nine designs at 191M and 374M. Every prompt starts with the beginning-of-sequence token. Paired bootstrap standard errors of the differences between rules are at most 0.003 on the mean. Bold marks the lowest loss per row.

Robustness to additional visits

Shared weights allow extra visits at inference, but excessive recurrence can degrade predictions, a problem known as overthinking ([4]). Ouro and the STARS study report this sensitivity beyond the trained loop count in language models ([38, 35]). We test whether Loopback reduces it without retraining, comparing with Plain Loop’s ordinary residual addition. Extra destinations reuse the last trained visit’s queries.

At 191M, 374M, and 1.4B, Loopback’s loss rises less across all tested extra counts, up to twice the trained count (Figure 6a–Figure 6c). At eight visits, the 191M loss increases by 0.060 for Loopback versus 0.454 for Plain Loop; at four visits, the 374M increases are 0.074 versus 0.270, and the 1.4B increases are 0.028 versus 0.248. On these curves, Loopback degrades less than Plain Loop when visits are added. At 3B, one extra iteration raises Loopback’s thirteen-task mean from 49.45% to 50.17% (Figure 6d); only Loopback is evaluated at extra depth, with Standard as a fixed-depth reference. The thirteen individual 3B task results are in Section H.

Figure 6. Robustness to additional visits without retraining. (a–c) Validation loss (nats/token), trained at 20, 32, and 25 layers; (d) thirteen-task mean accuracy, trained at 24 layers. Asterisks mark trained visit counts; the dashed line shows the fixed-depth Standard reference at 3B.

Ablations: decay, attention, and initialization

We test the contributions of decay and attention and whether Loopback’s gains persist across weight initializations. Decay controls accumulated update weight; attention allocates it by content. Table 4 tests these components separately and together, under three initialization schemes: fixed-scale Gaussian, GPT-2 residual scaling (Radford et al., [21]), and PyTorch defaults. Hyperloop reports that its advantage over Standard does not persist after switching to GPT-2-style initialization and retuning learning rates (Zeitoun et al., [36]), so we check Loopback’s sensitivity to initialization, holding learning rates fixed.

The complete design gives the lowest loss at both scales in panel (a): removing attention increases loss by 0.0033 and 0.0028 nats at 191M and 374M, respectively; removing decay increases it by 0.0434 and 0.0337. Thus content-based allocation improves on decay’s fixed weights, and controlling accumulation also benefits the learned reader. Across the six scale–initialization settings in panels (b–c), Loopback outperforms Standard and Plain Loop throughout and attains the lowest loss in five.

The default Gaussian scheme draws linear and embedding weights from N(0,0.022)\mathcal{N}(0, 0.02^2). GPT-2 additionally scales the standard deviation of attention-output and MLP down-projections by 1/2D1/\sqrt{2D}, where DD is the executed depth. PyTorch uses Kaiming-uniform linear weights with a=5a = \sqrt{5} and N(0,1)\mathcal{N}(0, 1) embeddings. These choices probe fixed-scale, depth-scaled, and fan-in-scaled linear weights, with different embedding scales.

Depth at fixed parameters

We ask whether Loopback’s validation-loss advantage persists when a fixed parameter budget is allocated to different widths and depths, instead of depending on one chosen topology. At 374M and 778M, we train every configuration from scratch, holding training tokens fixed and total parameters approximately fixed while trading width for depth from 16 to 64 layers. Our loop visits stay fixed at two. We compare the standard Transformer and the eight looped rules at every depth (Figure 7, Table 12, Section F). The effect of added depth differs across residual rules. The standard Transformer loses ground as it gets deeper and narrower: it is best at 24 layers and, at 64, is 0.076 worse at 374M and 0.035 worse at 778M. Among the looped rules, Loopback is lowest at nine of the ten points, and its best tested depth moves from 24 layers at 374M to 48 at 778M. At the sweep endpoints, its margin over the standard Transformer increases from 0.003 at 16 layers to 0.073 at 64 layers at 374M, and from 0.002 to 0.028 at 778M.

Figure 7. Depth at fixed parameters. ClimbMix validation loss after 20 TPP as each scale trades width for depth at approximately fixed total parameters and fixed training tokens. Looped rules make two visits through a shared stack that stores 56–58% of the standard baseline’s Transformer parameters.

Linear attention and recurrent memory in looped Transformers

We adapt the nine recurrent cores, Mesa, and a LaCT-inspired reader in Table 1 to visit boundaries at 191M and 374M, leaving token attention unchanged. Each reader uses either cumulative visit states or whole-visit updates; both interfaces initialize to Plain Loop (Section B.4).

The update interface gives lower validation loss in all 22 family–scale comparisons (Figure 8 and Table 11). Every update-based reader improves on Plain Loop, but the best readers still trail Loopback by 0.0302 nats at 191M and 0.0285 at 374M. These results favor updates over states as the memory value. Fixed-state recurrent cores keep one practical advantage for long loops: each new write is absorbed without rereading the history, and the state does not grow with the visit count. We have not tested whether optimized recurrent kernels turn this into lower memory traffic or higher throughput in deeply looped models.

Figure 8. Depth-memory readers. ClimbMix validation-loss reduction over Plain Loop at (a) 191M with 20 layers and (b) 374M with 24 layers for the update interface (solid) and the state interface (hatched) of each reader; the standard Transformer and Loopback are references.

Reader191M Updates191M States374M, 24 layers Updates374M, 24 layers States
Gated DeltaNet2.64892.66852.43082.4333
HGRN-22.65222.66192.42562.4333
RWKV-72.65322.67912.43712.4398
Mesa2.65392.66032.42772.4342
DeltaNet2.65422.66642.43102.4329
KDA2.65432.66122.42902.4295
Mamba-22.65442.67592.43082.4319
LaCT-inspired2.65582.67742.43082.4336
GLA2.66402.67732.43522.4419
RetNet2.66502.68272.43382.4396
Linear attention2.66972.68462.43462.4417
Reference models
Plain Loop2.68022.4415
Standard2.62342.4198
Loopback2.61872.3971

Table 11. Memory readers in looped Transformers. ClimbMix validation loss (↓\downarrow), with matched machine and budget within each scale (Sections D.1 and D.3). Bold marks the best reader in each column and the best reference model at each scale. Updates and states use one and eight heads, respectively, with different output blends (Section B.4).

Efficiency: compute, throughput, and memory

Loopback’s residual aggregation adds only 0.04–0.08% forward FLOPs over Standard at matched executed depth in the three measured configurations (Figure 9 and Table 15). These extra operations normalize and score stored updates, form their weighted sum, and apply decay and mixture replacement; they introduce no Q/K/V projections or extra block evaluations. With our fused implementation, 3B training throughput is 6% below Standard’s on the same eight H100 GPUs with the same batch layout. Detailed runtime and memory measurements are reported in § 18.

Figure 9. Compute and efficiency at 374M, 24 layers. Loopback and AttnRes run fused read kernels. (a) Per-GPU training throughput on eight H100s; bar labels give hours for the 20-TPP budget. (b) Validation loss versus forward FLOPs per token in excess of the standard baseline’s.

ModelTransformer params (M)Extra FLOPsTrain tok/s/GPUPeak GiB/GPUPrefill tok/s
191M, [2L → 4L (×4) → 2L], 3.82B tokens, 16 sequences per GPU
Standard141.59–222k38885k
Plain Loop56.64+0.00%226k38886k
mHC56.93+0.15%169k42594k
HC56.65+0.07%186k42671k
iHC57.04+0.17%116k67370k
Hyperloop56.79+0.08%186k42743k
MHAR56.64+0.01%204k41810k
Loopback56.65+0.08%167k42601k
374M, [2L → 10L (×2) → 2L], 7.47B tokens, 8 sequences per GPU
Standard308.33–128k26492k
Plain Loop179.86+0.00%130k26491k
mHC180.06+0.05%115k29422k
HC179.87+0.02%120k30439k
iHC180.53+0.15%68k52220k
Hyperloop179.96+0.03%120k29457k
MHAR179.86+0.01%126k27480k
Loopback179.88+0.08%98k32352k
778M, [2L → 8L (×2) → 6L], 15.60B tokens, 8 sequences per GPU
Standard679.55–75k38269k
Plain Loop453.04+0.00%76k37269k
mHC453.33+0.04%68k42240k
Loopback453.06+0.04%63k44217k

Table 15. Compute and efficiency of the residual rules. Parameters are Transformer-only; extra forward FLOPs are relative to Standard. Percentages are rounded to two decimal places, with the positive 374M MHAR overhead rounded up to 0.01%. Training throughput and peak memory are per GPU. Prefill uses one H100.

Related work

Parameter sharing and residual reuse. Models with shared depth use weight tying [7, 16], relaxed tying with low-rank adaptations [2], latent reasoning [22, 10], and adaptive recursion [3]. Hyperloop learns connections between loop visits [36]. Residual pathways use gating, addition, and concatenation [24, 13, 14], scaling [1, 30], depth mixtures [5], multi-stream mixing [38, 31], and attention-based aggregation [27, 17 b]. Storing updates is not new; what we add is the split of update coefficients into input strength, total weight, and allocation, and an account of how each design constrains them.

Attention and recurrent memory. Sequence memory includes attention and additive fast weights [28, 15, 23], retention and gating [25, 32, 20], structured state-space dynamics [12, 11, 6], delta-rule corrections [33, 34, 26, 19], and optimization-based memory [29, 37]. We adapt these operations along depth, retaining each block’s token attention; results concern residual interfaces, not the original architectures’ efficiency.

Conclusion

We present Loopback, a residual design for parameter-efficient looped Transformers. We connect residual computation with linear attention and recurrent memory by treating successive block applications as memory steps along depth: each application writes new computation, which later blocks access through the residual connection. Loopback stores each block’s update separately and reads this history with attention. The read starts from geometrically decayed weights and redistributes them according to content without changing their sum; the weighted updates are then added to the original loop input. Across the 191M, 374M, 778M, 1.4B, and 3B scales, Loopback improves aggregate quality over matched-depth standard Transformer baselines while using 40–75% of their Transformer parameters, excluding token embeddings and the output head. In looped models, how the results of shared computation are retained and reused matters as much as which blocks are shared.

Limitations and future work

Our evidence is limited to ClimbMix pretraining at the tested scales and budgets, with single-run large-scale comparisons. The input coefficient is fixed at one; we have not tried learned or layer-specific input injection. Explicit history adds memory traffic and activations, so parameter savings need not reduce latency or peak memory. Extra visits have not improved validation loss. Next steps are other pretraining corpora, controlled comparisons of the reader interfaces, compact memories, and training with a variable visit count.

Ethics statement

This work studies the architecture of language models. All models have at most 3B parameters and are trained on ClimbMix, a public pretraining corpus, with evaluation on public benchmarks; no new data were collected and no human subjects were involved. The models are research artifacts that inherit the biases of their training corpus and are not intended for deployment. The method reduces stored parameters at matched executed depth and does not alter the risks associated with language models in general.

Proofs for the depth-memory formulation

Coefficient ordering for cumulative states

For ordinary additive states hjadd=h0+∑i<jδih_j^{\mathrm{add}} = h_0 + \sum_{i<j} \delta_i, a nonnegative normalized read gives

∑j=0s+1ρjhjadd=h0+∑i=0s(∑j=i+1s+1ρj)δi,ρj≥0,∑jρj=1.(18)\sum_{j=0}^{s+1} \rho_j h_j^{\mathrm{add}} = h_0 + \sum_{i=0}^{s} \left(\sum_{j=i+1}^{s+1} \rho_j\right)\delta_i,\qquad\rho_j \ge0,\qquad\sum_j \rho_j = 1. \tag*{(18)}

The equality follows by exchanging the finite sums: update δi\delta_i appears in every state with j>ij > i. Hence cs,i=∑j=i+1s+1ρjc_{s,i} = \sum_{j=i+1}^{s+1} \rho_j and 1≥cs,0≥⋯≥cs,s≥01 \ge c_{s,0} \ge\cdots\ge c_{s,s} \ge0. Reading block updates separately removes this cumulative-sum restriction, but does not make the updates independent.

For a reweighted trajectory vj=ajh0+∑iCj,iδiv_j = a_j h_0 + \sum_i C_{j,i}\delta_i, the induced update weights are ρTC\rho^{\mathsf{T}}C; the ordering above need not hold.

Proof of the coefficient expansion

Repeated substitution in Eq. (10) gives

Ms+1=M0A0⋯As+∑i=0sδibiTAi+1⋯As.(19)M_{s+1} = M_0 A_0 \cdots A_s + \sum_{i=0}^{s} \delta_i b_i^{\mathsf{T}} A_{i+1} \cdots A_s. \tag*{(19)}

The base case s=0s = 0 is the update rule. Multiplying the previous expansion by AsA_s and adding δsbsT\delta_s b_s^{\mathsf{T}} proves the induction step. The factors retain their order, and empty products are identity matrices. Substituting M0=h0a0TM_0 = h_0 a_0^{\mathsf{T}} and multiplying by rsr_s yields Eq. (12). A separate position write eitiTe_i t_i^{\mathsf{T}} contributes (tiTAi+1⋯Asrs)ei(t_i^{\mathsf{T}} A_{i+1} \cdots A_s r_s)e_i by the same calculation.

The identity is evaluated on the actual trajectory. Keys, queries, and gates may depend on content; the resulting coefficients are still scalar. Projections of the values themselves require operator-valued coefficients; arbitrary nonlinear value maps are outside this expansion.

Residual adapters and reader implementations

Stream transport and explicit history

The block-level interface distinguishes a block update δs\delta_s from a full block output fk(xs)f_k(x_s), where k=s mod Kk = s \bmod K. Our iHC adapter writes the block update. Hyper-Connections, mHC, and Hyperloop instead operate at visit boundaries: Hyper-Connections writes the whole-visit update, while mHC and Hyperloop write the full stack output together with a position value. Applying Eq. (10) to the stack map FF gives Ar=Wr+prbrTA_r = W_r + p_r b_r^{\mathsf{T}} for the latter two, with the additional write erbrTe_r b_r^{\mathsf{T}}; rr indexes visits here. In value-by-stream storage, mHC uses Wr=HresTW_r = H_{\mathrm{res}}^{\mathsf{T}} and constrains this transport by Sinkhorn normalization; the additional input-write path is not part of that constraint. Hyperloop uses diagonal transport. iHC instead fixes As=Ws=IA_s = W_s = I and learns nonnegative write gates, input reads, and the final-coda read. Its original rule surrounds the attention and MLP computations separately; our adapter surrounds each complete block and writes δs=fk(xs)−xs\delta_s = f_k(x_s) - x_s.

For a history bank, appending δs\delta_s uses As=[I,0]A_s = [I, 0] and a write vector selecting the new storage location. Appending the full block output instead uses As=[I,ps]A_s = [I, p_s]. The sources and placements used by AttnRes, MHAR, and Loopback are specified in Section D.2.

Normalized attention and recurrent memory

Finite-feature attention. Linear attention summarizes values vi=vi(χ)v_i = v_i^{(\chi)} in sufficient statistics (Katharopoulos et al., [15]; Schlag et al., [23]). For a finite feature map ϕ:Rd→Rm\phi: \mathbb{R}^d \to\mathbb{R}^m with κ(q,k)=ϕ(q)Tϕ(k)≥0\kappa(q,k) = \phi(q)^{\mathsf{T}}\phi(k) \ge0, associativity gives

os=∑i=0sκ(qs,ki)vi∑i=0sκ(qs,ki)=Ssϕ(qs)zsTϕ(qs),Ss=∑i=0sviϕ(ki)T,zs=∑i=0sϕ(ki).(20)o_s = \frac{\sum_{i=0}^{s} \kappa(q_s,k_i)v_i}{\sum_{i=0}^{s} \kappa(q_s,k_i)} = \frac{S_s\phi(q_s)}{z_s^{\mathsf{T}}\phi(q_s)},\qquad S_s = \sum_{i=0}^{s} v_i\phi(k_i)^{\mathsf{T}},\qquad z_s = \sum_{i=0}^{s} \phi(k_i). \tag*{(20)}

Updating Ss∈Rd×mS_s \in\mathbb{R}^{d \times m} and zs∈Rmz_s \in\mathbb{R}^m avoids storing past values for either χ\chi. The denominator must be positive; this finite-feature identity is not a general softmax equivalence.

A destination-dependent softmax query changes earlier scores, so its previous read alone generally cannot summarize the history. Our evaluated recurrent adapters instead use the unnormalized memory update in Eq. (7a), omitting the kernel normalizer and allowing signed read coefficients.

Decay and key-specific correction. The delta rule corrects a memory prediction; it does not denote the block output-minus-input transformation. At block granularity, its target can be either a full block output or a block update. For these families, let Sˉi=Si−1Di\bar{S}_i = S_{i-1}D_i:

Si=Sˉi+ηi(vi−Sˉiki)kiT,Di={IDeltaNet,γiIGated DeltaNet,Diag⁡(gi)KDA.(21)S_i = \bar{S}_i + \eta_i(v_i - \bar{S}_i k_i)k_i^{\mathsf{T}}, \qquad D_i = \begin{cases} I & \text{DeltaNet},\\ \gamma_i I & \text{Gated DeltaNet},\\ \operatorname{Diag}(g_i) & \text{KDA}. \end{cases} \tag*{(21)}

Decay acts before correction toward the new value [33, 34, 26]. Thus KDA uses Wi=Diag⁡(gi)(I−ηikikiT)W_i = \operatorname{Diag}(g_i)(I - \eta_i k_i k_i^{\mathsf{T}}) in value-by-key orientation; reversing the factors generally changes the operator. RWKV-7 subtracts a rank-one correction from diagonal transport. Its normalized removal direction κs\kappa_s and coordinatewise strength asa_s need not align with the write key ksk_s [19].

Regression readers and their additional state

The regression readers in Table 1 require additional state: Mesa tracks inverse covariance, while the LaCT-inspired reader revisits a stored prefix.

Mesa. Let K=[k0,…,ks]K = [k_0, \ldots, k_s], V=[v0,…,vs]V = [v_0, \ldots, v_s], Ω=Diag⁡(ω0,…,ωs)\Omega= \operatorname{Diag}(\omega_0, \ldots, \omega_s) with ωi≥0\omega_i \ge0, and μ>0\mu> 0. Differentiating the weighted ridge objective gives

Ss=arg⁡min⁡W 12∥(WK−V)Ω1/2∥F2+μ2∥W∥F2=VΩKT(KΩKT+μI)−1.(22)\begin{aligned} S_s &= \underset{W}{\arg\min}\ \frac{1}{2}\lVert(WK - V)\Omega^{1/2}\rVert_F^2 + \frac{\mu}{2}\lVert W\rVert_F^2\\ &= V\Omega K^{\mathsf{T}}(K\Omega K^{\mathsf{T}} + \mu I)^{-1}. \tag*{(22)} \end{aligned}

Its recursive form starts from S−1=0S_{-1} = 0 and R−1=μ−1IR_{-1} = \mu^{-1}I:

ℓi=ωiRi−1ki1+ωikiTRi−1ki,Si=Si−1(I−kiℓiT)+viℓiT,Ri=Ri−1−Ri−1kiℓiT.(23)\ell_i = \frac{\omega_i R_{i-1}k_i}{1 + \omega_i k_i^{\mathsf{T}}R_{i-1}k_i}, \qquad S_i = S_{i-1}(I - k_i\ell_i^{\mathsf{T}}) + v_i\ell_i^{\mathsf{T}}, \qquad R_i = R_{i-1} - R_{i-1}k_i\ell_i^{\mathsf{T}}. \tag*{(23)}

The rank-one inverse identity yields Ri=(μI+∑j≤iωjkjkjT)−1R_i = (\mu I + \sum_{j \le i}\omega_j k_j k_j^{\mathsf{T}})^{-1}. Substituting this into the ridge normal equations gives the update for SiS_i. Hence Wi=I−kiℓiTW_i = I - k_i\ell_i^{\mathsf{T}} and bi=ℓib_i = \ell_i, with RiR_i as control state. Our implementation evaluates the equivalent weighted Gram solve.

LaCT-inspired reader. Let Ki=[k0,…,ki]K_i = [k_0, \ldots, k_i], Vi=[v0,…,vi]V_i = [v_0, \ldots, v_i], and Ei=Diag⁡(η0,…,ηi)E_i = \operatorname{Diag}(\eta_0, \ldots, \eta_i). A gradient step on the weighted prefix prediction error, followed by the implemented normalization, is

Sˉi=Si−1(I−KiEiKiT)+ViEiKiT,Si=νiSˉi,νi=(∥Sˉi∥F+ε)−1/2,S−1=0.(24)\begin{aligned} \bar{S}_i &= S_{i-1}(I - K_iE_iK_i^{\mathsf{T}}) + V_iE_iK_i^{\mathsf{T}},\\ S_i &= \nu_i\bar{S}_i,\qquad\nu_i = (\lVert\bar{S}_i\rVert_F + \varepsilon)^{-1/2},\qquad S_{-1}=0. \tag*{(24)} \end{aligned}

Because old values are written again, SiS_i alone is insufficient for a single-new-value update. Set M−=[Si−1,V<i]M^- = [S_{i-1},V_{<i}] and Gi=I−KiEiKiTG_i = I - K_iE_iK_i^{\mathsf{T}}. The augmented operators

Wi=[νiGi00νiE<iK<iTI0],biT=[viηikiT,0T,1].(25)W_i = \begin{bmatrix} \nu_iG_i & 0 & 0\\ \nu_iE_{<i}K_{<i}^{\mathsf{T}} & I & 0 \end{bmatrix}, \qquad b_i^{\mathsf{T}} = [v_i\eta_i k_i^{\mathsf{T}},0^{\mathsf{T}},1]. \tag*{(25)}

give M+=M−Wi+vibiT=[Si,Vi]M^+ = M^-W_i + v_i b_i^{\mathsf{T}} = [S_i,V_i] by direct multiplication, with read [qi;0][q_i;0] selecting SiqiS_iq_i. The initial augmented state is [0,∅][0,\varnothing]; prefix blocks are empty at i=0i = 0. This adapter uses linear predictions over the cumulative prefix, not the published LaCT layer. Its Frobenius normalization makes the update nonlinear, preserves the value span, and remains in the derivative graph.

Visit-level reader interfaces

The two interfaces in Table 11 read at visit boundaries and initialize to Plain Loop. Both use unprojected values, L2 normalization with an epsilon floor, and no learned Q/K/V projections.

Cumulative states and whole-visit updates

Write h~(r+1)K\widetilde{h}_{(r+1)K} for the output of visit rr before its boundary combination. The paired visit update is

Δr=h~(r+1)K−hrK=∑k=0K−1δrK+k.(26)\Delta_r = \widetilde{h}_{(r+1)K} - h_{rK} = \sum_{k=0}^{K-1} \delta_{rK+k}. \tag*{(26)}

The equality telescopes under within-visit addition, as used here; block-level reweighting would also enter the visit difference.

The state interface stores h0h_0 and each visit output before its boundary blend. Denote these stored states by vjv_j. Within each reader head,

kj=L2Norm⁡(vj),vj=vj,q=L2Norm⁡(h~(r+1)K).k_j = \operatorname{L2Norm}(v_j), \qquad v_j = v_j, \qquad q = \operatorname{L2Norm}(\widetilde{h}_{(r+1)K}).

The current key and query coincide, so their inner product is one when the normalization floor is inactive. The memory output oro_r is combined with the current proposal as

h(r+1)K=(1−ζ)h~(r+1)K+ζor,ζinit=0.(27)h_{(r+1)K} = (1-\zeta)\widetilde{h}_{(r+1)K} + \zeta o_r, \qquad\zeta_{\mathrm{init}} = 0. \tag*{(27)}

The update interface stores Δ0,…,Δr\Delta_0,\ldots,\Delta_r, with

kj=L2Norm⁡(Δj),vj=Δj,q=L2Norm⁡(h~(r+1)K).k_j = \operatorname{L2Norm}(\Delta_j), \qquad v_j = \Delta_j, \qquad q = \operatorname{L2Norm}(\widetilde{h}_{(r+1)K}).

Its output is

h(r+1)K=(1−ζ)h~(r+1)K+ζ(h0+or),ζinit=0.(28)h_{(r+1)K} = (1-\zeta)\widetilde{h}_{(r+1)K} + \zeta(h_0+o_r), \qquad\zeta_{\mathrm{init}} = 0. \tag*{(28)}

In both interfaces ζ\zeta is an unconstrained learned scalar initialized to zero, recovering Plain Loop.

Gates and memory evaluation

In our adapters, gates are content-independent learned parameters indexed by junction, head, and, when required, channel. Key-addressed families write along kjk_j; HGRN-2 writes the gate complement instead. For RWKV-7, the removal direction κj\kappa_j in Table 1 is the normalized visit update in the update interface and the normalized difference of adjacent stored states in the state interface. Mesa and the LaCT-inspired reader follow Section B.3, including the latter’s prefix normalization.

The implementations evaluate short histories using the factorization Ss=∑iviws,i⊤S_s = \sum_i v_iw_{s,i}^{\top}, where ws,iw_{s,i} is a write vector after subsequent transitions, or replay the required prefixes. These evaluate the stated reads without establishing an end-to-end recurrent speedup.

Loopback implementation and derivations

Mixture replacement and initialization

After appending the newest block update and applying anchored decay at visit ends, write the current proposal as hs+1base=h0+∑icˉs,iδih^{\mathrm{base}}_{s+1} = h_0 + \sum_i \bar{c}_{s,i}\delta_i. The implementation reaches the target in Eq. (17) by adding only the coefficient difference:

hs+1=hs+1base+∑i(cs,i−cˉs,i)δi.(29)h_{s+1} = h^{\mathrm{base}}_{s+1} + \sum_i (c_{s,i} - \bar{c}_{s,i})\delta_i. \tag*{(29)}

The implementation uses skip parameter β=1−λ\beta= 1-\lambda and tracks the actual coefficients after prior reads to avoid double-counting contributions. The explicit input path has unit gain, but the full Jacobian also includes the input dependence of the updates and routing weights.

To make the initial correction vanish, target coefficients are evaluated as

cs=cs0+Bs[softmax⁡(zs+log⁡cs0)−softmax⁡(log⁡cs0)].(30)c_s = c_s^0 + B_s\left[\operatorname{softmax}(z_s+\log c_s^0)-\operatorname{softmax}(\log c_s^0)\right]. \tag*{(30)}

For a positive unfloored prior this equals Bssoftmax⁡(zs+log⁡cs0)B_s\operatorname{softmax}(z_s+\log c_s^0) in exact arithmetic. Zero queries cancel the bracket; causal induction then recovers the decay-only trajectory of § 4.1. The first read has only one source and requires no query. The (KR−1)d(KR-1)d learned parameters total 11,520 at K=4K=4, R=4R=4, d=768d=768 (the 191M setting).

The actual log prior uses log⁡max⁡(cs,i0,10−8)\log\max(c_{s,i}^0,10^{-8}) in both softmax terms. If the floor activates, the centered form still preserves the total weight in exact arithmetic and cancels at zero queries, but its coefficients need not remain nonnegative. All reported configurations satisfy λR>10−8\lambda^R>10^{-8}, so the floor is inactive.

Allocation gradient

With the update bank, prior, and total BB fixed, and the prior floor inactive so that c=Bsoftmax⁡(z+log⁡c0)c=B\operatorname{softmax}(z+\log c^0) exactly, write πi=ci/B\pi_i=c_i/B and μ=∑jπjδj\mu=\sum_j\pi_j\delta_j, so h=h0+Bμh=h_0+B\mu. The softmax derivative ∂πj/∂zi=πj(1i=j−πi)\partial\pi_j/\partial z_i=\pi_j(1_{i=j}-\pi_i) gives ∂h/∂zi=ci(δi−μ)\partial h/\partial z_i=c_i(\delta_i-\mu). Contraction with g=∇hLg=\nabla_h\mathcal{L} yields

∂L∂zi=cig⊤(δi−μ),∑i∂L∂zi=0.(31)\frac{\partial\mathcal{L}}{\partial z_i}=c_i g^\top(\delta_i-\mu),\qquad\sum_i\frac{\partial\mathcal{L}}{\partial z_i}=0. \tag*{(31)}

The sum vanishes because ∑iciδi=Bμ\sum_i c_i\delta_i=B\mu. This local gradient redistributes a fixed total according to each update’s alignment with the downstream loss gradient. Actual score changes are coupled through the query; full training also changes the keys and updates.

Training and evaluation protocols

Table 5 lists the architecture, batch composition, and budget at every scale. We give the shared recipe and evaluation protocol once, then summarize larger-scale differences. The per-GPU sequence counts in Table 15 are these training micro-batches, and timing runs use the same micro-batch and accumulation.

ScaleWidthBlocksHeadsSwiGLULoopbackSeq./stepMicro × acc.StepsTokensLR (×10−3\times10^{-3})
191M76820122,048[2L → 4L (×4) → 2L]12816 × 17,2803.82B1 / 3
374M1,02424162,816[2L → 10L (×2) → 2L]2568 × 47,1207.47B2 / 2
778M1,53624244,096[2L → 8L (×2) → 6L]5128 × 87,44015.60B1 / 1.5
1.4B2,04825325,632[3L → 8L (×2) → 6L]9608 × 157,20028.31B1 / 1.5
3B3,07224488,192[6L → 6L (×2) → 6L]1,9204 × 607,60059.77B1 / 1

Table 5. Configuration at each scale. Blocks and heads apply to Standard and to every looped arm; head width is 64 throughout. Sequences per optimizer step are the per-GPU micro-batch times gradient-accumulation steps, on 8 ranks, at 4,096 tokens per sequence. Peak learning rates are Standard / looped arms.

Architecture and training protocol

Scale names use the standard baseline’s total parameters; reported Transformer counts exclude embeddings and the vocabulary head. We use the 20-TPP training setting, with the same token budget for all models within each comparison. At 191M, all nine models train from scratch for 7,280 steps using Table 6 and Table 7, differing only in topology, residual rule, and peak learning rate. An additional eight-block standard Transformer baseline matches Plain Loop’s stored parameters (56.64M), not its twenty applications; under the same protocol and tokens, its loss is 2.7376 versus Plain Loop’s 2.6802. Budgets are rounded to a 20-step grid. During the final tenth of training, the learning-rate multiplier is 1−u1-\sqrt{u} for decay progress u∈[0,1]u\in[0,1]. All results use the final checkpoint.

SettingValue
Standard structureTwenty independent blocks
Looped structure[2L→4L (×4)→2L][2\mathrm{L} \to4\mathrm{L}\ (\times4) \to2\mathrm{L}]; eight distinct blocks, twenty applications
AttentionFull-sequence causal attention; 12 heads of width 64; no GQA
Hidden / feed-forward width768 / 2,048; SwiGLU feed-forward layers
NormalizationPre-RMSNorm; ε=10−6\varepsilon= 10^{-6}
Position encodingRoPE; θ=106\theta= 10^{6}
Context / vocabulary4,096 tokens / 32,000 Llama tokens
Embedding / vocabulary headUntied; omitted from reported Transformer counts
Trunk initializationLinear and embedding weights: N(0,0.022)\mathcal{N}(0, 0.02^{2})
Other trunk optionsBias-free attention projections; no QK normalization, logit softcap, or zero-initialized residual-output projections

Table 6. 191M architecture. Looped topology: [PL→KL (×R)→CL][PL \to KL\ (\times R) \to CL].

SettingValue
Training dataClimbMix [8]; training shards 0–239
Input constructionOnline tokenization; concatenated documents packed into fixed 4,096-token sequences
Global batch128 sequences; 8 GPU ranks ×\times 16 sequences per rank
Gradient accumulation1; 524,288 input tokens per optimizer step
Training budget7,280 steps; 3,816,816,640 input tokens
OptimizerAdamW; (β1,β2)=(0.9,0.95)(\beta_{1}, \beta_{2}) = (0.9, 0.95); ε=10−15\varepsilon= 10^{-15}
Weight decay / gradient clip0 / global norm 1.0
Peak learning rateStandard: 10−310^{-3}; all looped arms: 3×10−33 \times10^{-3}
Learning-rate scheduleLinear warmup: 128 steps; stable peak: 6,424 steps; square-root decay to zero: 728 steps
Hardware allocationOne node with eight GPU ranks for every arm
Mixed precisionBF16 parameter computation; FP32 gradient reductions
ExecutionFSDP; no activation checkpointing or model compilation

Table 7. Training recipe at the 20-TPP endpoint. Only peak learning rates differ across arms.

The 374M, 778M, 1.4B, and 3B runs use the same optimizer, tokenization, normalization, schedule, and evaluation protocols. Table 5 gives their widths, topologies, batches, budgets, and peak learning rates. The 1.4B and 3B runs use training shards 0–559; smaller scales use shards 0–239. Within each comparison, training tokens and executed depth are matched. The depth-sweep variants are specified in Section F.

Residual rules and evaluation

Trunk weights are shared across visits; connections and queries are untied where required. The mHC, Hyper-Connections, and Hyperloop adapters act at visit boundaries with four streams; mHC uses 20 Sinkhorn iterations and Hyperloop uses zero residual bias. iHC wraps each middle block with four streams, reading the block input through gates before it and writing the block update back through gates after it; it uses magnitude 2, gate offset 10−610^{-6}, and a learned final readout. MHAR reads loop states at visit boundaries with eight heads and an outer blend. AttnRes reads the loop input and block states before token attention and before the MLP in each loop block, and once at loop exit, using Block AttnRes with N=8N = 8. Loopback reads block updates after each middle block with one head and λ=1/4\lambda= 1/4. The queries and gates of MHAR, AttnRes, and Loopback start at zero.

Held-out language modeling. Validation uses disjoint ClimbMix shard 3583. As in training, documents are tokenized with beginning-of-sequence markers, concatenated, and packed into 4,096-token sequences. All models use the first 1,024 sequences and their final checkpoints, evaluated in BF16. Loss averages next-token cross-entropy over 4,095 predicted positions per sequence after shifting labels. It is reported in nats; perplexity is its exponential.

Downstream tasks. The same final checkpoints are evaluated using lm-evaluation-harness 0.4.12 [9], zero-shot prompts, full evaluation sets, and 1,000 bootstrap iterations; every prompt starts with the beginning-of-sequence token, as training prepends it to every document. HellaSwag, OpenBookQA, ARC-Easy, and ARC-Challenge use length-normalized accuracy; the other accuracy tasks use the harness’s standard accuracy. MMLU is the harness aggregate over 57 subjects and counts as one task in the unweighted thirteen-task mean. WikiText word perplexity is reported separately and excluded from this mean.

Learning-rate selection

At 191M, Standard’s lowest loss among the four peak rates tested at 20 TPP is attained at 10−310^{-3} (Table 8); looped models use their best tested rate, 3×10−33 \times10^{-3}.

Peak learning rate5×10−45 \times10^{-4}7×10−47 \times10^{-4}10−310^{-3}2×10−32 \times10^{-3}
ClimbMix loss2.66662.64202.62342.6245

Table 8. 191M Standard learning-rate comparison. ClimbMix validation loss after 20 TPP on the same machine and training recipe. Bold marks the lowest loss.

For the final 20-TPP runs, 374M uses 2×10−32 \times10^{-3} for all arms, selected for Standard from {1,1.5,2}×10−3\{1, 1.5, 2\} \times10^{-3}. At 778M, Standard inherits 10−310^{-3}; Loopback uses 1.5×10−31.5 \times10^{-3} after a check against 2×10−32 \times10^{-3}. At 1.4B, Standard uses 10−310^{-3} from {0.5,1,1.5,2}×10−3\{0.5, 1, 1.5, 2\} \times10^{-3}, and Loopback uses 1.5×10−31.5 \times10^{-3} from {1,1.5}×10−3\{1, 1.5\} \times10^{-3}. At 3B, both use 10−310^{-3}, from {0.6,1,1.5}×10−3\{0.6, 1, 1.5\} \times10^{-3} for Standard and {0.7,1,1.5}×10−3\{0.7, 1, 1.5\} \times10^{-3} for Loopback. Screening used 5 TPP at 374M/778M and 2 TPP at 1.4B/3B; 1.4B additionally compared 10−310^{-3} and 1.5×10−31.5 \times10^{-3} at 20 TPP. Loopback’s 1.4B and 3B rate checks used [3L → 10L (×2) → 2L] and [2L → 10L (×2) → 2L], respectively; the final 3B topology was chosen in a separate 5-TPP screen. Short runs apply their own final decay, and continuations resume from the pre-decay checkpoint.

Full general-benchmark results at 1.4B and 3B

Table 9 gives the taskwise scores underlying the large-scale comparison, using the same beginning-of-sequence prompts and metric conventions as the smaller-scale evaluation.

At 3B, the bootstrap standard error of each model’s thirteen-task mean is 0.40 percentage points.

Run-to-run variation and memory-reader results

Run-to-run variation

Table 10 reports two training seeds for every 191M arm except AttnRes and four arms at 374M (24 layers). Each seed sets both the initialization and the data order. At 191M, Loopback has the lowest validation loss among the tested looped models in both runs. A third seed for the standard Transformer, Plain Loop, mHC, and Loopback keeps this ordering (Loopback −0.0110 against mHC and −0.0716 against Plain Loop). Its paired loss differences from the standard Transformer are −0.0047, +0.0011, and −0.0064 (mean −0.0033), so that comparison is not consistent across seeds. At 374M, Loopback has the lowest validation loss among the four tested models in both seeds.

Modelseed 42seed 1337
191M, 20 layers
Standard2.62342.6144
Plain Loop2.68022.6815
mHC2.62502.6292
HC2.64492.6427
iHC2.65642.6497
Hyperloop2.64892.6472
MHAR2.65642.6573
Loopback2.61872.6155
Loopback − Standard−0.0047+0.0011
Loopback − mHC−0.0063−0.0137
374M, 24 layers
Standard2.41982.4257
mHC2.40622.4079
Hyperloop2.41792.4175
Loopback2.39712.3996
Loopback − Standard−0.0227−0.0261
Loopback − mHC−0.0091−0.0083

Table 10. Validation-loss variation at 191M and 374M. ClimbMix loss (nats, lower is better) after 20 TPP. Each seed sets initialization and data order; difference rows compare models within a seed. Seed 42 is the run reported in the main text. Bold marks the best model in each column within each scale; shading marks Loopback. Validation loss ↓\downarrow

Memory-reader results

Depth sweep at fixed parameters

Table 12 gives the configurations and results for all nine rules, using the recipes in Section D.1. Width varies with depth to keep Standard’s total parameters within 5% of its 24-layer reference; training tokens remain fixed. Every run uses one node with eight 80 GB GPUs: H100s at 374M and A100s at 778M, including the 48-layer models in Table 2. Learning rates are 2×10−32 \times10^{-3} for all 374M arms, and 10−310^{-3} for Standard versus 1.5×10−31.5 \times10^{-3} for looped arms at 778M. We also check Standard’s rate: at 374M, 10−310^{-3} and 3×10−33 \times10^{-3} underperform 2×10−32 \times10^{-3} after 5 TPP at 48 and 64 layers (3×10−33 \times10^{-3} diverges at 48). At 778M, using 1.5×10−31.5 \times10^{-3} changes its 20-TPP loss by −0.0045-0.0045, −0.0020-0.0020, −0.0010-0.0010, and +0.0076+0.0076 at 16, 24, 32, and 48 layers.

Mathematical and code text

We use these evaluations to test whether parameter-efficient reuse also preserves mathematical and code solution modeling. The 1.4B and 3B results are in Table 3; Table 13 compares all nine designs at 191M and 374M. These use the checkpoints and matched training budgets of Table 2, including the 32-layer 374M setting.

Reference-solution loss. We evaluate GSM8K test (1,319 problems), MATH-500 (500), HumanEval (164), and sanitized MBPP test (257). Each prompt contains a beginning-of-sequence token and the problem; MBPP additionally includes the first test. GSM8K calculator annotations are removed and the final line is standardized to The answer is N. We score the reference solution with teacher forcing in a 4,096-token window using BF16 and the reference Loopback read, averaging NLL over solution tokens. The four-set mean weights datasets equally. Paired bootstrap resampling uses 2,000 draws over problems. This metric evaluates solution likelihood, not generated-answer or code-execution correctness.

Five-shot numeric-answer accuracy. The six sets are SingleOp (562), SingleEq (508), AddSub (395), MultiArith (600), the numeric-answer subset of ASDiv (2,084), and SVAMP (1,000), from the ToRA evaluation files. We use the first five standard-prompting exemplars of chain-of-thought prompting, formatted as Q: . . . A: The answer is NQ:\ .\ .\ .\ A:\ \textit{The answer is }N. A response is correct when each reference-number token is the greedy choice and the greedy token following the number is not a digit. The reported mean weights all six sets equally; the 1.4B and 3B means appear in Table 3. At 3B, Loopback is higher on each set: 11.9 versus 4.3 on SingleOp, 8.5 versus 3.3 on SingleEq, 5.6 versus 2.0 on AddSub, 1.3 versus 0.8 on MultiArith, 7.0 versus 3.4 on ASDiv, and 4.1 versus 1.9 on SVAMP. The low absolute accuracies limit claims about reliable mathematical problem solving.

3B task results with one extra visit

We evaluate the same 3B Loopback checkpoint at its trained two visits and at three visits without retraining. Additional destinations reuse the last trained visit’s query for the same block position; the shared blocks and coda are unchanged. We report all thirteen zero-shot tasks using the main evaluation protocol: HellaSwag, OpenBookQA, ARC-Easy, and ARC-Challenge use length-normalized accuracy; the other tasks use accuracy. The mean weights the thirteen tasks equally, with MMLU’s 57 subjects counted as one task.

Task2 visits (trained)3 visitsΔ\Delta (pp)
PIQA75.9075.30-0.60
HellaSwag63.7265.11+1.39
WinoGrande58.4858.96+0.48
OpenBookQA41.8042.80+1.00
CommonsenseQA21.8722.36+0.49
BoolQ64.9565.60+0.65
RACE36.1735.60-0.57
ARC-Easy71.1371.04-0.09
ARC-Challenge40.9640.02-0.94
MMLU (57 subjects)28.8629.69+0.83
ANLI-R132.9033.70+0.80
RTE56.6863.90+7.22
LAMBADA49.4548.09-1.36
Thirteen-task mean49.4550.17+0.72

Table 14. 3B Loopback with one extra visit. Zero-shot scores (%) at two visits (trained) and three visits, without retraining. Δ\Delta is the change in percentage points computed from the displayed scores; bold marks the higher score in each row.

The mean rises by 0.72 points, with gains on eight of thirteen tasks. RTE (277 items) contributes 0.56 points to the mean increase. These are single-checkpoint measurements; the improvement is not uniform across tasks.

Compute and efficiency measurements

FLOP accounting. Table 15 covers three scales. PyTorch’s operator counter measures forward FLOPs on one 4,096-token bf16 sequence, including the vocabulary head. Scaled-dot-product attention substitutes for the uncounted flash kernel; uncounted elementwise work is added analytically. For a read at destination ss over s+1s+1 stored updates, the score and weighted-sum matmuls contribute 4(s+1)d4(s+1)d FLOPs per token, excluding normalization, softmax, and mixture replacement. Anchored decay costs 3d3d per token at each visit end; AttnRes reads before token attention and before the MLP in each loop block, and once at loop exit, over the loop input and up to eight loop states; the stream rules cost O(n2d)O(n^2d) per connection for n=4n=4 streams.

Training measurements. Measurements use one eight-H100 node, FSDP, bf16 compute, and each scale’s micro-batch. Loopback uses fused reads measured in 160-step timing runs; other entries come from the training logs, including AttnRes in Figure 9, which trains with its fused read. For these three scales, throughput uses median step time, and memory is the trainer’s reported peak. Activations dominate sharded model state here, so fewer stored weights need not lower peak memory. At 3B, the mean logged training throughputs are 24.7k tokens/s/GPU for Standard and 23.3k for Loopback, a 6% gap after rounding. Both runs use the same eight-H100 node, four 4,096-token sequences per GPU, and 60 gradient accumulation steps. These means cover the logged intervals of the final 20-TPP training stage.

Inference measurements. One H100 runs each model eagerly with bf16 weights and no gradients; Loopback uses fused reads. Prefill is one forward pass over a batch of eight 4,096-token sequences that keeps only the last position’s logits and materializes no KV cache, so its measured peak covers weights and transient activations. For Loopback these include the depth history, which exists only within a forward pass and is never cached across decoding steps. Loopback’s prefill throughput relative to the standard Transformer is 0.68, 0.72, and 0.81 at 191M, 374M, and 778M. Table 16 instead counts persistent memory analytically: all stored weights plus the KV cache, whose capacity is equal at matched executed depth. The two views differ by construction: at 374M the measured prefill peak is 3.0 GiB for Loopback versus 2.1 GiB for the standard Transformer, whereas the persistent total for the same eight sequences is 3.46 versus 3.70 GiB.

ScaleModelTotal params (M)Weights (GiB)KV cache per sequence (MiB)Weights + KV (GiB) 1 sequenceWeights + KV (GiB) 8 sequences
191M, 20 layersStandard190.740.362400.592.23
Loopback105.800.202400.432.07
374M, 24 layersStandard373.870.703841.073.70
Loopback245.420.463840.833.46
778M, 24 layersStandard777.861.455762.015.95
Loopback551.361.035761.595.53
1.4B, 25 layersStandard1,415.682.648003.428.89
Loopback1,004.641.878002.658.12
3B, 24 layersStandard2,914.675.431,1526.5514.43
Loopback2,235.194.161,1525.2913.16

Table 16. Persistent inference memory of Standard and Loopback. Analytical bf16 storage for all weights, including embeddings and the vocabulary head, plus the KV cache at 4,096 tokens. Totals exclude temporary activations and workspaces; they are not measured peaks. Shading marks Loopback.

The fused read. Each read uses one Triton kernel, streaming updates once per token tile while accumulating the scored and prior softmaxes together. Backward saves one score per token and update rather than normalized keys, then reloads each update and writes its gradient once. Tests compare the kernel with a direct PyTorch implementation of Eq. (17). In FP32, the output, the installed coefficients, and every gradient (query, updates through the value and key paths, proposal, input, and previous coefficients) agree to within 10−610^{-6} relative error, measured as the largest absolute difference over the largest reference magnitude, across widths 768–3,072, 2–20 stored updates, ragged token counts, and boundary and in-visit reads. With bf16 inputs, both implementations are compared with an FP64 evaluation of the same rounded inputs, and the fused error is no larger than the reference’s. At zero queries the output and coefficients match the decay-only reference, which reduces to Plain Loop on only at λ=1\lambda= 1. An end-to-end comparison of the two implementations on a small model gives bitwise-identical results at initialization and agreement within 10−610^{-6} after perturbing the queries, forward and backward. Most remaining step-time overhead is in backward, as each update receives gradients from later reads. The kernel and tests will be released with the code.

References

  1. [1]Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in artificial intelligence, pp. 1352–1361. PMLR, 2021.
  2. [2]Sangmin Bae, Adam Fisch, Hraryr Harutyunyan, Ziwei Ji, Seungyeon Kim, and Tal Schuster. Relaxed recursive transformers: Effective parameter sharing with layer-wise lora. In International Conference on Learning Representations, volume 2025, pp. 34282–34327, 2025.arxiv.org/abs/2410.20672
  3. [3]Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hraryr Harutyunyan, Ziwei Ji, Aaron Courville, et al. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. Advances in Neural Information Processing Systems, 38:96572–96617, 2026.
  4. [4]Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In Advances in Neural Information Processing Systems, volume 35, pp. 20232–20242, 2022.
  5. [5]Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3028–3033, 2018.
  6. [6]Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024.
  7. [7]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018.arxiv.org/abs/1807.03819
  8. [8]Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. Advances in Neural Information Processing Systems, 38, 2026.
  9. [9]Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Zenodo, 2021.DOI
  10. [10]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. Advances in Neural Information Processing Systems, 38:41340–41391, 2026.arxiv.org/abs/2502.05171
  11. [11]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.arxiv.org/abs/2312.00752
  12. [12]Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.arxiv.org/abs/2111.00396
  13. [13]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.arxiv.org/abs/1512.03385
  14. [14]Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2018. URL https://arxiv.org/abs/1608.06993.
  15. [15]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156–5165. PMLR, 2020.arxiv.org/abs/2006.16236
  16. [16]Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.arxiv.org/abs/1909.11942
  17. [17]Cheng Luo, Zefan Cai, and Junjie Hu. Delta attention residuals. arXiv preprint arXiv:2605.18855, 2026a.arxiv.org/abs/2605.18855
  18. [18]Cheng Luo, Zefan Cai, and Junjie Hu. Multi-head attention residuals. arXiv preprint arXiv:2607.27230, 2026b.arxiv.org/abs/2607.27230
  19. [19]Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaj u Lin, Jiaxing Liu, Janna Lu, William Merrill, et al. Rwk-v7" goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456, 2025.arxiv.org/abs/2503.14456
  20. [20]Zhen Qin, Songlin Yang, Weixuan Sun, Guangyuan Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024.arxiv.org/abs/2404.07904
  21. [21]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  22. [22]Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. In International Conference on Learning Representations, volume 2025, pp. 14855–14881, 2025.arxiv.org/abs/2502.17416
  23. [23]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International conference on machine learning, pp. 9355–9366. PMLR, 2021.arxiv.org/abs/2102.11174
  24. [24]Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks, 2015. URL https://arxiv.org/abs/1505.00387.
  25. [25]Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.arxiv.org/abs/2307.08621
  26. [26]Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025.
  27. [27]Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, Yaoyu Wang, Yucheng Wang, Guanduo Chen, Bohong Yin, et al. Attention residuals. arXiv preprint arXiv:2603.15031, 2026.
  28. [28]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
  29. [29]Johannes Von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zuccher, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858, 2023.arxiv.org/abs/2309.05858
  30. [30]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761–6774, 2024.arxiv.org/abs/2203.00555
  31. [31]Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Kuai Yu, et al. mhc: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880, 2025.arxiv.org/abs/2512.24880
  32. [32]Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023.arxiv.org/abs/2312.06635
  33. [33]Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522, 2024.arxiv.org/abs/2406.06484
  34. [34]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In International Conference on Learning Representations, volume 2025, pp. 29687–29707, 2025.arxiv.org/abs/2412.06464
  35. [35]Xiao-Wen Yang, Ziyu Han, Xi-Hua Zhang, Wen-Da Wei, Jie-Jing Shao, Lan-Zhe Guo, and Yu-Feng Li. Stabilizing recurrent dynamics for test-time scalable latent reasoning in looped language models. arXiv preprint arXiv:2605.26733, 2026.arxiv.org/abs/2605.26733
  36. [36]Abbas Zeitoun, Lucas Torroba-Henningen, and Yoon Kim. Hyperloop transformers. arXiv preprint arXiv:2604.21254, 2026.arxiv.org/abs/2604.21254
  37. [37]Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William Freeman, and Hao Tan. Test-time training done right. In International Conference on Learning Representations, volume 2026, pp. 157604–157638, 2026.arxiv.org/abs/2505.23884
  38. [38]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaij ing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025b.arxiv.org/abs/2510.25741

Paper details

vlnc:W_qhgqeepy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 3

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents