WorksComputer Science
Loopback Transformer: Towards a Unified View of Residual Looped Transformers
Abstract
Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later reads weight it. A full block output contains both its input and its new update, so weighting that output also reweights the history already carried by the input. Reading individual updates instead, with the shared stack's input kept on a separate path, gives separate controls over input strength, total update weight, and allocation across updates. The Loopback Transformer implements these controls: geometric decay sets the total update weight and supplies a recency prior, and attention learns how to allocate that weight. On ClimbMix, Loopback reaches lower validation loss than standard Transformer baselines with 40-75% of their Transformer parameters (excluding token embeddings and the output head) at reference scales from 191M to 3B, with training tokens and executed depth matched. It also has higher mean accuracy over thirteen downstream tasks at 1.4B and 3B. At 191M, 374M, and 778M, it has the lowest validation loss and the highest thirteen-task mean among the standard Transformer and seven looped residual designs.
*Equal contribution
Corresponding author
Correspondence to Xuan Ouyang, Zefan Cai, and Junjie Hu ([email protected], [email protected], [email protected]).
Introduction
Looped Transformers reduce parameter count by repeatedly applying shared blocks to an evolving hidden representation ([7, 16]). Sharing separates computational depth from the number of distinct blocks and supports latent reasoning and inference-time scaling ([22, 10]). Each visit applies the same transformation to a different hidden state and so produces a different update. Parameter sharing fixes which transformation is reused; it leaves open how the updates from successive visits are stored, carried forward, and used.
Residual connections turn depth into a form of memory: each layer writes new computation into a residual stream carried forward to later layers. Standard residual addition accumulates block updates with fixed unit coefficients ([13]), whereas Attention Residuals (AttnRes) learns how to weight these updates and the input ([27]). Hyper-Connections (HC) and Hyperloop introduce learned pathways across residual streams or loop visits ([38, 36]).
This depth-memory view ([27]) connects residual design to linear attention and related recurrent models. Linear attention expresses attention over token history as writes to and reads from a recurrent memory ([15]); RetNet and Mamba-2 additionally control how that memory is retained across steps ([25, 6]). We use this write–transport–read perspective to analyze repeated computation along depth. Here, each block produces a new write from a representation assembled from earlier writes, so the memory rule also influences subsequent computation. We therefore examine what is stored and how memory transitions and readout determine each update’s contribution.
Seen this way, common residual designs tie together quantities that could be controlled separately. First, a full block output contains both its input and its new update. Weighting that output therefore also weights the history carried by the input. Storing updates separately lets the reader assign a coefficient directly to each update. Second, a joint softmax over the original input and stored updates makes them compete for a fixed total weight. Ordinary addition instead fixes the input coefficient but assigns every update weight one, so total update weight grows with the number of block applications. We therefore want separate controls over input strength, total update weight, and allocation across updates.
The Loopback Transformer is built around these three controls. It stores individual block updates rather than full block outputs and keeps the shared stack’s input on a separate path with coefficient one. Geometric decay assigns a baseline weight to each update. A softmax reader redistributes their total weight according to update content, using the decay weights as a recency prior. This separates total update weight from allocation without changing the shared blocks or their token-axis attention.
We compare against the standard Transformer, which uses distinct parameters at every layer (labeled Standard in tables and figures), and against seven looped residual designs, always at the same budget of training tokens and the same executed depth. On ClimbMix ([8]), Loopback reaches lower validation loss and a higher thirteen-task mean than Standard at 191M, 374M, 778M, 1.4B, and 3B while storing 40–75% of its Transformer parameters (Table 2, Table 4 and Table 9 and Figure 1). At 191M, 374M, and 778M, it also has the lowest validation loss and the highest thirteen-task mean of all nine designs (Table 2).
Figure 1. Loopback versus the standard Transformer from 191M to 3B. Each pair uses the same token budget and executed depth at the 20-TPP setting (tokens per reference-model parameter). (a) Validation loss and (c) thirteen-task mean accuracy against the standard Transformer’s estimated training FLOPs; (b) the same losses against Transformer parameters, where percentages give Loopback’s share of the standard Transformer’s. Lines interpolate the points.
Table 2: Residual adapters at matched tokens and depth. Validation loss, zero-shot accuracy (%) and WikiText perplexity at 191M (20 layers), 374M (32 layers), and 778M (48 layers) (Table 12); every prompt starts with the beginning-of-sequence token, as in training (Section D.2). Arrows mark lower-is-better metrics; bold marks the best result per row. Percentages compare Transformer parameters with Standard, excluding embeddings and the vocabulary head and counting shared weights once.
| Benchmark | Standard | Plain Loop | mHC | HC | iHC | Hyperloop | MHAR | AttnRes | Loopback |
| 191M, 20 layers, [2L → 4L (×4) → 2L], 40.0% of Standard, 3.82B tokens (20 TPP) | |||||||||
| Transformer params (M) | 141.59 | 56.64 | 56.93 | 56.65 | 57.04 | 56.79 | 56.64 | 56.66 | 56.65 |
| ClimbMix loss ↓ | 2.6234 | 2.6802 | 2.6250 | 2.6449 | 2.6564 | 2.6489 | 2.6564 | 2.6404 | 2.6187 |
| PIQA | 65.51 | 64.96 | 65.89 | 64.74 | 65.34 | 65.07 | 65.13 | 64.47 | 65.78 |
| HellaSwag | 33.26 | 32.14 | 33.79 | 33.18 | 32.74 | 33.04 | 33.12 | 33.07 | 33.83 |
| WinoGrande | 51.70 | 51.38 | 52.64 | 51.22 | 51.70 | 49.80 | 50.75 | 52.72 | 51.14 |
| OpenBookQA | 32.60 | 29.40 | 31.40 | 30.20 | 32.80 | 31.80 | 31.80 | 29.20 | 32.40 |
| CommonsenseQA | 21.54 | 19.98 | 19.90 | 18.67 | 19.98 | 21.95 | 18.92 | 20.48 | 21.13 |
| BoolQ | 62.72 | 46.82 | 61.99 | 59.60 | 62.17 | 60.58 | 61.22 | 60.73 | 61.44 |
| RACE | 28.04 | 29.09 | 27.75 | 29.86 | 29.67 | 28.61 | 30.14 | 29.95 | 28.33 |
| ARC-Easy | 47.90 | 46.93 | 46.51 | 47.14 | 46.42 | 45.66 | 46.21 | 44.28 | 47.22 |
| ARC-Challenge | 26.45 | 25.94 | 25.68 | 25.94 | 25.09 | 26.11 | 26.62 | 26.11 | 25.77 |
| MMLU (57 subjects) | 23.56 | 24.57 | 24.01 | 24.13 | 23.64 | 24.02 | 23.25 | 26.57 | 23.68 |
| ANLI-R1 | 30.40 | 32.20 | 27.30 | 32.40 | 31.30 | 32.90 | 33.00 | 31.60 | 30.70 |
| RTE | 49.46 | 50.90 | 55.60 | 53.43 | 53.43 | 53.79 | 53.79 | 53.07 | 54.15 |
| LAMBADA | 24.47 | 22.01 | 25.46 | 25.38 | 24.92 | 24.72 | 23.91 | 24.18 | 26.80 |
| WikiText word ppl ↓ | 46.30 | 50.30 | 45.86 | 47.52 | 48.72 | 47.87 | 48.30 | 47.01 | 45.74 |
| Mean (13 tasks) | 38.28 | 36.64 | 38.30 | 38.15 | 38.40 | 38.31 | 38.30 | 38.19 | 38.64 |
| 374M, 32 layers, [2L → 14L (×2) → 2L], 56.3% of Standard, 7.47B tokens (20 TPP) | |||||||||
| Transformer params (M) | 323.02 | 181.70 | 181.87 | 181.71 | 182.52 | 181.79 | 181.70 | 181.75 | 181.72 |
| ClimbMix loss ↓ | 2.4395 | 2.4371 | 2.4118 | 2.4273 | 2.4199 | 2.4147 | 2.4310 | 2.4166 | 2.4008 |
| PIQA | 68.82 | 68.23 | 68.39 | 68.88 | 69.48 | 69.21 | 69.10 | 68.72 | 69.10 |
| HellaSwag | 38.72 | 39.47 | 40.44 | 39.89 | 40.36 | 40.48 | 39.33 | 40.42 | 41.11 |
| WinoGrande | 51.93 | 52.96 | 52.88 | 52.17 | 52.01 | 52.49 | 52.88 | 51.14 | 52.09 |
| OpenBookQA | 30.80 | 34.00 | 33.60 | 32.80 | 31.20 | 32.00 | 33.60 | 33.20 | 34.60 |
| CommonsenseQA | 19.98 | 20.23 | 19.90 | 21.05 | 20.31 | 21.21 | 20.64 | 21.21 | 19.74 |
| BoolQ | 47.13 | 60.89 | 58.04 | 58.65 | 62.42 | 56.73 | 55.99 | 58.23 | 58.90 |
| RACE | 29.28 | 31.00 | 30.81 | 29.38 | 30.14 | 28.42 | 30.91 | 30.81 | 29.67 |
| ARC-Easy | 51.26 | 52.90 | 53.37 | 51.43 | 52.65 | 54.63 | 52.44 | 51.73 | 52.15 |
| ARC-Challenge | 27.65 | 28.58 | 29.01 | 28.41 | 27.82 | 28.41 | 29.69 | 28.16 | 28.07 |
| MMLU (57 subjects) | 25.25 | 23.64 | 25.32 | 25.76 | 24.85 | 25.81 | 26.25 | 25.37 | 26.24 |
| ANLI-R1 | 33.30 | 33.60 | 32.00 | 30.70 | 32.60 | 31.30 | 33.40 | 33.90 | 32.40 |
| RTE | 54.87 | 53.79 | 53.43 | 51.26 | 51.99 | 50.18 | 53.07 | 46.57 | 53.07 |
| LAMBADA | 30.29 | 31.69 | 31.94 | 31.85 | 32.54 | 32.76 | 31.19 | 33.05 | 34.43 |
| WikiText word ppl ↓ | 34.79 | 34.57 | 33.34 | 34.05 | 33.69 | 33.45 | 34.23 | 33.59 | 32.61 |
| Mean (13 tasks) | 39.18 | 40.84 | 40.70 | 40.17 | 40.64 | 40.28 | 40.65 | 40.19 | 40.89 |
| 778M, 48 layers, [3L → 21L (×2) → 3L], 56.3% of Standard, 15.60B tokens (20 TPP) | |||||||||
| Transformer params (M) | 708.68 | 398.63 | 398.84 | 398.65 | 400.11 | 398.74 | 398.64 | 398.73 | 398.68 |
| ClimbMix loss ↓ | 2.2644 | 2.2737 | 2.2454 | 2.2642 | 2.2523 | 2.2539 | 2.2646 | 2.2529 | 2.2404 |
| PIQA | 71.44 | 71.55 | 72.20 | 71.76 | 70.57 | 71.16 | 71.49 | 71.71 | 71.16 |
| HellaSwag | 47.07 | 47.07 | 49.03 | 47.74 | 48.56 | 48.08 | 48.10 | 48.72 | 48.97 |
| WinoGrande | 52.09 | 49.80 | 53.43 | 53.04 | 53.12 | 53.28 | 53.51 | 55.64 | 53.91 |
| OpenBookQA | 34.80 | 35.40 | 35.20 | 35.00 | 35.60 | 36.00 | 34.00 | 36.40 | 36.20 |
| CommonsenseQA | 19.16 | 19.41 | 18.10 | 19.82 | 18.76 | 20.97 | 20.15 | 21.95 | 20.80 |
| BoolQ | 59.97 | 58.99 | 55.17 | 60.28 | 58.93 | 60.37 | 57.98 | 57.92 | 58.93 |
| RACE | 31.48 | 32.54 | 31.96 | 32.06 | 32.34 | 32.44 | 33.49 | 33.21 | 33.30 |
| ARC-Easy | 60.44 | 57.58 | 61.20 | 60.56 | 60.98 | 58.63 | 61.41 | 58.42 | 58.92 |
| ARC-Challenge | 31.91 | 31.40 | 31.83 | 32.25 | 33.19 | 31.06 | 32.59 | 31.74 | 32.08 |
| MMLU (57 subjects) | 25.77 | 25.10 | 25.70 | 25.37 | 24.53 | 24.84 | 25.49 | 25.44 | 25.30 |
| ANLI-R1 | 31.30 | 32.00 | 33.70 | 33.20 | 30.70 | 33.30 | 33.20 | 33.30 | 35.20 |
| RTE | 56.32 | 57.40 | 52.71 | 51.26 | 53.79 | 53.79 | 50.18 | 53.07 | 51.99 |
| LAMBADA | 37.76 | 37.84 | 39.71 | 38.87 | 39.78 | 38.07 | 38.54 | 37.92 | 39.74 |
| WikiText word ppl ↓ | 26.39 | 26.78 | 25.68 | 26.49 | 25.84 | 26.01 | 26.35 | 25.95 | 25.33 |
| Mean (13 tasks) | 43.04 | 42.78 | 43.07 | 43.17 | 43.14 | 43.23 | 43.09 | 43.50 | 43.58 |
Table 2.
| (a) Decay and attention at two scales | ||||
| 191M (Standard: 2.6234) | 374M (Standard: 2.4198) | |||
| Decay | Fixed | Learned | Fixed | Learned |
| Off () | 2.6802 | 2.6621 | 2.4415 | 2.4308 |
| On () | 2.6220 | 2.6187 (−0.0047) | 2.3999 | 2.3971 (−0.0227) |
| (b) Weight initialization: 191M, 20 layers | ||||
| Model | Transformer params (M) | GPT-2 | PyTorch | |
| Standard | 141.59 | 2.6234 | 2.6419 | 2.7145 |
| Plain Loop | 56.64 | 2.6802 | 2.6977 | 2.6642 |
| mHC | 56.93 | 2.6250 | 2.6408 | 2.6278 |
| Loopback | 56.65 | 2.6187 (−0.0047) | 2.6178 (−0.0241) | 2.6138 (−0.1007) |
| (c) Weight initialization: 374M, 24 layers | ||||
| Model | Transformer params (M) | GPT-2 | PyTorch | |
| Standard | 308.33 | 2.4198 | 2.4252 | 2.4296 |
| Plain Loop | 179.86 | 2.4415 | 2.4272 | 2.4334 |
| mHC | 180.06 | 2.4062 | 2.4091 | 2.4093 |
| Loopback | 179.88 | 2.3971 (−0.0227) | 2.4117 (−0.0135) | 2.3938 (−0.0358) |
Table 4. Decay, attention, and weight initialization. Validation loss (↓) at 20 layers (191M) and 24 layers (374M); shading marks Loopback and green gives its difference from Standard. (a) Fixed allocation uses zero queries. (b–c) Three weight-initialization schemes with learning rates held fixed; bold marks the lowest loss in each initialization column. Parameter counts exclude embeddings and the vocabulary head.
| 1.4B, 25 layers, 28.31B tokens | 3B, 24 layers, 59.77B tokens | |||
| Standard | Loopback | Standard | Loopback | |
| Topology | – | [3L → 8L (×2) → 6L] | – | [6L → 6L (×2) → 6L] |
| Transformer params (M) | 1,284.61 | 873.57 | 2,718.06 | 2,038.58 |
| Relative to Standard | 100.0% | 68.0% | 100.0% | 75.0% |
| ClimbMix loss ↓ | 2.1283 | 2.1073 | 2.0005 | 1.9849 |
| PIQA | 73.88 | 74.48 | 77.04 | 75.90 |
| HellaSwag | 55.02 | 56.43 | 62.84 | 63.72 |
| WinoGrande | 56.51 | 56.99 | 57.22 | 58.48 |
| OpenBookQA | 37.80 | 36.60 | 40.40 | 41.80 |
| CommonsenseQA | 23.34 | 18.67 | 19.90 | 21.87 |
| BoolQ | 62.51 | 63.24 | 62.23 | 64.95 |
| RACE | 34.64 | 33.49 | 35.12 | 36.17 |
| ARC-Easy | 66.16 | 67.00 | 69.19 | 71.13 |
| ARC-Challenge | 36.35 | 37.03 | 40.53 | 40.96 |
| MMLU (57 subjects) | 24.70 | 26.81 | 26.26 | 28.86 |
| ANLI-R1 | 31.30 | 34.80 | 31.00 | 32.90 |
| RTE | 57.40 | 55.23 | 61.01 | 56.68 |
| LAMBADA | 42.64 | 43.72 | 48.50 | 49.45 |
| Thirteen-task mean (%) ↑ | 46.33 | 46.50 | 48.56 | 49.45 |
| WikiText perplexity ↓ | 21.09 | 20.29 | 17.28 | 16.74 |
Table 9. Standard and Loopback at 1.4B and 3B. Results after 20 TPP, one run per arm. Task scores are zero-shot accuracies (%) with beginning-of-sequence prompts; bold marks the better result within each scale. Parameter shares compare Transformer parameters, excluding embeddings and the vocabulary head.
We make three contributions:
We connect residual designs with linear attention and related recurrent models by adapting their memory-update rules from token sequences to network depth. The resulting framework describes what each block writes and how its contribution is transported and read.
Within this framework, we design the Loopback Transformer, which adds a weighted sum of block updates to the original loop input. Geometric decay keeps the total update weight bounded as visits increase at fixed stack size. Attention adjusts each update’s weight according to its content while preserving the total.
On ClimbMix, Loopback reaches lower validation loss and higher zero-shot task means than matched-depth standard Transformers at five scales from 191M to 3B, with 40–75% of their Transformer parameters.
Preliminaries
Looped Transformer
A looped Transformer shares parameters across depth by repeatedly applying the same sequence of Transformer blocks. Each block contains causal token attention followed by an MLP, with a residual connection around each computation. We call this shared sequence a stack, consisting of distinct blocks with transformations .
We adopt Huginn’s prefix–middle–coda layout (Geiping et al., [10]), with prefix blocks, a shared middle stack of blocks repeated for loop iterations, and coda blocks. Following Zeitoun et al. [36], we denote this topology by , where the suffix L denotes layers, counting each block application as one layer of executed depth. The model stores distinct blocks and executes block applications. Let be the prefix output entering the shared stack. Within this stack, indexes loop iterations, and identifies a block’s position within each iteration. The index counts block applications across all iterations. At application , we apply the shared block , where .
To describe how residual connections carry computation across depth, we distinguish a block’s full output from the update it produces. Let denote the aggregated hidden representation before application , and let be the input passed to block . A single-stream model passes one hidden representation between blocks, so ; a multi-stream model may form by mixing several parallel hidden states. Aggregating the block’s result with stored computation produces the next readout . The full block output is , and the block update is
We suppress batch and token indices; each token’s block update is in , where is the hidden width. The block update measures the change produced by the block itself, before residual aggregation.
Residual aggregation
We review ordinary residual addition, Attention Residuals, and Hyper-Connections to examine how block updates contribute to later computation. These designs accumulate updates directly, weight stored computation through attention, or maintain multiple residual streams.
Ordinary residual addition (He et al., [13]) takes and uses a fixed coefficient of one to accumulate each update:
Attention Residuals (AttnRes) (Team et al., [27]) learns content-dependent weights over the input and stored computation. Before block application , its block-level formulation reads the sources and for :
The learned query scores normalized sources, while the read combines their original values.
Hyper-Connections (HC) (Zhu et al., [38]) carries parallel -dimensional hidden states per token through the network, forming residual streams. Before block application , these states are stored side by side in . At block level, HC combines these states into one input and distributes the block update back to the streams:
Here combines the states into the block input, mixes the existing states, and controls how much of the block update each stream receives. These weights are learned and may depend on .
Linear attention and recurrent memory
Linear attention ([15]) summarizes token history in a recurrent memory. Each new token adds information to this memory, from which a query reads an output. A scalar-decay version of the unnormalized recurrence is
Here indexes tokens, is the value, and are the key and query, with feature dimension . The matrix stores the accumulated writes. The factor scales the previous memory, writes the current value, and reads the unnormalized output . With feature maps absorbed into the keys and queries, additive linear attention uses and divides the read by the positive normalizer , where . Related recurrent models modify transport: RetNet uses fixed decay ([25]), while Gated Linear Attention (GLA) and Mamba-2 make it input dependent ([32, 6]). Along depth, each token position can maintain a memory of earlier block computations. After a block runs, its result for that token updates the memory, whose readout helps form the next block’s input. Attention inside each block still operates over tokens.
Residual connections as depth memory
Following the depth-memory view ([27]), we describe residual connections and recurrent memory by the same write, transport, and read operations. We first compare full block outputs with individual block updates as the stored value, then expand the recurrence to see how the original input and earlier block updates enter later hidden states. The expansion is what we use to compare how existing residual designs reuse earlier computation, and it leads to the design of Loopback.
A common memory interface
To trace how block results contribute to later computation, we describe residual aggregation through a common memory formulation. For each token, let hold the representations retained before block application , where counts the -dimensional vectors used to represent memory. The block input is , with . Its result supplies a value to write into memory, such as the full block output or the block update:
The transport matrix scales or mixes the existing representations. The write term adds the current value to memory, with specifying its weight in each memory component. The read combines the updated representations using to produce . Thus, forms the input before the block runs, while reads the memory after it is updated; these weights may depend on content. Single-stream models pass to the next block. Multi-stream models form the next input from using . Unrolling this recurrence follows each written value through subsequent transport and read operations and gives its weight in a later readout, so different residual designs can be compared even when their memory representations differ. Table 1 summarizes their choices.
Table 1: Write, transport, and read operations for residual and attention memory, with applied per head where needed. Feature construction, output gates and normalization, feedthrough paths, and outer residual blends are not shown. The rows describe memory operations, not complete published architectures.
| Method | Transport | Write | Read |
| Addition ([13]) (Standard, Plain Loop) | |||
| Explicit-history attention | |||
| History banks start from and store each new value separately. At block granularity, the value is ; selects its new storage location. | |||
| AttnRes ([27]) | |||
| MHAR ([18]) | headwise | ||
| Linear-attention family and related recurrent models | |||
| The value memory starts from . Keys and queries include each rule’s feature mapping, when present. | |||
| Linear (normalized) ([15]) | |||
| RetNet ([25]) | |||
| GLA ([32]) | |||
| Mamba-2 ([6]) | |||
| HGRN-2 ([20]) | |||
| DeltaNet ([33]) | |||
| Gated DeltaNet ([34]) | |||
| Kimi Delta Attention (KDA) ([26]) | |||
| RWKV-7 ([19]) | |||
| Mesa ([29]) | |||
| The regression gain uses an additional inverse-covariance state. | |||
| LaCT-inspired ([37]) | Augmented prefix operators (Eq. (25)) | ||
| Hyper-connection family: ; read gives the next input . | |||
| HC ([38]) | Learned | Learned | |
| mHC ([31]) | |||
| Hyperloop ([36]) | |||
| iHC¹ |
Table 1.
The memory can store past values separately or combine them in a fixed-width state. An explicit history bank grows as new values are added, while recurrent memory updates a state of fixed size. Along depth, each block application supplies a value . Let denote the accumulated writes after application , starting from , and let be the query used to read after application . With feature maps absorbed into the write vectors and queries, the unnormalized recurrence and its expansion are
The second line follows by repeatedly substituting the first: each value is written by , transported through subsequent , and read by . Empty products are identity matrices. In the common interface, denotes the memory before application . For the zero-initialized case , the correspondence is , , and . This initializes the value memory, not the network input; an input path can be kept separately or included in a nonzero . For a general initial state, unrolling the common recurrence also contributes to the readout. The general block-update expansion below retains this initial-state contribution. When , these coefficients directly weight individual block updates. A full block output also contains its input, so its coefficient affects earlier computation as well.
Figure 2. Same shared blocks, three residual pathways. Repeated colors denote shared blocks emitting block updates. (a) Addition. (b) aggregates the input and updates computed so far using explicit history or a fixed-size recurrent memory. (c) Multiple streams with input read, transport, write, and final readout.
Outputs versus updates
A full block output contains both the current block update and earlier computation carried by its input. A read weight on this output therefore applies to both. Storing separately lets the read assign a coefficient directly to that block update.
Consider two successive full block outputs under ordinary residual addition: and . Let be the read weight on , with the remaining weight assigned to . Their weighted read is
Because appears in both stored outputs, its coefficient is always one. The update appears only in , so its coefficient is . More generally, a nonnegative normalized read over ordinary additive states assigns nonincreasing coefficients to successive updates (Section A.1). This ordering need not hold for reweighted trajectories. Reading updates separately removes the cumulative-sum restriction.
The example shows how a stored output carries updates already present in earlier states. In the common memory formulation, this occurs through the input term in . To make its effect explicit, let the binary indicator specify the stored value: selects the full block output, and selects the block update. The definition in Eq. (1) then gives
Using in Eq. (6) gives
Here is the effective transport when the recurrence is expressed in terms of block updates. With full-output writes, the additional term carries the input back into memory, since . With block-update writes, this term vanishes and . Constraints on alone therefore need not hold for the effective transport . The block update still depends on the input. An unrestricted transport operator can absorb the extra term; the distinction concerns parameterization and control, not a general expressivity advantage.
Input and update weights
Expanding Eq. (10) separates the original input from all subsequent block updates, including those contained in full block outputs. For , the vector specifies how the input is placed in memory (all ones, for example, copy it into every stream). The read becomes
Each combines write, subsequent transport, and final read; includes the input carried inside stored values. Empty products are identity matrices. All quantities are evaluated along the actual nonlinear trajectory: this is a value decomposition, not a linear model or Jacobian. It applies headwise; value projections require operator-valued coefficients (Section A).
For nonnegative coefficients, we can separate their total from their distribution. Suppressing the destination index and taking , write
Here is the input coefficient, is the total update weight, and specifies the fraction assigned to update . The total sums coefficients; it is not a representation norm. Ordinary addition fixes and gives every update coefficient one, so grows with the number of block applications. A joint softmax over the input and block updates instead enforces : increasing the total weight on updates reduces the input weight.
Residual and recurrent models
Table 1 contrasts explicit history, recurrent compression, and stream transport. Addition starts from . AttnRes and Multi-Head Attention Residuals (MHAR) ([18]) use explicit history banks; the block-level formulations in the table append block updates. Our evaluated adapters use Block AttnRes and a loop-state MHAR reader with an outer residual blend; their sources and placements are specified in Section D.2. Manifold-constrained Hyper-Connections (mHC) ([31]) constrains transport by applying Sinkhorn normalization to a learned matrix . Hyperloop’s transport is diagonal, and iHC1 uses identity transport.
For recurrent memory, scales the entire previous state, while scales its components separately; Diag forms a diagonal matrix from a vector. RetNet uses a fixed . In normalized linear attention, accumulates keys and is required. Our evaluated linear-attention adapters omit this kernel normalizer and use . GLA uses key-side gating, while Mamba-2 couples its scalar decay and write scale through , . The delta-rule families use to control a correction of the memory’s prediction of the new value; this correction is distinct from the block update . RWKV-7 uses a removal direction with elementwise strength , where denotes elementwise multiplication. Mesa additionally tracks inverse covariance; our LCT-inspired reader updates from the stored prefix. These extra states and the remaining implementation conventions are detailed in Section B.
Loopback Transformer
Storing block updates separately allows each read weight to act directly on one update. Loopback uses this representation to form each new hidden state from the original loop input and a weighted sum of past block updates (Figure 3). Geometric decay bounds their total weight as loop iterations increase, while attention adjusts their relative weights using update content. The input stays outside this allocation with coefficient one, preserving the usual residual identity path: if all updates vanish, the shared stack returns . Only the residual aggregation changes; the shared blocks are unchanged.
Figure 3. Loopback separates input, decay, and allocation. (a) Shared blocks with causal history reads. (b) Iteration-end decay sets the prior and total weight. (c) Attention redistributes that total over block updates, keeping the input coefficient at one.
Geometric decay sets the prior and total weight
High representation similarity has been observed across depth in Pre-Norm Transformers and across iterations in looped Transformers ([38, 36]). If successive block updates share a common component, ordinary residual addition reinforces that component at every iteration. Decay limits this accumulation by shrinking the weights of earlier updates. The decay-only reference accumulates updates within each loop iteration and applies a fixed retention at its end.
Each new update enters with coefficient one; at an iteration boundary, all update coefficients are multiplied by , while the input coefficient stays at one. For , the coefficients of the updates written so far, in application order, evolve as
After two iterations, the first iteration’s updates have crossed two boundaries and the second iteration’s updates only one. Repeated scaling therefore produces geometric weights. In this decay-only reference, an iteration maps its incoming state to
Decay follows every iteration, including the last before the coda. This reference defines Loopback’s prior and its trajectory at initialization; the learned reader below subsequently reweights updates after each block. No separate reference forward pass is evaluated.
This rule is a scalar instance of the recurrent memory in Eq. (7a). In the reference, write , with . The accumulated update and the prior coefficient of update after application follow
Here scales the new write and retains the previous accumulation. Both equal when and one otherwise. Each write is scaled on entry by and subsequently by the retention factors . For $0 \le i \le s < KR$, counting iteration boundaries from write through read gives
Superscript marks the prior, and is its total weight at read . For , the prior favors later iterations. After complete iterations, , which equals without decay and is bounded by for fixed . Thus a fixed retention controls accumulation as the number of iterations grows.
Softmax attention reweights the updates
The geometric prior assigns weights by loop iteration. Earlier updates may remain useful for forming the next hidden state, and updates from the same iteration can play different roles despite receiving equal prior weights. We therefore use attention to adjust their weights using each token’s update content while preserving the total set by decay. Loopback is single-stream, so . After application , we append , with , to its history. These updates come from the current Loopback trajectory; only the prior coefficients are fixed by the schedule above. For the read after this application, a learned vector scores normalized updates for . Using for numerical stability, the scores , weights , and next state are
Each baseline weight is multiplied by the content factor ; normalization then restores the total . Attention can therefore favor an older update while preserving . This total depends on depth but is unchanged by attention at each read. The query is specific to application and shared across tokens; content enters through the keys. There are no query, key, or value projections. The language-model loss trains the queries to redistribute weight (Section C.2). The readout replaces the current mixture of updates and retains the input coefficient of one. It feeds the next block, or the coda after the final application; weighting does not modify the stored updates. We initialize the queries to zero, giving and recovering the decay-only trajectory. With zero queries, also recovers ordinary addition. The first read has only one update and needs no learned query (Section C).
Loopback applies the write–transport–read view of linear attention and recurrent memory along depth: blocks supply the values, retention defines their prior weights, and attention reads them to form later block inputs. Its explicit history allows individual updates to be reweighted. The bank requires storage, and total read work is per token beyond block computation.
Experiments
We evaluate parameter efficiency, residual designs, and readers on ClimbMix (Diao et al., [8]). Main comparisons match training tokens and executed depth within each scale. Scale names and the 20-tokens-per-parameter (TPP) budget use Standard’s total parameters; Transformer counts exclude embeddings and the vocabulary head. Training and evaluation details are in Section D.
Baselines. The standard Transformer uses independent blocks; Plain Loop repeats shared blocks with ordinary addition. HC learns full stream mixing, mHC uses Sinkhorn-normalized transport, Hyperloop uses diagonal transport, and iHC uses identity transport with learned read and write gates (Table 1). AttnRes and MHAR are our loop-level adaptations of Attention Residuals and multi-head attention residuals (Team et al., [27]; Luo et al., [18]). Adapters act only inside the loop. Looped arms share the topology and learning rate within each setting; adapter placement and memory sources differ.
Parameter-efficient language modeling
We first test whether Loopback can achieve lower validation loss than depth-matched standard Transformers with fewer Transformer parameters. Table 2 compares all nine rules at 191M with 20 layers, 374M with 32 layers, and 778M with 48 layers. Loopback stores 40.0%, 56.3%, and 56.3% of the standard baseline’s Transformer parameters, respectively, and attains the lowest validation loss at all three. Its losses are 0.005, 0.039, and 0.024 nats lower than those of the standard Transformer, respectively. MHC is the closest looped baseline at all three, trailing Loopback by 0.006, 0.011, and 0.005 nats. At 374M with 24 layers, Loopback stores 58.3% of the standard baseline’s Transformer parameters and again has the lowest loss among the nine rules (Table 12). AttnRes trails Loopback by 0.022 nats at 191M and 0.009 nats at 374M with 24 layers. Figure 4 shows the training-loss curves of Loopback and the standard Transformer across all five model scales.
Figure 4. Training loss of the standard Transformer and Loopback. 100-step moving averages against tokens seen; insets show the last quarter of training; is the final training-loss difference, Loopback minus the standard Transformer, over the last 100 steps. The five panels use 20, 24, 24, 25, and 24 executed layers, respectively. Loopback stores 40–75% of the standard baseline’s Transformer parameters.
| Layers | Width | Heads | Looped topology | Standard total (M) | Standard | Loopback | Share | ||
| 374M, 7.47B tokens at 20 TPP (7,120 steps of 1,048,576 tokens) | |||||||||
| 16 | 1216 | 19 | [1L → 7L (×2) → 1L] | 366.75 | 288.92 | 162.54 | 56.3% | ||
| 24 | 1024 | 16 | [2L → 10L (×2) → 2L] | 373.87 | 308.33 | 179.88 | 58.3% | ||
| 32 | 896 | 14 | [2L → 14L (×2) → 2L] | 380.36 | 323.02 | 181.72 | 56.3% | ||
| 48 | 768 | 12 | [3L → 21L (×2) → 3L] | 388.97 | 339.81 | 191.18 | 56.3% | ||
| 64 | 640 | 10 | [4L → 28L (×2) → 4L] | 366.10 | 325.14 | 182.93 | 56.3% | ||
| 778M, 15.60B tokens at 20 TPP (7,440 steps of 2,097,152 tokens) | |||||||||
| 16 | 1856 | 29 | [1L → 7L (×2) → 1L] | 795.44 | 676.65 | 380.64 | 56.3% | ||
| 24 | 1536 | 24 | [2L → 10L (×2) → 2L] | 777.86 | 679.55 | 396.44 | 58.3% | ||
| 32 | 1344 | 21 | [2L → 14L (×2) → 2L] | 779.74 | 693.72 | 390.25 | 56.3% | ||
| 48 | 1088 | 17 | [3L → 21L (×2) → 3L] | 778.31 | 708.68 | 398.68 | 56.3% | ||
| 64 | 960 | 15 | [4L → 28L (×2) → 4L] | 769.35 | 707.91 | 398.25 | 56.3% | ||
| Layers | Standard | Plain Loop | mHC | HC | iHC | Hyperloop | MHAR | AttnRes | Loopback |
| 374M: ClimbMix loss ↓, 20 TPP | |||||||||
| 16 | 2.4218 | 2.4536 | 2.4234 | 2.4406 | 2.4355 | 2.4327 | 2.4453 | 2.4312 | 2.4191 |
| 24 | 2.4198 | 2.4415 | 2.4062 | 2.4249 | 2.4217 | 2.4179 | 2.4223 | 2.4065 | 2.3971 |
| 32 | 2.4395 | 2.4371 | 2.4118 | 2.4273 | 2.4199 | 2.4147 | 2.4310 | 2.4166 | 2.4008 |
| 48 | 2.4544 | 2.4425 | 2.4089 | 2.4327 | 2.4297 | 2.4154 | 2.4360 | 2.4168 | 2.4005 |
| 64 | 2.4958 | 2.4695 | 2.4328 | 2.4537 | 2.4607 | 2.4346 | 2.4515 | 2.4415 | 2.4224 |
| 778M: ClimbMix loss ↓, 20 TPP | |||||||||
| 16 | 2.2604 | 2.2857 | 2.2626 | 2.2741 | 2.2613 | 2.2696 | 2.2739 | 2.2735 | 2.2583 |
| 24 | 2.2441 | 2.2713 | 2.2432 | 2.2579 | 2.2461 | 2.2551 | 2.2595 | 2.2495 | 2.2425 |
| 32 | 2.2471 | 2.2737 | 2.2471 | 2.2627 | 2.2498 | 2.2536 | 2.2612 | 2.2453 | 2.2415 |
| 48 | 2.2644 | 2.2737 | 2.2454 | 2.2642 | 2.2523 | 2.2539 | 2.2646 | 2.2529 | 2.2404 |
| 64 | 2.2788 | 2.2816 | 2.2495 | 2.2684 | 2.2580 | 2.2614 | 2.2738 | 2.2507 | 2.2512 |
Table 12. Depth-sweep configurations and validation loss at 20 TPP. Training tokens are fixed; Standard’s total parameters stay within 5% of its 24-layer reference. (a) Configurations: heads are 64-dimensional and looped models make two visits. Transformer counts exclude embeddings and the vocabulary head; shared weights are counted once. Other looped arms are within 0.7% of Loopback’s Transformer count. (b) ClimbMix validation loss: bold marks the lowest loss per row; shading marks Loopback.
Loopback also has the highest thirteen-task mean and the lowest WikiText perplexity in all three settings of Table 2, although individual-task results remain mixed. Across the tested training seeds, Loopback retains lower validation loss than the looped baselines retrained at 191M and than the standard Transformer and mHC at 374M with 24 layers (Section E.1).
Parameter efficiency at larger scale
At 1.4B and 3B, Loopback uses and , respectively, matching the standard Transformer’s 25 and 24 executed layers with 68% and 75% of its Transformer parameters. We evaluate these 20-TPP checkpoints on general benchmarks and mathematical and code tasks. Tests of additional loop iterations also include 191M and 374M models for comparison with Plain Loop.
General benchmarks
Figure 5 summarizes Transformer parameter counts, validation loss, and the thirteen-task zero-shot mean. Loopback improves both aggregate metrics at each scale, and has lower WikiText perplexity: 20.29 versus 21.09 at 1.4B and 16.74 versus 17.28 at 3B. Individual tasks are mixed, with Loopback ahead on nine and eleven of the thirteen tasks, respectively; all task scores are in Table 9.
Figure 5. General benchmarks at 1.4B and 3B. Loopback uses 68% and 75% of Standard’s Transformer parameters while attaining lower validation loss and higher thirteen-task mean accuracy. Bars use the left parameter axis; connected points use the right metric axis. Training tokens and executed depth are matched within each scale, with one run per arm at 20 TPP.
Code and mathematical evaluation
We test whether shared computation improves the modeling of mathematical and code solutions with fewer Transformer parameters. At matched training tokens and executed depth, reference-solution loss on GSM8K, MATH-500, HumanEval, and MBPP measures how well the model predicts a worked solution given the problem. Five-shot numeric-answer accuracy on six word-problem sets separately tests final-answer correctness. Loopback has lower solution loss on all four datasets at both large scales (Table 3). At 3B, word-problem accuracy rises from 2.6% to 6.4%. The nine-design comparisons at 191M and 374M also give Loopback the lowest four-set mean (Table 13). Evaluation protocols are in Section G.
| Reference-solution loss | 1.4B | 3B | ||||
| Standard | Loopback | Standard | Loopback | |||
| GSM8K | 1.275 | 1.227 | -0.048 | 1.143 | 1.086 | -0.057 |
| MATH-500 | 1.421 | 1.391 | -0.030 | 1.289 | 1.259 | -0.030 |
| HumanEval | 1.065 | 1.035 | -0.030 | 0.906 | 0.868 | -0.038 |
| MBPP | 1.282 | 1.235 | -0.047 | 1.104 | 1.078 | -0.026 |
| Four-set mean | 1.261 | 1.222 | -0.039 | 1.110 | 1.073 | -0.037 |
| Word problems (%) | 2.0 | 2.6 | +0.6 | 2.6 | 6.4 | +3.8 |
Table 3. Mathematical and code evaluation at 1.4B and 3B. Reference-solution loss is teacher-forced NLL (nats per solution token), not generation or code-execution accuracy. Word problems use mean five-shot numeric-answer accuracy over six sets. is Loopback minus Standard (nats for loss; percentage points for accuracy). Training tokens and depth are matched, with one run per arm. Shading marks Loopback; bold scores are row-best, and green differences favor Loopback.
| Reference solutions ↓ | Standard | Plain Loop | mHC | HC | iHC | Hyperloop | MHAR | AttnRes | Loopback |
| Hyper-connection family | mHC | HC | iHC | Hyperloop | |||||
| Attention family | MHAR | AttnRes | Loopback | ||||||
| 191M, 20 layers, [2L → 4L (×4) → 2L] (3.82B tokens) | |||||||||
| GSM8K | 1.822 | 1.865 | 1.795 | 1.801 | 1.826 | 1.833 | 1.826 | 1.797 | 1.768 |
| MATH-500 | 1.967 | 2.028 | 1.961 | 1.999 | 2.023 | 2.024 | 1.994 | 1.962 | 1.972 |
| HumanEval | 1.827 | 1.935 | 1.861 | 1.900 | 1.924 | 1.905 | 1.897 | 1.895 | 1.842 |
| MBPP | 2.021 | 2.118 | 2.028 | 2.042 | 2.081 | 2.074 | 2.076 | 2.064 | 2.018 |
| Mean | 1.909 | 1.987 | 1.911 | 1.935 | 1.964 | 1.959 | 1.948 | 1.929 | 1.900 |
| 374M, 32 layers, [2L → 14L (×2) → 2L] (7.47B tokens) | |||||||||
| GSM8K | 1.600 | 1.612 | 1.564 | 1.596 | 1.554 | 1.552 | 1.592 | 1.563 | 1.529 |
| MATH-500 | 1.755 | 1.755 | 1.740 | 1.747 | 1.728 | 1.737 | 1.746 | 1.734 | 1.707 |
| HumanEval | 1.518 | 1.520 | 1.487 | 1.514 | 1.495 | 1.493 | 1.503 | 1.481 | 1.447 |
| MBPP | 1.690 | 1.707 | 1.664 | 1.684 | 1.661 | 1.680 | 1.691 | 1.675 | 1.643 |
| Mean | 1.641 | 1.648 | 1.614 | 1.635 | 1.610 | 1.615 | 1.633 | 1.613 | 1.582 |
Table 13. Reference solutions at matched tokens. Teacher-forced loss (nats per token, lower is better) of the reference solutions of GSM8K, MATH-500, HumanEval and MBPP given the problem, for all nine designs at 191M and 374M. Every prompt starts with the beginning-of-sequence token. Paired bootstrap standard errors of the differences between rules are at most 0.003 on the mean. Bold marks the lowest loss per row.
Robustness to additional visits
Shared weights allow extra visits at inference, but excessive recurrence can degrade predictions, a problem known as overthinking ([4]). Ouro and the STARS study report this sensitivity beyond the trained loop count in language models ([38, 35]). We test whether Loopback reduces it without retraining, comparing with Plain Loop’s ordinary residual addition. Extra destinations reuse the last trained visit’s queries.
At 191M, 374M, and 1.4B, Loopback’s loss rises less across all tested extra counts, up to twice the trained count (Figure 6a–Figure 6c). At eight visits, the 191M loss increases by 0.060 for Loopback versus 0.454 for Plain Loop; at four visits, the 374M increases are 0.074 versus 0.270, and the 1.4B increases are 0.028 versus 0.248. On these curves, Loopback degrades less than Plain Loop when visits are added. At 3B, one extra iteration raises Loopback’s thirteen-task mean from 49.45% to 50.17% (Figure 6d); only Loopback is evaluated at extra depth, with Standard as a fixed-depth reference. The thirteen individual 3B task results are in Section H.
Figure 6. Robustness to additional visits without retraining. (a–c) Validation loss (nats/token), trained at 20, 32, and 25 layers; (d) thirteen-task mean accuracy, trained at 24 layers. Asterisks mark trained visit counts; the dashed line shows the fixed-depth Standard reference at 3B.
Ablations: decay, attention, and initialization
We test the contributions of decay and attention and whether Loopback’s gains persist across weight initializations. Decay controls accumulated update weight; attention allocates it by content. Table 4 tests these components separately and together, under three initialization schemes: fixed-scale Gaussian, GPT-2 residual scaling (Radford et al., [21]), and PyTorch defaults. Hyperloop reports that its advantage over Standard does not persist after switching to GPT-2-style initialization and retuning learning rates (Zeitoun et al., [36]), so we check Loopback’s sensitivity to initialization, holding learning rates fixed.
The complete design gives the lowest loss at both scales in panel (a): removing attention increases loss by 0.0033 and 0.0028 nats at 191M and 374M, respectively; removing decay increases it by 0.0434 and 0.0337. Thus content-based allocation improves on decay’s fixed weights, and controlling accumulation also benefits the learned reader. Across the six scale–initialization settings in panels (b–c), Loopback outperforms Standard and Plain Loop throughout and attains the lowest loss in five.
The default Gaussian scheme draws linear and embedding weights from . GPT-2 additionally scales the standard deviation of attention-output and MLP down-projections by , where is the executed depth. PyTorch uses Kaiming-uniform linear weights with and embeddings. These choices probe fixed-scale, depth-scaled, and fan-in-scaled linear weights, with different embedding scales.
Depth at fixed parameters
We ask whether Loopback’s validation-loss advantage persists when a fixed parameter budget is allocated to different widths and depths, instead of depending on one chosen topology. At 374M and 778M, we train every configuration from scratch, holding training tokens fixed and total parameters approximately fixed while trading width for depth from 16 to 64 layers. Our loop visits stay fixed at two. We compare the standard Transformer and the eight looped rules at every depth (Figure 7, Table 12, Section F). The effect of added depth differs across residual rules. The standard Transformer loses ground as it gets deeper and narrower: it is best at 24 layers and, at 64, is 0.076 worse at 374M and 0.035 worse at 778M. Among the looped rules, Loopback is lowest at nine of the ten points, and its best tested depth moves from 24 layers at 374M to 48 at 778M. At the sweep endpoints, its margin over the standard Transformer increases from 0.003 at 16 layers to 0.073 at 64 layers at 374M, and from 0.002 to 0.028 at 778M.
Figure 7. Depth at fixed parameters. ClimbMix validation loss after 20 TPP as each scale trades width for depth at approximately fixed total parameters and fixed training tokens. Looped rules make two visits through a shared stack that stores 56–58% of the standard baseline’s Transformer parameters.
Linear attention and recurrent memory in looped Transformers
We adapt the nine recurrent cores, Mesa, and a LaCT-inspired reader in Table 1 to visit boundaries at 191M and 374M, leaving token attention unchanged. Each reader uses either cumulative visit states or whole-visit updates; both interfaces initialize to Plain Loop (Section B.4).
The update interface gives lower validation loss in all 22 family–scale comparisons (Figure 8 and Table 11). Every update-based reader improves on Plain Loop, but the best readers still trail Loopback by 0.0302 nats at 191M and 0.0285 at 374M. These results favor updates over states as the memory value. Fixed-state recurrent cores keep one practical advantage for long loops: each new write is absorbed without rereading the history, and the state does not grow with the visit count. We have not tested whether optimized recurrent kernels turn this into lower memory traffic or higher throughput in deeply looped models.
Figure 8. Depth-memory readers. ClimbMix validation-loss reduction over Plain Loop at (a) 191M with 20 layers and (b) 374M with 24 layers for the update interface (solid) and the state interface (hatched) of each reader; the standard Transformer and Loopback are references.
| Reader | 191M Updates | 191M States | 374M, 24 layers Updates | 374M, 24 layers States |
| Gated DeltaNet | 2.6489 | 2.6685 | 2.4308 | 2.4333 |
| HGRN-2 | 2.6522 | 2.6619 | 2.4256 | 2.4333 |
| RWKV-7 | 2.6532 | 2.6791 | 2.4371 | 2.4398 |
| Mesa | 2.6539 | 2.6603 | 2.4277 | 2.4342 |
| DeltaNet | 2.6542 | 2.6664 | 2.4310 | 2.4329 |
| KDA | 2.6543 | 2.6612 | 2.4290 | 2.4295 |
| Mamba-2 | 2.6544 | 2.6759 | 2.4308 | 2.4319 |
| LaCT-inspired | 2.6558 | 2.6774 | 2.4308 | 2.4336 |
| GLA | 2.6640 | 2.6773 | 2.4352 | 2.4419 |
| RetNet | 2.6650 | 2.6827 | 2.4338 | 2.4396 |
| Linear attention | 2.6697 | 2.6846 | 2.4346 | 2.4417 |
| Reference models | ||||
| Plain Loop | 2.6802 | 2.4415 | ||
| Standard | 2.6234 | 2.4198 | ||
| Loopback | 2.6187 | 2.3971 |
Table 11. Memory readers in looped Transformers. ClimbMix validation loss (), with matched machine and budget within each scale (Sections D.1 and D.3). Bold marks the best reader in each column and the best reference model at each scale. Updates and states use one and eight heads, respectively, with different output blends (Section B.4).
Efficiency: compute, throughput, and memory
Loopback’s residual aggregation adds only 0.04–0.08% forward FLOPs over Standard at matched executed depth in the three measured configurations (Figure 9 and Table 15). These extra operations normalize and score stored updates, form their weighted sum, and apply decay and mixture replacement; they introduce no Q/K/V projections or extra block evaluations. With our fused implementation, 3B training throughput is 6% below Standard’s on the same eight H100 GPUs with the same batch layout. Detailed runtime and memory measurements are reported in § 18.
Figure 9. Compute and efficiency at 374M, 24 layers. Loopback and AttnRes run fused read kernels. (a) Per-GPU training throughput on eight H100s; bar labels give hours for the 20-TPP budget. (b) Validation loss versus forward FLOPs per token in excess of the standard baseline’s.
| Model | Transformer params (M) | Extra FLOPs | Train tok/s/GPU | Peak GiB/GPU | Prefill tok/s |
| 191M, [2L → 4L (×4) → 2L], 3.82B tokens, 16 sequences per GPU | |||||
| Standard | 141.59 | – | 222k | 38 | 885k |
| Plain Loop | 56.64 | +0.00% | 226k | 38 | 886k |
| mHC | 56.93 | +0.15% | 169k | 42 | 594k |
| HC | 56.65 | +0.07% | 186k | 42 | 671k |
| iHC | 57.04 | +0.17% | 116k | 67 | 370k |
| Hyperloop | 56.79 | +0.08% | 186k | 42 | 743k |
| MHAR | 56.64 | +0.01% | 204k | 41 | 810k |
| Loopback | 56.65 | +0.08% | 167k | 42 | 601k |
| 374M, [2L → 10L (×2) → 2L], 7.47B tokens, 8 sequences per GPU | |||||
| Standard | 308.33 | – | 128k | 26 | 492k |
| Plain Loop | 179.86 | +0.00% | 130k | 26 | 491k |
| mHC | 180.06 | +0.05% | 115k | 29 | 422k |
| HC | 179.87 | +0.02% | 120k | 30 | 439k |
| iHC | 180.53 | +0.15% | 68k | 52 | 220k |
| Hyperloop | 179.96 | +0.03% | 120k | 29 | 457k |
| MHAR | 179.86 | +0.01% | 126k | 27 | 480k |
| Loopback | 179.88 | +0.08% | 98k | 32 | 352k |
| 778M, [2L → 8L (×2) → 6L], 15.60B tokens, 8 sequences per GPU | |||||
| Standard | 679.55 | – | 75k | 38 | 269k |
| Plain Loop | 453.04 | +0.00% | 76k | 37 | 269k |
| mHC | 453.33 | +0.04% | 68k | 42 | 240k |
| Loopback | 453.06 | +0.04% | 63k | 44 | 217k |
Table 15. Compute and efficiency of the residual rules. Parameters are Transformer-only; extra forward FLOPs are relative to Standard. Percentages are rounded to two decimal places, with the positive 374M MHAR overhead rounded up to 0.01%. Training throughput and peak memory are per GPU. Prefill uses one H100.
Related work
Parameter sharing and residual reuse. Models with shared depth use weight tying [7, 16], relaxed tying with low-rank adaptations [2], latent reasoning [22, 10], and adaptive recursion [3]. Hyperloop learns connections between loop visits [36]. Residual pathways use gating, addition, and concatenation [24, 13, 14], scaling [1, 30], depth mixtures [5], multi-stream mixing [38, 31], and attention-based aggregation [27, 17 b]. Storing updates is not new; what we add is the split of update coefficients into input strength, total weight, and allocation, and an account of how each design constrains them.
Attention and recurrent memory. Sequence memory includes attention and additive fast weights [28, 15, 23], retention and gating [25, 32, 20], structured state-space dynamics [12, 11, 6], delta-rule corrections [33, 34, 26, 19], and optimization-based memory [29, 37]. We adapt these operations along depth, retaining each block’s token attention; results concern residual interfaces, not the original architectures’ efficiency.
Conclusion
We present Loopback, a residual design for parameter-efficient looped Transformers. We connect residual computation with linear attention and recurrent memory by treating successive block applications as memory steps along depth: each application writes new computation, which later blocks access through the residual connection. Loopback stores each block’s update separately and reads this history with attention. The read starts from geometrically decayed weights and redistributes them according to content without changing their sum; the weighted updates are then added to the original loop input. Across the 191M, 374M, 778M, 1.4B, and 3B scales, Loopback improves aggregate quality over matched-depth standard Transformer baselines while using 40–75% of their Transformer parameters, excluding token embeddings and the output head. In looped models, how the results of shared computation are retained and reused matters as much as which blocks are shared.
Limitations and future work
Our evidence is limited to ClimbMix pretraining at the tested scales and budgets, with single-run large-scale comparisons. The input coefficient is fixed at one; we have not tried learned or layer-specific input injection. Explicit history adds memory traffic and activations, so parameter savings need not reduce latency or peak memory. Extra visits have not improved validation loss. Next steps are other pretraining corpora, controlled comparisons of the reader interfaces, compact memories, and training with a variable visit count.
Ethics statement
This work studies the architecture of language models. All models have at most 3B parameters and are trained on ClimbMix, a public pretraining corpus, with evaluation on public benchmarks; no new data were collected and no human subjects were involved. The models are research artifacts that inherit the biases of their training corpus and are not intended for deployment. The method reduces stored parameters at matched executed depth and does not alter the risks associated with language models in general.
Proofs for the depth-memory formulation
Coefficient ordering for cumulative states
For ordinary additive states , a nonnegative normalized read gives
The equality follows by exchanging the finite sums: update appears in every state with . Hence and . Reading block updates separately removes this cumulative-sum restriction, but does not make the updates independent.
For a reweighted trajectory , the induced update weights are ; the ordering above need not hold.
Proof of the coefficient expansion
Repeated substitution in Eq. (10) gives
The base case is the update rule. Multiplying the previous expansion by and adding proves the induction step. The factors retain their order, and empty products are identity matrices. Substituting and multiplying by yields Eq. (12). A separate position write contributes by the same calculation.
The identity is evaluated on the actual trajectory. Keys, queries, and gates may depend on content; the resulting coefficients are still scalar. Projections of the values themselves require operator-valued coefficients; arbitrary nonlinear value maps are outside this expansion.
Residual adapters and reader implementations
Stream transport and explicit history
The block-level interface distinguishes a block update from a full block output , where . Our iHC adapter writes the block update. Hyper-Connections, mHC, and Hyperloop instead operate at visit boundaries: Hyper-Connections writes the whole-visit update, while mHC and Hyperloop write the full stack output together with a position value. Applying Eq. (10) to the stack map gives for the latter two, with the additional write ; indexes visits here. In value-by-stream storage, mHC uses and constrains this transport by Sinkhorn normalization; the additional input-write path is not part of that constraint. Hyperloop uses diagonal transport. iHC instead fixes and learns nonnegative write gates, input reads, and the final-coda read. Its original rule surrounds the attention and MLP computations separately; our adapter surrounds each complete block and writes .
For a history bank, appending uses and a write vector selecting the new storage location. Appending the full block output instead uses . The sources and placements used by AttnRes, MHAR, and Loopback are specified in Section D.2.
Normalized attention and recurrent memory
Finite-feature attention. Linear attention summarizes values in sufficient statistics (Katharopoulos et al., [15]; Schlag et al., [23]). For a finite feature map with , associativity gives
Updating and avoids storing past values for either . The denominator must be positive; this finite-feature identity is not a general softmax equivalence.
A destination-dependent softmax query changes earlier scores, so its previous read alone generally cannot summarize the history. Our evaluated recurrent adapters instead use the unnormalized memory update in Eq. (7a), omitting the kernel normalizer and allowing signed read coefficients.
Decay and key-specific correction. The delta rule corrects a memory prediction; it does not denote the block output-minus-input transformation. At block granularity, its target can be either a full block output or a block update. For these families, let :
Decay acts before correction toward the new value [33, 34, 26]. Thus KDA uses in value-by-key orientation; reversing the factors generally changes the operator. RWKV-7 subtracts a rank-one correction from diagonal transport. Its normalized removal direction and coordinatewise strength need not align with the write key [19].
Regression readers and their additional state
The regression readers in Table 1 require additional state: Mesa tracks inverse covariance, while the LaCT-inspired reader revisits a stored prefix.
Mesa. Let , , with , and . Differentiating the weighted ridge objective gives
Its recursive form starts from and :
The rank-one inverse identity yields . Substituting this into the ridge normal equations gives the update for . Hence and , with as control state. Our implementation evaluates the equivalent weighted Gram solve.
LaCT-inspired reader. Let , , and . A gradient step on the weighted prefix prediction error, followed by the implemented normalization, is
Because old values are written again, alone is insufficient for a single-new-value update. Set and . The augmented operators
give by direct multiplication, with read selecting . The initial augmented state is ; prefix blocks are empty at . This adapter uses linear predictions over the cumulative prefix, not the published LaCT layer. Its Frobenius normalization makes the update nonlinear, preserves the value span, and remains in the derivative graph.
Visit-level reader interfaces
The two interfaces in Table 11 read at visit boundaries and initialize to Plain Loop. Both use unprojected values, L2 normalization with an epsilon floor, and no learned Q/K/V projections.
Cumulative states and whole-visit updates
Write for the output of visit before its boundary combination. The paired visit update is
The equality telescopes under within-visit addition, as used here; block-level reweighting would also enter the visit difference.
The state interface stores and each visit output before its boundary blend. Denote these stored states by . Within each reader head,
The current key and query coincide, so their inner product is one when the normalization floor is inactive. The memory output is combined with the current proposal as
The update interface stores , with
Its output is
In both interfaces is an unconstrained learned scalar initialized to zero, recovering Plain Loop.
Gates and memory evaluation
In our adapters, gates are content-independent learned parameters indexed by junction, head, and, when required, channel. Key-addressed families write along ; HGRN-2 writes the gate complement instead. For RWKV-7, the removal direction in Table 1 is the normalized visit update in the update interface and the normalized difference of adjacent stored states in the state interface. Mesa and the LaCT-inspired reader follow Section B.3, including the latter’s prefix normalization.
The implementations evaluate short histories using the factorization , where is a write vector after subsequent transitions, or replay the required prefixes. These evaluate the stated reads without establishing an end-to-end recurrent speedup.
Loopback implementation and derivations
Mixture replacement and initialization
After appending the newest block update and applying anchored decay at visit ends, write the current proposal as . The implementation reaches the target in Eq. (17) by adding only the coefficient difference:
The implementation uses skip parameter and tracks the actual coefficients after prior reads to avoid double-counting contributions. The explicit input path has unit gain, but the full Jacobian also includes the input dependence of the updates and routing weights.
To make the initial correction vanish, target coefficients are evaluated as
For a positive unfloored prior this equals in exact arithmetic. Zero queries cancel the bracket; causal induction then recovers the decay-only trajectory of § 4.1. The first read has only one source and requires no query. The learned parameters total 11,520 at , , (the 191M setting).
The actual log prior uses in both softmax terms. If the floor activates, the centered form still preserves the total weight in exact arithmetic and cancels at zero queries, but its coefficients need not remain nonnegative. All reported configurations satisfy , so the floor is inactive.
Allocation gradient
With the update bank, prior, and total fixed, and the prior floor inactive so that exactly, write and , so . The softmax derivative gives . Contraction with yields
The sum vanishes because . This local gradient redistributes a fixed total according to each update’s alignment with the downstream loss gradient. Actual score changes are coupled through the query; full training also changes the keys and updates.
Training and evaluation protocols
Table 5 lists the architecture, batch composition, and budget at every scale. We give the shared recipe and evaluation protocol once, then summarize larger-scale differences. The per-GPU sequence counts in Table 15 are these training micro-batches, and timing runs use the same micro-batch and accumulation.
| Scale | Width | Blocks | Heads | SwiGLU | Loopback | Seq./step | Micro × acc. | Steps | Tokens | LR () |
| 191M | 768 | 20 | 12 | 2,048 | [2L → 4L (×4) → 2L] | 128 | 16 × 1 | 7,280 | 3.82B | 1 / 3 |
| 374M | 1,024 | 24 | 16 | 2,816 | [2L → 10L (×2) → 2L] | 256 | 8 × 4 | 7,120 | 7.47B | 2 / 2 |
| 778M | 1,536 | 24 | 24 | 4,096 | [2L → 8L (×2) → 6L] | 512 | 8 × 8 | 7,440 | 15.60B | 1 / 1.5 |
| 1.4B | 2,048 | 25 | 32 | 5,632 | [3L → 8L (×2) → 6L] | 960 | 8 × 15 | 7,200 | 28.31B | 1 / 1.5 |
| 3B | 3,072 | 24 | 48 | 8,192 | [6L → 6L (×2) → 6L] | 1,920 | 4 × 60 | 7,600 | 59.77B | 1 / 1 |
Table 5. Configuration at each scale. Blocks and heads apply to Standard and to every looped arm; head width is 64 throughout. Sequences per optimizer step are the per-GPU micro-batch times gradient-accumulation steps, on 8 ranks, at 4,096 tokens per sequence. Peak learning rates are Standard / looped arms.
Architecture and training protocol
Scale names use the standard baseline’s total parameters; reported Transformer counts exclude embeddings and the vocabulary head. We use the 20-TPP training setting, with the same token budget for all models within each comparison. At 191M, all nine models train from scratch for 7,280 steps using Table 6 and Table 7, differing only in topology, residual rule, and peak learning rate. An additional eight-block standard Transformer baseline matches Plain Loop’s stored parameters (56.64M), not its twenty applications; under the same protocol and tokens, its loss is 2.7376 versus Plain Loop’s 2.6802. Budgets are rounded to a 20-step grid. During the final tenth of training, the learning-rate multiplier is for decay progress . All results use the final checkpoint.
| Setting | Value |
| Standard structure | Twenty independent blocks |
| Looped structure | ; eight distinct blocks, twenty applications |
| Attention | Full-sequence causal attention; 12 heads of width 64; no GQA |
| Hidden / feed-forward width | 768 / 2,048; SwiGLU feed-forward layers |
| Normalization | Pre-RMSNorm; |
| Position encoding | RoPE; |
| Context / vocabulary | 4,096 tokens / 32,000 Llama tokens |
| Embedding / vocabulary head | Untied; omitted from reported Transformer counts |
| Trunk initialization | Linear and embedding weights: |
| Other trunk options | Bias-free attention projections; no QK normalization, logit softcap, or zero-initialized residual-output projections |
Table 6. 191M architecture. Looped topology: .
| Setting | Value |
| Training data | ClimbMix [8]; training shards 0–239 |
| Input construction | Online tokenization; concatenated documents packed into fixed 4,096-token sequences |
| Global batch | 128 sequences; 8 GPU ranks 16 sequences per rank |
| Gradient accumulation | 1; 524,288 input tokens per optimizer step |
| Training budget | 7,280 steps; 3,816,816,640 input tokens |
| Optimizer | AdamW; ; |
| Weight decay / gradient clip | 0 / global norm 1.0 |
| Peak learning rate | Standard: ; all looped arms: |
| Learning-rate schedule | Linear warmup: 128 steps; stable peak: 6,424 steps; square-root decay to zero: 728 steps |
| Hardware allocation | One node with eight GPU ranks for every arm |
| Mixed precision | BF16 parameter computation; FP32 gradient reductions |
| Execution | FSDP; no activation checkpointing or model compilation |
Table 7. Training recipe at the 20-TPP endpoint. Only peak learning rates differ across arms.
The 374M, 778M, 1.4B, and 3B runs use the same optimizer, tokenization, normalization, schedule, and evaluation protocols. Table 5 gives their widths, topologies, batches, budgets, and peak learning rates. The 1.4B and 3B runs use training shards 0–559; smaller scales use shards 0–239. Within each comparison, training tokens and executed depth are matched. The depth-sweep variants are specified in Section F.
Residual rules and evaluation
Trunk weights are shared across visits; connections and queries are untied where required. The mHC, Hyper-Connections, and Hyperloop adapters act at visit boundaries with four streams; mHC uses 20 Sinkhorn iterations and Hyperloop uses zero residual bias. iHC wraps each middle block with four streams, reading the block input through gates before it and writing the block update back through gates after it; it uses magnitude 2, gate offset , and a learned final readout. MHAR reads loop states at visit boundaries with eight heads and an outer blend. AttnRes reads the loop input and block states before token attention and before the MLP in each loop block, and once at loop exit, using Block AttnRes with . Loopback reads block updates after each middle block with one head and . The queries and gates of MHAR, AttnRes, and Loopback start at zero.
Held-out language modeling. Validation uses disjoint ClimbMix shard 3583. As in training, documents are tokenized with beginning-of-sequence markers, concatenated, and packed into 4,096-token sequences. All models use the first 1,024 sequences and their final checkpoints, evaluated in BF16. Loss averages next-token cross-entropy over 4,095 predicted positions per sequence after shifting labels. It is reported in nats; perplexity is its exponential.
Downstream tasks. The same final checkpoints are evaluated using lm-evaluation-harness 0.4.12 [9], zero-shot prompts, full evaluation sets, and 1,000 bootstrap iterations; every prompt starts with the beginning-of-sequence token, as training prepends it to every document. HellaSwag, OpenBookQA, ARC-Easy, and ARC-Challenge use length-normalized accuracy; the other accuracy tasks use the harness’s standard accuracy. MMLU is the harness aggregate over 57 subjects and counts as one task in the unweighted thirteen-task mean. WikiText word perplexity is reported separately and excluded from this mean.
Learning-rate selection
At 191M, Standard’s lowest loss among the four peak rates tested at 20 TPP is attained at (Table 8); looped models use their best tested rate, .
| Peak learning rate | ||||
| ClimbMix loss | 2.6666 | 2.6420 | 2.6234 | 2.6245 |
Table 8. 191M Standard learning-rate comparison. ClimbMix validation loss after 20 TPP on the same machine and training recipe. Bold marks the lowest loss.
For the final 20-TPP runs, 374M uses for all arms, selected for Standard from . At 778M, Standard inherits ; Loopback uses after a check against . At 1.4B, Standard uses from , and Loopback uses from . At 3B, both use , from for Standard and for Loopback. Screening used 5 TPP at 374M/778M and 2 TPP at 1.4B/3B; 1.4B additionally compared and at 20 TPP. Loopback’s 1.4B and 3B rate checks used [3L → 10L (×2) → 2L] and [2L → 10L (×2) → 2L], respectively; the final 3B topology was chosen in a separate 5-TPP screen. Short runs apply their own final decay, and continuations resume from the pre-decay checkpoint.
Full general-benchmark results at 1.4B and 3B
Table 9 gives the taskwise scores underlying the large-scale comparison, using the same beginning-of-sequence prompts and metric conventions as the smaller-scale evaluation.
At 3B, the bootstrap standard error of each model’s thirteen-task mean is 0.40 percentage points.
Run-to-run variation and memory-reader results
Run-to-run variation
Table 10 reports two training seeds for every 191M arm except AttnRes and four arms at 374M (24 layers). Each seed sets both the initialization and the data order. At 191M, Loopback has the lowest validation loss among the tested looped models in both runs. A third seed for the standard Transformer, Plain Loop, mHC, and Loopback keeps this ordering (Loopback −0.0110 against mHC and −0.0716 against Plain Loop). Its paired loss differences from the standard Transformer are −0.0047, +0.0011, and −0.0064 (mean −0.0033), so that comparison is not consistent across seeds. At 374M, Loopback has the lowest validation loss among the four tested models in both seeds.
| Model | seed 42 | seed 1337 |
| 191M, 20 layers | ||
| Standard | 2.6234 | 2.6144 |
| Plain Loop | 2.6802 | 2.6815 |
| mHC | 2.6250 | 2.6292 |
| HC | 2.6449 | 2.6427 |
| iHC | 2.6564 | 2.6497 |
| Hyperloop | 2.6489 | 2.6472 |
| MHAR | 2.6564 | 2.6573 |
| Loopback | 2.6187 | 2.6155 |
| Loopback − Standard | −0.0047 | +0.0011 |
| Loopback − mHC | −0.0063 | −0.0137 |
| 374M, 24 layers | ||
| Standard | 2.4198 | 2.4257 |
| mHC | 2.4062 | 2.4079 |
| Hyperloop | 2.4179 | 2.4175 |
| Loopback | 2.3971 | 2.3996 |
| Loopback − Standard | −0.0227 | −0.0261 |
| Loopback − mHC | −0.0091 | −0.0083 |
Table 10. Validation-loss variation at 191M and 374M. ClimbMix loss (nats, lower is better) after 20 TPP. Each seed sets initialization and data order; difference rows compare models within a seed. Seed 42 is the run reported in the main text. Bold marks the best model in each column within each scale; shading marks Loopback. Validation loss
Memory-reader results
Depth sweep at fixed parameters
Table 12 gives the configurations and results for all nine rules, using the recipes in Section D.1. Width varies with depth to keep Standard’s total parameters within 5% of its 24-layer reference; training tokens remain fixed. Every run uses one node with eight 80 GB GPUs: H100s at 374M and A100s at 778M, including the 48-layer models in Table 2. Learning rates are for all 374M arms, and for Standard versus for looped arms at 778M. We also check Standard’s rate: at 374M, and underperform after 5 TPP at 48 and 64 layers ( diverges at 48). At 778M, using changes its 20-TPP loss by , , , and at 16, 24, 32, and 48 layers.
Mathematical and code text
We use these evaluations to test whether parameter-efficient reuse also preserves mathematical and code solution modeling. The 1.4B and 3B results are in Table 3; Table 13 compares all nine designs at 191M and 374M. These use the checkpoints and matched training budgets of Table 2, including the 32-layer 374M setting.
Reference-solution loss. We evaluate GSM8K test (1,319 problems), MATH-500 (500), HumanEval (164), and sanitized MBPP test (257). Each prompt contains a beginning-of-sequence token and the problem; MBPP additionally includes the first test. GSM8K calculator annotations are removed and the final line is standardized to The answer is N. We score the reference solution with teacher forcing in a 4,096-token window using BF16 and the reference Loopback read, averaging NLL over solution tokens. The four-set mean weights datasets equally. Paired bootstrap resampling uses 2,000 draws over problems. This metric evaluates solution likelihood, not generated-answer or code-execution correctness.
Five-shot numeric-answer accuracy. The six sets are SingleOp (562), SingleEq (508), AddSub (395), MultiArith (600), the numeric-answer subset of ASDiv (2,084), and SVAMP (1,000), from the ToRA evaluation files. We use the first five standard-prompting exemplars of chain-of-thought prompting, formatted as . A response is correct when each reference-number token is the greedy choice and the greedy token following the number is not a digit. The reported mean weights all six sets equally; the 1.4B and 3B means appear in Table 3. At 3B, Loopback is higher on each set: 11.9 versus 4.3 on SingleOp, 8.5 versus 3.3 on SingleEq, 5.6 versus 2.0 on AddSub, 1.3 versus 0.8 on MultiArith, 7.0 versus 3.4 on ASDiv, and 4.1 versus 1.9 on SVAMP. The low absolute accuracies limit claims about reliable mathematical problem solving.
3B task results with one extra visit
We evaluate the same 3B Loopback checkpoint at its trained two visits and at three visits without retraining. Additional destinations reuse the last trained visit’s query for the same block position; the shared blocks and coda are unchanged. We report all thirteen zero-shot tasks using the main evaluation protocol: HellaSwag, OpenBookQA, ARC-Easy, and ARC-Challenge use length-normalized accuracy; the other tasks use accuracy. The mean weights the thirteen tasks equally, with MMLU’s 57 subjects counted as one task.
| Task | 2 visits (trained) | 3 visits | (pp) |
| PIQA | 75.90 | 75.30 | -0.60 |
| HellaSwag | 63.72 | 65.11 | +1.39 |
| WinoGrande | 58.48 | 58.96 | +0.48 |
| OpenBookQA | 41.80 | 42.80 | +1.00 |
| CommonsenseQA | 21.87 | 22.36 | +0.49 |
| BoolQ | 64.95 | 65.60 | +0.65 |
| RACE | 36.17 | 35.60 | -0.57 |
| ARC-Easy | 71.13 | 71.04 | -0.09 |
| ARC-Challenge | 40.96 | 40.02 | -0.94 |
| MMLU (57 subjects) | 28.86 | 29.69 | +0.83 |
| ANLI-R1 | 32.90 | 33.70 | +0.80 |
| RTE | 56.68 | 63.90 | +7.22 |
| LAMBADA | 49.45 | 48.09 | -1.36 |
| Thirteen-task mean | 49.45 | 50.17 | +0.72 |
Table 14. 3B Loopback with one extra visit. Zero-shot scores (%) at two visits (trained) and three visits, without retraining. is the change in percentage points computed from the displayed scores; bold marks the higher score in each row.
The mean rises by 0.72 points, with gains on eight of thirteen tasks. RTE (277 items) contributes 0.56 points to the mean increase. These are single-checkpoint measurements; the improvement is not uniform across tasks.
Compute and efficiency measurements
FLOP accounting. Table 15 covers three scales. PyTorch’s operator counter measures forward FLOPs on one 4,096-token bf16 sequence, including the vocabulary head. Scaled-dot-product attention substitutes for the uncounted flash kernel; uncounted elementwise work is added analytically. For a read at destination over stored updates, the score and weighted-sum matmuls contribute FLOPs per token, excluding normalization, softmax, and mixture replacement. Anchored decay costs per token at each visit end; AttnRes reads before token attention and before the MLP in each loop block, and once at loop exit, over the loop input and up to eight loop states; the stream rules cost per connection for streams.
Training measurements. Measurements use one eight-H100 node, FSDP, bf16 compute, and each scale’s micro-batch. Loopback uses fused reads measured in 160-step timing runs; other entries come from the training logs, including AttnRes in Figure 9, which trains with its fused read. For these three scales, throughput uses median step time, and memory is the trainer’s reported peak. Activations dominate sharded model state here, so fewer stored weights need not lower peak memory. At 3B, the mean logged training throughputs are 24.7k tokens/s/GPU for Standard and 23.3k for Loopback, a 6% gap after rounding. Both runs use the same eight-H100 node, four 4,096-token sequences per GPU, and 60 gradient accumulation steps. These means cover the logged intervals of the final 20-TPP training stage.
Inference measurements. One H100 runs each model eagerly with bf16 weights and no gradients; Loopback uses fused reads. Prefill is one forward pass over a batch of eight 4,096-token sequences that keeps only the last position’s logits and materializes no KV cache, so its measured peak covers weights and transient activations. For Loopback these include the depth history, which exists only within a forward pass and is never cached across decoding steps. Loopback’s prefill throughput relative to the standard Transformer is 0.68, 0.72, and 0.81 at 191M, 374M, and 778M. Table 16 instead counts persistent memory analytically: all stored weights plus the KV cache, whose capacity is equal at matched executed depth. The two views differ by construction: at 374M the measured prefill peak is 3.0 GiB for Loopback versus 2.1 GiB for the standard Transformer, whereas the persistent total for the same eight sequences is 3.46 versus 3.70 GiB.
| Scale | Model | Total params (M) | Weights (GiB) | KV cache per sequence (MiB) | Weights + KV (GiB) 1 sequence | Weights + KV (GiB) 8 sequences |
| 191M, 20 layers | Standard | 190.74 | 0.36 | 240 | 0.59 | 2.23 |
| Loopback | 105.80 | 0.20 | 240 | 0.43 | 2.07 | |
| 374M, 24 layers | Standard | 373.87 | 0.70 | 384 | 1.07 | 3.70 |
| Loopback | 245.42 | 0.46 | 384 | 0.83 | 3.46 | |
| 778M, 24 layers | Standard | 777.86 | 1.45 | 576 | 2.01 | 5.95 |
| Loopback | 551.36 | 1.03 | 576 | 1.59 | 5.53 | |
| 1.4B, 25 layers | Standard | 1,415.68 | 2.64 | 800 | 3.42 | 8.89 |
| Loopback | 1,004.64 | 1.87 | 800 | 2.65 | 8.12 | |
| 3B, 24 layers | Standard | 2,914.67 | 5.43 | 1,152 | 6.55 | 14.43 |
| Loopback | 2,235.19 | 4.16 | 1,152 | 5.29 | 13.16 |
Table 16. Persistent inference memory of Standard and Loopback. Analytical bf16 storage for all weights, including embeddings and the vocabulary head, plus the KV cache at 4,096 tokens. Totals exclude temporary activations and workspaces; they are not measured peaks. Shading marks Loopback.
The fused read. Each read uses one Triton kernel, streaming updates once per token tile while accumulating the scored and prior softmaxes together. Backward saves one score per token and update rather than normalized keys, then reloads each update and writes its gradient once. Tests compare the kernel with a direct PyTorch implementation of Eq. (17). In FP32, the output, the installed coefficients, and every gradient (query, updates through the value and key paths, proposal, input, and previous coefficients) agree to within relative error, measured as the largest absolute difference over the largest reference magnitude, across widths 768–3,072, 2–20 stored updates, ragged token counts, and boundary and in-visit reads. With bf16 inputs, both implementations are compared with an FP64 evaluation of the same rounded inputs, and the fused error is no larger than the reference’s. At zero queries the output and coefficients match the decay-only reference, which reduces to Plain Loop on only at . An end-to-end comparison of the two implementations on a small model gives bitwise-identical results at initialization and agreement within after perturbing the queries, forward and backward. Most remaining step-time overhead is in backward, as each update receives gradients from later reads. The kernel and tests will be released with the code.
References
- [1]Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in artificial intelligence, pp. 1352–1361. PMLR, 2021.
- [2]Sangmin Bae, Adam Fisch, Hraryr Harutyunyan, Ziwei Ji, Seungyeon Kim, and Tal Schuster. Relaxed recursive transformers: Effective parameter sharing with layer-wise lora. In International Conference on Learning Representations, volume 2025, pp. 34282–34327, 2025.arxiv.org/abs/2410.20672
- [3]Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hraryr Harutyunyan, Ziwei Ji, Aaron Courville, et al. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. Advances in Neural Information Processing Systems, 38:96572–96617, 2026.
- [4]Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In Advances in Neural Information Processing Systems, volume 35, pp. 20232–20242, 2022.
- [5]Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3028–3033, 2018.
- [6]Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024.
- [7]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018.arxiv.org/abs/1807.03819
- [8]Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. Advances in Neural Information Processing Systems, 38, 2026.
- [9]Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Zenodo, 2021.DOI
- [10]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. Advances in Neural Information Processing Systems, 38:41340–41391, 2026.arxiv.org/abs/2502.05171
- [11]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.arxiv.org/abs/2312.00752
- [12]Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.arxiv.org/abs/2111.00396
- [13]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.arxiv.org/abs/1512.03385
- [14]Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2018. URL https://arxiv.org/abs/1608.06993.
- [15]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156–5165. PMLR, 2020.arxiv.org/abs/2006.16236
- [16]Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.arxiv.org/abs/1909.11942
- [17]Cheng Luo, Zefan Cai, and Junjie Hu. Delta attention residuals. arXiv preprint arXiv:2605.18855, 2026a.arxiv.org/abs/2605.18855
- [18]Cheng Luo, Zefan Cai, and Junjie Hu. Multi-head attention residuals. arXiv preprint arXiv:2607.27230, 2026b.arxiv.org/abs/2607.27230
- [19]Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaj u Lin, Jiaxing Liu, Janna Lu, William Merrill, et al. Rwk-v7" goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456, 2025.arxiv.org/abs/2503.14456
- [20]Zhen Qin, Songlin Yang, Weixuan Sun, Guangyuan Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024.arxiv.org/abs/2404.07904
- [21]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- [22]Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. In International Conference on Learning Representations, volume 2025, pp. 14855–14881, 2025.arxiv.org/abs/2502.17416
- [23]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International conference on machine learning, pp. 9355–9366. PMLR, 2021.arxiv.org/abs/2102.11174
- [24]Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks, 2015. URL https://arxiv.org/abs/1505.00387.
- [25]Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.arxiv.org/abs/2307.08621
- [26]Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025.
- [27]Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, Yaoyu Wang, Yucheng Wang, Guanduo Chen, Bohong Yin, et al. Attention residuals. arXiv preprint arXiv:2603.15031, 2026.
- [28]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
- [29]Johannes Von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zuccher, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858, 2023.arxiv.org/abs/2309.05858
- [30]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761–6774, 2024.arxiv.org/abs/2203.00555
- [31]Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Kuai Yu, et al. mhc: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880, 2025.arxiv.org/abs/2512.24880
- [32]Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023.arxiv.org/abs/2312.06635
- [33]Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522, 2024.arxiv.org/abs/2406.06484
- [34]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In International Conference on Learning Representations, volume 2025, pp. 29687–29707, 2025.arxiv.org/abs/2412.06464
- [35]Xiao-Wen Yang, Ziyu Han, Xi-Hua Zhang, Wen-Da Wei, Jie-Jing Shao, Lan-Zhe Guo, and Yu-Feng Li. Stabilizing recurrent dynamics for test-time scalable latent reasoning in looped language models. arXiv preprint arXiv:2605.26733, 2026.arxiv.org/abs/2605.26733
- [36]Abbas Zeitoun, Lucas Torroba-Henningen, and Yoon Kim. Hyperloop transformers. arXiv preprint arXiv:2604.21254, 2026.arxiv.org/abs/2604.21254
- [37]Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William Freeman, and Hao Tan. Test-time training done right. In International Conference on Learning Representations, volume 2026, pp. 157604–157638, 2026.arxiv.org/abs/2505.23884
- [38]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaij ing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025b.arxiv.org/abs/2510.25741








