Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

Introduction

Introduction

Looped transformers have attracted attention for combining strong performance with efficient use of parameters and training memory. By repeatedly applying a shared block of layers, they increase computational depth without increasing the number of trainable parameters [19, 28]. To reduce the cost of training these repeated blocks, several looped language models use truncated backpropagation through time (BPTT), propagating gradients only through the final loops while executing the earlier loops without tracking gradients [28, 37, 51, 59, 73]. Truncation introduces a new tradeoff: it saves backward computation and activation memory during training but can weaken the gains from looping [65]. To achieve the best performance under a fixed training compute budget, we must therefore decide how to divide compute among forward loops, backward loops, model size, and training tokens.

Existing scaling studies examine prescribed backpropagation recipes rather than jointly modeling forward and backward loop counts. Schwethelm et al. [65] compare full BPTT with truncation to half the loops, fitting a separate scaling law to each recipe. Parcae Prairie et al., [59] varies backward-loop counts in small-model experiments but fixes them to half the forward-loop count in its scaling study, leaving their joint optimization to future work. These studies characterize specific recipes but do not provide a unified law for choosing both loop counts across model sizes and token budgets (Table 1). We thus ask the following question:

Study	Arch.	$N$	$D$	$T$	$B$	Loop gain	BPTT	Regime
Schwethelm et al. [65]	dense	✓	✓	✓	✗	power law ($T^\phi$)	$T$; $\lceil T/2\rceil$	fixed depth
Parcae [59]	dense	✓	✓	✓	✗	linear ($T$)	$\lceil T/2\rceil$	fixed $N$
Chen et al. [14]	dense, MoE	✓	✓	✓	✗	bounded exp. ($T$)	full ($T$)	free $N$, mem.
SMELT [74]	MoE	✓	✓	✗	✗	per-arch. fit	/	matched $N$, KV
Lee et al. [46]	dense, MoE	✓	✗	✗	✗	per-arch. fit	/	free $N$
Chen et al. [15]	dense	✓	✓	✗	✗	per-arch. fit	full ($T$)	free $N$
Ouro† [82]	dense	✓	✓	✓	✗	additive ($T$)	/	/
Ours	dense	✓	✓	✓	✓	bounded ($T$, $B$)	$1 \le B \le T$	fixed & free $N$

How do model size, training tokens, and forward and backward loops jointly affect loss, and should we truncate backpropagation under a fixed training compute budget?

To characterize how these quantities affect loss, we conduct nearly 600 training runs, including a scaling suite spanning 58M–919M non-embedding parameters, over three orders of magnitude in training tokens per parameter, and up to 16 forward loops. We independently vary forward and backward loop counts across model sizes and token budgets and fit a unified scaling law. The scaling law expresses the benefit of looping through an effective parameter count—the size of a non-looped model that achieves the same loss at the same token budget. Fully trained loops increase the effective parameter count with diminishing returns, while truncation discounts this gain.

We then use the scaling law to decide whether to truncate, and the answer depends on whether the forward-loop count is prescribed. When it is, the law recovers the benefit of truncation reported in prior work [65]: the saved backward compute supports a larger model and more training tokens, which more than offset the discounted loop gain. When both loop counts can be chosen, however, the law favors full BPTT. Reducing the forward-loop count frees compute for additional backward loops until every remaining loop is trained, and under the fitted law this reallocation yields more effective parameters at the same compute than the original truncated setting. Separately fitted laws and matched-compute experiments support this advantage (§5).

Building on this preference for full BPTT, we additionally study how to allocate compute among loop count, training tokens, and model size. Consider first a setting where model size is predetermined, for example due to device memory constraints. At fixed model size, increasing the loop count is an effective way to increase the effective parameter count; however, additional loops offer diminishing gains while increasing the cost per token. The law precisely characterizes the optimal tradeoff between loop count and training tokens, allocating most additional compute to tokens and increasing the loop count only slowly. This result extends Parcae’s fixed-size allocation analysis [59] beyond a prescribed truncation rule. When model size is also free, compute can instead support more unique parameters; under our leading-FLOP approximation, the law favors a non-looped model and identifies two fully trained loops as the best looped setting (Figure 2). In experiments, models with two fully trained loops nearly match compute-optimal non-looped models at equal training compute while having substantially fewer non-embedding parameters. With a little more training compute, such a smaller model surpasses the compute-optimal non-looped model (§ 7).

Our contributions are threefold:

• A unified scaling law based on effective parameters. We develop and validate a law that expresses loop gains and truncation discounts through an effective parameter count, jointly modeling model size, training tokens, and independent forward and backward loop counts (§ 5).

• An explanation of when truncation pays. We show why truncation can pay at a prescribed forward-loop count and why joint forward–backward allocation favors full BPTT under the fitted law, supported by matched-compute experiments (§5).

• Compute-allocation rules and parameter-efficient training. We derive allocation rules for fixed and free model size and demonstrate that two fully trained loops nearly match compute-optimal non-looped models while having substantially fewer trainable parameters (§ 7).

Figure 1 previews the three results: loops add effective parameters and truncation discounts them; at fixed model size the best loop count grows with compute; two fully trained loops match the loss of a non-looped model with fewer parameters for a little more compute.

Related work

Looped transformers. Depth-wise parameter sharing in Universal Transformers and ALBERT decouples computational depth from the number of trainable parameters ([19, 45]). Theoretical and controlled studies show how recurrence supports iterative programs, in-context learning algorithms, and depth-sensitive reasoning ([31 Saunshi et al.,] 2025; Yang et al. [80]. In language model pretraining, Huginn uses a prelude–recurrent–coda architecture (Geiping et al. [28]), while Ouro scales looped pretraining to 7.7T tokens (Zhu et al. [82]). Other work adds recurrence to pretrained models (Bae et al. [3]; McLeish et al. [51]) or adapts depth at inference (Bae et al. [4]; Jeddi et al. [38]). Recent architectures improve recurrence through residual and hyper-connections (Fu et al. (2026); Marchenko et al. [50]; Prairie et al. [59]; Zeitoun et al. [81]) or MoE routing (Chen et al. [13]; Csordás et al. [18]; Gao et al. [27]; He et al. (2026)). These studies show that looping improves parameter efficiency across architectures; we instead fix the architecture and ask how to train its loops and allocate compute.

Training through recurrent depth. Truncated BPTT limits gradient propagation to a subset of recurrent steps, saving activation memory and backward compute at the cost of biased gradients (Tallec and Ollivier [68]; Williams and Peng [76]). It is the common recipe for looped language models, which backpropagate through the last few loops or half of them (Geiping et al. [28]; McLeish et al. [51]; Prairie et al. [59]; Schwethelm et al. [65]), and recursive reasoning models often keep only the last step (Jolicoeur-Martineau [39]; Wang et al. [72]). §A surveys these recipes in more detail. Implicit models likewise differentiate only a few steps at the fixed point (Bai et al. [7]; Geng et al. [30]), and truncated BPTT in looped transformers can be read as a truncated Neumann series of the fixed-point gradient (Huang et al. [37]; Movahedi et al. [52]). The few studies that measure what truncation costs compare training recipes at a fixed forward depth (Fein-Ashley and Rashidinejad [21]; Huang et al. [36]; Ren and Liu [61]; Salvatori et al. [62]). We instead vary the backward depth independently of the forward depth, weighing the compute that truncation saves against the loop gain it costs.

Scaling laws for recurrence. Size–data scaling laws quantify how parameters and training tokens jointly determine loss (Hoffmann et al. [35]; Kaplan et al. [41]; Videau et al. [71]). For looped models, some laws fit each architecture separately (Chen et al. [15]; Lee et al. [46]; Wang et al. [74]), while others model the loop count explicitly (Table 1) through additive loss terms (Zhu et al. [82]) or effective parameters with linear, power-law, or bounded gains (Chen et al. [14]; Prairie et al. [59]; Schwethelm et al. [65]). Closest to our work, Iso-Depth (Schwethelm et al. [65]) fits separate laws for full and half-length backpropagation and finds that truncation weakens the loop gain but lowers the loss at a fixed forward-loop count by freeing compute, while Parcae (Prairie et al. [59]) finds at fixed model size that loops and tokens should grow together with compute but ties the backward loops to half the forward loops and leaves their joint choice open. Our law instead treats the backward-loop count as a separate variable, combining coupled size–data scaling (Videau et al. [71]) with bounded loop gains (Chen et al. [14]). It recovers the truncation benefit at a prescribed forward-loop count but favors full BPTT when both counts are free (§5). Looped MoE models commonly use two loops (Gao et al. [27]; He et al. (2026); Wang et al. [74]). For dense models, our law ranks two fully trained loops first among looped settings, since they provide nearly the effective size of two untied copies (§6.2), and in our runs they nearly match compute-optimal models without looping with substantially fewer non-embedding parameters (§6.3).

Preliminaries

Looped transformers. We study the prelude–recurrent–coda architecture (Geiping et al. [28]; Schwethelm et al. [65]), illustrated in Figure 3 and detailed in §B. A prelude $\mathcal{P}$ of $\ell_p$ layers, including the input embedding, maps the input tokens to hidden states. A shared recurrent block $\mathcal{R}$ of $\ell_r$ layers then refines these states over $T$ recurrent iterations, which we call loops, and a coda $\mathcal{C}$ of $\ell_c$ layers, including the output head, turns the result into next-token predictions. For an input token sequence $x$ and loops $t=1,\ldots,T$, the model computes

Here $s_0,s_1,\ldots,s_T\in\mathbb{R}^d$ are the hidden states at each token position: the prelude output $s_0$ is the initial state, and $s_t$ is the state after loop $t$. Each loop concatenates $s_0$ with the previous state into $[s_0;s_{t-1}]$, projects the result back to width $d$ with $W_{\mathrm{inj}}\in\mathbb{R}^{d\times2d}$, and passes it through $\mathcal{R}$ and an RMSNorm to obtain $s_t$. This input injection feeds $s_0$ to every loop, and the RMSNorm at the end of each loop keeps the state passed to the next loop or the coda at a controlled scale, as in Schwethelm et al. [65]. We denote the loop map by $\Phi_\theta$, where $\theta$ collects the recurrent weights of $\mathcal{R}$ and $W_{\mathrm{inj}}$. The coda maps the final state $s_T$ to the next-token distribution $p$, and the cross-entropy loss applies only to $p$. We split the non-embedded parameters as $N=N_{\mathrm{once}}+N_{\mathrm{rec}}$, where $N_{\mathrm{once}}$ counts $\mathcal{P}$ and $\mathcal{C}$, which run once per token, and $N_{\mathrm{rec}}$ counts $\mathcal{R}$ and $W_{\mathrm{inj}}$, which are reused in every loop; Table 2 gives the share $N_{\mathrm{rec}}/N$ for each model size. Truncated BPTT. A loop setting $(T, B)$ with $1 \le B \le T$ runs $T$ loops forward and backpropagates through only the last $B$: the state $s_{T-B}$ is detached (Figure 3a), so the first $T-B$ loops carry no gradient and store no activations. The setting $B=T$ is full BPTT, $B<T$ is truncated BPTT, and $(1, 1)$ is a standard transformer. The recurrent weights $\theta$ and, through the input injection, the prelude receive only the gradient of the last $B$ loops. The coda’s gradient is exact. Evaluation always runs all $T$ loops. A backward loop costs twice the FLOPs of a forward loop (§B), so truncation trades backward compute and activation memory for a biased gradient [68, 76]. §B.2 writes out both gradients and discusses this bias.

Size	$d$	Layers	$N$	$N_{\mathrm{rec}}/N$
60M	768	2/4/2	57.8M	0.510
120M	1024	2/5/2	115.4M	0.564
240M	1280	3/6/3	239.2M	0.507
480M	1664	3/7/4	470.8M	0.506
1B	2176	4/8/4	918.7M	0.505

Baseline scaling law. To relate validation loss to non-embedding parameter count $N$ and training tokens $D$ for a non-looped model, we adopt the coupled scaling law of Videau et al. [71],

Here $E$ is the irreducible loss, $N_0$ and $D_0$ are fixed reference scales (we give token budgets in units of $D_0 = 2^{30}$ tokens), $\alpha,\beta> 0$ are the size and token exponents, and the outer exponent $\kappa$ couples the two terms ($\kappa= 1$ recovers the additive form of Hoffmann et al. [35]); we denote the data coefficient by $H$ to reserve $B$ for backward loops. § 5 extends (2) to looped models by replacing $N$ with an effective parameter count in which $(T,B)$ rescales $N_{\mathrm{rec}}$ by a looping coefficient $g(T,B)$.

A scaling law for forward and backward loops

Should a looped transformer truncate backpropagation? At a fixed model size and training compute, truncation saves backward computation that can be spent on more forward loops or more training data, but it also reduces the benefit of looping. We fit a scaling law that separates forward and backward loop counts to quantify this tradeoff. This section fits and validates the law and interprets it in terms of effective parameters. §5 then uses the law to decide whether to truncate, and §6 uses it to divide compute among loops, data, and model size.

Fitting and validating the scaling law

Experimental setup. We follow the architecture and training recipe of Schwethelm et al. [65], using the prelude–recurrent–coda architecture introduced in §3. We train a grid of five model sizes from 58M to 919M non-embedding parameters (Table 2) on the 100B-token FineWeb-Edu corpus [56]. We use MuonH for transformer-block and input-injection matrices and AdamW for embedding, output, and normalization parameters [40, 75]. We report validation cross-entropy in nats per token on deduplicated held-out shards; §B.3 gives the full training and evaluation setup, including the learning-rate and batch-size rules.

$d$: width; layers: $\ell_p/\ell_r/\ell_c$; $N$: non-embedding parameters (§B.3).

Errors in $\times10^{-3}$ nats/token over 230 configurations ($R^2 = 0.9992$; §C).

Our 592 training runs include a scaling suite of 347 distinct configurations spanning 1.1–1,291 tokens per parameter and fifteen $(T, B)$ combinations with $T \in\{1, 2, 4, 8, 16\}$, keeping each model’s parameter count fixed (Prairie et al. [59]). Figure 4 shows the training curves of three of these settings: as compute grows, larger models take over the frontier of run ends, as in Hoffmann et al. [35]. We fit the law on 230 configurations and reserve 117 for separate tests. For configurations with multiple accepted runs, we use their mean validation loss. Each configuration receives the same weight in the fit, regardless of its number of runs. Other experiments cover compute allocation, comparisons with untied models, learning-rate sweeps, and batch-size sweeps; Appendices § 11.3, § 12.3, § 14 and § 15 give the full experimental design and fitting details.

Effective-parameter scaling law. Only the recurrent block runs in every loop, so looping can enlarge only this block. We therefore write the loss as the coupled law of Videau et al. [71] with an effective parameter count. In it, the recurrent block, $N_{\mathrm{rec}}$, counts several times, and the prelude and coda, $N_{\mathrm{once}}$, count once (§3):

Here $g(T,B)$ is the looping coefficient: it multiplies the recurrent block’s parameter count, while the prelude and coda count once. With full BPTT, the looping coefficient $g(T,T)=\exp[c\cdot(1-T^{-(p+q)})]$ grows with $T$; truncation multiplies it by $g(T,B)/g(T,T)\le1$.

Design of the looping coefficient. We choose $g(T,B)$ to satisfy four modeling requirements:

1. Normalization $g(T,B)\ge0$, $g(1,1)=1$ The looping coefficient is nonnegative; a single loop recovers the non-looped model.

2. Monotonicity $\partial_T g>0$, $\partial_B g>0$ Increasing either loop count increases the looping coefficient.

3. Zero-backward limit $g(T,0)=0$ With no trained loop, the recurrent block adds no effective size.

4. Saturation $\sup_{1\le B\le T} g(T,B)<\infty$ Finite parameters store finite information, motivating a bounded gain.

The finite-ceiling assumption is supported by prior observations of saturation: Chen et al. [14] model a bounded effective-parameter gain from looping, while Parcae finds that adding test-time loops to a trained model approaches a loss floor ([59]). There is also an information-storage motivation. At a fixed parameter count and weight precision, the weights can store only a finite amount of information; additional loops provide more computation over those weights without increasing their storage capacity. We therefore treat saturation as an empirical modeling assumption and learn its ceiling from the data.

These requirements motivate the form in (3). Nonnegativity motivates an exponential parameterization, ensuring $g(T, B) > 0$ for $T, B > 0$. With $c,p,q > 0$, increasing either loop count decreases $T^{-p}$, $B^{-q}$, so $g(T, B)$ grows toward the finite ceiling $\exp(c)$. The shared coefficient $c$ enforces $g(1, 1) = 1$, and continuous extension as $B \to0^+$ gives $g(T, 0) = 0$. At fixed model size and token budget, $p$ governs the loss reduction from extra forward loops at fixed $B$, while $q$ governs the loss reduction from extra backward loops at fixed $T$. These three loop coefficients, $c,p,q$, together with the six coefficients $E,A,H,\alpha,\beta,\kappa$ of the coupled size–data law (2), give nine fitted coefficients.

Empirical looping coefficients. Figure 5 motivates coupling $T$ and $B$: the empirical looping coefficient has a smaller fitted log–log slope against $r = B/T$ at larger $T$. Figure 6 compares the shared looping-coefficient surface with empirical looping coefficients. For each looped run with loss $\ell$, we first use shape-preserving piecewise cubic Hermite interpolation (PCHIP) ([24]) to interpolate the non-looped measurements at the same target token budget to obtain the effective size $\hat{N}_{\mathrm{eff}}$ matching $\ell$. We then use the effective-size relation in (3): subtracting $N_{\mathrm{once}}$ and dividing by $N_{\mathrm{rec}}$ gives the empirical looping coefficient $\hat{g}$ (§C.2). Vertical segments join each empirical median to the fitted coefficient at the same loop setting. This agreement shows that $g(T, B)$ with just three parameters $c,p,q$ captures the observed loop dependence; Table 3 quantifies the law’s in-sample fit and leave-one-out (LOO) prediction errors.

Evaluation	Fits	RMSE	MAE
In-sample	1	6.14	4.91
One configuration out	230	6.40	5.11
One model size $N$ out	7	7.48	5.89
One token budget $D$ out	11	6.42	5.12
One $(T,B)$ setting out	15	8.67	6.74

Fitting procedure. We fit all nine parameters jointly by minimizing the sum of squared differences between predicted and measured validation losses, giving each configuration equal weight. The reference scales $N_0 = 2.4 \times10^8$ parameters and $D_0 = 2^{30}$ tokens are held fixed. Predictions use actual token counts $D$; budget holdouts group target token budgets. To reduce sensitivity to the initial guess, we fit the law 64 times from different random starting values and retain the solution with the smallest fitting error. All fits converge. Table 4 gives the parameters fitted on the 230 configurations used for the validation below. Further fitting details are in §C.

$E$	$A$	$H$	$\alpha$	$\beta$	$\kappa$	$c$	$p$	$q$
0.29865	6.48695	7.63343	0.30568	0.39281	0.32510	1.11198	0.23677	1.02181

Holdout validation. To test whether the law predicts measurements it has not been fitted to, we leave out selected configurations, fit all nine parameters again using the remaining measurements, and compare the predictions with the omitted measurements. For example, to test prediction at an unseen model size, we exclude every configuration at that size and fit the law using the other sizes. We repeat this for each size and for the other exclusions listed in Table 3.

Prediction accuracy. The law tracks the measured loss curves across sizes and token budgets (Figure 7), including the held-out configurations (marked with open circles). Figure 8 gives the complementary view across loop settings at a fixed model size and token budget: varying $T$ with $B = T$, varying $B$ at fixed $T$, and varying $T$ at fixed $B$. Both figures use the law fitted on the 230-configuration fitting set; the curves are evaluated on both fitted and held-out configurations. §G overlays the law on the five-size grid of the scaling suite.

Holding out a model size or token budget increases prediction error only modestly (Table 3), whereas unseen loop settings are harder to predict (§C). Extrapolation back to the non-looped baseline is the weakest loop-count check; §C reports the boundary errors and alternative gains. The table separates fit quality from held-out prediction error. §4.2 examines the remaining discrepancies in effective-parameter units (Figure 9), and § 7.1 checks their effect on compute allocation (Figure 13).

Illustrating the scaling law with effective parameters

How many parameters is a loop setting worth? (3) allows us to decompose the answer into the gain from fully trained loops, $g(T,T)$, and the discount for truncation, $g(T,B)/g(T,T)$. We use the empirical looping coefficients introduced above to compare the law with the runs in Figure 9; their construction is detailed in §C.2.

The looping coefficient saturates for large $T$. With full BPTT, $B=T$, and the gain is $g(T,T) = \exp\left[c \cdot\left(1-T^{-(p+q)}\right)\right]$. The runs in Figure 8a vary both loop counts together at fixed model size and token budget. Each doubling of trained loops adds less than the one before (Figure 8a and Figure 9a), as Chen et al. [14] also find with all loops trained. The power-law gain of prior work ([65]) keeps growing, whereas ours approaches $e^c$. Our runs support the bounded form over the power law, predicting held-out forward-loop counts more accurately (§C.4).

Doubling $B$ outweighs doubling $T$ alone. At fixed $T$, truncation reduces $B$ and hence $g(T,B)$, consistent with the smaller effective sizes in Figure 9b and higher losses in Figure 8b. Figure 9c clarifies the relative roles of $T$ and $B$ by grouping settings with the same $B$. The fitted bars and empirical estimates change relatively little as $T$ increases within a group, but show larger gains as $B$ increases across groups, including comparisons at the same $T$. The fitted exponents quantify this asymmetry: $q > p$ (Table 4) implies $g(T,2B) > g(2T,B)$ for $2B \le T$. From the same initial setting at fixed model size, doubling $B$ thus adds more effective parameters than doubling $T$ alone; the small loss gains from extra forward loops at fixed $B$ support the same conclusion (Figure 8c).

Truncate or not?

Truncation saves backward compute but reduces the effective parameter count (§4.2). Whether it lowers loss at a given compute depends on which quantities are held fixed. We first fix the forward-loop count, as prior scaling studies do, and then let both loop counts vary at equal compute.

Fixed forward loops: truncation can be optimal

Suppose the forward-loop count $T$ is prescribed, and the model size $N$, token count $D$, and backward-loop count $B$ are chosen to minimize loss at a training compute $C$, with the recurrent block keeping a fixed share of the parameters as $N$ varies. Truncation lowers the effective size $N_{\mathrm{eff}} = N_{\mathrm{once}} + g(T,B) \cdot N_{\mathrm{rec}}$, but it also lowers the training cost, which we express through the compute-equivalent size $N_{\mathrm{comp}}$:

We count two FLOPs per parameter and token for a forward pass and four for a backward pass, omitting attention products and the output projection. We also omit the optimizer update as its cost per step depends on the parameter count but not on $T$ or $B$. Under this count, the prelude and coda run one forward and one backward pass, costing $6N_{\mathrm{once}}$ FLOPs per token, and the recurrent block runs $T$ forward and $B$ backward passes, costing $(2T + 4B) \cdot N_{\mathrm{rec}}$. Their sum is $6N_{\mathrm{comp}}$. A looped model thus corresponds to two different non-looped sizes: trained on the same tokens, it reaches the loss of a non-looped model with $N_{\mathrm{eff}}$ parameters, and it costs the compute of a non-looped model with $N_{\mathrm{comp}}$ parameters. Their ratio $\chi= N_{\mathrm{comp}}/N_{\mathrm{eff}}$ is the compute paid per effective parameter. Therefore, our optimization at fixed compute reduces to choosing the backward-loop count that minimizes this ratio (see Lemma D.1 in §D.4 for a rigorous derivation):

Since $\chi$ does not involve $C$, neither does $B^*$. The stationary condition (D.7) gives continuous candidates for $B^*$; integer choices are evaluated directly using $\chi$ (Table 5).

$T$	$B^*$	$\chi(T,B^*)$	$\chi(T,T)$	$N^*$	$D^*$
1	1	1.000	1.000	1	1
2	2	1.031	1.031	1	1
4	2	1.204	1.430	1.27	1.08
8	2	1.580	2.376	1.51	1.20
16	2	2.343	4.339	1.70	1.31

Truncation reduces effective size, but frees compute for a larger model and more tokens. At $T = 8$, the law selects $B^* = 2$ and predicts a 34% compute saving at equal optimized loss (Table 5).

Separately fitted laws at $T=4,8$ also favor $B=2$ over $B=T$: at $T=8$, their compute-optimal frontiers predict a 33–35% saving within the range of the runs (Figure 10; fitting details in §D.4).

This agrees with the fixed-compute truncation benefit reported by [65]. Parcae uses truncated BPTT but leaves the compute-optimal choice of forward and backward loop counts to future work ([59], Appendix I). The benefit is conditional on prescribing $T$. We next ask whether it survives when both loop counts can be chosen.

Free forward-loop count: full BPTT is optimal

When $T$ can also be chosen, we optimize the split between forward and backward loops at equal compute. Fix the model architecture and token count $D$. In our FLOP accounting, each forward pass through the recurrent block costs one unit and each backward pass costs two (§D.1). The recurrent block thus costs $T+2B$ units per token; all other model components have the same cost across loop settings. Holding $T+2B$ fixed therefore holds total compute fixed: for example, $(T,B)=(4,4)$ and $(8,2)$ both cost 12 recurrent units per token.

Fix such a loop-compute count $S=T+2B$. With the architecture and $D$ fixed, the law (3) depends on the loop setting only through $N_{\mathrm{eff}}=N_{\mathrm{once}}+g(T,B)\cdot N_{\mathrm{rec}}$, and the loss decreases as $N_{\mathrm{eff}}$ grows. Since $N_{\mathrm{once}}$ and $N_{\mathrm{rec}}>0$ are fixed, minimizing the loss at this compute is equivalent to maximizing the looping coefficient $g(T,B)$, which we relax to continuous loop counts:

The optimal split uses full BPTT. Recall that $g(T,B)=\exp[c\cdot(1-T^{-p}\cdot B^{-q})]$ with $c>0$. Thus (5) is equivalent to minimizing $T^{-p}\cdot B^{-q}$ under the same affine constraints. This is a standard convex optimization problem: whenever $q>2p$, its unique solution is $T=B=S/3$ for every $S\ge3$, giving full BPTT (Lemma D.2 in §D.4.2). The fitted exponents in Table 4, $p\simeq0.237$ and $q\simeq1.022$, satisfy $q>2p\simeq0.474$ by a wide margin. Therefore, the law predicts that full BPTT is optimal at every compute if both $T$ and $B$ can be chosen continuously.¹

Separately fitted laws confirm the benefit of full BPTT. The compute-optimal frontiers of the separately fitted laws predict that $(4, 4)$ reaches the same loss with less compute than $(8, 2)$ where both optimal allocations lie within the ranges of their runs (Figure 11). Matched-compute runs agree: at $D=16D_{0}$, $(4, 4)$ achieves lower loss than $(8, 2)$ at every size (Table 6).

Size	$C$ ($10^{19}$)	$L_{8,2}$	$L_{4,4}$	$\Delta L$ ($10^{-2}$)
60M	2.41	2.5211	2.4933	2.78
120M	4.59	2.3843	2.3696	1.47
240M	8.28	2.2778	2.2642	1.36
480M	15.29	2.1845	2.1738	1.07
1B	28.38	2.1144	2.1057	0.87

Finding 2 Truncation helps at a prescribed forward-loop count; with both forward and backward loop counts tunable, full BPTT is optimal. When $T$ is prescribed, the law favors truncation, and separately fitted laws agree. When both loop counts can be chosen, full BPTT is optimal at every size and compute.

Allocating compute among loops, data, and model size

Given full BPTT, it remains to divide training compute among loops, tokens, and model size. Following Hoffmann et al. [35], we summarize each allocation by how its optimum scales with compute, $X^{*} \propto C^{\alpha}$. We first fix the model size (§6.1) and then let it vary (§ 7.2 and 6.3).

Fixed model size: loops versus data

Suppose a model must fit within a device’s memory budget, so adding parameters is not an option. With more training compute available, should we reuse those parameters through more loops per token, or train on more tokens? At fixed size, more loops raise effective size but leave fewer training tokens within the same compute budget; our law balances these two.

Asymptotic loop–token allocation under full BPTT. The allocation power laws below describe the large-compute limit of our fitted loss law, not a separate fit to measured optima. Fix the architecture and parameter counts ($N_{\mathrm{rec}} > 0$); setting $B = T$ in (3) gives the full-BPTT loss

Because $\kappa> 0$, minimizing (6) is equivalent to minimizing its inner sum, $J_N = (L_N - E)^{1/\kappa}$. As $T \to\infty$, the effective size approaches $N_\infty= N_{\mathrm{once}} + e^c \cdot N_{\mathrm{rec}}$. Expanding around this limit gives

where $J_\infty= A \cdot(N_\infty/N_0)^{-\alpha}$ and $K_N = \alpha\cdot c \cdot(N_{\mathrm{rec}} \cdot e^c/N_\infty) \cdot J_\infty> 0$ are fixed by the architecture and fitted coefficients; the leading loop term has exponent $p + q$ and the data term has exponent $\beta$.

Following the power-law allocation argument of Hoffmann et al. [35], we balance these two terms under a compute constraint. At fixed size, compute per token grows linearly with $T$ at large depth ($C \propto T \cdot D$). For continuous $T$, $D$, minimizing the leading terms in (7) gives the asymptotic scaling

These powers need not describe finite-compute optima, where overheads and integer loop counts remain consequential. §D.4.3 derives this limit from the exact stationary condition, including the compute overhead of the non-recurrent layers.

Loops grow slowly; most added compute goes to data. Substituting the fitted $p+q \simeq1.259$ and $\beta\simeq0.393$ into (8) gives, with rounded exponents,

A tenfold increase in compute therefore raises the optimal loop count by about 1.7 times and the token count by about 5.8 times. These exponents are asymptotic: because loop counts are integers, the best setting changes in steps. Between steps, added compute goes to tokens; at each step, the model switches to a deeper setting with fewer tokens. The runs show the same trend: at each size, the loop count minimizing loss at equal compute grows with compute as predicted (Figure 12a–Figure 12c).

Comparison with prior scaling laws. Parcae reports empirical exponents of approximately 0.40 for mean recurrence and 0.78 for training tokens [59 Section 5.2]. (9) agrees that tokens grow faster than loops, but predicts slower loop growth in the asymptotic regime. Parcae samples recurrence and uses truncated BPTT, so its reported optimum is conditional on that training recipe. Our full-BPTT result suggests that truncation may be suboptimal when both loop counts are free under our law and compute model; whether this also holds for Parcae requires a direct comparison. The law of Schwethelm et al. [65] instead lets the looping coefficient grow unbounded as $T^\rho$. Our gain saturates, and $p+q$ governs the approach to that ceiling, giving a different asymptotic allocation between loops and tokens.

Finding 3 At fixed model size, optimal loops and tokens both grow with compute; tokens grow faster. Under the fitted law with full BPTT, the compute-optimal loop count grows asymptotically as $C^{0.24}$ and the token count as $C^{0.76}$. The runs at every size agree.

Free model size: the law favors no looping

When model size is free, the compute spent repeating a block can instead train more unique parameters. Start with a $T$-loop model using full BPTT, for integer $T \ge2$, and replace its shared recurrent block with $T$ untied copies. The resulting non-looped model has the same compute $C$ and token budget $D$ under (4), but greater effective size:

The left side is the effective size of the looped model; the right side is that of the model with $T$ untied copies. The inequality follows from the fitted $g(T,T) < T$ (Figure 14; §D.4.4). The non-looped model therefore has greater effective size and lower predicted loss at every token budget. Figure 14 also shows that $g(2, 2)$ is close to 2; we examine this two-loop case in §6.3.

The best loop setting does not change with compute. With parameter proportions fixed and positive, continuous $N$, $D$ unrestricted, every loop setting shares the compute exponents of the model without looping. Optimizing the loop setting as well gives

The exponents are rounded predictions under the leading-FLOP count (§D.4.4). Under these assumptions, the ranking of loop settings is the same at every compute.

No looping is optimal, and two fully trained loops come closest. Among the fifteen settings, the law favors the model without looping at fixed compute (Figure 2), consistent with Schwethelm et al. [65]. The best looped setting is $(2, 2)$ (Table 5): $g(2, 2) \approx2$, so two shared loops provide nearly the effective size of two untied copies. Because $N_{\mathrm{eff}} > N$ for $(2, 2)$, the law predicts that it comes close to the model without looping with fewer trainable parameters. §6.3 tests this.

Finding 4 With free model size, no looping is optimal. At every compute, with model size and token count free, the law favors the model without looping (Figure 2).

Two fully trained loops are competitive with fewer parameters

Near parity at equal compute. At each of three training budgets, we train a model with two fully trained loops, $(2, 2)$, and a model without looping, each at the size and token count that minimize its loss for that budget, with three seeds per model (§E). At every budget, the looped model stays within 0.008 nats of the model without looping while having 27–32% fewer non-embedding parameters (Table 11). The gap shrinks as the budget grows, and at $2.1 \times10^{19}$ FLOPs$²$ it is smaller than its standard error across seeds. Both models sit near the flat minima of their IsoFLOP profiles, the looped one at a smaller size (Figure 12d).

Compute (FLOPs)	(1, 1)		(2, 2)
	Parameters	Loss	Parameters	Loss	Gap
$1.6 \times10^{18}$	115M	2.7098	79M	2.7177	$(7.9 \pm1.9) \times10^{-3}$
$6.0 \times10^{18}$	239M	2.5215	162M	2.5253	$(3.8 \pm1.8) \times10^{-3}$
$2.1 \times10^{19}$	471M	2.3663	344M	2.3677	$(1.4 \pm1.7) \times10^{-3}$
$2.2 \times10^{19}$ (+8%)	471M	2.3663	344M	2.3590	$(-7.3 \pm1.8) \times10^{-3}$

Ahead with a little more compute. Because the gap is so small, a little more compute lets the looped model pull ahead. We take the pair at $2.1 \times10^{19}$ FLOPs and train its $(2, 2)$ model on 8% more tokens, and so with 8% more compute. Its loss then falls 0.007 nats below that of the model without looping, well beyond the spread across seeds, while it still has 27% fewer non-embedding parameters (Figure 15; Table 11). Appendix § 14 also compares the looped model with an untied model, which gives each loop its own block, at a fixed number of training tokens.

Finding 5 Two fully trained loops nearly match the model without looping with 27–32% fewer parameters. In the runs, the compute-optimal $(2, 2)$ model stays within 0.008 nats of the compute-optimal model without looping at equal training compute, with 27–32% fewer non-embedding parameters; with 8% more compute, its loss falls below.

Conclusion

Truncated BPTT saves compute, but whether it lowers the loss depends on which quantities are held fixed. We fit a scaling law that expresses every loop setting as effective parameters, with a gain that saturates and shrinks under truncation (§ 5). Under this law, truncation helps only when the forward-loop count is prescribed. When both loop counts are chosen, full BPTT is optimal at every model size and compute we analyze (§5).

The law also determines how to allocate compute. At fixed model size, most additional compute should go to data, and the number of fully trained loops grows only as about $C^{0.24}$ (§ 7.1). When model size is free, the law favors no looping, but two fully trained loops come close: with 27–32% fewer non-embedding parameters they nearly match the compute-optimal non-looped model at equal compute, and with 27% fewer parameters they achieve lower loss at 8% more compute (§6.3).

Limitations. Our main limitation is the focus on dense models: we do not study mixture-of-experts (MoE) architectures, so whether our conclusions on truncation and compute allocation extend to looped MoE models remains open. Second, we measure validation loss only and do not establish whether the observed gains translate to downstream task performance or reasoning ability. Third, our experiments are capped at $T \le16$, so the gain ceiling and later switches to more loops remain extrapolations. Related scaling sweeps reach eight loops in Schwethelm et al. [65] and mean training loop counts of twelve in Parcae (Prairie et al. [59]). Much deeper recurrence can exacerbate training instability, including residual explosion and loss spikes ([59] (Fu et al. 2026)); developing stable training recipes for that regime is beyond our scope. Finally, we fix the ratio of non-embedding parameters in the combined prelude and coda to those in the recurrent block for each model size. Varying this ratio to study how parameter allocation affects the gains from looping and the compute-optimal loop counts is a direction for future work.

Acknowledgments

We thank Yuwei Niu, Yi Ge, and Tianyu Fu for valuable feedback and helpful discussions.

Extended related work

Training through recurrent depth. Truncated BPTT limits gradient propagation to a subset of recurrent steps, saving activation memory and backward compute at the cost of biased gradients [68, 76]. Looped language models often supervise the final output and backpropagate through only the last few loops. Huginn and retrofitted models use at most eight backward steps [28, 51], Parcae and Iso-Depth backpropagate through half of the loops [59, 65], URM runs the first loops without gradients [26], and looped MoE models supervise within recurrent segments (He et al., 2026). The same truncation trains looped models converted from pretrained language models [54], recurrent-depth models for robot control and dynamics forecasting [44, 70], and looped refinement in diffusion language models [67]. Recursive reasoning models use even shorter backward horizons. Hierarchical recurrent models backpropagate through only the final steps [33, 72, 73], and TRM and its successors run all but the last recursion without gradients [2, 5, 23, 32, 39, 53, 62, 66]. For algorithmic extrapolation, recurrent networks and looped transformers run a random number of iterations without gradients before the supervised ones, or compute the loss over a window of the last loops [9, 43, 80]. Implicit models replace backpropagation through the solver with implicit differentiation [7] or with one-step, Jacobian-free, and phantom gradients that differentiate only a few steps at the fixed point (Geng et al., 2021a,b; Wu Fung et al., 2022). Such approximations train implicit models for optical flow, diffusion, and language modeling [6, 8, 21, 57, 64], and truncated BPTT in looped transformers can be read as a truncated Neumann series of the fixed-point gradient [37, 52]. Few studies measure what truncation costs. Fein-Ashley and Rashidinejad [21] find that the one-step gradient nearly matches the perplexity of the exact implicit gradient at a fraction of its memory, Huang et al. [36] find that truncation lowers or raises Sudoku accuracy depending on the latent structure, Salvatori et al. [62] find that the required gradient window grows with the propagation distance of the graph, and Ren and Liu [61] argue that the one-step gradient delays the stabilization of hierarchical reasoning models. These studies compare training recipes at a fixed forward depth; we vary the backward depth independently of the forward depth, weighing the compute that truncation saves against the loop gain it costs.

Scaling laws for recurrence. Size–data scaling laws quantify how parameters and training tokens jointly determine loss [35, 41, 71], and their compute-optimal predictions depend on how the laws are fitted and compute is counted [11, 16, 55, 58] (Li et al. 2025a). Extensions account for architectural choices such as MoE sparsity [1, 17, 42]. For looped models, some laws fit each architecture separately [15, 46, 74], while others model the loop count explicitly Table 1 through additive loss terms [82] or effective parameters with linear, power-law, or bounded gains [14, 59, 65]. Among these scaling-law studies of looped transformers, Iso-Depth [65] and Parcae [59] are particularly related to our work. Iso-Depth fits separate laws for full and half-length backpropagation and finds that truncation weakens the loop gain but lowers the loss at a fixed forward-loop count by freeing compute. At fixed model size, Parcae finds that loops and tokens should grow together with compute, but ties the backward loops to half the forward loops and leaves their joint choice open. Our law instead treats the backward-loop count as a separate variable, combining coupled size–data scaling [71] with bounded loop gains [14]. It recovers the truncation benefit at a prescribed forward-loop count but favors full BPTT when both counts are free (§5). In looped MoE models, two loops are the common choice (He et al., 2026): SMELT finds them better than three or four under matched budgets [74], and Loopie finds the marginal benefit of recurrence largest at two [27]. For dense models, our law ranks two fully trained loops first among looped settings, since they provide nearly the effective size of two untied copies of the block while further loops add less (§ 7.2). In our runs, two-loop models nearly match compute-optimal models without looping at equal compute while having substantially fewer non-embedding parameters (§6.3).

Model architecture and experimental setup

Model architecture

Figure 3 draws the model of §3 for the loop setting $(4, 2)$. In a training step (Figure 3a), the prelude maps the tokens to $s_0$, every loop reads $s_0$ again through the input injection, and the state $s_{T-B}$ entering the last $B$ loops is treated as a constant. The gradient therefore reaches the coda, the last $B$ loops and, through the injection, the prelude; the first $T-B$ loops run forward only, and evaluation runs all $T$ loops. Each loop (Figure 3b) applies $W_{\mathrm{inj}}$ to $[s_0\,;\,s_{t-1}]$, the $\ell_r$ pre-norm transformer layers of $\mathcal{R}$ and an RMSNorm, with the same weights in every loop. At 2 FLOPs per parameter and token for a forward pass and 4 for a backward pass (Figure 3c), the prelude and coda cost 6, each forward loop 2 and each backward loop 4 more, giving $6N_{\mathrm{once}} + (2T + 4B)N_{\mathrm{rec}} = 6N_{\mathrm{comp}}$ FLOPs per token ((4)), before attention and output ((D.1)); the optimizer adds a cost per step ((D.3)).

Full and truncated backpropagation through the loops

This appendix writes out the gradients that full and truncated BPTT compute for the recurrence of ((1)), and the fixed-point argument that motivates truncation (§3).

The truncated gradient. A loop setting $(T, B)$ executes $T$ forward loops and backpropagates through only the last $B$: the state $s_{T-B}$ is detached from the computation graph, so the first $T-B$ loops run without gradients. To make this precise, we denote for loop $t$ of ((1)) the Jacobian with respect to the incoming state and the derivative with respect to $\theta$ with the incoming state fixed by

The loss $\mathcal{L}$ depends on $\theta$ only through $s_T$, so $\nabla_\theta\mathcal{L} = (\partial\mathcal{L}/\partial s_T)\cdot(\mathrm{d}s_T/\mathrm{d}\theta)$, and it suffices to compare $\mathrm{d}s_T/\mathrm{d}\theta$ under full and truncated BPTT. The chain rule sums one contribution per loop, and detaching $s_{T-B}$ keeps only the last $B$, giving the truncated gradient $g_B$:

where the empty product for $t=T$ is the identity. Truncation keeps the $B$ contributions with the fewest Jacobian products and drops the rest. The prelude gradient is truncated in the same way.

Truncation from a fixed point view. Whether the dropped contributions matter depends on the loop dynamics, and the deep-equilibrium view [7, 72] gives an idealized answer. Suppose the loops approach a fixed point $s^* = \Phi_\theta(s^*;s_0)$, so that $\mathcal{J}_t \approx\mathcal{J}^*$ and $\partial_\theta\Phi_t \approx\partial_\theta\Phi^*$ for the loops that carry gradient. If the spectral radius of $\mathcal{J}^*$ is below one, the implicit function theorem gives the gradient of the fixed point, and a Neumann series expands it into the same per-loop contributions:

Truncating to $B$ loops thus approximates the fixed-point gradient by the first $B$ terms of its Neumann series, whereas full BPTT uses the first $T$. The extreme case $B=1$ keeps only the leading term $\partial_\theta\Phi^\star$; this is the one-step gradient used to train the Hierarchical Reasoning Model Wang et al. [72]. When $\mathcal{J}^\star$ is contractive, the dropped terms shrink geometrically, which makes truncation a plausible approximation at a fraction of the backward compute. This idealized argument, however, does not settle whether truncation pays off in practice, for two reasons. First, it assumes that the loops reach a fixed point, whereas looped language models are usually trained with only a few loops [27, 74] (He et al. 2026). With so few iterations, the states need not come close to a fixed point, so the approximation $\mathcal{J}_t \approx\mathcal{J}^\star$ behind (B.2) can fail and the contributions dropped in (B.1) need not be small. Second, even at a fixed point, the dropped terms are nonzero for $B<T$, so $g_B$ is a biased estimate of the full gradient. The argument only bounds the error of a single gradient, whereas this bias enters every update, and how it accumulates over training and shapes the final model is not known. Prior measurements compare recipes at a fixed forward-loop count (§2) and so cannot separate the compute that truncation saves from the loop gain it costs. We therefore measure the net effect directly: the scaling law of § 5 asks whether the saved compute outweighs this bias in the loss that training actually reaches.

Experimental setup

Models. We take the architecture and the training recipe of Schwethelm et al. [65], the reference recipe below, which builds on the prelude–recurrent–coda layout of Huginn Geiping et al. [28] (§3). Its blocks use pre-norm attention with query–key normalization and rotary position embeddings, head dimension 128, a squared-ReLU feed-forward layer of width $4d$, no biases, and untied input and output embeddings. We change only what is held fixed. Schwethelm et al. [65] hold the executed depth fixed, so that more loops mean fewer unique layers; we hold the unique parameters fixed, as Prairie et al. [59] do, so that a loop setting changes the compute and not the model. The main grid spans five model sizes, from 58M to 919M non-embedding parameters (Table 2); width and depth grow together, and the recurrent block with the input injection holds 51% of the non-embedding parameters at four sizes and 56% at 120M. Short size names round $N$; including the untied input and output embeddings, total parameter counts run from about 107M to about 1.06B. We quote budgets per parameter as $D/N$. The scaling suite supplements this grid with two compute-matched pairs, adding the 79M and 162M models at one configuration each (§E).

Loop settings and budgets. The scaling suite spans 1.1–1,291 training tokens per parameter and uses fifteen loop settings. Ten core settings, $(1,1)$; $(2,1)$ and $(2,2)$; $(4,1)$, $(4,2)$, $(4,3)$ and $(4,4)$; and $(8,2)$, $(8,4)$ and $(8,8)$, cover $T \in\{1,2,4,8\}$ forward loops, full and truncated BPTT, and ratios $B/T$ from $\frac{1}{4}$ to 1. Five more settings extend them: $(8,1)$, and $T=16$ with $B=1,4,8$ and 16 at three sizes and short budgets (Appendices § 12.5 and § 16). The part of the training compute that depends on the loops grows with $T+2B$, since a backward loop costs twice a forward loop (§D.1); $(4,1)$ thus costs as much per token as $(2,2)$, and $(8,2)$ as much as $(4,4)$. Token budgets $D$ are given in units of $D_0=2^{30}\approx1.07\times10^9$ tokens. We group runs by their target token budget; actual token counts may differ slightly because of batch rounding. Predictions use the actual token count $D$. An experimental configuration is a choice of $(N,D,T,B)$; a run is one training execution of that configuration. Each configuration contributes one accepted validation loss to the analysis.

Data and evaluation. We train on the 100B-token sample of FineWeb-Edu [56], the corpus of the looped scaling study of Lee et al. [46], in a pre-shuffled release,³ tokenized with the Llama 2 tokenizer [69] of the reference recipe [65] into sequences of 2,048 tokens. We report cross-entropy in nats per token on the last two shards of the release, which no run trains on, after removing the documents whose exact text also appears in the training data (the same web page often occurs in several crawls).

Optimization. We use the optimizers of the reference recipe [65]. MuonH, a variant of Muon that keeps each weight matrix at a fixed Frobenius norm [40, 75], trains the hidden weight matrices (attention, feed-forward and input injection), and AdamW [49] trains the embedding, output and normalization parameters, both without weight decay. The MuonH update adds 0.06–5.7% to the compute of the forward and backward passes; the analysis leaves it out, and the measured compute we report includes it (§D.1). The peak learning rates follow the reference’s transfer rule [60, 65, 78], a power law in the batch and training tokens with the token exponent re-estimated for our budgets, and the global batch grows as $64\sqrt{D/D_0}$ sequences, following the square-root scaling of the loss-optimal batch in prior work [10, 22] (Li et al. 2025b) (§F). The learning rate warms up over the first 2% of steps and decays with a cosine to 10% of its peak; gradients are not clipped.

Runs and compute. The results in this paper come from 592 training runs with $2.7 \times10^{22}$ FLOPs of training compute, counted as in §D.1 with the optimizer update. The scaling suite contains 397 runs behind 347 configurations: 385 runs in the five-size grid (§G) and 12 runs in two compute-matched pairs (§E). Each configuration’s loss is the mean of its accepted runs. The remaining 195 distinct runs support the other experiments. These experiments comprise four more runs at 480M in Figure 13, nine further runs of the compute-optimal pairs and the 36 of the untied comparison (§E), and the 66 learning-rate and 124 batch-size runs of Figure 17 and Figure 18; 44 runs belong to two of these groups and are counted once. Counting cancelled runs, runs replaced by reruns, pilot runs and earlier calibration sweeps that the paper does not report, the study trained more than 1,100 runs with about $4.8 \times10^{22}$ FLOPs.

The law in detail

This appendix details the measured patterns behind the law, its fit and validation, the bounded gain, and the runs at sixteen loops.

What the runs show before any law is fitted

The measured losses separate the effects of forward and backward loops without a fitted law (§G). Forward loops that receive no gradient add almost nothing at first: at fixed $B$, doubling $T$ from $B$ to $2B$ changes the loss by less than 0.007 nats in all 24 size–budget combinations trained in all ten core settings, while doubling it again, from $2B$ to $4B$, lowers the loss in all 24. Spending the same compute per token on backward loops is better: (2, 2) beats (4, 1) and (4, 4) beats (8, 2) in all 24, by 0.023 nats on average for (2, 2) against (4, 1). Backward loops in turn have diminishing returns: with full BPTT, every doubling of $T = B$, from 1 to 8, lowers the loss, by less each time, in all 24 combinations where all four settings were trained.

Empirical looping coefficient

Empirical looping coefficient

The looping coefficient $g(T, B)$ in (3) rescales the recurrent block’s parameter count. To estimate it from the runs, we first find the non-looped model size that matches a looped run’s measured loss at the same token budget. This defines the empirical effective size, which we then convert to a looping coefficient using the run’s parameter counts.

Reference interpolation. Let $d = D_{\mathrm{target}}/D_0$ denote a target budget level, where $D_{\mathrm{target}}$ is the target token count and $D_0 = 2^{30}$ tokens. At each $d$, we collect the accepted non-looped $(T, B) = (1, 1)$ configurations and order their actual parameter counts and measured losses as $(n_j, \ell_j)_{j=1}^{m_d}$, where $n_1 < \cdots< n_{m_d}$. Each configuration contributes its accepted loss once, following the same aggregation rules as the fit. We require at least three distinct sizes and strictly decreasing losses, $\ell_1 > \cdots> \ell_{m_d}$, as holds for the eligible reference groups. We interpolate log size as a function of loss:

Here PCHIP denotes shape-preserving piecewise cubic Hermite interpolation (Fritsch and Butland, [24]), with reference points ordered by increasing loss. The interpolant in (C.1) passes through each measured point, $\phi_d(\ell_j) = \log n_j$, and preserves monotonicity between points. This loss-to-size conversion uses only the non-looped measurements, independent of the fitted coefficients of (3).

Coefficient of a looped configuration. For a looped configuration $i$, let $L_i$ be its measured loss, $d_i$ its target budget, and $N_{\mathrm{once},i}, N_{\mathrm{rec},i} > 0$ its once-run and recurrent counts; when its reference group is eligible and $L_i \in[\ell_{m_{d_i}},\ell_1]$, define

In (C.2), we subtract the once-run parameters and divide by the recurrent block’s size, as in (3). Each empirical coefficient depends on the run’s size and token budget; $g(T, B)$ is the law’s shared prediction. We report an estimate only when the reference group is eligible and the measured loss lies within its observed range, without extrapolation. Budget matching uses the target level $d$, keeping batch-rounded budgets in the same group. Actual consumed tokens differ from the target token budget by less than 0.2% in these runs; we leave these differences uncorrected.

For example, the 239.24M non-embedding-parameter run at $T = B = 2$ and $D = 4D_0$ has loss 2.4272 nats per token. The reference interpolation gives $\widehat{N}_{\mathrm{eff}} = 345.20\mathrm{M}$, hence $\widehat{g} = (345.20 - 117.98)/121.26 \simeq1.874$, using its own once-run and recurrent counts in millions of parameters.

Summaries across configurations. For each setting $(T, B)$, let $\mathcal{I}_{T,B}$ be the set of eligible size–budget configurations. Figure 6 shows the median of $\{\widehat{g}_i : i \in\mathcal{I}_{T,B}\}$, with the 25th and 75th percentiles as the interquartile range. Configurations have equal weight; for sorted estimates $x_{(1)}, \ldots, x_{(n)}$, a percentile at probability $u$ linearly interpolates adjacent order statistics at rank $1 + (n - 1)u$. Figure 9 uses the same loss-matched estimates, with total-size ratios shown where indicated. The surface figure includes 163 eligible configurations with $T \le8$: 121 from the fitting set and 42 from the separately held-out set. Reference configurations may also come from either set, and estimates that use the same references are dependent. These figures describe agreement with the runs rather than independent validation of the law. The interquartile ranges show variation across sizes and budgets, not confidence intervals. §C.3 reports the separate holdout evaluations in loss units.

Figure 5 uses the same estimator on the 150 eligible configurations with $T \in\{2, 4, 8\}$ and $1/4 \le r = B/T \le1$. Markers show medians and bars show minima and maxima. For each $T$, we fit an intercept $a_T$ and slope $s_T$ by ordinary least squares with equal weight per cell:

In (C.3), the fitted slopes are 0.823, 0.446 and 0.173 for $T = 2, 4, 8$, respectively; these lines summarize the empirical trend across the 150 configurations rather than a separate scaling law.

Form and fit

Functional form. The looping coefficient is

The normalization is exact: $g(1,1) = 1$, so $(T, B) = (1, 1)$ gives $N_{\mathrm{eff}} = N$. For $c, p, q > 0$, $g(T, B)$ increases in $T$ and in $B$ toward $e^c$. The fitted ceiling of the looping coefficient is $e^c \simeq3.04$; it is an extrapolation, not an observed asymptotic limit. With full BPTT the looping coefficient is $g(T,T) = \exp[c(1 - T^{-(p+q)})]$; for $B < T$ it is smaller by the factor $\exp[-cT^{-(p+q)}(r^{-q} - 1)]$, $r = B/T$, which approaches one as $T$ grows.

Data split. We split the 347 scaling-suite configurations into 230 for fitting and 117 for testing. The fitting set spans fifteen loop settings and includes 226 configurations from the five-size grid, covering five model sizes at approximately 9, 18, 37, or 74 training tokens per parameter, giving the sizes a common budget range. Of these 226 configurations, 194 use the eleven settings with at most eight loops (a few at the largest budgets were not run) and 32 use $T = 16$. The other four fitting configurations are also used to compare a looped model with a non-looped model at each of two training-compute budgets (§E), adding sizes 79M and 162M to the grid’s five sizes. Each configuration enters once through the mean loss of its accepted runs, aggregating repeated runs and seeds and removing duplicate exports. The other 117 configurations of the scaling suite, mostly at shorter or longer budgets, enter no fit and serve as separate tests. Predictions use actual consumed tokens and exact parameter counts. The fixed normalizations are $N_0 = 2.4 \times10^8$ and $D_0 = 2^{30}$.

Solver. We minimize equally weighted squared residuals in nats per token. All nine coefficients are refitted on every training fold using 64 independent uniform random starts, retaining the lowest-cost converged solution. The trust-region reflective solver uses an analytic Jacobian, Jacobian scaling, at most 4000 evaluations, and tolerances $\mathrm{ftol} = \mathrm{xtol} = \mathrm{gtol} = 10^{-12}$. Seeds are 20261010 plus the deterministic fold index. No full-data coefficients initialize a held-out fold, and no learned preprocessing, weighting or hyperparameter tuning is used.

In optimization coordinates, $E \in[0, 3]$, $\log A, \log H \in[-10, 15]$, $\alpha, \beta, \kappa\in[0.01, 3]$, $p, q \in[0.001, 10]$, and $\log c \in[\log0.001, \log10]$. Initial ranges are $E \in[0.3, 1.2]$, $\log A, \log H \in[0.5, 2.5]$, $\alpha, \beta\in[0.25, 0.65]$, $\kappa\in[0.15, 0.45]$, $p, q \in[0.1, 2]$, and $\log c \in[\log0.2, \log2]$. The power-law gain of §C.4 uses the same ranges for the first six coefficients and $\phi_T, \phi_B \in[-5, 5]$, initialized in $[0.05, 0.8]$. All 264 fits of each law converge with full-rank Jacobians (the largest-size holdout reaches $E = 0$).

Validation and aggregation. Residuals are prediction minus measurement. Point LOO holds out one entire accepted configuration. Size folds hold out all combinations at the selected model size. Budget folds group the target level $D_{\mathrm{target}}/D_0$, retaining batch-rounded versions together. Setting folds hold out an entire $(T, B)$ pair. Each family covers all 230 configurations. Point LOO trains on 229 configurations and tests on the remaining one, repeating this for all 230. Table 7 reports pooled errors, which weight held-out configurations equally. Every fold converges and none is dropped. §G overlays the law on the 226 fitted grid configurations; the four fitted configurations from the compute-matched pairs are not shown there.

Evaluation	Fits	RMSE	MAE	Bias	Max. error
In-sample	1	$6.14 \times10^{-3}$	$4.91 \times10^{-3}$	$-3.73 \times10^{-16}$	$2.04 \times10^{-2}$
One configuration out	230	$6.40 \times10^{-3}$	$5.11 \times10^{-3}$	$-3.30 \times10^{-6}$	$2.10 \times10^{-2}$
One model size $N$ out	7	$7.48 \times10^{-3}$	$5.89 \times10^{-3}$	$-2.16 \times10^{-4}$	$2.58 \times10^{-2}$
One token budget $D$ out	11	$6.42 \times10^{-3}$	$5.12 \times10^{-3}$	$-3.42 \times10^{-5}$	$2.08 \times10^{-2}$
One $(T, B)$ setting out	15	$8.67 \times10^{-3}$	$6.74 \times10^{-3}$	$8.86 \times10^{-4}$	$2.63 \times10^{-2}$

Why a bounded gain

Prior scaling laws for looped models let the looping gain grow as a power of the loop count, $T^\phi$ [65]. We extend this form to $T^{\phi_T}B^{\phi_B}$ in place of $g(T, B)$, keeping the rest of (3) unchanged. This form is the limit of our gain as $c \to\infty$ with $cp \to\phi_T$ and $cq \to\phi_B$; under full BPTT it reduces to $T^\phi$ with $\phi= \phi_T + \phi_B$. We fit both laws on the same configurations, folds and solver. The power-law gain is less accurate in every evaluation in Table 8, with nearly twice the error on a held-out $(T, B)$ setting ($1.65 \times10^{-2}$ vs. $8.67 \times10^{-3}$ nats); it keeps growing where the runs level off, and held out at sixteen loops its error is three times that of the bounded gain (§C.5). Our gain rises from $g(1,1) = 1$ toward $e^c$; with full BPTT and $p + q = 1$ it is the reciprocal exponential gain $\exp[c(1 - 1/T)]$, and free exponents separate the ceiling from the rate of approach.

Evaluation	Bounded gain	Power-law gain
In-sample	$6.14 \times10^{-3}$	$1.31 \times10^{-2}$
One configuration out	$6.40 \times10^{-3}$	$1.36 \times10^{-2}$
One model size $N$ out	$7.48 \times10^{-3}$	$1.40 \times10^{-2}$
One token budget $D$ out	$6.42 \times10^{-3}$	$1.35 \times10^{-2}$
One $(T, B)$ setting out	$8.67 \times10^{-3}$	$1.65 \times10^{-2}$

Setting	$g(T, B)$	$N_{\mathrm{eff}}/N$
$(1, 1)$	1.00000	1.00000
$(2, 1)$	1.18330	1.09294
$(2, 2)$	1.91023	1.46150
$(4, 1)$	1.36498	1.18505
$(4, 2)$	2.04945	1.53209
$(4, 4)$	2.50361	1.76236
$(8, 2)$	2.17552	1.59601
$(8, 4)$	2.57832	1.80024
$(8, 8)$	2.80330	1.91431
$(16, 16)$	2.93895	1.98308

The derivatives distinguish the empirical assumption from its algebraic consequences. At fixed coefficients, with $s = p + q$, the total derivatives below follow the full-BPTT path $B = T$:

For the fitted coefficients, $cs < s + 1$, so the full-BPTT gain is concave for $T \ge1$. Concavity is not automatic for every positive parameter choice. The effective size is an equivalence in validation loss, not in stored knowledge ([82]).

Runs at sixteen loops

The fit contains 32 configurations at $T = 16$, across 120M, 240M and 480M, with backward counts 1, 4, 8 and 16 and target token budgets from $2D_0$ to $8D_0$. Holding out this whole row leaves 198 training configurations and gives RMSE 0.0115 nats, against 0.0340 for the power-law gain of prior work, which keeps growing beyond the loop counts it is fitted on. The harder extrapolation is the row without looping, $T = 1$: RMSE 0.0172 (0.0207 for the power-law gain), the largest forward-row error. The four sixteen-loop configurations of 480M at $2D_0$ lie outside the fitting set.

Training compute and allocation

We first define the training-compute cost, then discuss the fixed-size switch points and their experimental checks. §D.4 gives the allocation derivations for prescribed forward loops, joint loop counts, and fixed or free model size.

Training compute

Training compute counts the FLOPs of the forward and backward passes. We use the usual convention of 2 FLOPs per parameter and token for the forward pass and 4 for the backward pass (Hoffmann et al., [35]; Kaplan et al., [41]). The prelude and coda run forward and backward once per token. Each forward loop runs the recurrent block forward once, and each backward loop adds a backward pass through it, which costs twice as much. The compute of the passes per token is

where $F_{\mathrm{attn}}$ counts the attention products of every executed layer and $F_{\mathrm{out}}$ the output projection.⁴ The attention products of the recurrent block follow the same count, one unit per forward loop and two more per backward loop, so every loop-dependent term grows with $T + 2B$, as in the accounting of truncated BPTT by Prairie et al. [59] and Schwethelm et al. [65]; for a given model, $(2, 2)$ and $(4, 1)$ therefore cost the same per token, as do $(4, 4)$ and $(8, 2)$.

For a model of size $N$ we write this compute per token as $F_N(T,B) = u_N + v_N(T + 2B)$, where $u_N$ collects the prelude, coda and output and $v_N > 0$ one forward pass through the recurrent block with its attention, so that $D$ training tokens cost

At fixed $N$ and $D$, the compute depends on the loop counts only through $T + 2B$. (4) is its leading term, without the attention and output costs.

The optimizer update is negligible. MuonH also spends FLOPs on each update: five Newton–Schulz iterations per weight matrix (Jordan et al., [40]), a cost $F_{\mathrm{opt}}$ per step that depends on the model but not on $T$ or $B$, so that a run of $n_{\mathrm{step}}$ steps costs

Across our runs the update adds 0.06–5.7% to the compute of the passes. It cannot change the split between forward and backward loops, since it does not depend on either; leaving it out moves the fixed-size switch points of § 7.1 by less than 0.5%, and it gives the models with more trainable parameters, the untied model and the model without looping at its compute-optimal size, 0.5–0.75% more compute, or about 0.001 nats in loss (§E). Like prior scaling studies (Prairie et al., [59]; Schwethelm et al., [65]), our analysis therefore omits it and uses (D.2); the measured compute in figures, tables and §B.3 includes it unless stated otherwise.

Switch points

At a fixed size, each setting traces a loss curve in $C$, and the best setting at each compute lies on their lower envelope (Table 10). Full BPTT reaches the envelope almost everywhere: among all fifteen trained settings, four of them with $T = 16$, the best setting at every compute is $(1,1)$, $(2,2)$, $(4,4)$, $(8,8)$ or $(16,16)$, apart from a narrow compute range between $(2,2)$ and $(4,4)$ where $(4,3)$ leads by at most 0.002 nats on our integer menu (§ 7.1). Truncation lowers the compute per token but gives up the gain of the loops it does not train, and at equal compute per token a setting that trains all of its fewer loops is the better buy, as the runs show directly (§C.1). Measured in the tokens per parameter that the model without looping trains on at the same compute, two loops become the best choice at about 14–17, four overtake two at about 312–413, and eight overtake four at about 7,000–9,600 (sixteen pay only beyond 147,000–210,000).

Best setting changes	60M	120M	240M	480M	1B
$(1,1) \rightarrow(2,2)$	16	17	16	15	14
$(2,2) \rightarrow(4,4)$	400	413	374†	345†	312†
$(4,4) \rightarrow(8,8)$	9,622†	9,648†	8,605†	7,833†	7,025†
$(8,8) \rightarrow(16,16)$	210,087‡	204,851‡	182,247‡	164,533‡	146,710‡

The runs at a fixed size

The runs confirm the first switch and the role of the trained loops (Figure 13; Figure 1c previews the 120M slice). Beyond about 30 tokens per parameter, $(2,2)$ ends below the model without looping at every point we measured, by about 0.01 nats at 50 tokens per parameter and by 0.025 at 200. Loops without gradient do not win: $(4,1)$, which costs as much per token as $(2,2)$ but trains only one of its four loops, stays above the model without looping up to 200 tokens per parameter. At 60M, 120M and 240M, where the runs reach several hundred tokens per parameter, $(4,4)$ overtakes $(2,2)$, as the law predicts, but already at about 200 tokens per parameter rather than at the law’s 374–413. At one compute per size far into this range, that of $(4,4)$ at $32D_0$ for 60M and 240M and at $24D_0$ for 120M (1,291, 533 and 335 tokens per parameter of the model without looping), the runs read 2.514, 2.389 and 2.247 for the model without looping, 2.479, 2.358 and 2.223 for $(2,2)$, and 2.460, 2.345 and 2.218 for $(4,4)$: $(4,4)$ is the lowest, 0.019, 0.012 and 0.006 nats below $(2,2)$, while $(4,3)$ at the same compute trails $(4,4)$ by 0.010, 0.008 and 0.006 and $(8,8)$ lies above it at all three sizes. At 480M the same compute is reached at 174 tokens per parameter, before the switch: the model without looping reads 2.141, $(2,2)$ 2.121 and $(4,4)$ 2.124, so $(2,2)$ and $(4,4)$ are level and both stay below the model without looping. The runs with $T = 16$, trained at $2D_0$–$8D_0$ (§C.5), lie well below the compute where eight or sixteen loops pay and stay above $(4,4)$ at equal compute.

Allocation identities and their assumptions

We follow the allocation regimes of § 6.1 to § 7.2: first prescribe $T$, then optimize the split between $T$, $B$, and finally allocate loops, data, and model size. The prescribed-$T$ and free-size results use the leading matrix-FLOP approximation (5.1); the joint-loop and fixed-size calculations use (D.2), with the attention and output costs. All omit the optimizer update (§D.1).

Fixed forward-loop count: optimize size, data, and backward loops

Proof. The proof reduces the looped allocation problem to a non-looped one. With a fixed recurrent fraction, the effective and compute-equivalent parameter counts are both constant multiples of $N$, so the loop setting enters the budget only through the factor $\chi(T,B)$. We first fix $B$ and solve the size–data allocation in closed form. The optimized loss then depends on $B$ only through a positive power of $\chi(T,B)$, multiplied by a factor in $C$ that is common to all backward counts, so comparing backward counts reduces to comparing $\chi$.

Step 1: the price of an effective parameter. Let us first fix the backward loop count $B$. Substituting $N_{\mathrm{rec}} = \rho N$ and $N_{\mathrm{once}} = (1-\rho)N$ into the effective parameter count of (3) and the compute-equivalent count of (4) makes both proportional to $N$:

The factor $N$ cancels in the ratio, so $\chi$ depends on the loop setting and the recurrent share $\rho$ but not on model size. Intuitively, $N_{\mathrm{eff}}$ measures what the model is worth in loss and $N_{\mathrm{comp}}$ what it costs in compute, so $\chi$ is the number of compute-equivalent parameters paid for each effective parameter. Because this price is the same at every size, choosing $N > 0$ is equivalent to choosing $N_{\mathrm{eff}} > 0$, and we optimize over the normalized effective size $x=N_{\mathrm{eff}}/N_0 > 0$ directly.

Step 2: the budget as a non-looped budget. Define $y=D/D_0$ and $\bar{C}=C/(6N_0D_0)$. Since $N_{\mathrm{comp}}=\chi N_{\mathrm{eff}}$, the budget $C=6DN_{\mathrm{comp}}$ becomes

A non-looped model has $\chi=1$ and buys effective size and tokens with $xy=\bar{C}$. A loop setting therefore acts like a non-looped model whose budget is divided by $\chi$: settings with $\chi<1$ stretch the budget and settings with $\chi>1$ shrink it. Once the size $x$ is chosen, the budget fixes the token count $y=\bar{C}/(\chi x)$, so a single free variable remains.

Step 3: a one-variable objective. Write $J=Ax^{-\alpha}+Hy^{-\beta}$ for the bracket in (3), so that $L=E+J^\kappa$. Since $z\mapsto E+z^\kappa$ is strictly increasing for $z>0$ when $\kappa>0$, minimizing $L$ and minimizing $J$ have the same minimizers. Eliminating $y$ with the budget gives

The first term is the size penalty and falls as the model grows. The second is the data penalty and rises with $x$, because a larger model leaves fewer tokens within the same budget. The optimum balances these two effects, which pull the model size in opposite directions.

Step 4: the balance condition and the optimal allocation. Both terms are power laws in $x$, so $\log x$ is the natural variable: in it, each term is an exponential. Holding $T$, $B$, $C$ fixed, write $X=Ax^{-\alpha}$ and $Y=H(\chi x/\bar{C})^\beta$. Then

The objective is strictly convex in $\log x$. It also diverges at both ends, since $X\to\infty$ as $x\to0$ and $Y\to\infty$ as $x\to\infty$. A strictly convex function that diverges at both ends has exactly one stationary point, and that point is its global minimum. The stationarity condition $\alpha X=\beta Y$ states that, at the optimum, a one-percent increase in model size lowers the size penalty by exactly as much as the resulting loss of tokens raises the data penalty. Writing it out,

Taking the positive root and substituting it into $y=\bar{C}/(\chi x)$ yields

As Step 2 anticipated, the loop setting enters only through the reduced budget $\bar{C}/\chi$: the optimal effective size and token count are those of a compute-optimal non-looped model trained with $\bar{C}/\chi$. The optimal size and token count are recovered as $N^*=N_0x^*/[1-\rho+\rho g(T,B)]$ and $D^*=D_0y^*$.

Step 5: the optimal loss. At the stationary point $Y^*=(\alpha/\beta)X^*$, so the two penalties stay in a fixed proportion and $J^*=X^*+Y^*=[(\alpha+\beta)/\beta]X^*$. Substituting $x^*$ into $X^*=A(x^*)^{-\alpha}$ gives

Define the positive constants

Raising $J^*$ to the power $\kappa$ restores the loss:

This is the compute-optimal loss of a non-looped model at budget $\bar{C}/\chi$.

Step 6: comparing backward counts. In (D.6), $K$ and $\eta$ depend only on the fitted coefficients and $\bar{C}^{-\eta}$ only on the budget; none of them depends on $B$. The backward count enters only through $\chi(T,B)^\eta$, which is strictly increasing in $\chi$ because $\eta>0$. The optimized losses over $B\in\{1,\ldots,T\}$ are therefore ordered exactly as the values of $\chi$, so the two have the same minimizers, including any ties. This proves (D.4). Changing the budget multiplies every optimized excess loss by the same factor $\bar{C}^{-\eta}$, which leaves their ranking, and hence the minimizing backward counts, unchanged whenever the admissible set of $B$ is fixed. In the limit $\rho=0$ the model has no recurrent block, so $\chi=1$ for every $B$ and all backward counts tie, as expected with nothing to truncate.

Joint forward–backward allocation at equal compute

For a restricted menu, minimize over its available backward counts. Figure 16 evaluates $\chi$ for every integer $B$ at $T = 2, 4, 8, 16$ (Table 5 lists the optima).

Separate laws for the contour comparison. To check the shared looping coefficient $g(T, B)$, we fit $L = E_s + [A_s(N/N_0)^{-\alpha_s} + H_s(D/D_0)^{-\beta_s}]^{\kappa_s}$ separately to each setting $s$ at $T = 4, 8$ in Figure 10. The contours use compute from (4) with $N_{\mathrm{rec}}/N \simeq0.507$; their minimum-compute points trace the compute-optimal frontiers. These are in-sample predictions of the separate laws.

Choosing the backward-loop count. For a continuous relaxation, an interior optimum balances the relative increase in compute-equivalent size against that in loss-equivalent size. The ratio is independent of model size, so we can differentiate it at fixed $N_{\mathrm{once}}, N_{\mathrm{rec}}$. By definition,

Differentiating $\log\chi= \log N_{\mathrm{comp}} - \log N_{\mathrm{eff}}$ and setting the derivative to zero yields

This is an implicit equation for a continuous candidate, as both parameter counts depend on $B$; boundary values $B = 1, T$ are also checked, and the discrete optimum evaluates $\chi$ at each integer.

The equation also gives an explicit asymptotic form when $N_{\mathrm{rec}} > 0$. Suppose $0 < p < 1$ and $q > 0$, as in the fitted law, and consider large $T$ with $B \ll T$. Then $g(T, B) \to e^c$, $N_{\mathrm{comp}} \sim T N_{\mathrm{rec}}/3$, and $N_{\mathrm{eff}} \to N_{\mathrm{once}} + e^c N_{\mathrm{rec}}$. Substituting these limits into (D.7) and rearranging gives

The exponent lies between zero and one, so this solution grows while $B^*/T \to0$, matching the assumed limit. This is a large-$T$ prediction; finite loop counts require discrete optimization.

Joint forward–backward allocation at equal compute

Proof. With the token count and architecture fixed, every split on the line $T+2B=S$ has the same compute, so only the looping coefficient $g(T,B)$ varies across feasible splits. We therefore maximize $g$ along this line, reduce the problem to a strictly convex function of $B$ alone, and show that its derivative is negative on the whole feasible interval when $q\ge2p$. The minimum then lies at the boundary $B=T$, which is full BPTT.

At fixed architecture and token count, (D.2) depends on the loop counts only through $S=T+2B$, including the attention products. Thus every feasible split has the same compute. Since the loss is strictly decreasing in $N_{\mathrm{eff}}$ and $N_{\mathrm{rec}}>0$, minimizing it is equivalent to maximizing $g(T,B)$. Using $g(T,B)=\exp[c(1-T^{-p}B^{-q})]$ with $c>0$, this is equivalent to minimizing $T^{-p}B^{-q}$, or its logarithm. Substituting $T=S-2B$ gives $1\le B\le S/3$ and the strictly convex objective

When $q\ge2p$, the constraint $B\le T$ gives

The derivative is strictly negative whenever $B<T$, including when $q=2p$. Hence the unique minimum is at $B=S/3$, where $T=B$. For $S=3$, this is also the only feasible split. □

The replacement preserves $N,D$ and strictly improves any truncated setting, so the conclusion also holds when either or both are optimized, provided the replacement is admissible. It does not require fixed parameter proportions across model sizes. Including the optimizer update does not change the result, since its cost is independent of $T,B$.

The condition is stronger than $q>p$: a backward loop costs twice a forward loop, so the comparison is $q$ versus $2p$. Our adopted fit satisfies $q>2p$. For integer or restricted loop menus, $S/3$ may be unavailable, so the continuous dominance argument does not replace the discrete check (D.8).

Fixed model size: allocate loops and data

With $N$ fixed, size cannot rebalance against data, and the loop setting determines how many tokens the compute buys: $D_N(C;T,B)=C/F_N(T,B)$ by (D.2). § 7.1 compares the fifteen observed settings $S$ at their respective token budgets:

The switch points quoted in § 7.1 are computed with the measured compute, optimizer update included; leaving it out moves them by less than 0.5% (§D.1).

For continuous loop counts, Lemma D.2 establishes the preference for full BPTT when $q\ge2p$, as in our adopted fit. We therefore set $B=T$ and first characterize the interior balance between loops and tokens, then derive its large-compute limit.

Throughout this derivation, a prime denotes a total derivative with respect to $T$ along $B=T$, holding the architecture and $C$ fixed. In particular,

Thus $N_{\mathrm{eff}}'$ measures how effective size changes with loop count ($N$ is fixed), and $F_N'(T,T)$ includes both loop counts as $B=T$ grows. Every interior optimum satisfies

Equivalently, the stationary points are parameterized by

:::

Proof. At fixed model size and compute, the loop count trades effective size against tokens: raising $T=B$ increases $N_{\mathrm{eff}}$ but, through the budget, reduces $D$. We write the loss as a function of $T$ alone by substituting $D=C/F_N(T,T)$, differentiate its size and data terms, and set the total derivative to zero, which gives (D.9). This first-order condition involves $C$ only through the factor $C^{-\beta}$, so it can be solved explicitly for the compute at which each $T$ is stationary, which gives (D.10).

Since $\kappa>0$, minimizing loss is equivalent to minimizing $J$; substituting the budget gives

Increasing $T$ raises the effective size and lowers $X$, but reduces the token budget and raises $Y$. At fixed $N,C$, differentiating the token budget gives

Consequently, the derivatives of the two loss terms are

Setting $J'=X'+Y'=0$ gives (D.9), and $N_{\mathrm{eff}}'=N_{\mathrm{rec}}cs\,g(T,T)T^{-s-1}>0$ with $F_N'=3v_N$ yields

All factors are positive, so solving for $C$ gives (D.10).

This is a necessary condition for an interior optimum; boundary values and all allowed integer settings must also be compared. On the finite menu, added compute goes to tokens until a deeper setting reaches a lower predicted loss, at which point the token budget drops.

Proof. The proof has two steps. First, we show that a global minimizer exists for every budget and grows without bound as compute increases, so at large compute it is interior and satisfies the stationarity relation (D.10) of Lemma D.3. Second, we take the large-$T$ limit of that relation, where $C(T)$ is asymptotically a power of $T$; inverting it gives $T^*(C)$, and the budget gives $D^*(C)$.

For each finite $C$, the continuous objective $J$ tends to infinity as $T \to\infty$, since its data term grows without bound, so a global minimizer exists. Moreover, $T^*(C) \to\infty$ as $C \to\infty$: for any fixed $M \ge1$, every $T \le M$ has $J(T;N,C) \ge X(M)$, whereas $J(2M;N,C) \to X(2M) < X(M)$. No minimizer stays in $[1,M]$ at large compute; it is eventually interior and satisfies (D.10).

Writing $s = p + q$, the large-$T$ limits are

The bracket in (D.10) is thus asymptotic to a positive constant times $T^{\beta-1}T^{s+1} = T^{\beta+s}$, so $C(T)$ is asymptotic to a positive constant times $T^{(\beta+s)/\beta}$. Evaluating this relation at $T^*(C)$ and inverting gives the first part of (D.11), and $D^*(C) \sim C/(3v_NT^*(C))$ gives the second, recovering (8).

The current coefficients give exponents approximately 0.238 and 0.762 ((9) rounds them to 0.24 and 0.76). These are asymptotic consequences of the fitted law along $B = T$, not a refit to measured optima. The dominance argument above applies within this law and compute model; its extrapolation to unbounded depth is not empirically established. Finite-depth overheads and allowed loop counts determine the practical switches.

Free model size: compare joint allocations

This subsection supports the comparisons of § 7.2 in two steps. Lemma D.5 compares a fully trained looped model with the non-looped model that unties its loops at the same leading compute. Corollary D.6 then lets the model size adapt to the budget and compares the compute-optimal frontiers of all loop settings.

Proof. Each untied copy of the recurrent block adds one full block of effective parameters, whereas each additional trained loop adds less than one once the gain begins to saturate. We make this precise by following the ratio $g(T,T)/T$, the effective size contributed per executed copy of the block. The first condition in (D.12) places this ratio below one at $T=2$, and the second makes it decrease for all $T \ge2$, so it stays below one. The statement about effective size then follows by comparing the two parameter counts term by term.

Along full BPTT, $\log g(T,T)=c(1-T^{-s})$, so

Because $s>0$, the factor $T^{-s}$ decreases in $T$, so for every $T\ge2$ we have $csT^{-s}\le cs\,2^{-s}<1$ and the derivative is negative. The ratio $g(T,T)/T$ is therefore strictly decreasing on $[2,\infty)$, and

For the second statement, the leading compute-equivalent size of the looped model is $N_{\mathrm{once}}+(T+2T)N_{\mathrm{rec}}/3=N_{\mathrm{once}}+TN_{\mathrm{rec}}$ by (4), which equals the parameter count, and hence the leading compute, of the non-looped replacement. Their effective sizes are $N_{\mathrm{once}}+g(T,T)N_{\mathrm{rec}}$ and $N_{\mathrm{once}}+TN_{\mathrm{rec}}$, which differ by $[T-g(T,T)]N_{\mathrm{rec}}$, positive by the first statement and $N_{\mathrm{rec}}>0$. □

The adopted coefficients satisfy both conditions: $g(2,2)\simeq1.910<2$ and $cs\,2^{-s}\simeq0.585<1$. The restriction to $T\ge2$ is needed, because the continuous extension has $g(T,T)>T$ just above $T=1$, where the derivative above is positive ($cs\simeq1.400>1$).

Proof. The proof of Lemma D.1 already contains the key fact: for a fixed loop setting, the budget constraint reads $C/\chi= 6N_{\mathrm{eff}}D$, so optimizing a looped model over its size and tokens is the non-looped problem with a reduced budget. Part (i) restates it for the non-looped optimum, and parts (ii) and (iii) follow by comparing the optimal losses and sizes, (D.6) and (D.5), at two budgets.

Part (i). By Step 1 of the proof of Lemma D.1, $N_{\mathrm{eff}}$ is the fixed multiple $1-\rho+\rho g(T,B)$ of $N$, so optimizing over $N$ is equivalent to optimizing over $N_{\mathrm{eff}}$, and $N_{\mathrm{comp}}=\chi N_{\mathrm{eff}}$ turns the budget into $C/\chi= 6N_{\mathrm{eff}}D$. In the variables $(N_{\mathrm{eff}},D)$, the loss (3) is the non-looped law, since $(1,1)$ has $g(1,1)=1$ and $N_{\mathrm{eff}}=N$, and the constraint is the non-looped budget $C/\chi$, since $\chi(1,1)=1$. The two problems therefore have the same minimum and the same unique minimizers (Step 4 of that proof), namely $L^*_{1,1}(C/\chi)$, with effective size $N^*_{1,1}(C/\chi)$ and token count $D^*_{1,1}(C/\chi)$. Dividing the effective size by $1-\rho+\rho g(T,B)$ recovers $N^*_{T,B}(C)$, which gives (D.13).

Part (ii). By (D.6), the excess loss of $(T,B)$ at budget $\lambda C$ is $K\bar{C}^{-\eta}\lambda^{-\eta}\chi^\eta$, and that of $(1,1)$ at budget $C$ is $K\bar{C}^{-\eta}$, because $\chi(1,1)=1$. Their ratio is $(\lambda/\chi)^{-\eta}$. Since $\eta>0$, it is below one exactly when $\lambda>\chi$. For the sizes, part (i) gives $N^*_{T,B}(\lambda C)=(N/N_{\mathrm{eff}})_{T,B}N^*_{1,1}(\lambda C/\chi)$, and (D.5) with $\chi=1$ shows that $N^*_{1,1}$ scales as the power $\beta/(\alpha+\beta)$ of the budget, so $N^*_{1,1}(\lambda C/\chi)/N^*_{1,1}(C)=(\lambda/\chi)^{\beta/(\alpha+\beta)}$.

Part (iii). Setting $\lambda=1$ in the loss ratio of part (ii) gives $L^*_{T,B}(C)-E=[L^*_{1,1}(C)-E]\chi^\eta$. Subtracting $L^*_{1,1}(C)-E$ from both sides yields (D.15).

The identities are exact within the stated approximation and have two consequences. First, by (D.5) every loop setting has $N^*\propto C^{\beta/(\alpha+\beta)}$ and $D^*\propto C^{\alpha/(\alpha+\beta)}$ (exponents 0.562 and 0.438 with the fitted coefficients), so the optimal pair minimizes $\chi(T,B)$ at every compute. Second, among the fifteen settings $(1,1)$ minimizes $\chi$ and $(2,2)$ is the best looped setting (Table 5; Figure 16b). For $(2,2)$, $N_{\mathrm{eff}}\simeq1.462N$, $N_{\mathrm{comp}}\simeq1.507N$ and $\chi\simeq1.031$, so (D.13) predicts equal loss at 3.1% more compute and 31.6% fewer parameters than $(1,1)$. By (D.14), 10% more compute predicts lower loss with about 29.0% fewer non-embedding parameters. At equal compute, with $\eta\simeq0.0559$, the bracketed multiplier in (D.15) is about $1.72\times10^{-3}$, or 0.172% of the non-looped model’s excess loss above $E$, rather than a constant gap in nats, making the compute convention consequential.

Limits of the scale-independent ratios. The identities describe an empirical loss law under a specified compute approximation; they do not establish a universal property of looped models. Size or data bounds can put an optimum on a boundary. A size-dependent recurrent share or the attention, output, and optimizer terms breaks the simple $6\chi N_{\mathrm{eff}}D$ factorization. Full-compute optimization therefore requires an explicit architectural scaling family and a fresh numerical envelope. Its results should be reported separately from the analytic ratios, using the same adopted fit.

Compute-optimal pairs and the untied model

The main text (§6.3) compares $(2,2)$ with a compute-optimal model without looping while allowing model size and token count to vary. Below we give the full table for that comparison, followed by a controlled comparison at fixed tokens against an untied model with a separate block per loop.

Compute-optimal pairs

We trained $(1,1)$ and $(2,2)$ each at the size and token count that minimize its loss for the same training compute, with three seeds each (Table 11). The first two pairs also belong to the scaling suite and its fitting set. At all three budgets, the looped model has 27–32% fewer non-embedding parameters and comes within $(1$–$8)\times10^{-3}$ nats of the model without looping. At $2.1\times10^{19}$ FLOPs ($2.0\times10^{19}$ for the forward and backward passes alone), training the 344M $(2,2)$ model on 8% more tokens, and so with 8% more compute, lowers its loss to 2.3590, $(7.3 \pm1.8) \times10^{-3}$ nats below the 471M model without looping (Figure 1d).

The untied model

The untied model of a looped model is the same network in which each loop has its own copy of the recurrent block, input injection and normalization. For two loops it holds $N_{\mathrm{once}} + 2N_{\mathrm{rec}}$ parameters, about 1.5 times as many, at the same compute per token. Tying layers to save parameters goes back to ALBERT (Lan et al. [45]), and Bae et al. [3] relax the tie of looped models converted from pretrained ones. Each copy is initialized like the looped model; with all copies set equal, the untied model reproduces the looped model’s loss and logits exactly.

Design and results. Each size trains at about 18 tokens per parameter of the looped model, close to the compute-optimal ratio of Hoffmann et al. [35]: 60M at $D_0$, 120M at $2D_0$, 240M at $4D_0$ and 480M at $8D_0$, with the recipe of its (2, 2) run and three seeds per model. Alongside the untied model, we train (2, 2) at the same width and (2, 2) widened to about 80% of the untied model’s non-embedding parameters, on the same tokens (Table 12). At the untied model’s width the looped model trails by $9 \times10^{-3}$–$1.7 \times10^{-2}$ nats $(1.3 \times10^{-2}$ averaged over the four sizes), against $(3$–$4) \times10^{-3}$ by the law, whose error at these (2, 2) runs is at most $4 \times10^{-3}$ nats. The wider looped model uses 15–22% more forward-and-backward compute per token than the untied model and reaches a lower loss by $5 \times10^{-3}$–$1.7 \times10^{-2}$ nats, in all twelve seed pairs.

Size	Model	Width	Parameters	Compute	Loss	− untied
60M	untied	768 (6 × 128)	87M	1.00	2.7843
	(2, 2)	768 (6 × 128)	58M	1.00	2.7931	$(+8.8 \pm3.0) \times10^{-3}$
	(2, 2), wider	840 (7 × 120)	69M	1.15	2.7668	$(-1.75 \pm0.19) \times10^{-2}$
120M	untied	1024 (8 × 128)	180M	1.00	2.5809
	(2, 2)	1024 (8 × 128)	115M	1.00	2.5982	$(+1.73 \pm0.14) \times10^{-2}$
	(2, 2), wider	1152 (9 × 128)	146M	1.22	2.5681	$(-1.28 \pm0.23) \times10^{-2}$
240M	untied	1280 (10 × 128)	360M	1.00	2.4147
	(2, 2)	1280 (10 × 128)	239M	1.00	2.4272	$(+1.25 \pm0.08) \times10^{-2}$
	(2, 2), wider	1408 (11 × 128)	289M	1.18	2.4044	$(-1.03 \pm0.17) \times10^{-2}$
480M	untied	1664 (13 × 128)	709M	1.00	2.2565
	(2, 2)	1664 (13 × 128)	471M	1.00	2.2700	$(+1.35 \pm0.19) \times10^{-2}$
	(2, 2), wider	1824 (12 × 152)	566M	1.18	2.2511	$(-5.4 \pm1.5) \times10^{-3}$

Learning rate and batch size

Our runs follow one learning-rate rule and one batch-size rule, both set by the token budget $D$ (Table 13). We test each rule against sweeps that vary only the learning rate or only the batch; a learning-rate sweep scales the MuonH and AdamW learning rates by one common factor. We call a value acceptable when its loss is within 0.25% of the optimum of its sweep, about a tenth of what (4, 4) gains over (1, 1) at the same model size and budget.

Budget $D/D_0$	$\frac{1}{2}$	1	2	4	8	16	32
Batch (sequences of 2,048 tokens)	48	64	88	128	176	256	352
Steps	5,464	8,192	11,920	16,384	23,832	32,768	47,664
Peak learning rate ($\times10^{-3}$)	10.7	9.25	8.11	8.01	7.03	6.36	6.09

Learning-rate rule

We take the transfer rule of the reference recipe (Schwethelm et al. [65]) for the peak MuonH learning rate of the hidden weight matrices,

where $b$ is the global batch in sequences of 2,048 tokens; the AdamW learning rates scale with width, and those of the embedding and output also with batch. The reference recipe uses the token exponent $\gamma= 0.32$ of HyperP [60], a transfer rule for MuonH. On our models this exponent is too small: relative to (F.1) with $\gamma= 0.32$, the optimal learning rate keeps falling as $D$ grows (Figure 17b). We thus re-estimate $\gamma$ and, as our models vary in depth, test for a depth term.

Token exponent: $\gamma= 0.40$. We swept the learning rate at the smallest and the largest model size, 60M at $D_0$, $4D_0$ and $16D_0$ and 1B at $4D_0$ and $16D_0$, each with (4, 4) and (4, 1), and fit a quadratic in $\log\eta$ to each sweep to locate its optimum (Figure 17a). The exponent that minimizes the rule’s largest loss above the optimum is $\gamma= 0.406$ (95% interval 0.400–0.412), and we use $\gamma= 0.40$. The rule is then within 0.06% of the optimum in every sweep, the largest gap being 60M with (4, 1) at $16D_0$, whereas $\gamma= 0.32$ ends up 0.45% above the optimum at 1B and $16D_0$.

No depth correction. Depth-wise transfer rules lower the learning rate as networks get deeper ([12, 20, 60, 79]). For updates normalized as in MuonH, keeping the change of the network per step fixed requires $\eta\propto\ell^{-1/2}$ with residual branches scaled by $\ell^{-1/2}$, or $\eta\propto\ell^{-1/2}$ without that scaling, where $\ell$ is the number of executed layers ([60 Theorem 3]). Our recipe scales neither the residual branches nor the learning rate with depth, although our models get deeper both with size (20 to 40 executed layers from 60M to 1B at $T = 4$) and with the number of forward loops (12 to 36 at 60M). Yet the optimum does not follow the depth (Figure 17c–Figure 17d): at $D_0$, the rule’s learning rate is the best of $1/\sqrt{2}$, 1 and $\sqrt{2}$ times itself in all six sweeps, and the fitted optima stay within about 7% of the rule, where the trend HyperP measures without residual scaling would lower them by more than half. The optimal learning rate follows the training tokens, not the depth, so the rule needs no depth term.

Batch-size rule

We set the global batch to

rounded to a ladder whose neighboring steps differ by about $\sqrt{2}$ (..., 32, 48, 64, 88, 128, 176, 256, 352). The loss-optimal batch grows about as $\sqrt{D}$, as in prior work [10, 22] (Li et al. 2025b), and in eleven batch sweeps with (4, 4) (60M at five budgets, 240M and 1B at three) the rule’s batch is acceptable in every one (Figure 18a).

Prefactor. The loss is flat over about a factor of two in batch, so the prefactor is a choice between neighboring ladder steps. One step lower, $45\sqrt{D/D_0}$, suits 1B with full BPTT as well as 64 suits 60M, but truncated BPTT prefers larger batches: at 1B and $16D_0$, the best batch is 176 for (4, 4) and 352 for (4, 1). Over all 33 sweeps with at least three batch sizes, which cover all ten core settings, only the rule is acceptable everywhere: it stays within 0.17% of the best batch, while one step lower reaches 0.30% and one step higher 0.42% (Figure 18b). The rule therefore changes comparisons between loop settings by less than 0.004 nats.

The law across the five-size grid

Figure 19 shows the validation loss of the 343 configurations in the suite’s five-size grid, one panel per loop setting. Filled markers denote the 226 configurations used to fit the law of § 5; open markers denote the rest. The suite’s four fitted configurations from the compute-matched pairs (120M, 79M, 240M and 162M) are listed in Table 11 and are not drawn here. The figure contains 290 configurations in the ten core settings, 17 with (8, 1) and 36 with $T = 16$. Each configuration shows the mean loss of its accepted runs (385 in all; §B.3); the law is drawn over each size’s measured budgets. The configurations span 1.1 to 1,291 tokens per parameter, while the fitted ones span 9 to 74. Per loop setting, the law’s RMSE on the fitted configurations is 0.0031 to 0.0093 nats; on the rest it is 0.0200, as most lie below 5 or above 125 tokens per parameter (§7). Figure 7 shows the four

full-BPTT settings at the 26 size–budget pairs where all four were trained: one set of coefficients tracks all four, with RMSE 0.0053 nats over the 68 fitted configurations (0.0148 over the 36 others).

Training curves

Figure 20 shows the training curves of the 385 runs behind the 343 configurations of §G, one panel per loop setting; Figure 4 in § 5.1 repeats its first row. Each curve is the training loss of every tenth step, the cross-entropy of that step’s batch, against the training compute spent so far, counted as in §D.1 with the optimizer update; it is not the validation loss on held-out documents that we report (§B.3). Each token budget is a separate run whose cosine schedule ends at that budget, so a run reaches its final loss only at its end. The highlighted line therefore joins run ends only (dots), those that end below every run with less compute, and needs no fit. As in the training-curve envelopes of Hoffmann et al. [35], larger models take over this frontier as compute grows; at the highest compute of each panel it follows the largest model of that setting, 1B, or 480M for $T = 16$.