betaFeedback

WorksComputer Science

Latent Feedback Is a Compute and Data Amplifier

Abstract

When high-quality data runs out, a common practice is to train on it for several epochs, but each repeat is worth less than the last. Latent feedback, which feeds each token's top-layer state into the next token's input, is another way to spend that compute, yet it has not been tested against a vanilla Transformer that spends the same compute on more epochs. We run this test across a ladder of model sizes, unique data and epochs, tuning both models at every size and amount of unique data. Latent feedback makes every FLOP and every token count for more except when unique data is scarce relative to model size, a regime that today's models are far from. We find that latent feedback serves as an amplifier: with unique data fixed, it reaches a Transformer's loss with less compute; with epochs fixed, it reaches that loss with less unique data. Furthermore, both savings grow with unique data per parameter and with repetition, and downstream tasks agree. To go beyond our grid, we fit one scaling law to both models, in which latent feedback enters as a Transformer with more parameters; it predicts that both savings keep growing with unique data and repetition. Fitted on models from 60M to 271M parameters, the law predicts the losses at a held-out 778M size to within 0.3% and the gap between the two models to within 0.05%. At most, latent feedback saves 48% of the compute and 47% of the unique data. When data, not compute, is the limit, latent feedback is the better use of a FLOP for a model of a given size.

Figure 1. Latent feedback saves compute and unique data. (a) Latent feedback: each token’s top-layer state hh enters the next token’s input through a map WW. (b) The share of compute that latent feedback saves at equal unique data, at a budget of two Transformer epochs; each bar is the median over model sizes, and each dot is one model size. (c) The share of unique data it saves when both models train one epoch on 50 unique tokens per parameter.

Introduction

High-quality text, not compute, now limits pretraining of large language models [1, 2]. Even web text, mostly not high quality, grows far more slowly than pretraining compute [3, 4]. So a common practice is to train on the best sources for several epochs [5, 6, 7, 8, 9, 10]. However, each repeat is worth less than the last, so additional epochs bring diminishing returns [7, 10, 11, 12].

Latent feedback offers another use for that compute. A latent feedback Transformer feeds each token’s top-layer state into the next token’s input. Prior work reports improved performance for extra training FLOPs [13, 14]. Yet these comparisons fix the training tokens, and none repeats data. With repeated data, the fair baseline is a Transformer that spends the same FLOPs on more epochs. Whether latent feedback’s gain survives at scale, against a vanilla Transformer given the same FLOPs and unique data, is still largely unknown. We therefore ask:

When data runs out, can latent feedback make every FLOP and every token count for more?

We answer it on a ladder of models from 60M to 778M parameters, each trained on 12.5 to 400 unique tokens per non-embedding parameter (UTPP) for one to four epochs, that is up to 1,600 training tokens per parameter with repetitions (TPP). Both models are tuned at every size and amount of unique data, and the pools of unique tokens are nested and deduplicated. The answer is yes, once each parameter has enough unique data, and today’s models have far more than enough. We measure the answer at a fixed model size in two ways, and then fit a law to extrapolate it.

  • Compute. With unique data fixed, the Transformer must train more epochs to match latent feedback; it needs κ\kappa times the FLOPs (Figure 2 (a)). The saving grows with UTPP and with epochs: at two epochs, latent feedback costs 8% more at 12.5 UTPP, but saves 15% to 18% at 100 and 44% to 48% at 400. Above about 75 UTPP it saves compute at every budget we train, and at two epochs already above 33%. Practice is far past this point: Chinchilla used 20 TPP [6], whereas today the median 1B–10B dense model trains on 1,040, at least 260 UTPP even over four epochs [15, 16, 17].1

Figure 2. The compute multiplier κ\kappa across unique data and model size. (a) 271M with U=200NU = 200N: each arrow runs from a latent feedback cell to the budget at which the Transformer reaches its loss, labelled with κ\kappa. (b)–(d) Dots are measured cells, filled with the κ\kappa read off the Transformer’s losses at F=1F = 1 to 44 interpolated in log FF; everywhere else the map shows the law of § 5.3. κ\kappa grows with unique tokens per parameter and with the budget FF, and its contours run nearly vertically: it depends on U/NU/N far more than on size.

  • Data. With epochs fixed, the Transformer needs μ\mu times the unique data to match latent feedback with the same parameters and key-value (KV) cache (Figure 3 (a)). At one epoch on 50 UTPP, latent feedback saves 16% of the unique data at 271M and 20% at 778M; with three epochs on 100 UTPP, it saves 41% at 271M.

Figure 3. The data multiplier μ\mu across unique data and model size. (a) 271M at two epochs: each arrow is the unique data latent feedback saves; the arrow at U=200NU = 200N is extrapolated. (b)–(d) Dots are measured cells, read off the Transformer’s losses across UU interpolated in log UU (Appendix § 11); everywhere else, the map shows the law of § 5.3. μ\mu grows with unique data and with epochs, and depends on U/NU/N far more than on size.

  • A fitted law for extrapolation. To go beyond our grid, we fit one scaling law to both models, with latent feedback entering as a Transformer with ν\nu times the parameters (§ 5.3). Fitted on 60M to 271M, it predicts the held-out 778M losses to within 0.3% and the gap between the models to within 0.05%. It then draws the boundary between the two regimes, a FLOP spent on another epoch or on latent feedback, and predicts that both savings grow with unique data and repetition beyond the grid.

Background

Latent Feedback Transformers feed each position’s top-layer state into the input of the next position [13, 14]. At decoding time, position tt enters the blocks ff with input utu_t and leaves as hth_t:

ut=et+W rms(ht−1).(1)u_t = e_t + W\,\mathrm{rms}(h_{t-1}). \tag*{(1)}

However, teacher-forced training cannot run the positions one by one to get top-layer states, so it runs passes over the whole sequence as fixed-point iterations:

[ut(p+1)=et+W rms(ht−1(p)),ht−1(p)=f(ut−1(p))]→p→∞ut(∗)=et+W rms(ht−1(∗)),(2)\left[u_t^{(p+1)} = e_t + W\,\mathrm{rms}\left(h_{t-1}^{(p)}\right),\quad h_{t-1}^{(p)} = f\left(u_{t-1}^{(p)}\right)\right]\xrightarrow{p\to\infty}u_t^{(*)}=e_t+W\,\mathrm{rms}\left(h_{t-1}^{(*)}\right), \tag*{(2)}

The first pass reads no feedback, ut(1)=etu_t^{(1)}=e_t, and pass p=2,3,…p=2,3,\ldots reads the previous pass’s top states, ideally till fixed points ht−1(∗)=ht−1h_{t-1}^{(*)}=h_{t-1}. In practice, two to three passes come close to the fixed point [14].

At inference, autoregressive decoding already has the previous token’s top state when it predicts the current token, so each generated token costs what a Transformer’s does, though prefilling a prompt needs additional passes of Equation (2). The KV cache is a Transformer’s plus one vector.

Training. Each update runs one pass with probability pp and two passes otherwise (Algorithm 1). The loss and gradient are computed only on the last pass. The map WW is drawn from a normal distribution with standard deviation 0.02. Because of the additional forward passes, a token with feedback costs c>1c>1 times a Transformer’s training FLOPs. So pp trades that cost against the feedback each update sees.

x = embed(tokens)
if rand() > p:
    with no_grad():
        s = blocks(x)
        # token t reads s[t−1]
        x = x + W(rms_norm(shift(s)))
loss = ce(head(blocks(x)), y)
loss.backward()

Scaling law

Budget. A run has NN non-embedding parameters [18, 19, 20], UU unique tokens and a compute budget FF; models are named by their total parameters. We count FF in Transformer epochs, its FLOPs over those of one Transformer epoch on UU, so a Transformer trains R=FR=F epochs. Latent feedback with a training schedule ss that costs cs>1c_s>1 Transformer FLOPs per token trains only R=F/csR=F/c_s epochs. So at equal compute, latent feedback sees fewer repeats than the Transformer. Appendix § 8 lists the notation.

Law. We extend the data-constrained law of Muennighoff et al. [12]:

Ls(N,U,F)=E+[(A(vsN)α)q+(BDsβ)q]1/q,Ds=U[1+R∗(1−e−(R−1)/R∗)],R=Fcs.(3)L_s(N,U,F)=E+\left[\left(\frac{A}{(v_sN)^\alpha}\right)^q+\left(\frac{B}{D_s^\beta}\right)^q\right]^{1/q},\qquad D_s=U\left[1+R^*\left(1-e^{-(R-1)/R^*}\right)\right],\qquad R=\frac{F}{c_s}. \tag*{(3)}

where EE is the irreducible loss, A/(vsN)αA/(v_sN)^\alpha is the capacity term, and B/DsβB/D_s^\beta is the data term. The factor vsv_s counts the effective parameters of training schedule ss, with v=1v=1 for the Transformer. DsD_s is the effective number of training tokens, where R∗R^* sets how fast repeats lose value, and qq couples the two terms.2

Repeats. Epoch kk is worth about e−(k−1)/R∗e^{-(k-1)/R^*} of a fresh one, so the effective data DsD_s saturates at U(1+R∗)U(1+R^*). Below one epoch we set Ds=RUD_s=RU. The two pieces join smoothly: near R=1R=1, 1−e−(R−1)/R∗≈(R−1)/R∗1-e^{-(R-1)/R^*}\approx(R-1)/R^*, so both give Ds≈RUD_s\approx RU with the same slope. Following Qin et al. [22], we let R∗∝(U/N)ρR^*\propto(U/N)^\rho.

Capacity. A schedule reruns the same blocks on a token, as a recurrent-depth model reruns its layers [23, 24]. Scaling laws for such models count the extra passes as effective parameters [25, 26]. We follow their practice and introduce vsv_s that multiplies the parameter count NN,

Table 9: Forms of Equation (3) on the cells at 60M, 134M and 271M, with E≥0.5E \ge0.5. RMSE over the cells and over the 64 equal-FLOPs gaps, in sample and leave-one-out (LOO). v(TPP)v(\mathrm{TPP}): vv grows as a power of TPP; v(TPP,N)v(\mathrm{TPP}, N): also of NN. Skaling equals the coupled form (k=1/qk = 1/q); Kaplan ties qq to 1/β1/\beta.

FormFitCellsCoefficientsRMSE ↓\downarrowLOO ↓\downarrowGap RMS ↓\downarrowGap LOO ↓\downarrow
Transformer only
Additivetwo-phase6470.00860.0103——
Additivejoint6470.00700.0080——
Coupledtwo-phase6480.00340.0037——
Coupledjoint6480.00190.0024——
Both arms
Additivejoint12880.00790.00870.00680.0082
Additive, v(TPP)v(\mathrm{TPP})joint12890.00760.00840.00500.0060
Coupledjoint12890.00270.00320.00350.0044
Coupled, v(TPP)v(\mathrm{TPP})joint128100.00250.00300.00310.0039
Coupled, v(TPP,N)v(\mathrm{TPP}, N)joint128110.00250.00300.00290.0037
Skaling, v(TPP,N)v(\mathrm{TPP}, N)joint128110.00250.00300.00290.0037
Kaplan, v(TPP,N)v(\mathrm{TPP}, N)joint128100.01490.01610.01390.0148

Table 9.

vs=v0(Ds100N)ω(N108)γ,(4)v_s = v_0 \left(\frac{D_s}{100N}\right)^\omega\left(\frac{N}{10^8}\right)^\gamma, \tag*{(4)}

where DsD_s is the effective data of Equation (3) and v0v_0, ω\omega and γ\gamma are fitted.3 The constants 100 and 10810^8 serve as a reference point: v0v_0 is the factor of a model with 10810^8 parameters trained on 100 effective tokens per parameter, and ω\omega and γ\gamma give how it grows with data per parameter and with size from there.

The law predicts a crossover. To first order in ln⁡vs\ln v_s and ln⁡cs\ln c_s, a schedule’s gap to the Transformer at equal FF is (Derivations in Appendix F)

Ls−LT≈S[−wαln⁡vs+(1−w)ϵˉβln⁡cs],w=XqXq+Yq,ϵˉ=ln⁡DT−ln⁡Dsln⁡cs∈(0,1],(5)L_s - L_T \approx S\left[-w\alpha\ln v_s + (1-w)\bar{\epsilon}\beta\ln c_s\right], \qquad w = \frac{X^q}{X^q + Y^q}, \qquad\bar{\epsilon} = \frac{\ln D_T - \ln D_s}{\ln c_s} \in(0,1], \tag*{(5)}

where LTL_T is the Transformer’s loss (Equation (3) with v=c=1v=c=1); X=A/NαX=A/N^\alpha and Y=B/DTβY=B/D_T^\beta are its two terms, S=(Xq+Yq)1/qS=(X^q+Y^q)^{1/q}, and DTD_T and DsD_s are the effective data of the Transformer and of the schedule.

The first term is the extra capacity, weighted by ww, which nears 1 when the capacity term dominates. The second term, weighted by 1−w1-w, is the data that latent feedback gives up. At equal FF it trains csc_s times fewer epochs, so its effective data DsD_s is smaller than the Transformer’s DTD_T. How much smaller depends on what the dropped epochs were worth, which ϵˉ\bar{\epsilon} measures: ϵˉ=1\bar{\epsilon}=1 when each dropped epoch was worth a fresh one, so Ds=DT/csD_s=D_T/c_s, and ϵˉ→0\bar{\epsilon}\to0 when repeats are worth nothing, so dropping them costs no effective data. With UU fixed, DD grows with FF (Equation (3)), so ww rises toward 1 and repeats saturate (ϵˉ→0\bar{\epsilon}\to0). As the budget grows, the capacity term thus counts for more and the data term for less. So feedback that costs more compute per token, may lose at small budgets but win past a crossover budget F∗F^*.

Two multipliers express latent feedback’s advantage as the compute or the unique data it saves. Both compare the two models at the same size NN and the same validation loss. The compute multiplier κ\kappa is the Transformer’s budget over latent feedback’s, at equal unique data UU, so latent feedback saves a share 1−1/κ1-1/\kappa of the compute. The data multiplier μ\mu is the Transformer’s unique data over latent feedback’s, at equal epochs RR, so latent feedback saves a share 1−1/μ1-1/\mu of the data:

κ=FTF with LT(N,U,FT)=Ls(N,U,F),μ=UTU with LT(N,UT,R)=Ls(N,U,csR).(6)\kappa= \frac{F_T}{F}\ \text{with}\ L_T(N,U,F_T)=L_s(N,U,F), \qquad\mu= \frac{U_T}{U}\ \text{with}\ L_T(N,U_T,R)=L_s(N,U,c_sR). \tag*{(6)}

Fitting. We fit all cells of both models jointly by least squares. Latent feedback differs from the Transformer only through its capacity factor vsv_s, with three fitted coefficients (v0v_0, ω\omega and γ\gamma), and its measured cost csc_s; all other coefficients are shared.4 To separate the effect of the size and data terms, our cells vary training tokens per parameter from 12.5 to 1,600 [28].

Setup

Models. Both models, the Transformer and the one with latent feedback, share one backbone at 60M, 134M, 271M and 778M parameters. The backbone is a modern decoder: pre-norm RMSNorm [29], rotary positions [30], QK-norm [31] and SwiGLU [32], with untied embeddings and OLMo 2’s initialization [33]. Its matrices train with NorMuon [34], a normalized variant of Muon [35], and its embeddings and output head with AdamW [36]; see Table 10 in Appendix § 14 for details. Latent feedback uses p=0.15p = 0.15 throughout5, so its cost cc is about 1.20 to 1.27 across the four sizes.

60M134M271M778M
Parameters60,041,728134,108,928271,094,784777,862,656
Non-embedding parameters27,273,72884,956,928205,558,784679,558,656
Layers8121624
Width51276810241536
Attention heads46812
MLP hidden width1536204828164096
U/NU/N12.5, 50, 100, 40012.5 to 40012.5 to 35012.5, 50, 100
Main run length FF, by U/NU/N4444, 2, 1
Tokens per updatea^{a}0.125M / 0.25M0.25M / 0.5M0.25M / 0.5M0.5M
β2\beta_{2}, AdamW and NorMuona^{a}0.987 / 0.9750.975 / 0.950.975 / 0.950.95
Warmup updates at 0.5M1152565171,484
Latent feedback: one-pass probability pp0.15
Latent feedback: cost cc1.19701.23171.25141.2677
Learning rate η\eta0.02520.01630.0111 / 0.01570.0043 / 0.0061
Tuning grid in ηb\eta^{b}η×2±1\eta\times\sqrt{2}^{\pm1}
Tuning grid in τb\tau^{b}τ×2±1\tau\times2^{\pm1}τ×2±1\tau\times2^{\pm1}τ×2−1,2−2\tau\times2^{-1},2^{-2}—
DataStratified Dolma 3 (Appendix § 16)
TokenizerLlama 2 (32,000 tokens)
Sequence length2048 tokens
Epoch orderOne fixed order, the same in every epoch
BlockPre-norm RMSNorm, rotary positions, QK-norm, SwiGLU; head width 128
EmbeddingsUntied input and output, no weight decay
InitializationOLMo 2 [33]
Matrix optimizerNorMuon, momentum 0.95, from 0.85 over the first 300 updates at 0.5M
Embedding and headAdamW, β1=0.9\beta_{1} = 0.9
ScheduleWarmup-stable-decay; linear decay to zero over the last 20% of each budget
OtherGradient clipping at 1.0; bfloat16

Table 10. Configuration of the runs; the latent feedback Transformer adds the feedback map WW (Equation (2)). Learning rate: centre of the grid, Transformer / latent feedback, one step of 2\sqrt{2} higher at U=12.5NU = 12.5N except at 134M. Warmup doubles with each halving of the batch.

Data. We take the Dolma 3 sample [9] and process it ourselves: we deduplicate it and remove the documents that share a paragraph with Paloma [37]. This keeps Paloma a clean held-out benchmark for a fair evaluation [38, 39]. We order its documents so that every prefix has the whole sample’s mixture of sources and topics with same ratios, and each run’s pool is the first UU tokens of this order (Appendix § 16). Pools are thus nested and differ only in size, from U=12.5NU = 12.5N to 400N400N.

Tuning. We call one setting of model size NN, unique data UU, and budget FF a cell. For each model, size and amount of unique data, we tune a main run over the learning rate η\eta and the timescale τ=1/(ηλT)\tau= 1/(\eta\lambda T) [40, 41], where λ\lambda is the weight-decay and TT the number of steps; other FF branch off the main run as decay runs and keep its hyperparameters. Each grid is centered on a predicted start point from earlier runs and extended wherever the best point sits on its edge; a cell’s value is the minimum of a quadratic fitted to its points. Where unique data is small, we also shrink the batch to stay near its optimum [41]. Each main run uses a warmup-stable-decay schedule [42] (Appendix § 14).

Evaluation. We measure on a held-out validation set of 47,274 documents (100M tokens), taken from the front of the same stratified order of the deduplicated Dolma 3 sample. It keeps the mixture ratio and excludes training documents. For downstream tasks, we score the Olmo 3 Base Easy suite [9], 109 tasks in 37 suites, as bits per byte (BPB) of the correct continuation on OLMES requests [43]. Each suite is weighted by the inverse of its seed variance, so noisy suites do not dominate (Appendix § 15).

Decoding latent feedback token by token (Equation (1)) is slow. We score it instead with 4 parallel passes of Equation (2), whose states h(4)h^{(4)} nearly match the decoding states, so more passes leave the loss almost unchanged (Table 5 in Appendix § 10).

Passes PPMean lossMean difference ↓\downarrowLargest difference ↓\downarrow
42.028630.000070.00013
82.028690.000010.00004
162.02870––

Table 5. Loss of latent feedback scored with PP parallel passes of Equation (2), on ten runs of an earlier ladder (one 271M, nine 778M) and a 10M-token Dolma 3 validation slice. Differences are taken per run against 16 passes.

Experiment and result

This section reads the grid of runs from § 4 on validation loss in three steps, and then checks it on downstream tasks. First, with unique data fixed, we measure the compute that latent feedback saves (κ\kappa, § 5.1). Second, with the epochs fixed, we measure the unique data that it saves (μ\mu, § 5.2). Third, we fit the law of § 3 (Equation (3)) jointly to both models, test it on the held-out 778M, and use it to explain the two savings and to predict them beyond the grid (§ 5.3). Finally, we check that the comparison is robust: repeats do not overfit, gaps are stable across seeds, and downstream tasks agree with the validation loss (§ 5.4).

Equal unique data: the compute multiplier

At equal size and unique data, latent feedback reaches a Transformer’s loss with less compute (Figure 2 (a)). Both models train the same model size NN on the same UU unique tokens, and only the budget FF varies: the Transformer trains FF epochs and latent feedback, about 1.2 times as costly per token, trains F/csF/c_s. Latent feedback saves a share 1−1/κ1 - 1/\kappa of the compute (Equation (6)).

The saving grows with UTPP and with epochs (Figure 2). Along UTPP, latent feedback costs more compute than the Transformer when unique data is scarce and saves compute when it is plentiful. The unique data at which both models need the same compute (κ=1\kappa= 1) falls as the budget grows: about 75 UTPP at F=1F = 1 and 33 at F=2F = 2. Above ≈75 UTPP, latent feedback saves compute at every budget we train. Along epochs, the saving grows with the budget FF. At 400 UTPP and F=2F = 2, latent feedback saves nearly half the compute (κ=1.91\kappa= 1.91 at 60M); Table 7 in Appendix § 11 lists every measured κ\kappa.

UTPPF=1F = 1F=2F = 2F=3F = 3
60M134M271M778M60M134M271M778M60M134M271M778M
12.5————0.920.920.920.930.980.980.980.97
25—————0.990.98——1.081.07—
50————1.051.101.06—1.171.221.17—
1001.061.021.04—1.231.181.18——1.331.30—
200—1.221.19——1.461.38—————
350——1.38———1.68—————
4001.451.45——1.911.79——————
UU1 epoch2 epochs3 epochs
60M134M271M778M60M134M271M60M134M271M
12.5NN0.991.021.051.091.081.121.141.141.181.20
25NN—1.101.14——1.191.23—1.261.30
50NN1.111.161.191.251.251.351.341.351.471.46
100NN1.271.291.34—1.461.491.531.601.671.69
200NN—1.541.54——1.88————

Table 7. Measured multipliers. Top: compute multiplier κ\kappa ↑ at equal size and unique data. Bottom: data multiplier μ\mu ↑ at equal size and epochs. —: no such cell, or the read leaves the measured range (one to four Transformer epochs for κ\kappa, the largest UU for μ\mu).

Equal epochs: the data multiplier

At equal size and epochs, latent feedback reaches a Transformer’s loss with up to half the unique data (Figure 3 (a)). With size and epochs equal, both serve with the same parameters and KV cache; latent feedback pays only about 1.2 times the training FLOPs per token, and saves 1−1/μ1 - 1/\mu of the unique data. At 271M and two epochs, for example, μ=1.34\mu= 1.34 at U=50NU = 50N, so latent feedback needs 25% less unique data than the vanilla Transformer (Figure 3 (a)).

The saving grows with U/NU/N and with epochs (Figure 3). At the smallest UU the two models need about the same data. The largest measured saving at 271M is 41% of the unique data, with U=100NU = 100N and three epochs (μ=1.69\mu= 1.69). At one epoch it also grows with size. Table 7 in Appendix § 11 lists every measured μ\mu.

One law for both models

The two multipliers above are measured only inside the grid, up to 778M, 400 UTPP and four epochs. To explain them and to go beyond the grid, we fit Equation (3) jointly to both models’ losses at 60M, 134M and 271M, which gives

L=0.5+[(1.75(vn)0.112)3.75+(1.40d0.126)3.75]1/3.75,R∗=14.6(un)0.45,(7)L = 0.5 + \left[\left(\frac{1.75}{(v n)^{0.112}}\right)^{3.75} + \left(\frac{1.40}{d^{0.126}}\right)^{3.75}\right]^{1/3.75}, \qquad R^* = 14.6\left(\frac{u}{n}\right)^{0.45}, \tag*{(7)}
v=1 for the Transformer,v=1.14(dn)0.010n0.007 for latent feedback,v = 1 \text{ for the Transformer}, \qquad v = 1.14\left(\frac{d}{n}\right)^{0.010} n^{0.007} \text{ for latent feedback},

with n=N/108n = N/10^{8}, u=U/1010u = U/10^{10} and d=D/1010d = D/10^{10}, where the effective data DD follows Equation (3).6

The law predicts a held-out size (Table 1). The 778M model is 3.3 times the size of the largest fitted one, yet the law predicts its losses with a mean absolute error of 0.13% and at most 0.29%. This error is below those that recent scaling laws report on their own held-out runs, 0.50% for Farseer and 0.60% for Skaling [21, 44]. The law also picks the winner: it is off by about the same amount for both models, so predicts the gap to within 0.05% in every held-out row, and gives the measured sign in 6 of 7, with the seventh a tie, and in all but two of the fitted cells, both at U=12.5NU = 12.5N and F=4F = 4, where the predicted gap is within 0.001 of zero.

UUFFTransformerLatent feedback
MeasuredPredictedError (%)MeasuredPredictedError (%)
12.5N12.5N12.20542.2112+0.262.22112.2276+0.29
12.5N12.5N22.14342.1460+0.122.15012.1532+0.15
12.5N12.5N32.11362.1172+0.172.11622.1195+0.16
12.5N12.5N42.09502.1005+0.262.09502.0996+0.22
50N50N12.08782.0873−0.032.08932.0897+0.02
50N50N22.04522.0454+0.012.04212.0413−0.04
100N100N12.04482.0441−0.032.04132.0406−0.03

Table 1. Held-out test of the law at 778M. Fitted on 60M to 271M only. Error: (predicted − measured) / measured.

SymbolMeaningValue or source
Inputs
ssTraining scheduleTransformer, or latent feedback at p=0.15p = 0.15
NNNon-embedding parameters27.3M, 85.0M, 205.6M (60M, 134M, 271M)
UUUnique tokens12.5N12.5N to 400N400N
FFCompute budget, in Transformer epochs1 to 4
Fitted coefficients
EEIrreducible loss0.5 (its lower bound)
AAScale of the parameter term1.755
α\alphaExponent of the parameter term0.112
BBScale of the data term1.401
β\betaExponent of the data term0.126
qqCoupling of the two terms; q=1q = 1 is additive3.748
RT∗R_T^{*}R∗R^{*} at U=100NU = 100N14.60
ρ\rhoGrowth of R∗R^{*} with U/NU/N+0.450
v0v_0Capacity factor vsv_s at the reference run1.139
ω\omegaGrowth of vsv_s with effective tokens per parameter0.010
γ\gammaGrowth of vsv_s with model size0.007
Fixed constants
csc_sTraining FLOPs per token over a Transformer’s1.197, 1.232, 1.251 (60M, 134M, 271M)
N0N_0, D0D_0Reference run: parameters and effective tokens10810^8 and 101010^{10}
Derived
RREpochs within the budget FFF/csF/c_s
nn, uu, dsd_sNN, UU and DsD_s in reference unitsN/N0N/N_0, U/D0U/D_0, Ds/D0D_s/D_0
DsD_sEffective data, in fresh tokensEquation (8)
LsL_sPredicted validation loss at the tuned optimumEquation (8)

Table 4. Symbols of Equation (8), with the joint fit on the 128 cells. Appendix E gives the equation.

With the law checked at 778M, we use it to explain the two savings, to draw the boundary between the two regimes, and to predict the savings beyond the grid.

First, the law explains the two savings: latent feedback trades epochs for capacity. Equation (7) has a capacity term and a data term, and latent feedback moves both. It lowers the capacity term, because it fits as a Transformer with 1.14 times the parameters, a factor that grows slowly with data per parameter and with size. However, it raises the data term, because its cost leaves it about 1.2 times fewer epochs, and repeated epochs still count: in the coupled fits, four epochs at 134M are worth 3.0 to 3.9 fresh ones. Latent feedback therefore wins where capacity limits the loss, and loses where data limits it. This is why both savings grow with unique data and with more epochs: each moves the Transformer toward its capacity floor, which latent feedback lowers.

We want to point out the savings hold at a fixed model size. Latent feedback spends cc times a Transformer’s training FLOPs per token, as much as a Transformer cc times larger, yet the law counts it as a Transformer only vv times larger, and vv stays below cc (1.10 to 1.19 against 1.20 to 1.27). So when model size is free, simply spending the same FLOPs on a Transformer cc times larger, with the same unique data, gives a loss 0.006 to 0.012 lower in every fitted cell. That Transformer, however, costs cc times more per decoding step and holds a larger KV cache, while latent feedback serves at the cost of the smaller model.

Second, the law draws the boundary between the two regimes (the line κ=1\kappa= 1 in Figure 2).7 On one side of this line, a FLOP is worth more as another Transformer epoch; on the other, it is worth more as latent feedback. The boundary moves toward less unique data as the budget grows: latent feedback wins from about 75 UTPP at F=1F = 1, 33 at F=2F = 2, 20 at F=3F = 3 and 12 at F=4F = 4.

Third, the law predicts that both savings keep growing beyond the grid (Figure 4; Table 8 in Appendix § 12). Each panel extends one variable beyond the measured range. Along size, the savings rise because α<β\alpha< \beta makes larger models more capacity-limited, and further through γ\gamma: at 100B, 100 UTPP and F=2F = 2, the law predicts κ=1.83\kappa= 1.83, and 1.41 with γ=0\gamma= 0. Within 60M to 271M, γ\gamma is not significantly above zero (p=0.054p = 0.054), but the held-out 778M favors it: without γ\gamma, the law underrates latent feedback in all seven held-out rows. The dashed curves in Figure 4 show the savings if vv does not grow with size. More unique data per parameter raises them fast, until the Transformer cannot catch up at any budget: it nears its capacity floor, while latent feedback’s floor is that of a Transformer about 1.2 times larger. More repeats push the compute saving the same way, since repeats saturate for both models and only the capacity gap remains; the data saving instead levels off at a height set by the unique data. The curves only shift with the other variables, so latent feedback passes the crossover at any unique data once a run repeats enough, as long as repeats do not overfit (true up to four epochs; Figure 6 in Appendix § 11).

Figure 4. The savings beyond the grid, as the law of § 5.3 predicts them. Every curve comes from one fit of the law to all cells; the dots are the few of those cells that lie on each curve. Each panel extends one variable beyond its measured range (shaded), with one curve per value of a second variable; in (a) and (d), dashed curves set γ=0\gamma= 0, so vv does not grow with size. Top: compute saved at equal unique data. Bottom: unique data saved at equal epochs. A saving of 100% means the Transformer cannot reach latent feedback’s loss with any number of epochs or any amount of unique data.

Validation loss over epochs for four model sizes

Figure 6. Repeats up to four epochs do not overfit. Validation loss after one to four epochs at the scarcest unique data, U=12.5NU = 12.5N, one panel per model size; both curves keep falling. Latent feedback costs c>1c > 1 Transformer FLOPs per token, so at equal compute it trains fewer epochs and its points sit to the left.

Table 8: The fitted law beyond the grid: predicted losses along one curve of Figure 4 (a) to (c), with κ\kappa. Each block extends one axis from a grid cell (its first row); sizes above 778M count non-embedding parameters. ∞\infty: no number of epochs lets the Transformer match.

Axis extendedSizeUUFFTransformer ↓\downarrowFeedback ↓\downarrowκ↑\kappa\uparrow
Larger model271M100N100N22.23562.22661.22
778M100N100N22.01162.00251.27
3B100N100N21.77351.76391.39
10B100N100N21.60851.59881.51
100B100N100N21.35051.34101.83
More unique data271M350N350N12.20952.19671.41
271M1,000N1{,}000N12.17542.15612.50
271M4,000N4{,}000N12.14852.1227>10>10
271M16,000N16{,}000N12.13412.1035∞\infty
More repeats271M50N50N42.23992.23021.27
271M50N50N82.21502.19991.97
271M50N50N162.19972.1803∞\infty
271M50N50N322.19262.1700∞\infty

Table 8.

Figure 7. The loss gap at equal compute across unique data and budget. Markers are measured cells; everywhere else, the map is extrapolated with the law of § 5.3. Latent feedback wins above the solid line, the crossover F∗F^{*}, which falls as unique data grows and hardly moves with size.

Robustness: repeats, seeds and downstream tasks

Repeats up to four epochs do not overfit either model (Figure 6 in Appendix § 11). Even at the scarcest unique data, validation loss keeps falling through four epochs at every size. The gaps are stable across seeds: at 134M, a gap’s standard deviation is below 0.002 and none changes sign (Appendix § 11).

Downstream tasks agree well with the validation loss (Table 2). In Table 2, the Transformer has the lower BPB only at the smallest unique data (U≤25NU \le25N) and budgets of F≤2F \le2; latent feedback has it everywhere else, so the winner switches at almost the same place as in Figure 2. At F=4F = 4, latent feedback has the lower BPB in all 17 finished cells. On downstream tasks, latent feedback starts to win slightly earlier than on validation loss, and the two metrics pick the same winner in 59 of 71 paired cells (Table 6 in Appendix § 11 gives the validation losses).

SizeUUF=1F=1 TransformerF=1F=1 FeedbackF=2F=2 TransformerF=2F=2 FeedbackF=3F=3 TransformerF=3F=3 FeedbackF=4F=4 TransformerF=4F=4 Feedback
60M12.5N12.5N1.23241.24531.16231.16801.13991.13881.12791.1186
50N50N1.10511.10141.06641.05391.05111.03711.04131.0278
100N100N1.06731.05661.03731.02051.02201.00721.01630.9999
400N400N1.01160.99170.99700.97080.98670.96340.98220.9615
134M12.5N12.5N1.03741.05140.99070.98890.96770.96240.95590.9466
25N25N0.98650.98350.94280.94230.92630.92250.91510.9088
50N50N0.94440.93800.91600.90620.90130.88800.89300.8766
100N100N0.90710.90200.88490.87670.87390.86020.86480.8544
200N200N0.88380.87250.86370.84840.85270.83890.85010.8309
400N400N0.86340.85410.85060.83480.84120.82820.83700.8213
271M12.5N12.5N0.90960.91730.86650.86750.85080.84700.83800.8337
25N25N0.86780.86900.83250.82820.81790.81190.80730.8012
50N50N0.83360.83090.80450.80090.79020.78540.78440.7729
100N100N0.80210.79800.77920.77230.76750.75820.76190.7502
200N200N0.77830.77120.76030.74990.74990.73920.74300.7308
350N350N0.76520.75250.74690.73650.73940.72650.73750.7217
778M12.5N12.5N0.76700.77150.73010.73220.71340.71060.70090.6985
50N50N0.69590.69490.66730.6662————
100N100N0.66830.6613——————

Table 2. Downstream tasks at equal model size, unique data and compute. Bits per byte (BPB ↓) of both models at budgets of F=1F=1 to 4 Transformer epochs. The lower of each pair is shaded. BPB averages 37 task suites, each weighted by its inverse seed variance (Appendix § 15).

SizeUUF=1F = 1F=2F = 2F=3F = 3F=4F = 4
TransformerFeedbackTransformerFeedbackTransformerFeedbackTransformerFeedback
60M12.5N12.5N3.01413.05082.90602.91832.86312.86472.84042.8343
50N50N2.81102.81992.74562.74242.71662.70742.69962.6866
100N100N2.74282.73872.69182.68032.66892.65362.65532.6376
400N400N2.65072.63432.61992.59872.60592.58262.59732.5737
134M12.5N12.5N2.69272.72022.60732.61782.56972.57182.54832.5440
25N25N2.59992.61262.53202.53332.50142.49642.48262.4743
50N50N2.52812.53412.47162.46562.44522.43432.42922.4159
100N100N2.46852.46702.42382.41552.40302.39052.39042.3750
200N200N2.42342.41322.38802.37222.37112.35342.36102.3421
400N400N2.38902.37412.36102.34282.34782.32812.33972.3190
271M12.5N12.5N2.46912.49072.39572.40452.36292.36422.34282.3394
25N25N2.38932.39932.32922.33082.30152.29742.28422.2777
50N50N2.32832.33232.27742.27392.25402.24622.23972.2288
100N100N2.27692.27442.23632.22832.21692.20582.20482.1921
200N200N2.23742.22912.20422.19162.18852.17382.17862.1626
350N350N2.21142.19882.18412.16812.17122.15332.16332.1444
778M12.5N12.5N2.20542.22112.14342.15012.11362.11622.09502.0950
50N50N2.08782.08932.04522.0421————
100N100N2.04482.0413——————

Table 6. Validation loss ↓\downarrow of every cell at F=1F = 1 to 4. Both models, at equal size, unique data and budget FF; Feedback is the latent feedback Transformer. The lower of each pair is shaded: purple where latent feedback leads, grey where the Transformer does. 778M is held out of the law; —: its grid has no such cell.

Related work

Spending compute on scarce data. With unique data scarce, prior work fits a scaling law and, given unique data, compute, and a fixed architecture, chooses the model size and number of epochs [12, 22, 27]. When taking architecture into account, sparsity in models delays the point where repeats saturate [45]; masked diffusion [46] overtakes autoregressive training once compute is abundant, across unique data and epochs. However, its gain needs excessive epochs: on 100M unique tokens its best loss comes at 500 epochs, against 50 for autoregressive training, whose hyperparameters follow a prior recipe instead of being tuned per run. The looped model Parcae [47] spends extra FLOPs on loops at a fixed parameter count, yet weighs them against fresh tokens rather than epochs. Latent feedback pays off within four epochs once each parameter has enough unique data, and we price it against epochs on fixed unique data, tuning both models at every size and amount of unique data.

Recurrence in depth. Looped models buy more compute per token by running their layers several times within each token [23, 24, 48], and their scaling laws count the extra loops as effective parameters [25, 26,

[47]. Our law takes the same view: latent feedback enters it as a Transformer with vv times the parameters. However, a looped model still pays its extra compute again at every decoding step, even if it allows KV cache sharing [49]. Latent feedback recurs across tokens instead: it pays for its extra passes in training and in prefill, and generates each token like a Transformer with one more cached vector.

Feedback across tokens. Feedback memory [13] first fed deep states to later tokens but trained token by token; recent models feed back a deeper layer [14, 50, 51, 52] or a frozen teacher’s state [53]. At equal compute or training time, some beat a Transformer [14, 50] and others trail it [51, 53]. Yet none varies data repetition: even Wang et al. [14], who motivate feedback by scarce unique data, give the Transformer more tokens rather than more epochs. We compare against more epochs, across model size, unique data and repetition, and fit all three in one scaling law.

Conclusion

We show that latent feedback makes every FLOP and every token count for more: once each parameter has enough unique data, it reaches a given loss with less compute than more epochs, and with less unique data at equal epochs. Both savings grow with unique data and repetition, and one scaling law, fitted to latent feedback and the Transformer together, predicts them at a held-out size. As a takeaway, when high-quality data, not compute, limits a run of a given model size, feeding back the top-layer state uses compute better than training more epochs.

Limitation. So far, our ladder stops at a relatively small scale and uses a single backbone; larger models and data budgets, as well as other architectures may shift the crossover. We also train latent feedback with a single fixed schedule, and further optimizations are possible. Lastly, a comparison with the recurrent depth models will be of great value. We leave these to future work.

Acknowledgment. We sincerely thank Zitian Gao and Xi Wang for many valuable discussions.

Notation

Table 3 lists the quantities of the main text, and Table 4 the symbols of the fitted law.

QuantityMeaningDefined in
TPPTraining tokens per parameter, repeats included§ 1
UTPPUnique tokens per non-embedding parameter, U/NU/N§ 1
Latent feedback TransformerAlgorithm 1: one or two passes, no gradient in the first§ 2
ppProbability that an update runs one pass§ 2
FFCompute budget, in Transformer epochs§ 3
csc_sTraining FLOPs per token of schedule ss over a Transformer’s§ 3
RREpochs trained within the budget, F/csF/c_s§ 3
F∗F^{*}Crossover budget, above which latent feedback wins§ 3
κ\kappaCompute multiplier, at equal size and unique data§ 3
μ\muData multiplier, at equal size and epochs§ 3

Table 3. Quantities of the main text, in the order they are defined.

Repeated data in practice

Models train far past the compute-optimal ratio (Figure 5). Chinchilla trained on 20 tokens per parameter [6]. We checked 257 base models whose reports state both counts, each against its source. The median dense model trains on about 6,800 tokens per parameter below 1B. It uses 1,040 from 1B to 10B, and 210 from 10B to 100B. The median mixture-of-experts (MoE) model uses 1,640 per active parameter.

Figure 5. Training tokens per parameter of base models. Sources are each developer’s report or model card [2, 7, 8, 9, 10, 11, 15, 16, 17, 33, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68]. Colour gives the total size, and hatched bars are MoE models. MoE bars count active parameters; the tick marks total parameters.

Parallel passes

Each pass feeds position tt only the states of earlier positions, so after pp passes the first pp positions equal decoding, and the fixed point of Equation (2) is unique and equals decoding. More parallel passes than the 4 we evaluate with barely move the loss, far less than the gaps between the two models (Table 5).

Every cell

Table 6 lists the validation loss of every cell at F=1F = 1 to 4, and Table 2 its downstream BPB.

Seeds. At 134M we ran three seeds at U=12.5NU = 12.5N and 100N, for F=1F = 1 to 4. The standard deviation of a cell’s loss is 0.0003 to 0.0014. That of a gap is 0.0004 to 0.0017, and every gap keeps its sign.

Multipliers. Table 7 (top) lists the measured κ\kappa. Each latent feedback cell’s loss is read off the Transformer’s losses at the same size and UU, interpolated in log FF between F=1F = 1 and 4. Table 7 (bottom) lists the measured μ\mu. Latent feedback’s loss at R=1,2,3R = 1, 2, 3 is interpolated in log RR between its branches at R=F/cR = F/c. We then read μ\mu off the Transformer’s losses across UU, interpolated in log UU between levels.

The fitted law

Equation 8 is Equation 3 in its best form (Table 9). It is the coupled form, with latent feedback’s capacity growing with TPP and with size. It is fitted jointly to both models’ 128 cells at 60M, 134M and 271M. Each symbol has one of four kinds (Table 4). Inputs are chosen by whoever uses the fitted law. Fitted coefficients are estimated once, jointly, from the 128 cells. Fixed constants are measured or set by design and never fitted. Black symbols are derived: computed from the other three kinds.

Ls(N,U,F)=E+[(A(vsn)α)q+(Bdsβ)q]1/q,vs=v0(dsn)ωnγ,(8)L_{\mathrm{s}}(N,U,F)=E+\left[\left(\frac{A}{(v_{\mathrm{s}}n)^{\alpha}}\right)^{q}+\left(\frac{B}{d_{\mathrm{s}}^{\beta}}\right)^{q}\right]^{1/q},\qquad v_{\mathrm{s}}=v_{0}\left(\frac{d_{\mathrm{s}}}{n}\right)^{\omega}n^{\gamma}, \tag*{(8)}
Ds=U[1+R∗(1−e−(R−1)/R∗)],R=Fcs,R∗=RT∗(un)ρ,D_{\mathrm{s}}=U\left[1+R^{*}\left(1-e^{-(R-1)/R^{*}}\right)\right],\qquad R=\frac{F}{c_{\mathrm{s}}},\qquad R^{*}=R_{\mathrm{T}}^{*}\left(\frac{u}{n}\right)^{\rho},
n=NN0,u=UD0,ds=DsD0,N0=108,D0=100N0=1010.n=\frac{N}{N_{0}},\qquad u=\frac{U}{D_{0}},\qquad d_{\mathrm{s}}=\frac{D_{\mathrm{s}}}{D_{0}},\qquad N_{0}=10^{8},\qquad D_{0}=100N_{0}=10^{10}.

Below one epoch the bracket is replaced by RR. The Transformer TT has vT=1v_{\mathrm{T}}=1. Every size is measured against one reference run: N0=108N_{0}=10^{8} non-embedding parameters trained on D0=1010D_{0}=10^{10} effective tokens, that is, 100 tokens per parameter. So ds/nd_{\mathrm{s}}/n counts effective tokens per parameter in units of 100, and AA, BB, v0v_{0} and RT∗R_{\mathrm{T}}^{*} are the values of their terms at the reference run. The reference is a choice of units: moving it rescales these four coefficients and changes no prediction.

  • One fit per model family. EE, AA, α\alpha, BB, β\beta and qq belong to one shape ladder, dataset, tokenizer, optimizer and tuning recipe. Every schedule shares them, and a new family refits them. The batch is not fixed: at small UU it follows the optimum of Bergsma et al. [41]. Its β2\beta_{2} then moves to keep its half-life in tokens [69] (Appendix § 14).

  • The count of NN. We count non-embedding parameters in both U/NU/N and the size term [18, 19, 20, 44]. Pearce and Song [70] trace part of the disagreement between Kaplan and Chinchilla to this count. The count matters at small scale: at 60M, embeddings are 55% of the parameters. Counting all parameters, or adding the output head, fits the Transformer cells about as well. The joint coupled RMSE is then 0.0014 against 0.0018, but α\alpha moves to 0.24 or 0.18.

  • The size term. Fitting γ\gamma barely changes the in-sample error, and an F-test of γ\gamma against γ=0\gamma= 0 gives p=0.054p = 0.054. Without γ\gamma, however, the law underrates latent feedback in all seven held-out 778M rows, and its gap error grows 2.5 times. These rows come from three main runs, so they give only three independent signs, not seven. We are training larger models to test the size term further.

  • The loss floor. Three sizes do not pin it: EE sits at its lower bound of 0.5 in every form. Freed from its bound, EE falls to 0 to 0.3 at equal error. The exponent α\alpha then moves from 0.11 to 0.07 to 0.10.

Derivation of the crossover

This appendix derives Equation (5), gives the exact condition for the crossover budget F∗F^*, and shows that the law has at most one crossover.

Setup. Fix NN, UU and FF. The Transformer has v=1v = 1 and trains R=FR = F epochs; its terms are X=A/NαX = A/N^\alpha and Y=B/D(F)βY = B/D(F)^\beta, and its reducible loss is S=(Xq+Yq)1/qS = (X^q + Y^q)^{1/q}. A schedule ss multiplies each term by a factor:

Xs=Xvs−α,Ys=Y(D(F)D(F/cs))β=Ycsεˉβ,εˉ=1ln⁡cs∫F/csFε(R) dln⁡R.(9)X_s = Xv_s^{-\alpha}, \qquad Y_s = Y\left(\frac{D(F)}{D(F/c_s)}\right)^\beta= Yc_s^{\bar{\varepsilon}\beta}, \qquad\bar{\varepsilon} = \frac{1}{\ln c_s}\int_{F/c_s}^{F}\varepsilon(R)\,\mathrm{d}\ln R. \tag*{(9)}

Here ε(R)=dln⁡D/dln⁡R\varepsilon(R) = \mathrm{d}\ln D/\mathrm{d}\ln R is the elasticity of effective data to epochs. So the εˉ\bar{\varepsilon} of Equation (5), with DT=D(F)D_T = D(F) and Ds=D(F/cs)D_s = D(F/c_s), is the mean of this elasticity over the epochs that the schedule gives up.

First order. Because ln⁡S=1qln⁡(Xq+Yq)\ln S = \frac{1}{q}\ln(X^q + Y^q), its derivatives are ∂ln⁡S/∂ln⁡X=w\partial\ln S/\partial\ln X = w and ∂ln⁡S/∂ln⁡Y=1−w\partial\ln S/\partial\ln Y = 1-w, with w=Xq/(Xq+Yq)w = X^q/(X^q + Y^q). So, to first order in ln⁡vs\ln v_s and ln⁡cs\ln c_s,

ln⁡SsS≈−wαln⁡vs+(1−w)εˉβln⁡cs.(10)\ln\frac{S_s}{S} \approx-w\alpha\ln v_s + (1-w)\bar{\varepsilon}\beta\ln c_s. \tag*{(10)}

Equation (5) follows, since Ls−LT=Ss−S≈Sln⁡(Ss/S)L_s - L_T = S_s - S \approx S\ln(S_s/S).

Elasticity. From Equation (3), with x=(R−1)/R∗x = (R-1)/R^*,

ε(R)=Re−x1+R∗(1−e−x) for R≥1,ε(R)=1 for R≤1.(11)\varepsilon(R) = \frac{Re^{-x}}{1+R^*(1-e^{-x})}\ \text{for } R \ge1,\qquad\varepsilon(R)=1\ \text{for } R\le1. \tag*{(11)}

It is 1 at one epoch and falls to 0 as repeats saturate. It never rises on the way: dln⁡ε/dln⁡R=1−R/R∗−ε\mathrm{d}\ln\varepsilon/\mathrm{d}\ln R = 1-R/R^*-\varepsilon, and ε≥e−x≥1−x>1−R/R∗\varepsilon\ge e^{-x} \ge1-x > 1-R/R^*, because the denominator is at most 1+R∗x=R1+R^*x=R.

Exact crossover. The two losses are equal when Xsq+Ysq=Xq+YqX_s^q + Y_s^q = X^q + Y^q. With Equation (9), this is

(XY)q=csqεˉβ−11−vs−qα.(12)\left(\frac{X}{Y}\right)^q = \frac{c_s^{q\bar{\varepsilon}\beta}-1}{1-v_s^{-q\alpha}}. \tag*{(12)}

No approximation enters. To first order, its right side is εˉβln⁡cs/(αln⁡vs)\bar{\varepsilon}\beta\ln c_s/(\alpha\ln v_s): data cost over capacity gain.

One crossover. Hold NN and UU and raise FF. The left side of Equation (12) rises, because D(F)D(F) grows, so YY falls while XX stays. The right side falls: εˉ\bar{\varepsilon} falls because ε\varepsilon does, and vsv_s does not shrink, since ω>0\omega> 0. Therefore the two sides meet at most once, and the schedule loses below F∗F^* and wins above it. Moreover, the right side tends to 0 as repeats saturate while the left side stays positive. So under the law, a costlier schedule with vs>1v_s > 1 wins at a large enough budget.

Data per parameter. More unique data lowers YY at every FF, which raises the left side and moves F∗F^* earlier. However, it also raises R∗R^*, which slows saturation and moves F∗F^* later. In the fitted law the first effect wins: at 134M, F∗F^* falls from 4.1 at U=12.5NU = 12.5N to 1.5 at 50N50N (Figure 7).

Accuracy. Equation (12) reproduces every crossover of Figure 7. Equation (5) stays within 0.002 of the fitted law’s gap for U=12.5NU = 12.5N to 100N100N and F=1F = 1 to 4, always in favour of latent feedback. So its crossover comes 7% to 12% early: 3.8 against 4.1 at 134M and U=12.5NU = 12.5N. The bias arises because the first-order ratio lies below the exact right side of Equation (12).

Model and training configuration

All sizes share one recipe (Table 10). They differ only in shape, learning rate, weight decay, schedule lengths and, at small UU, the batch.

Learning rate. The learning rate η\eta is that of the matrix optimizer. The embedding and the output head train at 0.905η0.905\eta and 0.08η0.08\eta.

Weight decay. It follows a timescale rule τ=raTPPb\tau= r a \mathrm{TPP}^{b} in training tokens per parameter (TPP). The rule has (a,b)=(0.403,−0.483)(a,b) = (0.403,-0.483) for the Transformer and (0.751,−0.577)(0.751,-0.577) for latent feedback. We re-centre rr per arm on the 134M cells.

Tuning grid. Each model gets three learning rates and two or three τ\tau levels per cell (Table 10). We add the next point wherever the best one sits on an edge of the grid. A cell’s value is the minimum of a quadratic fitted to its points if that minimum lies inside the grid, and otherwise the best point.

Decay branches. Each branch keeps its main run’s η\eta and λ\lambda, as in other warmup-stable-decay scaling studies [42]. On an earlier pool we checked this against a separate ladder, whose runs were tuned on their own grids for about two epochs. In seven cells at two sizes and 1 to 2.3 epochs, five of the Transformer and two of latent feedback, the branches match those runs within 0.0012, with a mean difference of 0.0003.

Batch. The batch keeps each cell near the optimum of Bergsma et al. [41]. It stays between 0.46 and 1.72 times that optimum at every size. At 134M and 271M it is 0.25M tokens up to U=25NU = 25N and 0.5M from 50N; 778M uses 0.5M (Table 10). Both models share each cell’s batch; one 60M cell run at both batches moves the gap by 0.002. We set β2\beta_{2} by its half-life in tokens [69].

Choice of pp. On earlier pools, pp from 0.01 to 0.31 changed the loss at equal compute by under 0.003. With p=0p = 0, the passes fell into a two-cycle, so decoding failed.

Downstream suites

Weighting the downstream suites. Table 2 lists the downstream BPB of every cell. A plain mean over the 37 suites is dominated by yes/no and arithmetic suites, whose seed noise reaches 0.24 BPB. So we weight each suite by the inverse of its seed variance. Let br,jb_{r,j} be run rr’s BPB on suite jj, the mean over the suite’s tasks. The seed variance σj2\sigma_j^2 is the variance of br,jb_{r,j} across seeds 0 to 2, averaged over 16 groups of runs at 134M (both models, U=12.5NU = 12.5N and 100N, F=1F = 1 to 4). A run’s downstream score is then

br=∑jwjbr,j∑jwj,wj=1σj2.(13)b_r = \frac{\sum_j w_j b_{r,j}}{\sum_j w_j}, \qquad w_j = \frac{1}{\sigma_j^2}. \tag*{(13)}

Nested pools

Construction. Each raw file of the Dolma 3 sample holds one of 224 source-topic buckets. We keep the first copy of each exact duplicate, a document with the same token sequence. We drop documents that share a paragraph of 13+ words with Paloma’s validation or test sets [37]. These are 0.16% of the documents. We shuffle documents within each bucket. We then merge the buckets so that each advances along the stream at the rate of its token share. Every prefix is then a stratified sample of the stream, so pools differ only in size. From 1B tokens on, a prefix’s bucket mixture lies within 0.003 of the whole stream’s in total variation.

References

  1. [1]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv:2204.02311, 2022.arxiv.org/abs/2204.02311
  2. [2]Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al. Kimi K2: Open agentic intelligence. arXiv:2507.20534, 2025.
  3. [3]Konwoo Kim, Suhas Kotha, Percy Liang, and Tatsunori Hashimoto. Pre-training under infinite compute. arXiv:2509.14786, 2025.arxiv.org/abs/2509.14786
  4. [4]Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobhahn. Will we run out of data? limits of LLM scaling based on human-generated data. arXiv:2211.04325, 2024. Version 2.arxiv.org/abs/2211.04325
  5. [5]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv:2005.14165, 2020.arxiv.org/abs/2005.14165
  6. [6]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  7. [7]Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. StarCoder 2 and the stack v2: The next generation. arXiv:2402.19173, 2024.arxiv.org/abs/2402.19173
  8. [8]Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv:2412.08905, 2024.arxiv.org/abs/2412.08905
  9. [9]Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groen-eveld, Faeze Brahman, Finbarr Timbers, Hamish Visin, et al. Olmo 3. arXiv:2512.13961, 2025.
  10. [10]The Microsoft AI Team. MAI-Thinking-1: Building a hill-climbing machine. https://microsoft.ai/pdf/mai-thinking-1.pdf, 2026. Technical report.
  11. [11]Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. SmolLM2: When smol goes big – data-centric training of a small language model. arXiv:2502.02737, 2025.arxiv.org/abs/2502.02737
  12. [12]Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv:2305.16264, 2023.arxiv.org/abs/2305.16264
  13. [13]Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar. Addressing some limitations of transformers with feedback memory. arXiv:2002.09402, 2020.arxiv.org/abs/2002.09402
  14. [14]Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, and John Langford. Full-bandwidth transformer. arXiv:2608.08888, 2026.arxiv.org/abs/2608.08888
  15. [15]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, et al. The Llama 3 herd of models. arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  16. [16]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen3 technical report. arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  17. [17]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, et al. Gemma 3 technical report. arXiv:2503.19786, 2025.
  18. [18]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
  19. [19]Houyi Li et al. Predictable scale: Part I, Step Law – optimal hyperparameter scaling law in large language model pretraining. arXiv:2503.04715, 2025.arxiv.org/abs/2503.04715
  20. [20]Kimi Team. Kimi Linear: An expressive, efficient attention architecture. arXiv:2510.26692, 2025.
  21. [21]Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz, and Kartik Ahuja. Scaling: Chinchilla’s exponents meet Kaplan’s coupling. arXiv:2608.07222, 2026.arxiv.org/abs/2608.07222
  22. [22]Tian Qin, Kimia Hamidieh, and David Alvarez-Melis. Bridging compute- and data-optimal pretraining. arXiv:2607.25271, 2026.arxiv.org/abs/2607.25271
  23. [23]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019.arxiv.org/abs/1807.03819
  24. [24]Jonas Geiping et al. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv:2502.05171, 2025.arxiv.org/abs/2502.05171
  25. [25]Kristian Schwethelm, Daniel Rückert, et al. How much is one recurrence worth? iso-depth scaling laws for looped language models. arXiv:2604.21106, 2026.arxiv.org/abs/2604.21106
  26. [26]Xuyuan Xiong, Siyu Chen, Zihan Shao, Yue Su, Weiyang Jin, Ping Luo, Zhuoran Yang, and Xihui Liu. Truncate or not? a scaling study of looped transformers with truncated backpropagation. arXiv:2610.26030, 2026.arxiv.org/abs/2610.26030
  27. [27]Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, and Kilian Q. Weinberger. Prescriptive scaling laws for data constrained training. arXiv:2605.01640, 2026.arxiv.org/abs/2605.01640
  28. [28]Joshua Shay Kricheli, Alexander Lawrence Reid, Soumaj yoti Sarkar, Venkata Gandikota, and Paulo Shakarian. Tokens-per-parameter coverage is critical for robust LLM scaling law extrapolation. arXiv:2605.08541, 2026.arxiv.org/abs/2605.08541
  29. [29]Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, 2019.
  30. [30]Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. arXiv:2104.09864, 2021.arxiv.org/abs/2104.09864
  31. [31]Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.arxiv.org/abs/2010.04245
  32. [32]Noam Shazeer. GLU variants improve transformer. arXiv:2002.05202, 2020.arxiv.org/abs/2002.05202
  33. [33]Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groenveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 OLMo 2 furious. arXiv preprint arXiv:2501.00656, 2025.arxiv.org/abs/2501.00656
  34. [34]Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. NorMuon: Making Muon more efficient and scalable. arXiv:2510.05491, 2025.arxiv.org/abs/2510.05491
  35. [35]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024.
  36. [36]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  37. [37]Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groenveld, Iz Beltagy, Hanneane Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. arXiv:2312.10523, 2023.arxiv.org/abs/2312.10523
  38. [38]Luca Soldaini, Rodney Kinney, Akshita Bhagia, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. arXiv:2402.00159, 2024.arxiv.org/abs/2402.00159
  39. [39]Dirk Groenveld, Iz Beltagy, Pete Walsh, et al. OLMo: Accelerating the science of language models. arXiv:2402.00838, 2024.
  40. [40]Xi Wang and Laurence Aitchison. How to set AdamW’s weight decay as you scale model and dataset size. arXiv:2405.13698, 2024.arxiv.org/abs/2405.13698
  41. [41]Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Power lines: Scaling laws for weight decay and batch size in LLM pre-training. arXiv:2505.13738, 2025.arxiv.org/abs/2505.13738
  42. [42]Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv:2405.18392, 2024.arxiv.org/abs/2405.18392
  43. [43]Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hanneane Hajishirzi. OLMES: A standard for language model evaluations. arXiv:2406.08446, 2024.arxiv.org/abs/2406.08446
  44. [44]Houyi Li et al. Predictable scale: Part II, Farseer: A refined scaling law in large language models. arXiv:2506.10972, 2025.arxiv.org/abs/2506.10972
  45. [45]Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, and Decebal Constantin Mocanu. When data is scarce: Scaling sparse language models with repeated training. In International Conference on Machine Learning, 2026. arXiv:2606.01155.arxiv.org/abs/2606.01155
  46. [46]Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffusion beats autoregressive in data-constrained settings. arXiv:2507.15857, 2025.arxiv.org/abs/2507.15857
  47. [47]Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, and Daniel Y. Fu. Parcae: Scaling laws for stable looped language models. arXiv:2604.12946, 2026.arxiv.org/abs/2604.12946
  48. [48]Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, and Xuezhe Ma. Towards looped models done right, part i: Topology, input injection, recurrent-state design. arXiv preprint arXiv:2610.23036, 2026.arxiv.org/abs/2610.23036
  49. [49]Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, and Xuezhe Ma. Towards looped models done right, part ii: Rethinking at fixed points. arXiv preprint arXiv:2610.06833, 2026.arxiv.org/abs/2610.06833
  50. [50]Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, and Yelong Shen. Latent recurrent transformer: Architecture exploration, training strategies, and scaling behavior. arXiv:2605.26797, 2026.arxiv.org/abs/2605.26797
  51. [51]Ziyang Cai, Xingyu Zhu, Yihe Dong, Yinghui He, and Sanjeev Arora. T²MLR: Transformer with temporal middle-layer recurrence. arXiv:2607.15178, 2026.arxiv.org/abs/2607.15178
  52. [52]Michael C Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, and Rosanne Liu. Recirculation. arXiv preprint arXiv:2608.17981, 2026.arxiv.org/abs/2608.17981
  53. [53]Dor Tirosh, Ido Amos, and Mor Geva. Pretraining latent information feedback transformers with teacher supervision. arXiv:2609.38149, 2026.arxiv.org/abs/2609.38149
  54. [54]Qwen Team, An Yang, Baosong Yang, Beichen Zhang, et al. Qwen2.5 technical report. arXiv:2412.15115, 2024.
  55. [55]Meta. Llama 3.2 model card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md, 2024.
  56. [56]Aili Chen, Aonian Li, Baichuan Zhou, Bangwei Gong, Binyang Jiang, et al. The MiniMax-M2 series: Mini activations unleashing max real-world intelligence. arXiv:2605.26494, 2026.arxiv.org/abs/2605.26494
  57. [57]Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLlama: An open-source small language model. arXiv:2401.02385, 2024.arxiv.org/abs/2401.02385
  58. [58]Meta. Llama 3 model card. https://github.com/meta-llama/llama-models/blob/main/models/llama3/MODEL_CARD.md, 2024.
  59. [59]NVIDIA, Aaron Blakeman, Aarti Basant, Abhinav Khattar, et al. Nemotron-H: A family of accurate and efficient hybrid Mamba-Transformer models. arXiv:2504.03624, 2025.
  60. [60]MiniCPM Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, et al. MiniCPM4: Ultra-efficient LLMs on end devices. arXiv:2506.07900, 2025.
  61. [61]Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv:2404.14219, 2024.arxiv.org/abs/2404.14219
  62. [62]Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, et al. Gemma 2: Improving open language models at a practical size. arXiv:2408.00118, 2024.
  63. [63]GLM-4.5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, et al. GLM-4.5: Agentic, reasoning, and coding (ARC) foundation models. arXiv:2508.06471, 2025.
  64. [64]DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, et al. DeepSeek-V3 technical report. arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
  65. [65]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023.arxiv.org/abs/2307.09288
  66. [66]Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv:2304.01373, 2023.arxiv.org/abs/2304.01373
  67. [67]DeepSeek-AI, Xiao Bi, Deli Chen, Guanting Chen, et al. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv:2401.02954, 2024.
  68. [68]Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, et al. The Falcon series of open language models. arXiv:2311.16867, 2023.arxiv.org/abs/2311.16867
  69. [69]Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Goldblum. Small batch size training for language models: When vanilla SGD works, and why gradient accumulation is wasteful. arXiv:2507.07101, 2025.arxiv.org/abs/2507.07101
  70. [70]Tim Pearce and Jinyeop Song. Reconciling Kaplan and Chinchilla scaling laws. arXiv:2406.12907, 2024.arxiv.org/abs/2406.12907

Paper details

vlnc:W_wu7dhj2y
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents