WorksComputer Science
ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories
Abstract
Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.
Introduction
Softmax attention scales poorly to long contexts: its compute grows quadratically with sequence length, while its key–value cache grows linearly. Recurrent language models address both problems by processing the sequence in linear time and carrying a fixed-size state while decoding. Moreover, modern recurrent architectures such as gated linear attention, Gated DeltaNet, and Mamba-2 [61, 63, 11] admit an efficiently parallelizable structure commonly referred to as the chunkwise-parallel form, in which tokens within each chunk are processed in parallel using dense matrix multiplications, while only the recurrent state is propagated sequentially across chunks. Given this scalable structure, the central question for this class of models is how to make the recurrent memory more expressive without sacrificing parallel evaluation.
Test-time training (TTT) addresses the expressivity side of this question by treating the recurrent state as the weights of a small model that learns from the sequence through online gradient descent [49]. With a linear memory and a squared reconstruction loss, this recovers the delta rule of Widrow & Hoff [58], the update used by DeltaNet [46, 62], whereas a nonlinear memory can adapt its features as well as its readout. TTT-MLP, Titans, and
LaCT (Sun et al., [49]; Behrouz et al., [6]; Zhang et al., [64]) explore such neural memories, but their gradients depend nonlinearly on the current weights: each token changes the weights at which the next token’s gradient should be evaluated, creating a sequential dependency. To recover parallelism, existing methods group gradient evaluations into blocks and evaluate every gradient in a block at the weights entering it. This creates a fidelity–speed tradeoff: larger blocks reduce the sequential depth but more coarsely approximate token-by-token learning, while smaller blocks are more faithful but require more sequential steps.
In this paper, we introduce ParaTTT (PTTT), an algorithm that evaluates an MLP-based TTT memory in parallel (Figure 1). The difficulty in parallelizing the recurrence is the inner weight trajectory: the gradient step at each token is taken at weights that depend on every earlier step, so the trajectory cannot be computed without stepping through it sequentially. PTTT instead refines an estimate of this trajectory, represented by the gradients of the inner loss with respect to the hidden-layer pre-activations. Each refinement consists of three parallel passes: a forward pass that constructs the features implied by the current gradient estimate, a readout pass that computes each token’s loss gradient with respect to its features, and a backward pass that propagates these gradients through the hidden layers to form a new estimate. Crucially, each refinement step is a composition of modified instances of existing parallel recurrences such as Gated DeltaNet (Yang et al., [62]; Yang et al., [63]), Mamba-2 (Dao & Gu, [11]), or gated linear attention (Yang et al., [61]). The exact nonlinear sequential recurrence is a fixed point of the iteration.

Figure 1. Parallelism and approximation in nonlinear TTT. Each cell represents a hidden pre-activation gradient , with the layer index omitted. Columns index tokens; downward steps mark computational dependencies, not approximation error or elapsed time. (a) Exact gradients use the latest online weights , including decay. (b) Block-wise TTT, as used by TTT-MLP (Sun et al., [49]) and Titans (Behrouz et al., [6]), evaluates gradients at shared block-entry weights for , omitting earlier within-block updates. (c) ParaTTT refines the full gradient trajectory using three chunkwise-parallel passes per round. The block size and refinement count control the respective approximations; the exact trajectory is a fixed point of the ParaTTT iteration.
Our technical contributions can be summarized as follows:
We introduce ParaTTT (PTTT), an algorithm for parallelizing MLP-based recurrent memories of arbitrary depth, and prove geometric convergence for a two-layer variant under sufficient conditions (§ 3).
We demonstrate PTTT’s superior speed–accuracy trade-off over the block-based evaluation of TTT-MLP: at 4K context, two PTTT refinements match the output error of TTT-MLP’s block approximation while running 32× faster, and full 340M models train 4.9× faster than TTT-MLP (§ 4.2 and § 4.3).
At 340M and 1.3B parameters, PTTT attains lower perplexity and stronger recall on SWDE and FDA than Gated DeltaNet and Mamba-2, showing that the proposed nonlinear memory improves over the evaluated linear recurrent baselines (§ 4.1).
Background
We write for the input sequence of a layer and , , for the per-token query, key, and value projections; the fast-weight state is a matrix or, for a deep memory, a set of weights ; is the chunk size of the parallel kernels. Heads are independent, and we suppress the head index. Section A extends this section.
Linear recurrences
Linear recurrences retain the associative-memory view of attention while replacing its growing key–value cache with a fixed-size state. Linear attention (Katharopoulos et al., [29]; Schlag et al., [46]Yang et al., 2024a) decays the state before the write, $S_t = \alpha_t S_{t-1} + v_t k_t^{\mathsf{T}}$, using a diagonal gate, while Mamba-2 ({citet}Dao & Gu, 2024) is the scalar-gated case. DeltaNet ({citet}Schlag et al., 2021; {citet}Yang et al., 2024b) updates the state by replacing the additive write with one step of gradient descent on $\frac{1}{2}\lVert S k_t - v_t\rVert^2$, $S_t = S_{t-1}(I-\beta_t k_t k_t^{\mathsf{T}})+\beta_t v_t k_t^{\mathsf{T}}$, which erases the value stored along $k_t$ before writing the new one. Gated DeltaNet ({citet}Yang et al., 2025`) combines this delta-rule update with decay. All of these updates are linear in the recurrent state. Consequently, the state at the end of a chunk of tokens is an affine function of the state entering the chunk. This allows tokens within each chunk to be evaluated in parallel with dense matrix multiplications, while only the chunk-boundary states are propagated sequentially with a short scan. This exact evaluation is commonly referred to as the chunkwise-parallel form.
Test-time training
Test-time training (Sun et al., [49]) makes the memory more expressive than a single matrix by interpreting the recurrent state as the parameters of a small model trained online. At token , the model takes a gradient step on the inner loss and produces the layer output . The projections, gates, and learning rates that define this inner problem are trained end-to-end by backpropagating through the inner updates.
TTT-linear. The simplest choice of inner model recovers the delta-rule recurrence described above. With , one gradient step on the squared reconstruction loss is exactly the DeltaNet update, while adding a decay term gives Gated DeltaNet. Because this update remains linear in the recurrent state, it retains the exact chunkwise-parallel form described above. We exploit this identity for the linear readout of a nonlinear memory.
TTT-nonlinear. A nonlinear inner model can adapt the features it reads and writes with, rather than only the readout; this additional expressivity motivates deep memories. TTT-MLP (Sun et al., [49]), for example, uses a two-layer MLP as . Exact online gradient descent evaluates each token’s gradient at the weights produced by all preceding updates, which for a nonlinear memory prevents the chunkwise-parallel rearrangement of the linear case, so TTT-MLP uses a mini-batch (dual) approximation: for a block of tokens starting at position , every gradient is evaluated at the weights entering the block,
The causal partial sum lets tokens within the block be evaluated in parallel, but the gradients are stale: updates from earlier tokens in the block do not change the weights at which later gradients are evaluated. Blocks remain sequential, requiring steps, so larger blocks reduce sequential depth at the cost of a coarser approximation to token-by-token learning. Titans (Behrouz et al., [6]) adds momentum and weight decay under the same block form, while LaCT (Zhang et al., [64]) uses blocks of thousands of tokens together with a sliding-window attention branch for local context.
Other nonlinear recurrent approaches
Sequential nonlinear RNNs. Classic recurrent networks (Elman [13]; Hochreiter & Schmidhuber [23]; Cho et al. [9]) carry a vector state through a nonlinear map. Their standard evaluation is sequential over tokens, limiting parallelism across the sequence. This motivates attention (Vaswani et al. [53]) and parallel linear recurrences, but the choice of transition also affects expressivity. Under their precision and architectural assumptions, Merrill & Sabharwal [35]; Merrill et al. [37] place transformers and certain SSMs in . This rules out general permutation composition if . The xLSTM family (Beck et al. [4]) includes both a sequential sLSTM cell and a parallelizable mLSTM cell. More recently, MRNN (Mishra et al. [38]) uses a nonlinear transition with a matrix-valued state, but pays for it with sequential forward and backward passes.
Parallelizing nonlinear recurrences. DEER (Lim et al. [33]) and ParaRNN (Danieli et al. [10]) cast a nonlinear recurrence as a fixed-point problem over the state trajectory and solve it with Newton iterations, where each iteration reduces to a linear recurrence evaluated by a parallel scan; deep equilibrium models (Bai et al. [3]) share the fixed-point perspective. A generic Newton step uses a transition Jacobian whose size is quadratic in the state dimension; ParaRNN makes this tractable with structured Jacobians. ParaTTT instead iterates on the pre-activation gradient at each token and hidden layer. For hidden width , each gradient is a -dimensional vector, compared with the fast-weight state. Conditioned on these gradients, the fast-weight trajectory becomes a linear recurrence, so each refinement can be evaluated with chunkwise linear recurrent kernels without ever forming the full weight trajectory or its transition Jacobian.
Method
Test-time training with a deep memory
We consider a single head and omit the head index. As in Gated DeltaNet, the input token is mapped to a key, a query and a value, and , by linear projections followed by a short causal convolution; keys and queries are RMS-normalized, . The token also produces a decay gate and, for each layer of the memory defined below, a learning rate . The gate follows Yang et al. [63], each is the sigmoid of a learned linear function of , and all layers of the memory share the decay.
Memory. The recurrent state is the set of weights of an -layer MLP. For given weights , with pre-activations and activations , the memory computes
where is a pointwise nonlinearity and are learned biases. The final linear layer is the readout, and its input defines the features. The features are additionally projected onto the unit ball, , which we absorb into the last . This projection is used in all experiments. Our experiments use , as in Sun et al. [49], but the method applies to any . Each weight is decomposed as , where is a parameter trained by backpropagation and is the recurrent state, initialized to zero at the start of every sequence.
Sequential recurrence. Token first decays the state and then takes one gradient step on the reconstruction loss , with the layer’s learning rate as the step size. For each layer ,
The layer output is : the key is evaluated under the decayed weights, before its own gradient step, and the query after it, exactly as in Gated DeltaNet, whose memory is the linear case of Eq. (1).
Equation (2) is the exact token-by-token recurrence that we seek to evaluate. Computing it directly requires sequential gradient steps because the gradient at each token depends on all preceding updates. The block approximation used by TTT-MLP, described in § 2.2, reduces this sequential depth by evaluating multiple gradients at shared weights, but changes the recurrence. Our goal is instead to parallelize the trajectory while retaining the exact online recurrence as the fixed point.
ParaTTT: Iterated refinement of the inner-loop weight trajectory
We first describe the ParaTTT iteration, which uses a linear attention update for the hidden layers and an uncentered readout. The practical design choices used by our models are introduced in Section 3.4.
The central difficulty in parallelizing the nonlinear MLP recurrence in Eq. (2) is its inner weight trajectory: the gradient step at token is taken at weights that depend on every earlier step, so the trajectory cannot be computed without stepping through it sequentially. ParaTTT instead iteratively refines an estimate of this trajectory, with each refinement parallel across the sequence length. The estimate is represented by a guess of the pre-activation gradients of the hidden layers. Given such a guess, a parallel forward pass evaluates the weight trajectory these gradients define (Step 1), the readout is evaluated exactly on the resulting features (Step 2), and a parallel backward pass propagates the resulting feature gradients through the trajectory to obtain a new guess (Step 3). At convergence, the gradients produced by the backward pass agree with those used in the forward pass (Figure 2).

Figure 2. One iteration of ParaTTT; the recurrences are evaluated in chunkwise-parallel form over all tokens. (1) A parallel forward pass evaluates the hidden layers from the current estimate of their pre-activation gradients. (2) The readout runs exactly on the resulting features and DELTARULEGRAD returns each token’s feature gradient. (3) A parallel backward pass propagates it through the hidden layers using shifted reads of the transposed recurrences (see Step 3 in § 3.2 for details) and updates the estimate. (4) After iterations a final forward pass and readout produce the output.
To see why each pass can be parallelized, write for the activations of under the decayed weights and obtain by backpropagation through Eq. (1) at . The gradient step in Eq. (2) is the outer product of the two, so in terms of the state recurrence reads
Given a guess of the hidden-layer gradients , one vector per token and hidden layer, Eq. (3) is linear in its state with known coefficients. Conditioned on , the nonlinear recurrence therefore reduces to a collection of linear recurrences. For each hidden layer, Eq. (3) is exactly gated linear attention with scalar decay, the state update of Mamba-2 (Yang et al., 2024; Dao & Gu, [11]), with keys , values and gates ; for the readout, where is the readout’s own error, it is a Gated DeltaNet (Yang et al., [63]). Both admit exact chunkwise-parallel kernels. The nonlinearity is thus confined to the self-consistency condition that determines the state trajectory while the resulting trajectory determines . ParaTTT solves this fixed-point problem by iteration from (Figure 2; pseudocode in Algorithm 1, Section C). Since the vectors of enter the scans as values, we also call them pseudo-labels.
Step 1: parallel forward pass. Given the current estimate , the forward pass evaluates the memory MLP at all tokens, each under its own decayed weights, layer by layer and without ever forming the fast weights. Unrolling Eq. (3) for a hidden layer ,
which is gated linear attention (Yang et al., [61]) with keys , values and gates ; the sum excludes because key is read before its own write. We call a scan shifted when token reads the state before its own update, , and unshifted, the standard form, when it reads the state after the update, ; keys are read shifted and queries unshifted. One chunkwise scan thus gives the fast contribution to the pre-activations of layer at all tokens; adding and applying gives the inputs of the next layer. The pass ends with the features and stores the Jacobians for Step 3. When on the first forward pass, the fast scan is identically zero and can be skipped.
Step 2: readout trajectory and its gradient. Unlike the hidden layers, the readout needs no trajectory estimate. Once its input features are fixed, its fast weights follow a Gated DeltaNet with keys , corrected values , gates and step sizes . The chunkwise gated-delta-rule kernel (Yang et al., [62, 63]) computes this recurrence exactly; the slow readout is retained in the feature gradient and added to the final output (Section C.3).
The backward pass in Step 3 additionally requires each token’s loss gradient with respect to its features. We obtain this during the readout pass:
where is the update token writes to the readout. The DELTARULEGRAD kernel computes this quantity as a small modification of the existing GDN kernel (Section C.3).
Step 3: parallel backward pass. The backward pass propagates each token’s feature gradient through the hidden layers along the trajectory from Step 1, for all tokens at once, giving a recomputed gradient trajectory . It requires the transposed Jacobian of and the transposed weights; the only operation involving a fast weight is the transpose of Eq. (4),
a shifted linear-attention read with the pseudo-labels as queries and keys and the layer inputs as values, computed by the same chunkwise scan. The estimate is then updated by a damped Picard step,
Output. After iterations, a final forward pass computes the features of the keys using shifted reads and of the queries using unshifted reads, since each query is evaluated under the updated weights. One gated-delta-rule pass with these features as keys and queries and corrected values produces the fast contribution; adding gives the output. § 3.3 shows that the exact sequential recurrence is a fixed point of this iteration and gives conditions under which the iterates converge to it.
Cost, training and decoding. Each iteration requires a constant number of chunkwise scans per memory layer, with sequential depth for chunk size (Section G). We use unless stated otherwise; § 4.3 shows empirically that two iterations already closely approximate the exact sequential recurrence. All parameters are trained by backpropagation through the iterations. Decoding requires no iteration: with a single new token, Eq. (2) is one exact gradient step from the cached weights, so affects prefill only.
Fixed point and convergence
Proposition 1 (Fixed point and finite-step exactness, informal). For any memory depth , the gradient sequence of the sequential recurrence Eq. (2) is a fixed point of the iteration of § 3.2, and the parallel evaluation at reproduces the sequential recurrence. Without damping (), every iteration makes the estimate at one more token exact, so iterations reproduce the sequential recurrence exactly.
Proposition 1, proved in Section B, needs no contraction assumption. For a geometric rate we analyze the iteration with , the linear-attention hidden update Eq. (4) and the uncentered readout, with the unit-ball projection replaced, for the analysis only, by the soft clip , so that the features , , have a globally Lipschitz Jacobian (Section B restates the setting). Write and for one iteration’s map .
Theorem 1 (Convergence of ParaTTT with soft clipping, ). Assume , , , a shared decay , a shared learning rate , and , . With and , is globally Lipschitz in with constant
If , has a unique fixed point , the gradient sequence of the sequential recurrence Eq. (2) with in place of the projection, and for any initialization and damping the iteration Eq. (7) satisfies ; the features, readout states and outputs converge at the same rate, up to constants.
The proof (Section B) controls the features and their changing Jacobians; SiLU satisfies the derivative assumptions, and is independent of the sequence length. The bound does not cover the hard projection, the hidden delta rule or the centering of § 3.4, so § 4.3 measures the trained models’ convergence directly.
Practical design choices
The analysis above uses the simplest form of the iteration. Our reported models use two design choices that substantially improve practical performance: a hidden-layer delta rule that preserves the target sequential recurrence, and centering, which changes the target to the sequential recurrence with the centered readout.
Delta rule for the hidden layers. Step 1 of the iteration evolves each hidden layer’s fast weights by gated linear attention Eq. (4), which sums the writes with a scalar decay. The same weights can instead be evolved by a gated delta rule whose target is built from the previous iteration’s read. The update is Eq. (3) plus a correction that erases stale writes along the iterates (Section C.2). This correction vanishes at the fixed point, so the target sequential recurrence is unchanged. The delta rule costs slightly more but gives substantially better recall (Table 1), and we therefore use it throughout.
| Model | Wiki | Lamb | PIQA | Hella | ARC-e | Wino | ARC-c | BoolQ | SciQ | SWDE | FDA | TQA | DROP | 4K | 8K | 16K | 32K | |
| 340M, 24 layers, 15.7B tokens | ||||||||||||||||||
| GDN | – | 26.57 | 29.66 | 65.2 | 36.5 | 47.1 | 51.5 | 24.1 | 56.6 | 78.4 | 32.5 | 17.8 | 35.7 | 15.3 | 25.9 | 17.0 | 13.7 | 9.3 |
| Mamba2 | – | 26.99 | 45.87 | 65.7 | 35.1 | 45.5 | 51.9 | 23.4 | 57.8 | 75.2 | 25.8 | 5.7 | 30.6 | 12.6 | 8.1 | 3.0 | 1.1 | 1.2 |
| Transformer | – | 25.34 | 39.25 | 64.3 | 35.8 | 46.3 | 52.8 | 22.4 | 60.8 | 76.7 | 54.6 | 28.4 | 40.5 | 14.5 | 38.6 | 23.2 | 16.3 | 10.5 |
| TTT-MLP | – | 26.12 | 40.35 | 65.2 | 35.1 | 46.3 | 50.8 | 23.3 | 58.5 | 77.2 | 33.1 | 16.2 | 35.2 | 13.9 | 31.5 | 20.4 | 13.0 | 12.4 |
| LaCT (branch only) | – | 463.2 | 53.7 | 25.0 | 28.8 | 50.1 | 24.1 | 60.1 | 21.2 | 0.1 | 0.0 | 0.1 | 7.6 | 0.0 | 0.0 | 0.0 | 0.0 | |
| PTTT (ours) | 2 | 25.46 | 26.93 | 65.3 | 37.7 | 48.6 | 51.2 | 24.1 | 59.8 | 79.0 | 41.9 | 20.3 | 39.6 | 14.4 | 32.6 | 20.7 | 15.6 | 16.0 |
| zero-init readout | 2 | 26.06 | 32.62 | 64.5 | 36.8 | 48.4 | 50.6 | 25.3 | 58.2 | 78.9 | 36.6 | 19.6 | 37.0 | 14.3 | 29.0 | 21.1 | 15.0 | 12.2 |
| linear-attention body | 2 | 25.62 | 25.88 | 65.8 | 37.2 | 48.7 | 53.2 | 23.5 | 61.0 | 78.9 | 28.4 | 12.9 | 33.2 | 11.2 | 7.2 | 5.5 | 4.0 | 2.8 |
| trained at | 5 | 25.47 | 28.44 | 64.9 | 37.6 | 46.2 | 51.4 | 24.5 | 58.2 | 79.3 | 45.0 | 22.0 | 41.9 | 15.6 | 40.1 | 25.9 | 15.9 | 5.8 |
| 1.3B, 24 layers, 100B tokens | ||||||||||||||||||
| GDN | – | 15.78 | 10.56 | 72.2 | 54.9 | 60.0 | 58.9 | 28.1 | 53.9 | 86.7 | 59.0 | 27.0 | 52.9 | 15.2 | 50.8 | 34.1 | 18.8 | 17.5 |
| Mamba2 | – | 15.42 | 11.62 | 73.6 | 53.5 | 58.0 | 54.1 | 28.2 | 53.9 | 87.7 | 56.7 | 21.5 | 54.5 | 14.8 | 33.4 | 19.6 | 11.3 | 4.9 |
| Transformer | – | 15.04 | 10.91 | 71.5 | 53.5 | 58.0 | 56.4 | 29.9 | 57.7 | 88.3 | 75.3 | 31.9 | 53.7 | 15.2 | 78.1 | 56.7 | 43.4 | 17.4 |
| PTTT (ours) | 2 | 15.21 | 9.00 | 71.9 | 56.0 | 61.0 | 58.6 | 29.9 | 61.3 | 89.2 | 62.8 | 28.1 | 53.7 | 15.0 | 53.8 | 37.1 | 17.5 | 14.7 |
Table 1. Main results at 340M and 1.3B. Recall follows the generation protocol of Arora et al. [1]; NIAH is the RULER needle average at the given length. Best per column and size in bold, second best underlined. A couple of notes: (1) LaCT fast-weight branch only is an ablation of the published hybrid. (2) The zero-init readout removes both centering and the learned initialization; Table 6 isolates their effects.
Centering. The hidden layers receive gradient only through the readout. By Eq. (5), is proportional to , so a zero-initialized readout gives the hidden layers no gradient at the start of a sequence. Conversely, a nonzero provides a nonzero gradient immediately, but also adds a persistent bias to the output. We use centering to obtain a nonzero readout Jacobian at initialization while keeping the initial output zero. Centering trains in place of (Chizat et al., [8]); related zero-output initializations appear in Woodworth et al. [59]; Hu et al. [26]. We subtract from the readout its value on the features of the slow MLP, that is, . At the start of a sequence, so the readout is zero, while its Jacobian is nonzero from the first token. Centering changes only the values passed to Step 2 and adds one term to the output (Section C.5).
Other choices. We use with a hidden width of , SiLU as , and a learned per-head gain on the fast part of the hidden pre-activation. Setting the hidden learning rates to zero yields a Gated DeltaNet on the features of the frozen MLP, the control.
Empirical study
Setup. All models share a backbone of 24 pre-norm residual blocks, each consisting of a token-mixing layer followed by a SwiGLU MLP, with at 340M parameters and at 1.3B, and a default head dimension of 128. TTT-MLP and LaCT use the separate head layouts detailed in Section D. The token-mixing layer is ParaTTT, Gated DeltaNet, Mamba-2, attention, TTT-MLP or LaCT. We use short convolutions on the QKV projections, as in Gated DeltaNet (Yang et al., [63]). ParaTTT uses one hidden layer with width equal to the head dimension, together with the hidden-layer delta rule and centered readout of § 3.4, and is trained with iterations unless stated otherwise. We include LaCT’s fast-weight branch alone as an ablation, with its sliding-window attention branch removed; Section F.2 separately evaluates the published hybrid with sliding-window attention and discusses the role of this branch in prior TTT architectures. Section D gives every configuration.
Language modeling performance
Model set and evaluation. At 340M parameters, we compare Gated DeltaNet, Mamba-2, attention, TTT-MLP and ParaTTT, together with a LaCT fast-weight-branch-only ablation, two ParaTTT ablations (zero-init readout and a linear-attention body in place of the hidden-layer delta rule), and a ParaTTT model trained with . At 1.3B parameters, we compare Gated DeltaNet, Mamba-2, attention, and ParaTTT. All models are trained on SlimPajama with the same training recipe, data order, and seed, for 15.7B tokens at 340M and 100B tokens at 1.3B (Section D). We evaluate perplexity on WikiText-103 and LAMBADA, zero-shot accuracy with lm-evaluation-harness (Gao et al., [15]), recall on SWDE, FDA, TriviaQA, and DROP under the generation protocol of Arora et al. [1], and RULER needle-in-a-haystack (Hsieh et al., [24]) from 4K to 32K tokens, using YaRN (Peng et al., [43]) for attention. Section E gives the full evaluation protocol.
Main results. Table 1 gives the language-modeling results. ParaTTT has the best perplexity of the recurrent models at both scales, the best LAMBADA perplexity among the main model comparisons, and stronger recall than both Gated DeltaNet and Mamba-2 on SWDE and FDA at both scales. We note that ParaTTT performs well on recall tasks using exact recurrent gradient-descent decoding despite being trained with the iterated approximation, suggesting that the prefill–decode discrepancy does not prevent strong recall. Attention, unsurprisingly, performs best on many of the recall and long-context retrieval metrics. TTT-MLP, which evaluates its nonlinear memory in blocks at a much higher training cost, matches Gated DeltaNet on WikiText-103 but is far behind on LAMBADA and provides little improvement on recall. Despite the practical design choices from
Section 3.4: the zero-init readout ablation, which removes centering and sets the initial readout to zero, degrades perplexity and recall and greatly reduces the gains from iteration (Figure 3(c,Figure 3d)), while replacing the hidden-layer delta rule with linear attention leaves perplexity similar but substantially degrades recall. Table 6 separately isolates centering from the learned initialization. Training with improves all four recall tasks at a proportionally higher cost per step, but raises LAMBADA perplexity from 26.93 to 28.44 and reduces 32K NIAH accuracy from 16.0 to 5.8. The LaCT fast-weight-branch-only ablation performs substantially worse than the published hybrid; Section F.2 discusses the reliance of prior TTT layers on this branch.
Figure 3. (a,b) Perplexity along 698 held-out SlimPajama documents at 340M and 1.3B. (c,d) LAMBADA perplexity and SWDE accuracy of the 340M ParaTTT models against the number of iterations at evaluation; filled marker: training ; dotted line: Gated DeltaNet; off-chart loop-off values are printed at the edge. The uncentered model uses a zero-initialized readout.
| Readout | Perplexity ↓ | Zero-shot acc. ↑ | Recall ↑ | NIAH ↑ | LongBench ↑ | |||||||||||||||||||
| Wiki | Lamb | PIQA | Hella | ARC-e | Wino | ARC-c | BoolQ | SciQ | SWDE | FDA | TQA | DROP | 4K | 8K | 16K | 32K | 4K | 8K | 16K | 32K | ||||
| Gated DeltaNet | 26.57 | 29.66 | 65.2 | 36.5 | 47.1 | 51.5 | 24.1 | 56.6 | 78.4 | 32.5 | 17.8 | 35.7 | 15.3 | 25.9 | 17.0 | 13.7 | 9.3 | 11.9 | 12.1 | 12.3 | 12.2 | |||
| Zero-init | 26.06 | 32.62 | 64.5 | 36.8 | 48.4 | 50.6 | 25.3 | 58.2 | 78.9 | 36.6 | 19.6 | 37.0 | 14.3 | 29.0 | 21.1 | 15.0 | 12.2 | 12.6 | 12.6 | 12.5 | 12.5 | |||
| Learned init, uncentered | 26.13 | 29.06 | 65.8 | 36.7 | 46.1 | 52.6 | 24.4 | 59.7 | 80.4 | 37.7 | 19.2 | 37.1 | 13.5 | 32.5 | 19.2 | 14.4 | 14.4 | 11.6 | 11.2 | 11.2 | 11.4 | |||
| Centered (PTTT) | 25.46 | 26.93 | 65.3 | 37.7 | 48.6 | 51.2 | 24.1 | 59.8 | 79.0 | 41.9 | 20.3 | 39.6 | 14.4 | 32.6 | 20.7 | 15.6 | 16.0 | 12.4 | 12.5 | 12.4 | 12.1 |
Table 6. Centering ablation at 340M. Three parameterizations of the readout layer, trained with the same recipe, initialization and data order; the gated DeltaNet baseline is shown for reference. Columns as in Table 1; best of the three PTTT variants per column in bold.
Perplexity as a function of token position. Figure 3(a,Figure 3b) follows perplexity along held-out SlimPajama documents. ParaTTT has the lowest perplexity at every position at both scales, and its lead over Gated DeltaNet persists to the end of the 4096-token window, indicating that the gain is not limited to the beginning of the sequence. Attention has the worst cold start and only catches up with the recurrent models after a few hundred tokens.
Performance as a function of the evaluation-time iteration count. Figure 3(c,Figure 3d) evaluates the 340M ParaTTT models at iteration counts other than their training . Without iteration, the centered models perform poorly (LAMBADA perplexity above 300), whereas the zero-init readout ablation performs well but gains little from additional iterations, suggesting that centering is necessary to unlock the full gains from refinement. Models perform best near their training : over-iterating the model trained at degrades performance slowly, while the model trained at exhibits little degradation. This suggests that the model adapts to the finite- prefill used during training, even though exact recurrent decoding remains strong on the recall tasks.
Efficiency
On MI325X GPUs in bf16, we time one token-mixing layer (forward and backward, , 8 heads of dimension 128, batch 8) as a function of sequence length and head dimension, and measure the training throughput of the full 340M models on one node of eight GPUs (Section G). We compare
ParaTTT at with Gated DeltaNet (Yang et al., [63]), Mamba-2 (Dao & Gu, [11]), softmax attention (FlashAttention), TTT-MLP with its default block size of 16 (Sun et al., [49]), and the published LaCT configuration with a 2048-token sliding-window attention branch (Zhang et al., [64]). Figure 4 shows the results.
Figure 4. Efficiency. Forward+backward time of one token-mixing layer against (a) sequence length and (b) head dimension; (c) training throughput of the full 340M models on eight MI325X GPUs. The fused TTT-MLP implementation used in our benchmarks supports head dimension 64; hollow markers show a compiled scan at larger dimensions.
Results. ParaTTT scales linearly in like the linear recurrences it is built from: one layer at costs a GDN layer, and each additional iteration adds a fixed number of chunkwise scans. TTT-MLP’s block-sequential form is slower than ParaTTT at and at . At the model level, ParaTTT at trains at 37.9k tokens/s per node against 7.8k for TTT-MLP, a speedup, while retaining roughly one-third the throughput of the linear baselines.
Convergence analysis
ParaTTT better approximates the exact online recurrence than TTT-MLP. We compare ParaTTT’s -iteration approximation with TTT-MLP’s sequential block approximation at block size on one fixed random inner MLP (, eight heads, constant step size 0.2 and decay 0.98). Figure 5(a) plots each configuration’s output error relative to exact gradient descent against the time required to compute the recurrence, with TTT-MLP timed using either its fused kernel or a compiled version of its scan, whichever is faster. TTT-MLP’s error is determined by the block size and can only be reduced by using smaller blocks and a longer sequential loop, whereas ParaTTT’s error decreases with . At , matches the error of TTT-MLP at in less time, and approaches the bf16 noise floor. Section I repeats the comparison at other sequence lengths.
Figure 5. Convergence to the sequential recurrence. (a) One random inner MLP at : relative output error against exact online GD versus the time to compute one 340M layer’s recurrence, for ParaTTT at – and TTT-MLP’s block form at – (filled: fused kernel; hollow: compiled scan); the arrow marks equal error to . (b,c) Trained models on real text: top-1 agreement and total-variation distance between the prefill at iterations and the exact recurrence.
Trained models converge quickly on real text. For the trained models, we use the exact token-by-token recurrence as the reference. Figure 5(b,c) compares the chunkwise prefill at iterations against this reference using top-1 agreement and total-variation distance between the next-token distributions. No looping gets about 60% agreement, one iteration raises this to roughly 90%, two to about 94%, and further iterations saturate near 98% at the bf16 noise floor. Thus, even for the practical models outside the assumptions of Theorem 1, only a few iterations are needed to closely approximate the exact recurrence.
Conclusion
In this work, we introduce ParaTTT (PTTT), an algorithm for parallelizing test-time training layers with arbitrarily deep memories through iterative refinement of the inner-loop gradient trajectory. We show that the exact online gradient descent recurrence is a fixed point of the iteration and prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, only a few refinement steps are needed: at 4K context, full 340M models train 4.9× faster than TTT-MLP, while models at 340M and 1.3B achieve strong language-modeling performance and stronger recall on most tasks than the linear recurrent baselines. These results narrow the long-standing tradeoff between the expressivity of nonlinear recurrence and the efficiency of parallel sequence processing.
Acknowledgements
This work was supported by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Extended related work
Linear recurrences. The lineage in § 2.1 runs from linear attention [29] through gated linear attention [61], Mamba-2 [11], DeltaNet [46, 62] and Gated DeltaNet [63]; Wang et al. (2025) give the unifying view of all of these as test-time regression. The RWKV family follows the same path from a vector-valued to a matrix-valued state [41] and, in RWKV-7, to a generalized delta rule with vector-valued gates and in-context learning rates [42]. The most recent linear recurrences refine the delta rule’s write and forget. KDA [30] replaces Gated DeltaNet’s scalar decay by a channel-wise one; Gated DeltaNet-2 [22] further decouples the erase gate from the write gate, recovering both KDA and Gated DeltaNet as special cases; Mamba-3 [32] revisits the discretization of the state-space recurrence and adds complex-valued and multi-input multi-output updates; DeltaProduct [48] takes several delta steps per token so that the transition is a product of generalized Householder matrices; Comba [25] adds output feedback to the bilinear state transition; Lattice [28] writes only the component of a token orthogonal to the stored memory slot, with an approximate chunkwise evaluation; Momentum DeltaNet [27] gives the online gradient step a momentum term; Preconditioned DeltaNet [52] preconditions it with an online estimate of the curvature; and MesaNet [54] solves the in-context regression to optimality with a conjugate-gradient solver. These methods retain matrix-valued memories [40] but use different update and evaluation schemes: momentum introduces an auxiliary state, MesaNet adds a per-query solve, and Lattice approximates a state-dependent update. ParaTTT uses chunkwise linear recurrent kernels as building blocks. Extending its readout to these other updates is a possible direction, but would require checking the corresponding solver and gradient computation.
Test-time training. TTT-Linear and TTT-MLP [49] introduced the layer whose state is the weights of a small model trained online, evaluated in the mini-batch form of § 2.2. It is worth writing this form out, since it is the approximation that ParaTTT removes. Let a block consist of tokens and let be the weights entering it (TTT-MLP uses no decay, so we drop ). Exact online descent evaluates the gradient of token at the weights left by , ; the block form evaluates every gradient of the block at the weights that entered it and scores token with the causal partial sum of these stale gradients,
For a squared-loss MLP with a linear readout, the readout update has an attention-style form inside the block. In the next display, denotes the readout weights only. With the features of token , computed with the block-start hidden weights, the gradient at is the outer product of a residual with a feature. Let denote the query features under the causally updated hidden weights at token ; then the output is
an attention-style readout with keys , values the residuals and queries , so the readout sum can be evaluated with matrix multiplications once the features are given. The price is in the residuals: exact online descent would use the preceding readout weights and the current online features, so that the write of token erases what earlier tokens of the block stored under similar keys (the delta rule), whereas the block form’s residuals see none of the block’s earlier writes and the writes simply add up, as in linear attention without a delta rule. The hidden-layer gradients are also evaluated at block-start weights. This freezes the features used to compute those gradients, but does not freeze the hidden weights or query features used for each token’s output. At the block form recovers online descent; at all gradient evaluations share the initial weights. TTT-MLP’s default and LaCT’s larger blocks trade sequential depth against update frequency, and Figure 5(a) measures the resulting error against the exact trajectory. ParaTTT instead targets the recurrence by iterating on the gradients.
Liu et al. [34] study LaCT-style and vision TTT layers. They report that adding inner steps only at inference can hurt downstream performance, that retraining with gradient ascent can preserve performance, and that replacing queries with keys can have little effect in their experiments. Their attention-style representation allows updated hidden features; it is not limited to a frozen-feature model. They also obtain parallel variants through further simplifications, including restricting adaptation to the final layer. These findings motivate testing which parts of nonlinear adaptation matter. Our experiments address this question for the online recurrence, but do not establish that the block approximation explains their observations.
Titans (Behrouz et al., [6]) adds momentum and weight decay to the inner optimizer and ATLAS (Behrouz et al., [5]) optimizes the memory over a window of past tokens rather than the current one; LaCT (Zhang et al., [64]) makes the block very large so that the inner update is a dense matrix multiplication, and pairs it with a sliding-window attention branch (Section F.2). Two recent lines move the fast weights into the backbone. In-place TTT (Feng et al., [14]) treats the down-projection of every MLP block as the fast weights, trains it with an objective aligned with next-token prediction, and updates it chunk-wise; end-to-end TTT (Tandon et al., [50]) keeps a standard transformer with sliding-window attention and adapts selected MLP weights on the context by next-token prediction, meta-learned so that the test-time updates help. In its reported configuration, adaptation is restricted to the last quarter of the blocks. Both inherit the block structure: the weights are frozen inside a chunk or a window and updated between them. ParaTTT is orthogonal to these choices of what to adapt and with which loss; it changes how the inner trajectory of a nonlinear memory is computed, so that every token’s update is taken at the weights left by the previous token, and it does so with the parallel kernels of the linear recurrences above.
Sequential nonlinear RNNs. Nonlinear recurrent networks ([13, 23, 9, 51]) are trained by backpropagation through time (BPTT) ([44, 57]), which unrolls the recurrence over the sequence and runs the backward pass through every step in reverse. Their straightforward forward evaluation and BPTT are sequential over , limiting sequence parallelism compared with attention (Vaswani et al., [53]). The sLSTM cell of xLSTM (Beck et al., [4]) keeps a sequential nonlinear transition next to a parallel linear one. M²RNN (Mishra et al., [38]) is the most direct recent attempt to make the sequential path competitive: it keeps a matrix-valued state, updated by a tanh of a linear map of the previous state plus an outer-product write with a learned forget gate, so that the state size scales as it does in linear attention, and it evaluates the recurrence sequentially in a Triton kernel that parallelizes over batch and heads while looping over time; the backward pass is full BPTT over the 4096-token training sequence, recomputing the forward states and running the reverse loop over every step. The nonlinear transition prevents direct evaluation by the linear scans used by SSMs. The paper reports perfect length generalization on its evaluated state-tracking tasks, with training throughput below its linear recurrent baselines. ParaTTT targets the same expressivity, a nonlinear transition on a large state, but it obtains a parallel evaluation of it, and its nonlinearity acts on the features that the memory reads and writes with rather than on the state transition directly.
Parallelizing nonlinear recurrences and its limits. DEER (Lim et al., [33]) cast the evaluation of a nonlinear recurrence as a fixed-point problem over the whole trajectory and solved it with Newton’s method, each step a linear recurrence evaluated by a parallel scan; ELK and quasi-DEER (Gonzalez et al., [18]) stabilize the Newton iteration through its connection to Kalman smoothing and make it scale by using diagonal Jacobians; ParaRNN (Danieli et al., [10]) designs LSTM and GRU cells with structured Jacobians so that the Newton step is cheap and trains 7B-parameter nonlinear RNNs this way; Gonzalez et al. [19] show that the number of iterations these methods need is governed by the predictability of the dynamics, measured by the largest Lyapunov exponent, and is polylogarithmic in the sequence length for contractive systems. Our Theorem 1 is a statement of the same kind for our iteration: the contraction condition bounds the learning rate times the memory horizon, which is the quantity that controls how far a token’s update can propagate. A small iteration count improves parallelism, but its expressivity requires a separate analysis. An associative scan generally has logarithmic circuit depth; a fixed number of scans alone does not establish a upper bound. The bounds of Merrill & Sabharwal [35]; Merrill et al. [37] apply to the architectures and precision assumptions they analyze. For transformers, Merrill & Sabharwal [36] show that logarithmic depth suffices to recognize regular languages. We do not claim a circuit-complexity classification for ParaTTT. Our finite-step result instead shows that recovers tial recurrence in the undamped case (Proposition 1). The practical question, which § 4.3 addresses, is how many iterations are needed for the tasks that matter.
Looped and implicit models. A second family iterates a block of layers in depth rather than a recurrence in time. Universal transformers (Dehghani et al. [12]) apply the same layer repeatedly with adaptive halting; deep equilibrium models (Bai et al. [3]) take the limit and define the output as the fixed point of a layer, found by a root solver and differentiated implicitly; looped transformers were shown to be programmable computers (Giannou et al. [17]) and to match the reasoning ability of much deeper unlooped models (Saunshi et al. [45]), and recent language models scale test-time compute by unrolling a recurrent block to a chosen depth (Geiping et al. [16]; Zhu et al. [65]). Fixed-point reasoners (Movahedi et al. [39]) use convergence to a fixed point as the halting criterion of a looped transformer and add damping to remove oscillations of the iterates, and Labovich (2026) analyzes when the fixed points of a looped transformer are reachable, input-dependent and stable under backpropagation, and finds that recall connections with outer normalization improve stability and generalization in their studied architectures. Closest to our work, Schöne et al. [47] iterate a state-space model over the whole sequence until convergence to a fixed point and show that the implicit model implements the nonlinear state transition of an RNN; like ParaTTT, they observe that approximate convergence suffices during training, and they scale the approach to 1.3B parameters. The essential difference is at decoding time. In recurrent-depth models, the chosen number of loops is part of the token computation; in fixed-point models, the output is defined by an equilibrium. Both require repeated computation per generated token, including in the sequential mode of Schöne et al. [47]. For ParaTTT the loop is only a method for computing a prefill; the object being computed is the exact sequential recurrence of Eq. (2), which for one new token is a single gradient step from the cached weights. Decoding therefore requires no iteration at all (§ 3.2), and affects the cost and the accuracy of the prefill only.
Parallelizing continuous chain of thought. A third family feeds a hidden state of one decoding step back into the input of the next, a nonlinear recurrence in time whose state is a full residual-stream vector rather than a token. Coconut (Hao et al. [21]) feeds the last hidden state of a “continuous thought” position back as the next input embedding; because each thought depends on the previous one, thoughts cost sequential forward passes at training time, which the authors note is the main obstacle to parallelism. Parallel continuous chain of thought (Wu et al. [60]) removes the sequential dependence with the same tool we use: starting from dummy thought inputs, all thought positions are updated in parallel by nonlinear Jacobi iteration, each pass feeding the previous pass’s hidden states forward by one position, and the iteration is exact after as many passes as there are thoughts (their Theorem 1 is the finite-step exactness of our Proposition 1). They train and decode with a few extra passes (three, for 24 thoughts) and observe that the thought vectors do not in fact converge after training and that training with more than about six passes becomes unstable. Two recent language models apply the same construction to the whole sequence. T2MLR (Cai et al. [7]) fuses a cache of a middle-layer representation from the previous token into an earlier layer of the current token, and the full-bandwidth transformer (Wang et al. [56]) fuses the previous token’s top-layer state with the next token embedding through a gated linear unit. Both train by Jacobi iteration over the sequence, re-running the affected layers in parallel with the previous pass’s states shifted right by one position, and both decode sequentially with one pass per token, as ParaTTT does. They differ from each other in how the iteration is optimized: the full-bandwidth transformer uses at most three passes with gradients through all of them, on a schedule in which most batches use a single pass, whereas T2MLR runs sixteen passes but backpropagates only through the last four, detaching the earlier ones in the manner of truncated backpropagation through time, and reports that this approximation matches exact recurrent training to within 0.005 in validation loss at 135M to 1B parameters. The recurrences in these models run through many layers of a transformer, so each Jacobi pass is a forward pass of the network and the number of passes is kept small. In ParaTTT the recurrence is a gradient step on the weights of one fast-weight layer, the iteration acts on that layer’s pre-activation gradients, and each pass is a chunkwise scan inside one layer’s kernel, so the iteration is cheap enough that we backpropagate through all passes and the analysis of § 3.3 applies to the converged object.
Proofs
Setting, restated. We analyze the iteration with , the linear-attention hidden update Eq. (4), and the uncentered readout. For this analysis only, replace the unit-ball projection used in the experiments with the soft clip . This keeps the features bounded and, under the activation assumptions below, makes their Jacobian globally Lipschitz. Write , where acts coordinatewise. The features of this smooth variant are
Step 3 uses the exact Jacobian at the current pre-activation. Write and for one iteration’s map .
Theorem (Theorem 1, restated). For the soft-clipped variant defined above, assume , , and ; shared decay and shared learning rates ; and with and for every . Set and . Then is globally Lipschitz in with constant
For , . If , has a unique fixed point , the gradient sequence of the sequential recurrence Eq. (2) with in place of the projection. For any initialization and damping , Eq. (7) converges geometrically:
The features, readout states and outputs converge to those of the sequential recurrence at the same rate, up to constant factors.
Proof strategy. We compare two guesses of the hidden-gradient sequence and follow their differences through one iteration. First, the hidden scan turns the guesses into pre-activations and features. Next, these features determine the readout trajectory and its feature gradients. Finally, the backward pass multiplies each feature gradient by the activation Jacobian. Both factors in this last product change between the two guesses, so we must control both changes. The lemmas below give these bounds in the order they are needed.
Notation. We use the Euclidean norm for vectors and the operator norm for matrices. For a sequence, is the maximum of the vector norms over tokens. There is one hidden layer, so we omit its layer index and write . A prime denotes the quantity obtained from a second guess , with the keys, values and model parameters held fixed.
Past writes are multiplied by the decay factors
The assumption bounds their total contribution, independently of sequence length:
We will use this geometric-series bound for both the hidden and readout states. For brevity, write throughout the proof.
Lemma 1 (Soft clip). For , the output and Jacobian satisfy
Proof. The output bound follows directly from the definition. To bound the derivatives, set and . Differentiation gives
Along , this matrix has eigenvalue ; in directions perpendicular to , it has eigenvalue . At it is the identity. Thus , which also shows that is nonexpansive.
It remains to bound the change in the Jacobian. In a direction , the derivative of the scalar factor has magnitude at most . The derivative of has norm at most . Applying the product rule therefore gives
Here we used . Integrating this bound along the segment from to proves the final inequality.
Lemma 2 (Composite activation). The full activation satisfies
Proof. The chain rule factors the Jacobian into the derivatives of the soft clip and the coordinatewise activation:
Since and , their product has norm at most . For its variation, the activation assumptions and Lemma 1 give
Now split the difference of the products as . The first term contributes at most , and the second contributes at most . Adding them proves the claim.
Lemma 3 (Features). For any two gradient guesses, let . Then
Proof. Only the fast weights depend on the gradient guess. Their contribution to the pre-activation is, by Eq. (4),
The normalized keys satisfy . Subtracting the two sums and using gives
Applying the -Lipschitz map gives the feature-difference bound. The bound follows from the soft clip.
Lemma 4 (Readout trajectory). Let be the readout’s fast weights for features satisfying . Then
For two such feature sequences, writing , we also have
Proof. Separate each readout update into the part carried from the previous state and the new write:
We first bound the state itself, then compare the states generated by different features. State size. Because , the matrix has operator norm at most one. The new write has norm at most . Hence
Starting from zero and summing the geometric series gives . Adding the slow weights then gives
Sensitivity to the features. Write . Two elementary bounds control the feature-dependent terms in Eq. (11):
The first follows by changing one factor of the outer product at a time. For the second, the value term contributes and the quadratic feature term contributes . Subtracting the two recurrences now gives
In the last line we used the state-size bound and . Since , another geometric sum yields .
Lemma 5 (Feature gradient). For features in the unit ball, let
Its size and sensitivity to the feature sequence satisfy
where
Proof. Fix a token and abbreviate its decayed readout by and its residual by . Set . By Lemma 4,
Therefore , using .
For the difference bound, there are two sources of change: the readout weights and the features at this token. Writing and , we have
Splitting the gradient difference in the same way gives
The readout lemma supplies . Substituting this and yields
Proof of Theorem 1. Bounding one iteration. Let . The backward pass returns , where . The preceding lemmas give the four bounds needed to compare these products:
Here bounds the sensitivity of the feature gradient and bounds its size, as defined in Lemma 5. The change in the gradient contributes the first term below; the change in the Jacobian contributes the second:
Substituting and and collecting terms gives exactly the constant in Eq. (8). Taking the maximum over tokens proves . The simpler bound for follows from
Identifying the limit. If , the map is a contraction on the complete space . Banach’s theorem therefore gives a unique fixed point . At that point, every guessed gradient equals the actual gradient returned by the backward pass. This identifies the resulting trajectory with the sequential recurrence using the soft clip: both start from zero fast weights; if their states agree before a token, their features and gradients agree at that token, so their updates give the same next states. Induction over tokens proves agreement of the whole trajectory.
Damping and the convergence rate. The damped map is . It has the same fixed point and satisfies
Applying this inequality repeatedly with gives the stated geometric rate.
Features, states and outputs. The feature and readout lemmas show that these quantities change by at most a constant times the error in . The same is true for query-side features: their unshifted hidden scan has gain at most , followed by the -Lipschitz map . Finally, the output is , a product of bounded quantities that are Lipschitz in . Thus their errors all inherit the same geometric rate, up to constant factors.
SiLU satisfies the assumptions. For with , and . Since and , valid global bounds are and .
Fixed point and finite-step exactness. This is the formal version of Proposition 1.
Proposition 2 (Finite-step exactness of the undamped iteration). For the iteration of § 3.2 with and any initialization, the estimated hidden gradients at tokens equal their sequential values after iterations. Consequently, for , the final evaluation reproduces the sequential recurrence exactly in exact arithmetic. The argument applies at any memory depth, with either projection or soft clipping, using the same derivative convention in the parallel and sequential evaluations, and requires no contraction condition. For , finite-step exactness fails in general: already at token , .
Proof. Group all hidden-layer gradient estimates at token into . The iteration is strictly causal: depends only on . To see this at arbitrary depth, proceed layer by layer in the forward pass. At token , each hidden scan uses only writes , and its layer inputs depend only on estimates at tokens preceding their own positions. Thus all key-side features through token depend only on . The readout before token is determined by earlier features and updates. The backward pass uses these same hidden and readout states, so it introduces no dependence on or later estimates.
The sequential gradient sequence satisfies : at the shifted scans of Step 1 sum the writes of tokens and reproduce the decayed weights of Eq. (2) at every token, Step 2 reproduces , and Step 3 is the backpropagation of the sequential model, so ; the parallel evaluation at therefore coincides with the sequential recurrence, and damping changes the iterates but not this fixed point. Since has no dependence on , the first undamped iteration gives . Suppose the first tokens are exact after iterations. For every , all arguments of then agree with those of , hence . Induction proves the prefix claim and gives whenever . The final forward and readout passes therefore reproduce the sequential states and outputs. The argument uses only causality and identical local derivative evaluations, so it applies to either clipping choice; at a nondifferentiable point of the hard projection, both evaluations must use the same fixed derivative convention. □
Algorithm details
Algorithm 1 ParaTTT with linear-attention hidden updates and an uncentered readout (one head; all operations act on the full sequence of tokens). Practical modifications are in § 3.4.
Require: keys , queries , values ; gates , learning rates ; weights , ; iterations ; damping
for every hidden layer ▷ initial estimate of the backward gradients
for do
Step 1: parallel forward pass
for do
; store the Jacobian
end for
Step 2: readout trajectory and its gradient
▷ Eq. (5); includes slow offset
Step 3: parallel backward pass
for do
if then
end if
end for
for every hidden layer
end for
Output
features of from a final Step 1 (shifted); features of from the same scans, unshifted
return
Shifted and unshifted scans
Every scan in Algorithm 1 is one of two forms of the same recurrence, which differ only in whether token reads the state before or after its own write. For a gated linear attention with keys , values and gates , the state is , and the two reads of a query are
The unshifted read is the standard one, in which the sum includes the current token ; the shifted read excludes it. We need both because the sequential recurrence Eq. (2) evaluates everything about the key of token at the decayed weights , before its own gradient step, and the query at , after it. The hidden-layer pre-activation of key , Eq. (4), is therefore a shifted read with and ; the transposed read of Step 3, Eq. (6), is a shifted read with the roles of keys and values exchanged; DELTARULEGRAD reads the readout state before token ’s write, Eq. (21), which is again shifted; and the final output pass reads the query-side features under the updated weights, which is unshifted. The shift is also what makes the fixed-point argument of § 3.3 work: with a shifted read no token’s update depends on its own write, so all updates can be evaluated simultaneously from an estimate of ; with an unshifted read the update of token would depend on itself and the parallel evaluation would not reproduce the sequential recurrence. The use of shifted kernels for keys is consistent with standard formulation of online gradient descent used in GDN and TTT-MLP, where the gradient (on the keys) is evaluated after applying the decay (shifted), and the readout from the query is applied using the updated state (unshifted).
In the chunkwise form the difference is confined to the intra-chunk term. Writing the chunk’s queries, keys and values as matrices and for the matrix of decay ratios, the two reads are
where is the strictly lower-triangular part. The boundary state contains only tokens of earlier chunks, so the inter-chunk term is identical in both forms; only the causal mask of the intra-chunk attention changes, from to . This is how the shifted kernels are implemented: we copied FLA’s chunkwise gated-linear-attention and gated-delta-rule kernels and changed the intra-chunk mask in the forward output kernel and its two backward kernels (for the query/key gradients and for the value gradient) from a causal to a strictly causal comparison of the token indices, leaving the inter-chunk state recurrence, the WY solve and the chunk-boundary bookkeeping untouched. The same one-line change applied to the delta-rule kernel’s final read gives the shifted delta-rule scan used for the hidden layer in Section C.2 and, with the query and value operands exchanged, DELTARULEGRAD. At decode time () the shifted read is the read of the cached state before the token’s update, so no special case is needed.
Algorithm 1 is implemented with three chunkwise kernels of the FLA library and their transposes: the gated-linear-attention scan for the hidden layers (Eq. (4)) and its transpose (Eq. (6)), the gated-delta-rule scan for the hidden layer in the delta-rule form of Eq. (14), and the gated-delta-rule kernel with the query and value operands of its final read exchanged for DELTARULEGRAD (Section C.3). The shift is implemented inside the chunk: the intra-chunk attention is strictly lower-triangular for the key-side read and lower-triangular including the diagonal for the query-side read, and the inter-chunk state is the same for both. Every iteration recomputes the hidden-layer scans from the current pseudo-labels. In the reported one-hidden-layer configuration, the slow hidden contribution and the iteration-0 features are computed once. For deeper memories, the inputs of subsequent hidden layers change with the iteration, so their slow contributions are recomputed. Training backpropagates through all iterations with the kernels’ chunkwise backward passes; iteration stores the pseudo-labels, the features and the Jacobians of , and the readout states at chunk boundaries are recomputed in the backward pass rather than stored.
Delta rule for the hidden layers
Step 1 evolves each hidden layer’s fast weights by the gated linear attention Eq. (4), which sums the writes without ever erasing. The same weights can be evolved by a gated delta rule. With the unit key and write rate , the delta rule is
where is the read of the key from the decayed weights and the target is built from , the same read taken on the trajectory of the previous iteration. Substituting ,
which is Eq. (3) plus a correction that erases the part of the previous iteration’s read that the current trajectory does not reproduce. At the fixed point and the correction vanishes, so the two forms agree there; along the iterates the delta-rule form erases stale writes rather than accumulating them. It is computed with a chunkwise gated-delta-rule kernel; Table 1 compares the two.
The readout gradient
Following the notation of Yang et al. [62]; Tumma et al. [52], write for the fast part of the readout, which decays as , and keep the slow readout as a fixed offset, so that the weight token reads is . (Equivalently, the full state decays toward : .) With the corrected values (or their centered form, Section C.5) and the value that token writes, the fast readout’s recurrence is , and token ’s inner loss and the gradient that Step 3 needs are
The chunkwise gated-delta-rule kernel of Yang et al. [62, 63], as implemented in FLA, already forms every quantity on the right: the chunk-boundary states, the intra-chunk decays and the corrected values of a chunk, which it obtains jointly from the WY representation. DELTARULEGRAD therefore reuses that kernel and changes only its final read, exchanging the roles of the queries and the values; its cost equals one delta-rule pass. The chunkwise form follows.
Chunkwise computation of the readout gradient
We use the chunk notation of Yang et al. [62, 63]; Tumma et al. [52]. Partition the sequence into chunks of tokens and, for a chunk, stack its features, values, step sizes and outputs as , , and , with the readout state at the chunk’s start and the cumulative decay within the chunk. Write , , , and . In this transposed convention the gated delta rule advances the state by , and the transitions of a chunk, applied in order from the left, have the WY form with
a triangular solve against the Gram matrix of the chunk’s features. Including the decay gives the solve
The rows of the corrected values
are exactly the writes of Section C.3, expressed without the token-by-token state, and the state entering token of the chunk is
The kernel computes, per chunk, , , the next boundary state (the inter-chunk recurrence, a short scan over chunks), and the output
where the tril mask is causal; this is Eq. (19) substituted into (the slow term is added outside the kernel), a product with the boundary state plus a causal intra-chunk attention with scores and values .
The feature gradient of Eq. (16) substitutes the same Eq. (19) and gives, per token and per chunk,
a product with the boundary state plus a causal intra-chunk attention in which the corrected values act as both queries and keys and the features as values. Comparing Eqs. (20) and (21): the two expressions share , , and , and differ only in which operands enter the final read, for the output against for the gradient, and in the strict lower-triangular mask, which excludes and implements the shifted arrangement. DELTARULEGRAD is therefore the gated-delta-rule kernel with the query and value operands of its final read exchanged, with the offset entering its final read only; its cost equals that of one delta-rule pass. Because depends on the features, which change with every iteration, it is recomputed in each of the iterations and in the final scans. In the reported one-hidden-layer configuration, the hidden delta-rule solve (Section C.2) is computed once because its keys are the fixed inputs . In deeper memories, this reuse applies only to the first hidden layer; subsequent hidden-layer inputs change with the iteration. The transposed reads of Step 3, Eq. (6), have the same structure with plain linear attention: values in place of , and no triangular solve.
Centering
The centered readout of § 3.4 is
the fast readout applied to the features plus the slow readout applied to their drift from the slow features. With it, the inner loss of token is . Writing for the corrected values, this is the uncentered loss with replaced by , so the fast readout follows the same gated delta rule as before on the corrected values, and its trajectory is computed by the same kernel. The feature gradient of Eq. (5) becomes
which DELTARULEGRAD returns when its readout weight is offset by ; the offset enters the kernel’s final read only. The output is , one gated-delta-rule pass on the corrected values plus a matrix product. The slow features of the keys are exactly those produced by iteration 0, where , so they cost nothing extra; the slow features of the queries need one additional pass through the slow MLP. The fixed point of § 3.3 is unchanged in form: it is the gradient sequence of the sequential recurrence with the centered readout. is trained by backpropagation like every other slow parameter; because the forward pass depends on it through , its gradient is a true gradient and no stop-gradient is involved.
Model configurations and training recipe
The reported experiments use the hard unit-ball projection specified below. The soft clip is used only for the theoretical analysis in Sections B and 3.3.
All models share one backbone: a stack of pre-norm residual blocks, each a token-mixing layer followed by a SwiGLU MLP, with RMSNorm and tied embeddings, 24 layers at both scales, at 340M and at 1.3B. The default head layouts are 8 heads of dimension 128 at 340M and 16 heads of dimension 128 at 1.3B; TTT-MLP and LaCT use the separate layouts below. The token-mixing layer is ParaTTT, Gated DeltaNet, Mamba-2, softmax attention with RoPE, or TTT-MLP; the recurrent layers apply a short causal convolution of width 4 to the query, key and value projections and an output gate. Mamba-2 uses expansion 1 and 4 groups, TTT-MLP its published layout of 16 heads of dimension 64 with a block of 16 tokens, and LaCT its published recipe with a chunk of 2048 tokens, without the sliding-window branch, so that it is a pure fast-weight model. ParaTTT uses the memory of § 3.1 with one hidden layer of width and SiLU, the delta rule for the hidden layer and centering (§ 3.4), a per-token learning rate that is the sigmoid of a linear function of the input, the decay gate of Gated DeltaNet shared by both layers, damping , and iterations during training unless stated. Every ParaTTT model starts from the initialization of the Gated DeltaNet model of the same size, so that the two differ only in the memory. Table 2 lists the models of Table 1. All models are trained with FLAME on SlimPajama (parquet shards of the 100B-token subset) with the Llama tokenizer, sequence length 4096, 8 sequences per GPU with two accumulation steps on 8 GPUs (0.5M tokens per step), AdamW (, , weight decay 0.01), peak learning rate with 1024 warmup steps and cosine decay to , gradient clipping at norm 1, bf16, seed 42 and the same data order. The 340M models train for 30k steps and the 1.3B models for 190k steps; the 1.3B ParaTTT model trains on two nodes with a sharding that reproduces the one-node data order. Every ParaTTT model is initialized from the Gated DeltaNet initialization of the same size (the slow weights of the memory are new parameters, initialized with the standard scheme and zero fast weights), so that at initialization the two models differ only in the memory.
| Model | Params | Mixing layer | Memory / state | Notes |
| 340M, 24 layers, | ||||
| Gated DeltaNet | 342M | 8 heads 128 | 128 128 per head | conv 4, output gate |
| Mamba-2 | 342M | 8 heads 128, expand 1, 4 groups | 128 128 per head | scalar decay |
| Transformer | 341M | 8 heads 128, RoPE | KV cache | YaRN for RULER |
| TTT-MLP | 349M | 16 heads 64, block 16 | MLP 64 256 64 per head | inner LR base 0.1 |
| LaCT fast-weight branch only | 340M | 4 heads, chunk 2048 | SwiGLU fast weights | Muon updates, conv 4 |
| ParaTTT | 348M | 8 heads 128 | MLP 128 128 128 per head | , , delta-rule body, centered |
| 1.3B, 24 layers, | ||||
| Gated DeltaNet | 1.30B | 16 heads 128 | ||
| Mamba-2 | 1.30B | 16 heads 128, expand 1, 4 groups | ||
| Transformer | 1.30B | 16 heads 128, RoPE | ||
| ParaTTT | 1.31B | 16 heads 128 | MLP 128 128 128 per head |
Table 2. Models of Table 1. Parameter counts include embeddings. The ParaTTT memory has a hidden layer of width with SiLU, a bias, the unit-ball projection on the features, and a learned per-head gain on the fast pre-activation; its learning rates are per head and layer, its decay is the Gated DeltaNet gate shared by both layers.
Evaluation protocol
Perplexity and zero-shot. WikiText-103 word perplexity and LAMBADA (OpenAI) perplexity, and accuracy on PIQA, HellaSwag (normalized), ARC-easy, ARC-challenge (normalized), Wino-Grande, BoolQ and SciQ, all zero-shot with lm-evaluation-harness (Gao et al. [15]). ParaTTT models are evaluated at their training with the chunkwise prefill.
Recall. SWDE, FDA, TriviaQA (the cloze version of Arora et al. [2]) and DROP are generation tasks: the model greedily decodes an answer after the prompt and the score is whether the gold answer appears in the generation, case-insensitively, on the full evaluation sets. ParaTTT prefills the prompt with the chunkwise iteration at its training and decodes token by token with the exact recurrence; TTT-MLP decodes with a block-faithful cache at batch 1; the other models use their standard cached decoding.
RULER. The eight needle-in-a-haystack tasks of RULER [24] (single 1–3, multikey 1–3, multiquery, multivalue) at 4K, 8K, 16K and 32K tokens with 100 samples per task; the table reports the average. Attention models use YaRN [43] with a scale of the evaluation length over 4096.
Synthetic tasks. Two-layer models of width 256 (head dimension 128) with the same backbone as the language models, trained from scratch for 4k steps with AdamW, batch 128, over learning rates and three seeds; the long-horizon settings (parity at 512 and 1024, MQAR at 512 with 32 pairs) are also trained for 8k steps at , and MQAR with 64 pairs at width 512. Figure 6 reports, per model and setting, the learning rate (and budget) with the best mean final accuracy over the three seeds. ParaTTT models train and evaluate at (or ).
Figure 6. Synthetic nonlinear-state tasks (two layers, width 256 except width 512 for MQAR with 64 pairs; mean over three seeds at the best learning rate). (a) Cumulative parity against sequence length; the dotted line is chance. (b) -hop pointer following at . (c) Multi-query associative recall for increasing sequence length and number of key-value pairs.
Per-position perplexity. 698 held-out SlimPajama documents (validation and test splits) of at least 4097 tokens, each contributing its first 4096 tokens, so that every position is the same number of tokens into a document; the mean NLL per position is binned in 27 log-spaced bins and reported as perplexity with one standard error.
Additional results
Synthetic nonlinear-state tasks
Three synthetic tasks isolate what a nonlinear memory adds, each learned from scratch by two-layer models of width 256 (width 512 for MQAR with 64 pairs) with the backbone of Section D (protocol in Section E; Figure 6). Cumulative parity asks, at every position, for the parity of the number of ones seen so far; k-hop asks for the token reached by following pointer links back through the sequence; multi-query associative recall (MQAR) asks for the values of previously stored keys.
Parity probes whether a memory can maintain a state that flips with selected inputs. It is not a universal separation between nonlinear and linear memories: negative transition eigenvalues allow linear RNNs to solve it. The relevant limitation concerns finite-precision linear RNNs restricted to nonnegative real eigenvalues at arbitrary sequence lengths [20]. Our finite-length experiments test the architectures and training settings used here. In Figure 6(a), Gated DeltaNet and Mamba-2 fall from 74–77% at length 128 to 63% at 256 and 52–55% at 1024, approaching chance, and the two-layer attention model stays below 63% at every length, whereas ParaTTT reaches 96–100% at lengths up to 256, 79–90% at 512 and 78–80% at 1024, with the model trained at best at every length. Turning the loop off () drops ParaTTT to chance, indicating that the iteration is important in this setup. TTT-MLP sits between the two groups (87% at length 128, 58% at 1024). Its block form evaluates gradients at the weights entering each 16-token block, although each token’s readout incorporates preceding within-block updates [49]. The frozen gradient evaluations differ from the online recurrence that ParaTTT targets; these results alone do not establish that this difference causes the parity gap.
The other two tasks are solvable by a linear memory and serve as controls. On -hop (Figure 6b) the recurrent models with an erasing update, ParaTTT and Gated DeltaNet, solve every , Mamba-2 breaks at , and the two-layer attention model cannot compose hops at this depth. MQAR (Figure 6c) is a capacity test: Gated DeltaNet and ParaTTT solve it up to 32 pairs, with ParaTTT more seed-sensitive at length 1024 and the model the more reliable of the two, and only attention solves 64 pairs, which exceeds the empirical capacity of the recurrent models in this setting. ParaTTT thus keeps the linear recurrences’ recall while achieving stronger state-tracking performance in these experiments.
Evaluation at other iteration counts. Table 3 gives the full sweeps behind Figure 3(c,Figure 3d), including the 1.3B model, on WikiText-103, LAMBADA and SWDE.
| Model | Metric | 1 | 2 | 3 | 5 | 7 | 10 | |
| 340M, trained at | WikiText-103 | 61.3 | 25.76 | 25.46 | 25.54 | 25.67 | 25.75 | 25.87 |
| LAMBADA | 332 | 28.43 | 26.93 | 27.37 | 28.21 | 28.40 | 28.60 | |
| SWDE | 7.4 | 39.3 | 41.9 | 40.8 | 39.4 | 39.6 | 38.7 | |
| 340M, trained at | WikiText-103 | 61.7 | 26.29 | 25.63 | 25.51 | 25.47 | 25.49 | 25.52 |
| LAMBADA | 375 | 31.60 | 29.31 | 28.69 | 28.44 | 28.52 | 28.72 | |
| SWDE | 5.0 | 33.8 | 43.1 | 44.0 | 45.0 | 44.2 | 43.8 | |
| 340M, zero-init readout | WikiText-103 | 26.28 | 26.09 | 26.06 | 26.06 | 26.08 | 26.09 | 26.12 |
| LAMBADA | 33.84 | 32.97 | 32.62 | 32.40 | 32.14 | 32.05 | 32.01 | |
| SWDE | 34.5 | 36.7 | 36.6 | 36.8 | 36.2 | 35.8 | 35.8 | |
| 340M, linear-attention body | WikiText-103 | 55.4 | 26.43 | 25.62 | 25.75 | 26.00 | 25.74 | 25.64 |
| LAMBADA | 179 | 26.76 | 25.88 | 26.08 | 26.11 | 26.13 | 26.15 | |
| SWDE | 7.7 | 29.4 | 28.4 | 28.7 | 28.9 | 28.7 | 27.7 | |
| 1.3B, trained at | WikiText-103 | 73.1 | 15.36 | 15.21 | 15.32 | 15.49 | 15.58 | 15.70 |
| LAMBADA | 348 | 8.94 | 9.00 | 9.40 | 9.91 | 10.16 | 10.38 | |
| SWDE | 6.8 | 62.4 | 62.8 | 58.2 | 53.6 | 48.9 | 44.4 |
Table 3. Perplexity (WikiText-103, LAMBADA) and SWDE accuracy of the ParaTTT models as a function of the number of iterations at evaluation. Bold: best for the centered models.
LongBench. Table 4 reports LongBench at four context lengths for every model in Table 1.
| Model | 4K | 8K | 16K | 32K |
| 340M, 24 layers, 15.7B tokens | ||||
| GDN | 11.9 | 12.1 | 12.3 | 12.2 |
| Mamba2 | 11.8 | 11.9 | 11.9 | 11.9 |
| Transformer | 10.9 | 11.5 | 10.9 | 10.2 |
| TTT-MLP | 9.8 | 9.0 | 8.1 | 8.1 |
| LaCT fast-weight branch only | 4.9 | 4.5 | 4.2 | 4.2 |
| PTTT (ours) | 12.4 | 12.5 | 12.4 | 12.1 |
| zero-init readout | 12.6 | 12.6 | 12.5 | 12.5 |
| linear-attention body | 9.5 | 9.6 | 9.3 | 9.7 |
| trained at | 12.7 | 12.7 | 12.5 | 12.4 |
| 1.3B, 24 layers, 100B tokens | ||||
| GDN | 17.4 | 17.7 | 17.8 | 17.7 |
| Mamba2 | 15.1 | 15.1 | 15.1 | 15.2 |
| Transformer | 19.3 | 20.8 | 21.0 | 19.2 |
| PTTT (ours) | 16.4 | 16.9 | 17.2 | 17.3 |
Table 4. LongBench: average score over the 16 English tasks (100 samples per task) at context lengths 4K–32K, for the models of Table 1. Best per column and size in bold, second best underlined. All models trained at 4K; ParaTTT is within a point of Gated DeltaNet at every length and scale, and attention leads at 1.3B.
Prior TTT layers rely on a sliding-window attention branch
The published TTT architectures that scale beyond the original TTT-Linear and TTT-MLP do not evaluate their memory alone. LaCT (Zhang et al., [64]) pairs its large-chunk fast-weight update with a sliding-window attention branch in every layer and reports only the combination; Titans (Behrouz et al., [6]) reports its long-context results for the hybrids that add attention to the memory (MAC, MAG and MAL). A window of 2048 tokens covers half of a 4096-token training context, so the attention branch alone already accounts for local modeling, and the fast-weight memory only has to contribute beyond the window.
Table 5 measures how much of LaCT’s performance comes from this branch, at 340M with the training recipe of Section D. With its sliding window LaCT is a competitive language model. The fast-weight-branch-only ablation, trained with the sliding window removed, with the same chunk size of 2048, yields a model that is essentially context-free: WikiText-103 perplexity in the hundreds, LAMBADA at chance, and zero recall (Table 1). The large-chunk update itself is the cause, since every token of a 2048-token chunk is scored with the weights from before the chunk, so within a chunk the memory sees nothing of the current text and the model has to learn the local statistics from attention. The sliding window therefore serves as a required local-context branch for this large-chunk approximation. ParaTTT uses no attention branch: every ParaTTT model in Table 1 is a pure recurrence, its memory is updated at every token, and it is trained and evaluated without a local branch of any kind.
| Model | Perplexity ↓ Wiki | Lamb | Zero-shot acc. ↑ PIQA | Hella | ARC-e | Wino | ARC-c | BoolQ | SciQ |
| LaCT with SWA-2048 (as published) | 25.51 | 33.3 | 66.2 | 35.8 | 45.1 | 51.7 | 23.6 | 61.1 | 79.3 |
| LaCT fast-weight branch only (Table 1) | 463.2 | 53.7 | 25.0 | 28.8 | 50.1 | 24.1 | 60.1 | 21.2 | |
| PTTT (ours), no attention | 25.46 | 26.9 | 65.3 | 37.7 | 48.6 | 51.2 | 24.1 | 59.8 | 79.0 |
Table 5. LaCT at 340M with and without its sliding-window attention branch, same recipe and data as Table 1; recall for the fast-weight-branch-only ablation is in Table 1.
Centering ablation
Table 6 isolates the readout parameterization. All three models share the recipe, initialization and data order of Section D and differ only in how the readout layer is parameterized:
Zero-initialized readout. and , the standard delta-rule state that starts at zero and decays toward zero (the “zero-init readout” row of Table 1). By Eq. (5) the hidden layers receive no gradient until the fast readout has grown.
Learned initial readout, uncentered. A slow is learned and : the full weights decay toward instead of zero, so the hidden layers receive gradient from the first token, but the output carries the bias before any test-time learning has taken place. This model has exactly the parameters of the centered one.
Centered readout. Eq. (22), the ParaTTT default.
Read against the Gated DeltaNet reference, the three rows form a ladder. The zero-initialized loop already improves recall over the linear recurrence (SWDE +4, NIAH at 4K +3) but costs 3 points of LAMBADA perplexity. The learned initial readout repairs LAMBADA to the Gated DeltaNet level and leaves recall essentially unchanged, showing that giving the hidden layers a gradient at the start of a sequence matters. Centering buys the rest with the same parameters: 0.7 WikiText-103 and 2.1 LAMBADA perplexity, 4 points of SWDE and 2.5 of TriviaQA, and the largest NIAH gain at 32K. The commonsense tasks move within noise across the ladder, and each row is a single seed. The gain of centering over the uncentered learned readout is therefore not explained by the extra slow readout: it comes from removing the bias so that the readout at the start of a sequence is zero while its Jacobian is not.
Centering for TTT-MLP. We also trained TTT-MLP with the centered readout of Eq. (22). It does not require centering: the results moved by about two points in either direction (SWDE 33.1 → 35.7, FDA 16.2 → 18.2, perplexity unchanged within noise), so we report the original recipe.
Kernel implementation and efficiency details
The ParaTTT layer is built from the chunkwise Triton kernels of FLA with a chunk size of 64: the gated-linear-attention and gated-delta-rule forward and backward kernels, the transposed reads of Eq. (6) (the same kernel with the roles of keys and values exchanged) and DELTARULEGRAD, which is the gated-delta-rule kernel with the final read transposed; the pointwise epilogues (nonlinearity, Jacobian, projection, damping) are fused into two Triton kernels described in Section G.1. Per iteration and hidden layer the layer stores the pseudo-labels, the features and the Jacobian, each per head, and the readout states at chunk boundaries are recomputed in the backward pass; peak memory at is 2.5× that of Gated DeltaNet, both for one layer at batch 8 (6.5 against 2.6 GiB at ) and for the full 340M model in training (Table 7). For Figure 4(a,Figure 4b) we time one token-mixing layer in isolation (forward and backward, bf16, batch 8, 20 iterations after 3 warm-up iterations) on one MI325X; The fused Triton implementation of TTT-MLP used in our benchmarks supports head dimension 64 and block size 16; elsewhere we use a torch.compiled version of its PyTorch scan, which is 2.2× slower than the fused kernel where both are available. For (c) we measure the training throughput of the full 340M models on one node of eight MI325X GPUs with the training recipe of Section D, averaged over 50 steps after warm-up.
| Variant | Layer forward+backward (ms) at 1K | 2K | 4K | 8K | 16K | Training, 8 GPUs tokens/s () | GiB/GPU |
| Readout parameterization (delta-rule body, optimized kernels) | |||||||
| Zero-init readout | 7.3 | 13.1 | 26.2 | 52.2 | 104.1 | 42.7 | 131 |
| Learned-init readout, uncentered | 8.3 | 14.8 | 29.4 | 58.8 | 116.7 | 38.7 | 136 |
| Centered readout (PTTT) | 8.5 | 15.1 | 29.8 | 59.8 | 118.8 | 38.0 | 145 |
| Hidden-layer update (centered readout, reference implementation) | |||||||
| Linear-attention body | 8.4 | 15.3 | 30.4 | 60.7 | 120.3 | 37.4 | 107 |
| Delta-rule body (PTTT) | 10.0 | 18.0 | 35.6 | 70.9 | 140.3 | 32.5 | 138 |
| Reference | |||||||
| Gated DeltaNet | 2.1 | 3.3 | 6.3 | 11.9 | 23.2 | 112.1 | 58 |
Table 7. Cost of the hidden-layer update and of the readout parameterization at 340M, . Left: forward+backward time of one token-mixing layer (bf16, batch 8, , 8 heads of dimension 128, one MI325X), the setting of Figure 4(a). Right: training throughput and peak memory of the full model on one node of eight MI325X GPUs with the recipe of Section D. The readout rows use the optimized kernels of Section G; the two bodies are timed in the reference implementation, which supports both updates, so the body rows are comparable with each other but slower than the readout rows.
The fused epilogue kernels
With one hidden layer () the only operations of an iteration that are not scans are pointwise in the token: the nonlinear and projection that turn the hidden pre-activation into the features (Step 1), and their transposed Jacobians, which are all that Step 3 needs, because the transposed fast weight of Eq. (6) is only required to reach a layer below the first. Both are implemented as one Triton kernel each, with one program per (sequence, token, head) that holds the -vector of its row in fp32 registers; every reduction is over this vector, so the kernels are memory-bound and read and write each activation once. We write for the sigmoid, for SiLU and for the unit-ball projection.
Forward epilogue. The hidden pre-activation of token is the sum of a slow part and a fast part,
where is the learned per-head gain of § 3.4. The slow part does not depend on the pseudo-labels, so it is computed once per layer forward and shared by all iterations; the scan output is read in bf16 straight from the layout the scan writes, without the copy that separating the two arms of a SwiGLU would need (the value arm is the constant 1 in our models). The kernel stores , and in fp32, stores in bf16 for the readout kernels, and saves and for the backward epilogue; is recomputed there from .
Backward epilogue. DELTARULEGRAD returns Eq. (5). The backward epilogue turns it into the value the hidden scan consumes in the next iteration, in five register-resident steps:
with and . The second step is the transposed Jacobian of the projection, outside the unit ball and the identity inside it, and the third that of SiLU; together they are the transposed Jacobian of § 3.3, with the hard projection in place of the soft clip. The value the scan receives, , attaches the hidden learning rate and the sign with which the pseudo-label enters as a value. The damped Picard step Eq. (7) is fused into the same store, , with handled without a read, and the result is written directly in the layout the scan reads, so an iteration is four kernel launches per layer: the hidden scan, the forward epilogue, DELTARULEGRAD and the backward epilogue. The kernels also implement a pre-activation RMSNorm, a per-token norm clip of (hard or soft), a skip connection and a second SwiGLU arm, none of which the reported models use.
Training through the iteration. End-to-end training backpropagates through all iterations, so each epilogue is an autograd function whose backward is itself a fused kernel: the vector-Jacobian product of Eq. (24) (through and the projection Jacobian) and that of Eq. (25), which differentiates the SiLU Jacobian a second time () and the learning rates. For decoding, the hidden scan, the forward epilogue and the DELTARULEGRAD step of one token are fused into a single kernel per (sequence, head) with the states resident in registers, since at that length the launch latency of the chunkwise kernels dominates.
Cost of the hidden-layer update and of centering
Table 7 measures what the two design choices of § 3.4 cost, in the two settings of Figure 4: one token-mixing layer in isolation and the full 340M model in training.
Readout parameterization. The three readouts of Table 6 run on the same kernels and differ only in the readout offset and in the extra terms of Eq. (22). Relative to the zero-initialized readout, the uncentered learned readout adds the offset to the final read of DELTARULEGRAD and costs 12% of the layer time at 4K and 9% of training throughput. Centering adds to this one more pass of the slow MLP over the queries (the features of Section C.5) and the corrected values, which is a further 1–2% of layer time and throughput and 9 GiB of activation memory. Centering therefore costs about 11% of training throughput in total, against the gains of Table 6.
Hidden-layer update. The delta-rule body replaces the gated-linear-attention scan of the hidden layer with a gated-delta-rule scan of the same shape, plus the transposed reads of Eq. (5) that the delta rule’s gradient needs (Section C.2); the readout side of the layer is unchanged. In the reference implementation, which supports both updates, the delta-rule body costs 17% of the layer time at every length (35.6 against 30.4 ms at 4K) and 1.7 GiB more activation memory per layer for the extra chunk states; in training this is 13% of throughput (32.5k against 37.4k tokens/s) and 30 GiB of peak memory per GPU. Together with Table 1 (SWDE 28.4 → 41.9, NIAH at 4K 7.2 → 32.6), this is the more cost-effective of the two design choices.
Convergence versus compute at other sequence lengths
Figure 7 shows the comparison of Figure 5(a) at every sequence length from 256 to 4096 with the same inner MLP, step size and decay. The picture is the same at every length: TTT-MLP’s error is set by its block size and its time by the number of sequential blocks , whereas ParaTTT’s error is set by the iteration count and its time, being a fixed number of chunkwise-parallel scans, is nearly independent of at batch 1, so the gap widens with the sequence length.

Figure 7. Relative error against exact online GD versus the time to compute the forward recurrence of one 340M layer (batch 1), at five sequence lengths; same setting and legend as Figure 5(a).
References
- [1]Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. In International Conference on Machine Learning, 2024a. URL https://proceedings.mlr.press/v235/arora24a.html.proceedings.mlr.press/v235/arora24a.html
- [2]Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: Closing the recall gap for recurrent language models. arXiv preprint arXiv:2407.05483, 2024b. URL https://arxiv.org/abs/2407.05483.
- [3]Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/01386bd6d8e091c2ab4c7c7de644d37b-Abstract.html.proceedings.neurips.cc/paper/2019/hash/01386bd6d8e091c2ab4c7c7de644d37b-Abstract.html
- [4]Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short-term memory. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/c2ce2f2701c10a2b2f2ea0bfa43cfaa3-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2024/hash/c2ce2f2701c10a2b2f2ea0bfa43cfaa3-Abstract-Conference.html
- [5]Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, and Vahab Mirrokni. ATLAS: Learning to optimally memorize the context at test time. arXiv preprint arXiv:2505.23735, 2025a. URL https://arxiv.org/abs/2505.23735.
- [6]Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. In Advances in Neural Information Processing Systems, 2025b. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/a4ca07aa108036f80cbb5b82285fd4b1-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/a4ca07aa108036f80cbb5b82285fd4b1-Abstract-Conference.html
- [7]Ziyang Cai, Xingyu Zhu, Yihe Dong, Yinghui He, and Sanjeev Arora. T2MLR: Transformer with temporal middle-layer recurrence. arXiv preprint arXiv:2607.15178, 2026. URL https://arxiv.org/abs/2607.15178.
- [8]Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.html.proceedings.neurips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.html
- [9]Kyunghyun Cho, Bart van Merriënboer, Çağlar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724–1734, 2014. doi: 10.3115/v1/D14-1179. URL https://aclanthology.org/D14-1179/.DOI
- [10]Federico Danieli, Pau Rodríguez, Miguel Sarabia, Xavier Suau, and Luca Zappella. ParaRNN: Unlocking parallel training of nonlinear RNNs for large language models. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=mX8b64iUaa.openreview.net/forum?id=mX8b64iUaa
- [11]Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning, 2024. URL https://proceedings.mlr.press/v235/dao24a.html.proceedings.mlr.press/v235/dao24a.html
- [12]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1807.03819.
- [13]Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179–211, 1990. URL https://doi.org/10.1207/s15516709cog1402_1.
- [14]Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, and Tianle Cai. In-place test-time training. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/ba6f5879d7f382241316f7dda7e6b8ac-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2026/hash/ba6f5879d7f382241316f7dda7e6b8ac-Abstract-Conference.html
- [15]Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., … Zou, A. (2023). A framework for few-shot language model evaluation. https://zenodo.org/records/10256836zenodo.org/records/10256836
- [16]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In Advances in Neural Information Processing Systems, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/3b01972cf31e6fa0fe29e4b8b5c2a0a1-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/3b01972cf31e6fa0fe29e4b8b5c2a0a1-Abstract-Conference.html
- [17]Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning, 2023. URL https://proceedings.mlr.press/v202/giannou23a.html.proceedings.mlr.press/v202/giannou23a.html
- [18]Xavier Gonzalez, Andrew Warrington, Jimmy T. H. Smith, and Scott W. Linderman. Towards scalable and stable parallelization of nonlinear RNNs. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2407.19115.
- [19]Gonzalez, X., Kozachkov, L., Zoltowski, D. M., Clarkson, K. L., & Linderman, S. W. (2025). Predictability enables parallelization of nonlinear state space models. Advances in Neural Information Processing Systems. https://proceedings.neurips.cc/paper_files/paper/2025/hash/1b7a239766f041abcd4286e5548037-Abstract-Conference.htmlproceedings.neurips.cc/paper_files/paper/2025/hash/1b7a239766f041abcd4286e5548037-Abstract-Conference.html
- [20]Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K. H. Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear RNNs through negative eigenvalues. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/5a0ce3abb720b740419e193c87afd080-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2025/hash/5a0ce3abb720b740419e193c87afd080-Abstract-Conference.html
- [21]Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. In Second Conference on Language Modeling (COLM), 2025. URL https://arxiv.org/abs/2412.06769.
- [22]Ali Hatamizadeh, Yejin Choi, and Jan Kautz. Gated DeltaNet-2: Decoupling erase and write in linear attention. arXiv preprint arXiv:2605.22791, 2026. URL https://arxiv.org/abs/2605.22791.
- [23]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.
- [24]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling (COLM), 2024. URL https://openreview.net/forum?id=kIoBbc76Sy.openreview.net/forum?id=kIoBbc76Sy
- [25]Jiaxi Hu, Yongqi Pan, Jusen Du, Disen Lan, Xiaqiang Tang, Qingsong Wen, Yuxuan Liang, and Weigao Sun. Improving bilinear RNNs with closed-loop control. In Advances in Neural Information Processing Systems, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/9a439efaa34fe37177eba00737624824-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/9a439efaa34fe37177eba00737624824-Abstract-Conference.html
- [26]Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=Hke3gyHYwH.openreview.net/forum?id=Hke3gyHYwH
- [27]Yulong Huang, Xiang Liu, Hongxiang Huang, Xiaopeng Lin, Zunchang Liu, Xiaowen Chu, Zeke Xie, and Bojun Cheng. MDN: Parallelizing stepwise momentum for delta linear attention. In International Conference on Machine Learning, 2026. URL https://icml.cc/virtual/2026/poster/63901.icml.cc/virtual/2026/poster/63901
- [28]Mahdi Karami, Razvan Pascanu, and Vahab Mirrokni. Lattice: Learning to efficiently compress the memory. arXiv preprint arXiv:2504.05646, 2025. URL https://arxiv.org/abs/2504.05646.
- [29]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html.proceedings.mlr.press/v119/katharopoulos20a.html
- [30]Team, K. (2025). Kimi Linear: An expressive, efficient attention architecture. In arXiv preprint. https://arxiv.org/abs/2510.26692
- [32]Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, and Albert Gu. Mamba-3: Improved sequence modeling using state space principles. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/8abd2043b71a074278d5f687947bff9c-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2026/hash/8abd2043b71a074278d5f687947bff9c-Abstract-Conference.html
- [33]Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing nonlinear sequential models over the sequence length. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=E34AlVLN0v.openreview.net/forum?id=E34AlVLN0v
- [34]Liu, J., Elflein, S., Litany, O., Gojcic, Z., & Li, R. (2026). Test-time training with KV binding is secretly linear attention. International Conference on Machine Learning. https://icml.cc/virtual/2026/poster/64516icml.cc/virtual/2026/poster/64516
- [35]William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics, 11:531–545, 2023. URL https://aclanthology.org/2023.tacl-1.31/.aclanthology.org/2023.tacl-1.31
- [36]William Merrill and Ashish Sabharwal. A little depth goes a long way: The expressive power of log-depth transformers. In Advances in Neural Information Processing Systems, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/88dd7aa6979e352fda7c4952ca8eaec59-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/88dd7aa6979e352fda7c4952ca8ea c59-Abstract-Conference.html
- [37]William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In International Conference on Machine Learning, 2024. URL https://proceedings.mlr.press/v235/merrill24a.html.proceedings.mlr.press/v235/merrill24a.html
- [38]Mishra, M., Tan, S., Stoica, I., Gonzalez, J. E., & Dao, T. (2026). M²RNN: Nonlinear RNNs with matrix-valued states for scalable language modeling. In arXiv preprint. https://arxiv.org/abs/2603.14360
- [39]Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin, Alexander Theus, Thomas Hofmann, Valentina Boeva, T. Konstantin Rusch, and Antonio Orvieto. Fixed-point reasoners: Stable and adaptive deep looped transformers. arXiv preprint arXiv:2606.18206, 2026. URL https://arxiv.org/abs/2606.18206.
- [40]Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, and Stefano Massaroli. Quantifying memory utilization with effective state-size, 2025. URL https://arxiv.org/abs/2504.19561.
- [41]Peng, B., Goldstein, D., Anthony, Q., Albalak, A., Alcaide, E., Biderman, S., Cheah, E., Ferdinan, T., Hou, H., Kiran GV, K., Krishna, S., McClelland, R. J., Muennighoff, N., Obeid, F., Saito, A., Song, G., Tu, H., Zhang, R., Zhao, B., … Zhu, R.-J. (n.d.). Eagle and Finch: RWKV with matrix-valued states and dynamic recurrence. First Conference on Language Modeling (COLM). Retrieved https://openreview.net/pdf/9f3d829735b231725665ff226a4f732293ad4aa8.pdfopenreview.net/pdf/9f3d829735b231725665ff226a4f732293ad4aa8.pdf
- [42]Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. RWKV-7 “Goose” with expressive dynamic state evolution. In Second Conference on Language Modeling (COLM), 2025. URL https://openreview.net/forum?id=ayB1PACN5j.openreview.net/forum?id=ayB1PACN5j
- [43]Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=wHBfxhZu1u.openreview.net/forum?id=wHBfxhZu1u
- [44]Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by backpropagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0
- [45]Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/2676109d49d1eb26d6bc584a8f556305-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2025/hash/2676109d49d1eb26d6bc584a8f556305-Abstract-Conference.html
- [46]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, 2021. URL https://proceedings.mlr.press/v139/schlag21a.html.proceedings.mlr.press/v139/schlag21a.html
- [47]Mark Schöne, Babak Rahmani, Heiner Kremer, Fabian Falck, Hitesh Ballani, and Jannes Gladrow. Implicit language models are RNNs: Balancing parallelization and expressivity. In International Conference on Machine Learning, 2025. URL https://proceedings.mlr.press/v267/schone25a.html.proceedings.mlr.press/v267/schone25a.html
- [48]Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. DeltaProduct: Improving state-tracking in linear RNNs via Householder products. In Advances in Neural Information Processing Systems, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/e1ea2520fb9cd1600b287dde67e0a3c-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/e1ea2520fb9cd1600b287dde67e0a3c-Abstract-Conference.html
- [49]Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): RNNs with expressive hidden states. In International Conference on Machine Learning, 2025. URL https://proceedings.mlr.press/v267/sun25h.html.proceedings.mlr.press/v267/sun25h.html
- [50]Tandon, A., Dalal, K., Li, X., Koceja, D., Rød, M., Buchanan, S., Wang, X., Leskovec, J., Koyejo, S., Hashimoto, T., Guestrin, C., McCaleb, J., Choi, Y., & Sun, Y. (2025). End-to-end test-time training for long context. In arXiv preprint. https://arxiv.org/abs/2512.23675
- [51]Neehal Tumma, Mathias Lechner, Noel Loo, Ramin Hasani, and Daniela Rus. Leveraging low-rank and sparse recurrent connectivity for robust closed-loop control, 2023. URL https://arxiv.org/abs/2310.03915.
- [52]Neehal Tumma, Noel Loo, and Daniela Rus. Preconditioned DeltaNet: Curvature-aware sequence modeling for linear recurrences. In International Conference on Machine Learning, 2026. URL https://icml.cc/virtual/2026/poster/63733.icml.cc/virtual/2026/poster/63733
- [53]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
- [54]Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Aguera y Arcas, and João Sacramento. MesaNet: Sequence modeling by locally optimal test-time training. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/2c3d5a4eff20d24390a445762d61bfbb-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2026/hash/2c3d5a4eff20d24390a445762d61bfbb-Abstract-Conference.html
- [56]Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, and John Langford. Full-bandwidth transformer. arXiv preprint arXiv:2608.08888, 2026. URL https://arxiv.org/abs/2608.08888.
- [57]Paul J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560, 1990. URL https://doi.org/10.1109/5.58337.
- [58]Bernard Widrow and Marcian E. Hoff, Jr. Adaptive switching circuits. In 1960 IRE WESCON Convention Record, Part 4, pp. 96–104, 1960. URL https://isl.stanford.edu/~widrow/papers/c1960adaptiveswitching.pdf.DOI
- [59]Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., & Srebro, N. (2020). Kernel and rich regimes in overparametrized models. Conference on Learning Theory. https://proceedings.mlr.press/v125/woodworth20a.htmlproceedings.mlr.press/v125/woodworth20a.html
- [60]Haoyi Wu, Zhihao Teng, and Kewei Tu. Parallel continuous chain-of-thought with Jacobi iteration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 914–926, 2025. doi: 10.18653/v1/2025.emnlp-main.47. URL https://aclanthology.org/2025.emnlp-main.47/.DOI
- [61]Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In International Conference on Machine Learning, 2024a. URL https://proceedings.mlr.press/v235/yang24ab.html.proceedings.mlr.press/v235/yang24ab.html
- [62]Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In Advances in Neural Information Processing Systems, 2024b. URL https://arxiv.org/abs/2406.06484.
- [63]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=r8H7xhYPwz.openreview.net/forum?id=r8H7xhYPwz
- [64]Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T Freeman, and Hao Tan. Test-time training done right. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/ffbfff81fb78e4bb558273b91e12e318-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2026/hash/ffbfff81fb78e4bb558273b91e12e318-Abstract-Conference.html
- [65]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025. URL https://arxiv.org/abs/2510.25741.



