! GitHub
Introduction
Softmax attention scales poorly to long contexts: its compute grows quadratically with sequence length, while its key–value cache grows linearly. Recurrent language models address both problems by processing the sequence in linear time and carrying a fixed-size state while decoding. Moreover, modern recurrent architectures such as gated linear attention, Gated DeltaNet, and Mamba-2 [62 , 64 , 11 ] admit an efficiently parallelizable structure commonly referred to as the chunkwise-parallel form, in which tokens within each chunk are processed in parallel using dense matrix multiplications, while only the recurrent state is propagated sequentially across chunks. Given this scalable structure, the central question for this class of models is how to make the recurrent memory more expressive without sacrificing parallel evaluation.
Test-time training (TTT) addresses the expressivity side of this question by treating the recurrent state as the weights of a small model that learns from the sequence through online gradient descent [50 ] . With a linear memory and a squared reconstruction loss, this recovers the delta rule of [59 ] , the update used by DeltaNet [47 , 63 ] , whereas a nonlinear memory can adapt its features as well as its readout. TTT-MLP, Titans, and
LaCT (Sun et al., [50 ] ; Behrouz et al., 2025b; Zhang et al., [65 ] ) explore such neural memories, but their gradients depend nonlinearly on the current weights: each token changes the weights at which the next token’s gradient should be evaluated, creating a sequential dependency. To recover parallelism, existing methods group gradient evaluations into blocks and evaluate every gradient in a block at the weights entering it. This creates a fidelity–speed tradeoff: larger blocks reduce the sequential depth but more coarsely approximate token-by-token learning, while smaller blocks are more faithful but require more sequential steps.
In this paper, we introduce ParaTTT (PTTT), an algorithm that evaluates an MLP-based TTT memory in parallel (Figure 1 ). The difficulty in parallelizing the recurrence is the inner weight trajectory: the gradient step at each token is taken at weights that depend on every earlier step, so the trajectory cannot be computed without stepping through it sequentially. PTTT instead refines an estimate of this trajectory, represented by the gradients of the inner loss with respect to the hidden-layer pre-activations. Each refinement consists of three parallel passes: a forward pass that constructs the features implied by the current gradient estimate, a readout pass that computes each token’s loss gradient with respect to its features, and a backward pass that propagates these gradients through the hidden layers to form a new estimate. Crucially, each refinement step is a composition of modified instances of existing parallel recurrences such as Gated DeltaNet (Yang et al., [63 ] ; Yang et al., [64 ] ), Mamba-2 (Dao & Gu, [11 ] ), or gated linear attention (Yang et al., [62 ] ). The exact nonlinear sequential recurrence is a fixed point of the iteration. Training and prefill use a finite number of these iterations, while decoding applies the exact token-by-token update.
Figure 1. Parallelism and approximation in nonlinear TTT. Each cell represents a hidden pre-activation gradient δ z t ( W ) = ∂ ℓ t ∂ z t ∣ W \delta z_t(W) = \left.\frac{\partial\ell_t}{\partial z_t}\right|_W δ z t ( W ) = ∂ z t ∂ ℓ t W , with the layer index omitted. Columns index tokens; downward steps mark computational dependencies, not approximation error or elapsed time. (a) Exact gradients δ z t ∗ \delta z_t^* δ z t ∗ use the latest online weights W ˉ t − 1 \bar{W}_{t-1} W ˉ t − 1 , including decay. (b) Block-wise TTT, as used by TTT-MLP (Sun et al., [50 ] ) and Titans (Behrouz et al., 2025b), evaluates gradients at shared block-entry weights W s b l k W_s^{\mathrm{blk}} W s blk for s < t ≤ s + b s < t \le s + b s < t ≤ s + b , omitting earlier within-block updates. (c) ParaTTT refines the full gradient trajectory using three chunkwise-parallel passes per round. The block size b b b and refinement count N N N control the respective approximations; the exact trajectory is a fixed point of the ParaTTT iteration.
Our technical contributions can be summarized as follows:
We demonstrate PTTT’s superior speed–accuracy trade-off over the block-based evaluation of TTT-MLP: at 4K context, two PTTT refinements match the output error of TTT-MLP’s block approximation while running 32× faster, and full 340M models train 4.9× faster than TTT-MLP (§ 4.2 and § 4.3 ).
At 340M and 1.3B parameters, PTTT attains lower perplexity and stronger recall on SWDE and FDA than Gated DeltaNet and Mamba-2, showing that the proposed nonlinear memory improves over the evaluated linear recurrent baselines § 4.1 .
Background
We write x 1 : T x_{1:T} x 1 : T for the input sequence of a layer and q t q_t q t , k t k_t k t , v t v_t v t for the per-token query, key, and value projections; the fast-weight state is a matrix S t S_t S t or, for a deep memory, a set of weights W t W_t W t ; C C C is the chunk size of the parallel kernels. Heads are independent, and we suppress the head index. Section A extends this section.
Linear recurrences
Linear recurrences retain the associative-memory view of attention while replacing its growing key–value cache with a fixed-size state. Linear attention ([29 , 47 ] ) does this by dropping the softmax: it maintains a matrix state updated by an outer product, S t = S t − 1 + v t k t T S_t = S_{t-1} + v_t k_t^{\mathsf{T}} S t = S t − 1 + v t k t T , and reads it with the query, o t = S t q t o_t = S_t q_t o t = S t q t . Because this state only accumulates, practical variants introduce forgetting. Gated linear attention ([62 ] ) decays the state before the write, S t = α t S t − 1 + v t k t T S_t = \alpha_t S_{t-1} + v_t k_t^{\mathsf{T}} S t = α t S t − 1 + v t k t T , using a diagonal gate, while Mamba-2 ([11 ] ) is the scalar-gated case. DeltaNet ([47 , 63 ] ) updates the state by replacing the additive write with one step of gradient descent on 1 2 ∥ S k t − v t ∥ 2 \frac{1}{2}\lVert S k_t - v_t\rVert^2 2 1 ∥ S k t − v t ∥ 2 , S t = S t − 1 ( I − β t k t k t T ) + β t v t k t T S_t = S_{t-1}(I-\beta_t k_t k_t^{\mathsf{T}})+\beta_t v_t k_t^{\mathsf{T}} S t = S t − 1 ( I − β t k t k t T ) + β t v t k t T , which erases the value stored along k t k_t k t before writing the new one. Gated DeltaNet ([64 ] ) combines this delta-rule update with decay. All of these updates are linear in the recurrent state. Consequently, the state at the end of a chunk of C C C tokens is an affine function of the state entering the chunk. This allows tokens within each chunk to be evaluated in parallel with dense matrix multiplications, while only the chunk-boundary states are propagated sequentially with a short scan. This exact evaluation is commonly referred to as the chunkwise-parallel form.
Test-time training
Test-time training ([50 ] ) makes the memory more expressive than a single matrix by interpreting the recurrent state as the parameters W t W_t W t of a small model f W f_W f W trained online. At token t t t , the model takes a gradient step on the inner loss ℓ t ( W ) = 1 2 ∥ f W ( k t ) − v t ∥ 2 \ell_t(W) = \frac{1}{2}\lVert f_W(k_t) - v_t\rVert^2 ℓ t ( W ) = 2 1 ∥ f W ( k t ) − v t ∥ 2 and produces the layer output f W t ( q t ) f_{W_t}(q_t) f W t ( q t ) . The projections, gates, and learning rates that define this inner problem are trained end-to-end by backpropagating through the inner updates.
TTT-linear. The simplest choice of inner model recovers the delta-rule recurrence described above. With f W ( k ) = W k f_W(k) = Wk f W ( k ) = W k , one gradient step on the squared reconstruction loss is exactly the DeltaNet update, while adding a decay term gives Gated DeltaNet. Because this update remains linear in the recurrent state, it retains the exact chunkwise-parallel form described above. We exploit this identity for the linear readout of a nonlinear memory.
TTT-nonlinear. A nonlinear inner model can adapt the features it reads and writes with, rather than only the readout; this additional expressivity motivates deep memories. TTT-MLP ([50 ] , for example, uses a two-layer MLP as f W f_W f W . Exact online gradient descent evaluates each token’s gradient at the weights produced by all preceding updates, which for a nonlinear memory prevents the chunkwise-parallel rearrangement of the linear case, so TTT-MLP uses a mini-batch (dual) approximation: for a block of b b b tokens starting at position s s s , every gradient is evaluated at the weights W s W_s W s entering the block,
exact: W j = W j − 1 − η j ∇ ℓ j ( W j − 1 ) , block: W t b l k = W s − ∑ j = s + 1 t η j ∇ ℓ j ( W s ) , s < t ≤ s + b . \text{exact: } W_j = W_{j-1} - \eta_j \nabla\ell_j(W_{j-1}), \qquad\text{block: } W_t^{\mathrm{blk}} = W_s - \sum_{j=s+1}^{t} \eta_j \nabla\ell_j(W_s), \qquad s < t \le s+b. exact: W j = W j − 1 − η j ∇ ℓ j ( W j − 1 ) , block: W t blk = W s − j = s + 1 ∑ t η j ∇ ℓ j ( W s ) , s < t ≤ s + b .
The causal partial sum lets tokens within the block be evaluated in parallel, but the gradients are stale: updates from earlier tokens in the block do not change the weights at which later gradients are evaluated. Blocks remain sequential, requiring T / b T/b T / b steps, so larger blocks reduce sequential depth at the cost of a coarser approximation to token-by-token learning. Titans ((Behrouz et al., 2025b)) adds momentum and weight decay under the same block form, while LaCT ([65 ] ) uses blocks of thousands of tokens together with a sliding-window attention branch for local context.
Other nonlinear recurrent approaches
Sequential nonlinear RNNs. Classic recurrent networks [13 , 23 , 9 ] carry a vector state through a nonlinear map. Their standard evaluation is sequential over tokens, limiting parallelism across the sequence. This motivates attention [54 ] and parallel linear recurrences, but the choice of transition also affects expressivity. Under their precision and architectural assumptions, Merrill & Sabharwal [36 ] ; Merrill et al. [38 ] place transformers and certain SSMs in T C 0 \mathrm{TC}^{0} TC 0 . This rules out general permutation composition if T C 0 ≠ N C 1 \mathrm{TC}^{0} \ne\mathrm{NC}^{1} TC 0 = NC 1 . The xLSTM family [4 ] includes both a sequential sLSTM cell and a parallelizable mLSTM cell. More recently, M2 ^{2} 2 RNN [39 ] uses a nonlinear transition with a matrix-valued state, but pays for it with sequential forward and backward passes.
Parallelizing nonlinear recurrences. DEER [33 ] and ParaRNN [10 ] cast a nonlinear recurrence h t = g ( h t − 1 , x t ) h_t = g(h_{t-1}, x_t) h t = g ( h t − 1 , x t ) as a fixed-point problem over the state trajectory and solve it with Newton iterations, where each iteration reduces to a linear recurrence evaluated by a parallel scan; deep equilibrium models [3 ] share the fixed-point perspective. A generic Newton step uses a transition Jacobian whose size is quadratic in the state dimension; ParaRNN makes this tractable with structured Jacobians. ParaTTT instead iterates on the pre-activation gradient at each token and hidden layer. For hidden width d d d , each gradient is a d d d -dimensional vector, compared with the O ( d 2 ) O(d^2) O ( d 2 ) fast-weight state. Conditioned on these gradients, the fast-weight trajectory becomes a linear recurrence, so each refinement can be evaluated with chunkwise linear recurrent kernels without ever forming the full weight trajectory or its transition Jacobian.
Method
Test-time training with a deep memory
We consider a single head and omit the head index. As in Gated DeltaNet, the input token x t x_t x t is mapped to a key, a query and a value, k t , q t ∈ R d k_t, q_t \in\mathbb{R}^{d} k t , q t ∈ R d and v t ∈ R d v v_t \in\mathbb{R}^{d_v} v t ∈ R d v , by linear projections followed by a short causal convolution; keys and queries are RMS-normalized, ∥ k t ∥ = ∥ q t ∥ = d \lVert k_t\rVert= \lVert q_t\rVert= \sqrt{d} ∥ k t ∥ = ∥ q t ∥ = d . The token also produces a decay gate α t ∈ ( 0 , 1 ] \alpha_t \in(0, 1] α t ∈ ( 0 , 1 ] and, for each layer l l l of the memory defined below, a learning rate η t ( l ) ∈ ( 0 , 1 ) \eta_t^{(l)} \in(0, 1) η t ( l ) ∈ ( 0 , 1 ) . The gate follows Yang et al. [64 ] , each η t ( l ) \eta_t^{(l)} η t ( l ) is the sigmoid of a learned linear function of x t x_t x t , and all layers of the memory share the decay.
Memory. The recurrent state is the set of weights W = ( W ( 1 ) , … , W ( L ) ) \mathbf{W} = (\mathbf{W}^{(1)}, \ldots, \mathbf{W}^{(L)}) W = ( W ( 1 ) , … , W ( L ) ) of an L L L -layer MLP. For given weights W \mathbf{W} W , with pre-activations z ( l ) \mathbf{z}^{(l)} z ( l ) and activations h ( l ) \mathbf{h}^{(l)} h ( l ) , the memory computes
z ( l ) = W ( l ) h ( l − 1 ) + b ( l ) , h ( l ) = σ ( z ( l ) ) ( 1 ≤ l < L ) , h ( 0 ) = k , f W ( k ) = W ( L ) h ( L − 1 ) , (1) \mathbf{z}^{(l)} = \mathbf{W}^{(l)}\mathbf{h}^{(l-1)} + \mathbf{b}^{(l)}, \qquad\mathbf{h}^{(l)} = \sigma(\mathbf{z}^{(l)}) \quad(1 \le l < L), \qquad\mathbf{h}^{(0)} = \mathbf{k}, \qquad f_{\mathbf{W}}(\mathbf{k}) = \mathbf{W}^{(L)}\mathbf{h}^{(L-1)}, \tag*{(1)} z ( l ) = W ( l ) h ( l − 1 ) + b ( l ) , h ( l ) = σ ( z ( l ) ) ( 1 ≤ l < L ) , h ( 0 ) = k , f W ( k ) = W ( L ) h ( L − 1 ) , (1)
where σ \sigma σ is a pointwise nonlinearity and b ( l ) \mathbf{b}^{(l)} b ( l ) are learned biases. The final linear layer is the readout , and its input ϕ t = h t ( L − 1 ) \boldsymbol{\phi}_t = \mathbf{h}_t^{(L-1)} ϕ t = h t ( L − 1 ) defines the features . The features are additionally projected onto the unit ball, n ( x ) = x / max ( 1 , ∥ x ∥ ) \mathfrak{n}(\mathbf{x}) = \mathbf{x}/\max(1,\lVert\mathbf{x}\rVert) n ( x ) = x / max ( 1 , ∥ x ∥) , which we absorb into the last σ \sigma σ . This projection is used in all experiments. Our experiments use L = 2 L = 2 L = 2 , as in Sun et al. [50 ] , but the method applies to any L L L . Each weight is decomposed as W t ( l ) = W 0 ( l ) + Δ W t ( l ) \mathbf{W}_t^{(l)} = \mathbf{W}_0^{(l)} + \Delta\mathbf{W}_t^{(l)} W t ( l ) = W 0 ( l ) + Δ W t ( l ) , where W 0 ( l ) \mathbf{W}_0^{(l)} W 0 ( l ) is a parameter trained by backpropagation and Δ W t ( l ) \Delta\mathbf{W}_t^{(l)} Δ W t ( l ) is the recurrent state, initialized to zero at the start of every sequence.
Sequential recurrence. Token t t t first decays the state and then takes one gradient step on the reconstruction loss ℓ t ( W ) = 1 2 ∥ f W ( k t ) − v t ∥ 2 \ell_t(\mathbf{W}) = \frac{1}{2}\lVert f_{\mathbf{W}}(\mathbf{k}_t) - \mathbf{v}_t\rVert^2 ℓ t ( W ) = 2 1 ∥ f W ( k t ) − v t ∥ 2 , with the layer’s learning rate as the step size. For each layer l l l ,
W ˉ t − 1 ( l ) = W 0 ( l ) + α t Δ W t − 1 ( l ) (decay) , W t ( l ) = W ˉ t − 1 ( l ) − η t ( l ) ∂ ℓ t ∂ W ( l ) ∣ W ˉ t − 1 (gradient step) . (2) \bar{\mathbf{W}}_{t-1}^{(l)} = \mathbf{W}_0^{(l)} + \alpha_t \Delta\mathbf{W}_{t-1}^{(l)} \quad\text{(decay)}, \qquad\mathbf{W}_t^{(l)} = \bar{\mathbf{W}}_{t-1}^{(l)} - \eta_t^{(l)} \left.\frac{\partial\ell_t}{\partial\mathbf{W}^{(l)}}\right|_{\bar{\mathbf{W}}_{t-1}} \quad\text{(gradient step)}. \tag*{(2)} W ˉ t − 1 ( l ) = W 0 ( l ) + α t Δ W t − 1 ( l ) (decay) , W t ( l ) = W ˉ t − 1 ( l ) − η t ( l ) ∂ W ( l ) ∂ ℓ t W ˉ t − 1 (gradient step) . (2)
The layer output is o t = f W ( q t ) \mathbf{o}_t = f_{\mathbf{W}}(\mathbf{q}_t) o t = f W ( q t ) : the key is evaluated under the decayed weights, before its own gradient step, and the query after it, exactly as in Gated DeltaNet, whose memory is the linear case L = 1 L = 1 L = 1 of Eq. (1) .
Equation (2) is the exact token-by-token recurrence that we seek to evaluate. Computing it directly requires T T T sequential gradient steps because the gradient at each token t t t depends on all preceding updates. The block approximation used by TTT-MLP, described in § 2.2 , reduces this sequential depth by evaluating multiple gradients at shared weights, but changes the recurrence. Our goal is instead to parallelize the trajectory while retaining the exact online recurrence as the fixed point.
ParaTTT: Iterated refinement of the inner-loop weight trajectory
We first describe the ParaTTT iteration, which uses a linear attention update for the hidden layers and an uncentered readout. The practical design choices used by our models are introduced in § 3.4 .
The central difficulty in parallelizing the nonlinear MLP recurrence in Eq. (2) is its inner weight trajectory: the gradient step at token t t t is taken at weights that depend on every earlier step, so the trajectory cannot be computed without stepping through it sequentially. ParaTTT instead iteratively refines an estimate of this trajectory, with each refinement parallel across the sequence length. The estimate is represented by a guess of the pre-activation gradients δ z t ( l ) = ∂ ℓ t / ∂ z t ( l ) \delta\mathbf{z}_{t}^{(l)} = \partial\ell_t/\partial\mathbf{z}_{t}^{(l)} δ z t ( l ) = ∂ ℓ t / ∂ z t ( l ) of the hidden layers. Given such a guess, a parallel forward pass evaluates the weight trajectory these gradients define (Step 1), the readout is evaluated exactly on the resulting features (Step 2), and a parallel backward pass propagates the resulting feature gradients through the trajectory to obtain a new guess (Step 3). At convergence, the gradients produced by the backward pass agree with those used in the forward pass (Figure 2 ).
Figure 2. One iteration of ParaTTT; the recurrences are evaluated in chunkwise-parallel form over all T T T tokens. (1) A parallel forward pass evaluates the hidden layers from the current estimate of their pre-activation gradients. (2) The readout runs exactly on the resulting features and DELTARULEGRAD returns each token’s feature gradient. (3) A parallel backward pass propagates it through the hidden layers using shifted reads of the transposed recurrences (see Step 3 in § 3.2 for details) and updates the estimate. (4) After N N N iterations a final forward pass and readout produce the output.
To see why each pass can be parallelized, write h t ( l ) \mathbf{h}_{t}^{(l)} h t ( l ) for the activations of k t \mathbf{k}_t k t under the decayed weights W ‾ t − 1 \overline{\mathbf{W}}_{t-1} W t − 1 and obtain δ z t ( l ) \delta\mathbf{z}_{t}^{(l)} δ z t ( l ) by backpropagation through Eq. (1) at W ‾ t − 1 \overline{\mathbf{W}}_{t-1} W t − 1 . The gradient step in Eq. (2) is the outer product of the two, so in terms of the state recurrence reads
Δ W t ( l ) = α t Δ W t − 1 ( l ) − η t ( l ) δ z t ( l ) h t ( l − 1 ) T ( l = 1 , … , L ) . (3) \Delta\mathbf{W}_{t}^{(l)} = \alpha_t \Delta\mathbf{W}_{t-1}^{(l)} - \eta_t^{(l)} \delta\mathbf{z}_{t}^{(l)} \mathbf{h}_{t}^{(l-1)\mathsf{T}} \qquad(l = 1,\ldots,L). \tag*{(3)} Δ W t ( l ) = α t Δ W t − 1 ( l ) − η t ( l ) δ z t ( l ) h t ( l − 1 ) T ( l = 1 , … , L ) . (3)
Given a guess of the hidden-layer gradients D = ( δ z 1 : T ( l ) ) l < L \mathbf{D} = (\delta\mathbf{z}_{1:T}^{(l)})_{l<L} D = ( δ z 1 : T ( l ) ) l < L , one vector per token and hidden layer, Eq. (3) is linear in its state with known coefficients. Conditioned on D \mathbf{D} D , the nonlinear recurrence therefore reduces to a collection of linear recurrences. For each hidden layer, Eq. (3) is exactly gated linear attention with scalar decay, the state update of Mamba-2 ([62 ] ; [11 ] ), with keys h t ( l − 1 ) \mathbf{h}_{t}^{(l-1)} h t ( l − 1 ) , values − η t ( l ) δ z t ( l ) -\eta_t^{(l)} \delta\mathbf{z}_{t}^{(l)} − η t ( l ) δ z t ( l ) and gates α t \alpha_t α t ; for the readout, where δ z t ( L ) \delta\mathbf{z}_{t}^{(L)} δ z t ( L ) is the readout’s own error, it is a Gated DeltaNet ([64 ] ). Both admit exact chunkwise-parallel kernels. The nonlinearity is thus confined to the self-consistency condition that D \mathbf{D} D determines the state trajectory while the resulting trajectory determines D \mathbf{D} D . ParaTTT solves this fixed-point problem by iteration from D = 0 \mathbf{D} = 0 D = 0 (Figure 2 ; pseudocode in Algorithm 1, Section C). Since the vectors of D \mathbf{D} D enter the scans as values, we also call them pseudo-labels .
Step 1: parallel forward pass. Given the current estimate D \mathbf{D} D , the forward pass evaluates the memory MLP at all T T T tokens, each under its own decayed weights, layer by layer and without ever forming the fast weights. Unrolling Eq. (3) for a hidden layer l < L l < L l < L ,
α t Δ W t − 1 ( l ) h t ( l − 1 ) = − ∑ j < t ( ∏ i = j + 1 t α i ) η j ( l ) δ z j ( l ) ( h j ( l − 1 ) T h t ( l − 1 ) ) , (4) \alpha_t \Delta W^{(l)}_{t-1} h_t^{(l-1)} = -\sum_{j<t}\left(\prod_{i=j+1}^{t}\alpha_i\right)\eta_j^{(l)}\delta z_j^{(l)}\left(h_j^{(l-1)\mathsf{T}}h_t^{(l-1)}\right), \tag*{(4)} α t Δ W t − 1 ( l ) h t ( l − 1 ) = − j < t ∑ ( i = j + 1 ∏ t α i ) η j ( l ) δ z j ( l ) ( h j ( l − 1 ) T h t ( l − 1 ) ) , (4)
which is gated linear attention [62 ] with keys h j ( l − 1 ) h_j^{(l-1)} h j ( l − 1 ) , values − η j ( l ) δ z j ( l ) -\eta_j^{(l)}\delta z_j^{(l)} − η j ( l ) δ z j ( l ) and gates α j \alpha_j α j ; the sum excludes j = t j=t j = t because key t t t is read before its own write. We call a scan shifted when token t t t reads the state before its own update, W ‾ t − 1 \overline{\mathbf{W}}_{t-1} W t − 1 , and unshifted , the standard form, when it reads the state after the update, W t \mathbf{W}_t W t ; keys are read shifted and queries unshifted. One chunkwise scan thus gives the fast contribution to the pre-activations of layer l l l at all T T T tokens; adding W 0 ( l ) h t ( l − 1 ) + b ( l ) \mathbf{W}_0^{(l)}h_t^{(l-1)}+b^{(l)} W 0 ( l ) h t ( l − 1 ) + b ( l ) and applying σ \sigma σ gives the inputs of the next layer. The pass ends with the features ϕ 1 : T \phi_{1:T} ϕ 1 : T and stores the Jacobians J t ( l ) \mathbf{J}_t^{(l)} J t ( l ) for Step 3. When D = 0 \mathbf{D}=0 D = 0 on the first forward pass, the fast scan is identically zero and can be skipped.
Step 2: readout trajectory and its gradient. Unlike the hidden layers, the readout needs no trajectory estimate. Once its input features are fixed, its fast weights follow a Gated DeltaNet with keys ϕ t \phi_t ϕ t , corrected values v ~ t = v t − W 0 ( L ) ϕ t \tilde{\mathbf{v}}_t=\mathbf{v}_t-\mathbf{W}_0^{(L)}\phi_t v ~ t = v t − W 0 ( L ) ϕ t , gates α t \alpha_t α t and step sizes η t ( L ) \eta_t^{(L)} η t ( L ) . The chunkwise gated-delta-rule kernel [63 , 64 ] computes this recurrence exactly, while the slow readout W 0 ( L ) \mathbf{W}_0^{(L)} W 0 ( L ) is retained in the feature gradient and added to the final output (Section C.3).
The backward pass in Step 3 additionally requires each token’s loss gradient with respect to its features. We obtain this during the readout pass:
e t = η t ( L ) W ‾ t − 1 ( L ) ( W ‾ t − 1 ( L ) ϕ t − v t ) = η t ( L ) ∂ ℓ t ∂ ϕ t = − W ‾ t − 1 ( L ) T u t , (5) \mathbf{e}_t=\eta_t^{(L)}\overline{\mathbf{W}}_{t-1}^{(L)}\left(\overline{\mathbf{W}}_{t-1}^{(L)}\phi_t-\mathbf{v}_t\right)=\eta_t^{(L)}\frac{\partial\ell_t}{\partial\phi_t}=-\overline{\mathbf{W}}_{t-1}^{(L)\mathsf{T}}\mathbf{u}_t, \tag*{(5)} e t = η t ( L ) W t − 1 ( L ) ( W t − 1 ( L ) ϕ t − v t ) = η t ( L ) ∂ ϕ t ∂ ℓ t = − W t − 1 ( L ) T u t , (5)
where u t = η t ( L ) ( v t − W ‾ t − 1 ( L ) ϕ t ) \mathbf{u}_t=\eta_t^{(L)}\left(\mathbf{v}_t-\overline{\mathbf{W}}_{t-1}^{(L)}\phi_t\right) u t = η t ( L ) ( v t − W t − 1 ( L ) ϕ t ) is the update token t t t writes to the readout. The DELTARULEGRAD kernel computes this quantity as a small modification of the existing GDN kernel (Section C.3).
Step 3: parallel backward pass. The backward pass propagates each token’s feature gradient through the hidden layers along the trajectory from Step 1, for all T T T tokens at once, giving a recomputed gradient trajectory D ^ = ( δ z ^ t ( l ) ) l < L \hat{\mathbf{D}}=(\delta\hat{\mathbf{z}}_t^{(l)})_{l<L} D ^ = ( δ z ^ t ( l ) ) l < L . It requires the transposed Jacobian of σ \sigma σ and the transposed weights; the only operation involving a fast weight is the transpose of Eq. (4) ,
( α t Δ W t − 1 ( l ) ) T δ z t = − ∑ j < t ( ∏ i = j + 1 t α i ) η j ( l ) ( δ z j ( l ) T δ z t ) h j ( l − 1 ) , (6) \left(\alpha_t\Delta\mathbf{W}_{t-1}^{(l)}\right)^{\mathsf{T}}\delta\mathbf{z}_t=-\sum_{j<t}\left(\prod_{i=j+1}^{t}\alpha_i\right)\eta_j^{(l)}\left(\delta\mathbf{z}_j^{(l)\mathsf{T}}\delta\mathbf{z}_t\right)h_j^{(l-1)}, \tag*{(6)} ( α t Δ W t − 1 ( l ) ) T δ z t = − j < t ∑ ( i = j + 1 ∏ t α i ) η j ( l ) ( δ z j ( l ) T δ z t ) h j ( l − 1 ) , (6)
a shifted linear-attention read with the pseudo-labels as queries and keys and the layer inputs as values, computed by the same chunkwise scan. The estimate is then updated by a damped Picard step,
D [ n ] = D [ n − 1 ] + λ ( D ^ − D [ n − 1 ] ) , λ = 0.8. (7) \mathbf{D}^{[n]}=\mathbf{D}^{[n-1]}+\lambda\left(\hat{\mathbf{D}}-\mathbf{D}^{[n-1]}\right),\qquad\lambda=0.8. \tag*{(7)} D [ n ] = D [ n − 1 ] + λ ( D ^ − D [ n − 1 ] ) , λ = 0.8. (7)
Output. After N N N iterations, a final forward pass computes the features of the keys using shifted reads and of the queries using unshifted reads, since each query is evaluated under the updated weights. One gated-delta-rule pass with these features as keys and queries and corrected values v ~ t = v t − W 0 ( L ) ϕ k , t \tilde{\mathbf{v}}_t=\mathbf{v}_t-\mathbf{W}_0^{(L)}\phi_{k,t} v ~ t = v t − W 0 ( L ) ϕ k , t produces the fast contribution. Finally, adding W 0 ( L ) ϕ q , t \mathbf{W}_0^{(L)}\phi_{q,t} W 0 ( L ) ϕ q , t gives the output. § 3.3 shows that the exact sequential recurrence is a fixed point of this iteration and gives conditions under which the iterates converge to it.
Cost, training and decoding. Each iteration requires a constant number of chunkwise scans per memory layer, with sequential depth O ( T / C ) O(T/C) O ( T / C ) for chunk size C C C (Section G). We use N = 2 N=2 N = 2 unless stated otherwise and train all parameters by backpropagation through the N N N iterations. This finite-iteration computation approximates the sequential recurrence; its agreement with exact evaluation depends on the sequence length and training recipe (§ 4.3 and F.4). Decoding requires no iteration: with a single new token, Eq. (2) is one exact gradient step from the cached weights, so N N N affects prefill only.
Fixed point and convergence
Proposition 1 (Fixed point and finite-step exactness, informal). For any memory depth L L L , the gradient sequence D ⋆ \mathbf{D}^{\star} D ⋆ of the sequential recurrence Eq. (2) is a fixed point of the iteration of § 3.2 , and the parallel evaluation at D ⋆ \mathbf{D}^{\star} D ⋆ reproduces the sequential recurrence. Without damping (λ = 1 \lambda= 1 λ = 1 ), every iteration makes the estimate at one more token exact, so N ≥ T N \ge T N ≥ T iterations reproduce the sequential recurrence exactly.
Proposition 1, proved in Section B, needs no contraction assumption. For a geometric rate we analyze the iteration with L = 2 L = 2 L = 2 , the linear-attention hidden update Eq. (4) and the uncentered readout, with the unit-ball projection replaced, for the analysis only, by the soft clip n s o f t ( x ) = x / 1 + ∥ x ∥ 2 n_{\mathrm{soft}}(x) = x/\sqrt{1+\lVert x\rVert^2} n soft ( x ) = x / 1 + ∥ x ∥ 2 , so that the features ϕ t = ψ ( z t ) \phi_t = \psi(z_t) ϕ t = ψ ( z t ) , ψ = n s o f t ∘ σ \psi= n_{\mathrm{soft}}\circ\sigma ψ = n soft ∘ σ , have a globally Lipschitz Jacobian (Section B restates the setting). Write ∥ D ∥ ∞ = max t ∥ δ z t ( 1 ) ∥ \lVert\mathbf{D}\rVert_{\infty} = \max_t\lVert\delta z_t^{(1)}\rVert ∥ D ∥ ∞ = max t ∥ δ z t ( 1 ) ∥ and F F F for one iteration’s map D ↦ D ^ \mathbf{D}\mapsto\widehat{\mathbf{D}} D ↦ D .
Theorem 1 (Convergence of ParaTTT with soft clipping, L = 2 L = 2 L = 2 ). Assume ∥ k t ∥ ≤ 1 \lVert k_t\rVert\le1 ∥ k t ∥ ≤ 1 , ∥ v t ∥ ≤ V \lVert v_t\rVert\le V ∥ v t ∥ ≤ V , ∥ W 0 ( 2 ) ∥ o p ≤ w 0 \lVert W_0^{(2)}\rVert_{\mathrm{op}} \le w_0 ∥ W 0 ( 2 ) ∥ op ≤ w 0 , a shared decay α t ≤ α ˉ < 1 \alpha_t \le\bar{\alpha} < 1 α t ≤ α ˉ < 1 , a shared learning rate η t ( 1 ) = η t ( 2 ) ≤ η ˉ ≤ 1 \eta_t^{(1)} = \eta_t^{(2)} \le\bar{\eta} \le1 η t ( 1 ) = η t ( 2 ) ≤ η ˉ ≤ 1 , and ∣ σ ′ ∣ ≤ c σ |\sigma'| \le c_{\sigma} ∣ σ ′ ∣ ≤ c σ , ∣ σ ′ ′ ∣ ≤ h σ |\sigma''| \le h_{\sigma} ∣ σ ′′ ∣ ≤ h σ . With ϵ = η ˉ / ( 1 − α ˉ ) \epsilon= \bar{\eta}/(1-\bar{\alpha}) ϵ = η ˉ / ( 1 − α ˉ ) and Q = V + w 0 Q = V+w_0 Q = V + w 0 , F F F is globally Lipschitz in ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_{\infty} ∥ ⋅ ∥ ∞ with constant
ρ = ϵ ( 1 + ϵ ) Q 2 [ c σ 2 ( 1 + ϵ ) ( 5 + 4 ϵ ) + h σ ( 2 + ϵ ) ] ≤ ( 10 c σ 2 + 3 h σ ) ϵ Q 2 for ϵ ≤ 1 4 . (8) \rho= \epsilon(1+\epsilon)Q^2\left[c_{\sigma}^2(1+\epsilon)(5+4\epsilon)+h_{\sigma}(2+\epsilon)\right]\le(10c_{\sigma}^2+3h_{\sigma})\epsilon Q^2\qquad\text{for }\epsilon\le\frac{1}{4}. \tag*{(8)} ρ = ϵ ( 1 + ϵ ) Q 2 [ c σ 2 ( 1 + ϵ ) ( 5 + 4 ϵ ) + h σ ( 2 + ϵ ) ] ≤ ( 10 c σ 2 + 3 h σ ) ϵ Q 2 for ϵ ≤ 4 1 . (8) If ρ < 1 \rho< 1 ρ < 1 , F F F has a unique fixed point D ⋆ \mathbf{D}^{\star} D ⋆ , the gradient sequence of the sequential recurrence Eq. (2) with n s o f t n_{\mathrm{soft}} n soft in place of the projection, and for any initialization and damping λ ∈ ( 0 , 1 ] \lambda\in(0,1] λ ∈ ( 0 , 1 ] the iteration Eq. (7) satisfies ∥ D [ n ] − D ⋆ ∥ ∞ ≤ ( 1 − λ ( 1 − ρ ) ) n ∥ D [ 0 ] − D ⋆ ∥ ∞ \lVert\mathbf{D}^{[n]}-\mathbf{D}^{\star}\rVert_{\infty}\le(1-\lambda(1-\rho))^n\lVert\mathbf{D}^{[0]}-\mathbf{D}^{\star}\rVert_{\infty} ∥ D [ n ] − D ⋆ ∥ ∞ ≤ ( 1 − λ ( 1 − ρ ) ) n ∥ D [ 0 ] − D ⋆ ∥ ∞ ; the features, readout states and outputs converge at the same rate, up to constants.
The proof (Section B) controls the features and their changing Jacobians; SiLU satisfies the derivative assumptions, and ρ < 1 \rho< 1 ρ < 1 is independent of the sequence length. Section B extends the analysis to the hidden-layer delta rule (Theorem 2) and centering (Corollary 1), while retaining the soft clip and sufficient contraction conditions. These results identify a regime in which convergence is guaranteed. The trained models use RMS-normalized keys and a hard projection, and their learned rates and gates lie outside the contraction conditions (Section F.5). We measure their agreement with the exact recurrence directly in § 4.3 and F.4.
Practical design choices
The analysis above uses the simplest form of the iteration. Our reported models use two design choices that substantially improve practical performance: a hidden-layer delta rule that preserves the target sequential recurrence, and centering, which changes the target to the sequential recurrence with the centered readout.
Delta rule for the hidden layers. Step 1 of the iteration evolves each hidden layer’s fast weights by gated linear attention Eq. (4) , which sums the writes − η t ( l ) δ z t ( l ) h t ( l − 1 ) T -\eta_t^{(l)}\delta z_t^{(l)}h_t^{(l-1)\mathsf{T}} − η t ( l ) δ z t ( l ) h t ( l − 1 ) T with a scalar decay. The same weights can instead be evolved by a gated delta rule whose target is built from the previous iteration’s read. The update is Eq. (3) plus a correction that erases stale writes along the iterates (Section C.2). This correction vanishes at the fixed point, so the target sequential recurrence is unchanged. The delta rule costs slightly more but gives substantially better recall (Table 1 ), and we therefore use it throughout.
Table 1. Main results at 340M and 1.3B. Recall follows the generation protocol of Arora et al. [1 ] ; NIAH is the RULER needle average at the given length. Best per column and size in bold, second best underlined.
Centering. The hidden layers receive gradient only through the readout. By Eq. (5) , e t e_t e t is proportional to W ‾ t − 1 ( L ) T \overline{W}_{t-1}^{(L)\mathsf{T}} W t − 1 ( L ) T , so a zero-initialized readout gives the hidden layers no gradient at the start of a sequence. Conversely, a nonzero W 0 ( L ) W_0^{(L)} W 0 ( L ) provides a nonzero gradient immediately, but also adds a persistent bias W 0 ( L ) ϕ t W_0^{(L)}\phi_t W 0 ( L ) ϕ t to the output. We use centering to obtain a nonzero readout Jacobian at initialization while keeping the initial output zero. Centering trains f θ − f θ 0 f_{\theta}-f_{\theta_0} f θ − f θ 0 in place of f θ f_{\theta} f θ ([8 ] ); related zero-output initializations appear in [60 , 26 ] . We subtract from the readout its value on the features ϕ 0 , t \phi_{0,t} ϕ 0 , t of the slow MLP, that is, f W ( k t ) = W ( L ) ϕ t − W 0 ( L ) ϕ 0 , t f_W(k_t)=W^{(L)}\phi_t-W_0^{(L)}\phi_{0,t} f W ( k t ) = W ( L ) ϕ t − W 0 ( L ) ϕ 0 , t . At the start of a sequence, ϕ t = ϕ 0 , t \phi_t=\phi_{0,t} ϕ t = ϕ 0 , t so the readout is zero, while its Jacobian W 0 ( L ) + Δ W ( L ) W_0^{(L)}+\Delta W^{(L)} W 0 ( L ) + Δ W ( L ) is nonzero from the first token. Centering changes only the values passed to Step 2 and adds one term to the output (Section C.5).
Other choices. We use L = 2 L = 2 L = 2 with a hidden width of d d d , SiLU as σ \sigma σ , and a learned per-head gain on the fast part of the hidden pre-activation. Setting the hidden learning rates to zero yields a Gated DeltaNet on the features of the frozen MLP, the N = 0 N = 0 N = 0 control.
Empirical study
Setup. All models use 24 pre-norm residual blocks, each with a token-mixing layer and SwiGLU MLP. The backbone width is d = 1024 d = 1024 d = 1024 at 340M and d = 2048 d = 2048 d = 2048 at 1.3B, with a default head dimension of 128; TTT-MLP and LaCT use the layouts in Section D. We use short QKV convolutions as in Gated DeltaNet (Yang et al., [64 ] ). ParaTTT has one hidden layer of width equal to the head dimension, the hidden-layer delta rule and centered readout (§ 3.4 ), and N = 2 N = 2 N = 2 training iterations unless stated otherwise. We report the no-SWA LaCT configuration in Section D and compare LaCT, Gated DeltaNet and ParaTTT with the same sliding window in Section F.2.
Language modeling performance
Model set and evaluation. At 340M, we compare Gated DeltaNet, Mamba-2, attention, TTT-MLP, LaCT without SWA, and ParaTTT, including its zero-init readout, linear-attention body and N = 5 N = 5 N = 5 variants. At 1.3B, we compare Gated DeltaNet, Mamba-2, attention and ParaTTT. Models share the training recipe, data order and seed, using 15.7B SlimPajama tokens at 340M and 100B at 1.3B (Section D). Comparisons match data and steps, with one seed per ParaTTT configuration; compute and state size are not matched. We evaluate WikiText-103 and LAMBADA perplexity, zero-shot accuracy with lm-evaluation-harness (Gao et al. [15 ] ), recall on SWDE, FDA, TriviaQA and DROP following Arora et al. [1 ] , and RULER NIAH (Hsieh et al. [24 ] ) at 4K–32K tokens, using YaRN (Peng et al., [44 ] ) for attention. Section E details the protocol.
Main results. Table 1 gives the language-modeling results. ParaTTT has the best perplexity of the recurrent models at both scales, the best LAMBADA perplexity among the main model comparisons, and stronger recall than both Gated DeltaNet and Mamba-2 on SWDE and FDA at both scales. Recall tasks use exact recurrent decoding, so the prefill–decode discrepancy does not prevent strong recall. Attention performs best on many recall and long-context tasks. Despite its higher training cost, TTT-MLP matches Gated DeltaNet on WikiText-103, falls far behind on LAMBADA and offers little improvement on recall. We ablate the practical design choices from § 3.4 : the zero-init readout ablation, which removes centering and sets the initial readout to zero, degrades perplexity and recall and greatly reduces the gains from iteration (Figure 3 (c,Figure 3 d)), while replacing the hidden-layer delta rule with linear attention leaves perplexity similar but substantially degrades recall. Table 7 separately isolates centering from the learned initialization. Training with N = 5 N = 5 N = 5 improves all four recall tasks at a proportionally higher cost per step, but raises LAMBADA perplexity from 26.93 to 28.44 and reduces 32K NIAH accuracy from 16.0 to 5.8. The LaCT configuration without SWA performs substantially worse than the published hybrid. Section F.2 compares LaCT, Gated DeltaNet and ParaTTT with the same sliding window (Table 6 ). We find that ParaTTT is ahead of both LaCT variants on perplexity, most recall tasks, LongBench and NIAH.
Figure 3. (a,b) Perplexity along 698 held-out SlimPajama documents at 340M and 1.3B. (c,d) LAMBADA perplexity and SWDE accuracy of the 340M ParaTTT models against the number of iterations at evaluation; filled marker: training N N N ; dotted line: Gated DeltaNet; off-chart loop-off values are printed at the edge. The uncentered model uses a zero-initialized readout.
Table 6. 340M models with and without a 2048-token sliding-window attention branch; same recipe, data and seed as Table 1 . “+ norm” applies RMS normalization to the attention’s queries, keys and values. LaCT without a window is the configuration of Table 1 , detailed in Section D. LaCT (as published) is a second run of the published hybrid configuration; the first run reaches 25.51 on WikiText-103 and 33.3 on LAMBADA. Best per column among the models with a window in bold.
Table 7. Centering ablation at 340M. Three parameterizations of the readout layer, trained with the same recipe, initialization and data order; the gated DeltaNet baseline is shown for reference. Columns as in Table 1 ; best of the three PTTT variants per column in bold.
Perplexity as a function of token position. Figure 3 (a,Figure 3 b) shows that ParaTTT has the lowest perplexity throughout the 4096-token window at both scales, so its advantage over Gated DeltaNet persists beyond the initial tokens. Attention has the worst cold start and catches up with the recurrent models after a few hundred tokens.
Performance as a function of the evaluation-time iteration count. Figure 3 (c,Figure 3 d) evaluates the 340M ParaTTT models at iteration counts other than their training N N N . Without iteration, the centered models perform poorly (LAMBADA perplexity above 300), whereas the zero-init readout ablation performs well but gains little from additional iterations, suggesting that centering is necessary to unlock the full gains from refinement. Models perform best near their training N N N : over-iterating the model trained at N = 2 N = 2 N = 2 degrades performance slowly, while the model trained at N = 5 N = 5 N = 5 exhibits little degradation. This suggests that the model adapts to the finite-N N N prefill used during training, even though exact recurrent decoding remains strong on the recall tasks. The stronger degradation at 1.3B (Table 4 ) motivates training the model with more than one iteration count. We test two recipes: one switches to N = 5 N = 5 N = 5 near the end of training, while the other samples N N N independently per layer throughout training (details in Section F.4). Both reduce sensitivity to the evaluation-time N N N , with the randomized recipe giving the more stable sweep. At the usual N = 2 N = 2 N = 2 evaluation, the randomized model also improves long-context retrieval and exceeds Gated DeltaNet on NIAH and LongBench at every tested length; the final phase at N = 5 N = 5 N = 5 improves SWDE (Table 1 and Table 5 ). These gains come with some trade-offs: both variants score lower on FDA, and randomized N N N raises LAMBADA perplexity relative to the original N = 2 N = 2 N = 2 -trained model. The extra training cost is modest (Section G).
Table 4. Perplexity (WikiText-103, LAMBADA) and SWDE accuracy of the ParaTTT models as a function of the number of iterations at evaluation. Bold: best N N N for the centered models.
Table 5. LongBench: average score over the 16 English tasks (100 samples per task) at context lengths 4K–32K, for the models of Table 1 . Best per column and size in bold, second best underlined. All models trained at 4K; ParaTTT is within a point of Gated DeltaNet at every length and scale for the original N = 2 N = 2 N = 2 models, and attention leads at 1.3B. The two 1.3B variants are those of Table 1 , evaluated at N = 2 N = 2 N = 2 ; the Poisson-N N N variant is above Gated DeltaNet at every length.
Efficiency
On MI325X GPUs in bf16, we measure forward+backward time for one token-mixing layer (d = 1024 d = 1024 d = 1024 , 8 heads of dimension 128, batch 8) and full-model training throughput on eight GPUs (Section G). Figure 4 compares ParaTTT at N ∈ { 1 , 2 , 3 , 5 } N \in\{1, 2, 3, 5\} N ∈ { 1 , 2 , 3 , 5 } with Gated DeltaNet (Yang et al., [64 ] ), Mamba-2 (Dao & Gu, [11 ] ), FlashAttention, TTT-MLP at its default block size of 16 (Sun et al., [50 ] ), and published LaCT with a 2048-token sliding window (Zhang et al., [65 ] ).
Figure 4. Efficiency. Forward+backward time of one token-mixing layer against (a) sequence length and (b) head dimension; (c) training throughput of the full 340M models on eight MI325X GPUs. The fused TTT-MLP implementation used in our benchmarks supports head dimension 64; hollow markers show a compiled scan at larger dimensions.
Results. ParaTTT scales linearly in T T T like the linear recurrences it is built from: one layer at N = 2 N = 2 N = 2 costs 4.7× a GDN layer, and each additional iteration adds a fixed number of chunkwise scans. TTT-MLP’s block-sequential form is 5.5× slower than ParaTTT at T = 4096 T = 4096 T = 4096 and 8.5× at T = 16,384 T = 16{,}384 T = 16 , 384 . At the model level, ParaTTT at N = 2 N = 2 N = 2 trains at 37.9k tokens/s per node against 7.8k for TTT-MLP, a 4.9× speedup, while retaining roughly one-third the throughput of the linear baselines. During decode, the iterations are replaced by one exact recurrent update per token. At batch size 1, ParaTTT’s latency remains nearly constant across the tested context lengths and is close to Gated DeltaNet’s, while remaining less than half that of TTT-MLP and LaCT (Section G.1).
Convergence analysis
ParaTTT better approximates the exact online recurrence than TTT-MLP. Figure 5 (a) compares the output error relative to exact online gradient descent against recurrence evaluation time for ParaTTT and TTT-MLP on the same random inner MLP (d = 64 d = 64 d = 64 , eight heads, step size 0.2, decay 0.98). We use the faster of TTT-MLP’s fused kernel and compiled scan at each block size. Reducing TTT-MLP’s error requires smaller blocks and a longer sequential loop; ParaTTT instead increases the number of parallel refinements. At T = 4096 T = 4096 T = 4096 , N = 2 N = 2 N = 2 matches TTT-MLP’s error at b ≈ 5 b \approx5 b ≈ 5 in 32× less time, and N = 8 N = 8 N = 8 approaches the bf16 noise floor. § 15 repeats this comparison at other sequence lengths.
Figure 5. Convergence to the sequential recurrence. (a) One random inner MLP at T = 4096 T = 4096 T = 4096 : relative output error against exact online GD versus the time to compute one 340M layer’s recurrence, for ParaTTT at N = 1 N = 1 N = 1 –8 8 8 and TTT-MLP’s block form at b = 4 b = 4 b = 4 –256 256 256 (filled: fused kernel; hollow: compiled scan); the arrow marks equal error to N = 2 N = 2 N = 2 . (b,c) Trained models on real text: top-1 agreement and total-variation distance between the prefill at N N N iterations and the exact recurrence.
Agreement with the exact recurrence in trained models. Figure 5 (b,Figure 5 c) compares finite-iteration prefill with the exact token-by-token recurrence on 512-token sequences, using top-1 agreement and total-variation distance between next-token distributions. A few iterations give close agreement, with diminishing gains thereafter, despite the trained models lying outside Theorem 1’s assumptions. The longer-document tests in Section F.4 show a larger discrepancy that increasing N N N only partly recovers over the tested range. Varying N N N during training reduces the perplexity penalty from exact recurrent decoding, but does not eliminate the mismatch.
Conclusion
We introduce ParaTTT, an algorithm for parallelizing test-time training layers with arbitrarily deep memories through iterative refinement of the inner-loop gradient trajectory. The exact online gradient descent recurrence is a fixed point, and we prove geometric convergence for a two-layer variant under sufficient contraction conditions. Training and prefill use a finite-iteration approximation, while decoding applies exact recurrent updates. With two iterations at 4K context, full 340M models train 4.9× faster than TTT-MLP, while models at 340M and 1.3B achieve strong language-modeling performance and stronger recall on most tasks than the linear recurrent baselines. These results narrow the long-standing tradeoff between the expressivity of nonlinear recurrence and the efficiency of parallel sequence processing.
Acknowledgements
This work was supported by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Extended related work
Linear recurrences. The lineage in § 2.1 runs from linear attention [29 ] through gated linear attention [62 ] , Mamba-2 [11 ] , DeltaNet [47 , 63 ] and Gated DeltaNet [64 ] ; Wang et al. [56 ] give the unifying view of all of these as test-time regression. The RWKV family follows the same path from a vector-valued to a matrix-valued state [42 ] and, in RWKV-7, to a generalized delta rule with vector-valued gates and in-context learning rates [43 ] . The most recent linear recurrences refine the delta rule’s write and forget. KDA [30 ] replaces Gated DeltaNet’s scalar decay by a channel-wise one; Gated DeltaNet-2 [22 ] further decouples the erase gate from the write gate, recovering both KDA and Gated DeltaNet as special cases; Mamba-3 [32 ] revisits the discretization of the state-space recurrence and adds complex-valued and multi-input multi-output updates; DeltaProduct [49 ] takes several delta steps per token so that the transition is a product of generalized Householder matrices; Comba [25 ] adds output feedback to the bilinear state transition; Lattice [28 ] writes only the component of a token orthogonal to the stored memory slot, with an approximate chunkwise evaluation; Momentum DeltaNet [27 ] gives the online gradient step a momentum term; Preconditioned DeltaNet [53 ] preconditions it with an online estimate of the curvature; and MesaNet [55 ] solves the in-context regression to optimality with a conjugate-gradient solver. These methods retain matrix-valued memories [41 ] but use different update and evaluation schemes: momentum introduces an auxiliary state, MesaNet adds a per-query solve, and Lattice approximates a state-dependent update. ParaTTT uses chunkwise linear recurrent kernels as building blocks. Extending its readout to these other updates is a possible direction, but would require checking the corresponding solver and gradient computation.
Test-time training. TTT-Linear and TTT-MLP [50 ] introduced the layer whose state is the weights of a small model trained online, evaluated in the mini-batch form of § 2.2 . It is worth writing this form out, since it is the approximation that ParaTTT removes. Let a block consist of tokens s + 1 , … , s + b s+1,\ldots,s+b s + 1 , … , s + b and let W s \mathbf{W}_s W s be the weights entering it (TTT-MLP uses no decay, so we drop α \alpha α ). Exact online descent evaluates the gradient of token j j j at the weights left by j − 1 j-1 j − 1 , W j = W j − 1 − η j ∇ ℓ j ( W j − 1 ) \mathbf{W}_j=\mathbf{W}_{j-1}-\eta_j\nabla\ell_j(\mathbf{W}_{j-1}) W j = W j − 1 − η j ∇ ℓ j ( W j − 1 ) ; the block form evaluates every gradient of the block at the weights that entered it and scores token t t t with the causal partial sum of these stale gradients,
W t b l k = W s − ∑ j = s + 1 t η j ∇ ℓ j ( W s ) , s < t ≤ s + b . (9) \mathbf{W}^{\mathrm{blk}}_t=\mathbf{W}_s-\sum_{j=s+1}^{t}\eta_j\nabla\ell_j(\mathbf{W}_s),\qquad s<t\leq s+b. \tag*{(9)} W t blk = W s − j = s + 1 ∑ t η j ∇ ℓ j ( W s ) , s < t ≤ s + b . (9)
For a squared-loss MLP with a linear readout, the readout update has an attention-style form inside the block. In the next display, W \mathbf{W} W denotes the readout weights only. With ϕ j \phi_j ϕ j the features of token j j j , computed with the block-start hidden weights, the gradient at W s \mathbf{W}_s W s is the outer product ( W s ϕ j − v j ) ϕ j ⊤ (\mathbf{W}_s\phi_j-v_j)\phi_j^\top ( W s ϕ j − v j ) ϕ j ⊤ of a residual with a feature. Let ϕ q , t \phi_{q,t} ϕ q , t denote the query features under the causally updated hidden weights at token t t t ; then the output is
o t = W t b l k ϕ q , t = W s ϕ q , t − ∑ j = s + 1 t η j ( W s ϕ j − v j ) ( ϕ j ⊤ ϕ q , t ) , (10) \mathbf{o}_t=\mathbf{W}^{\mathrm{blk}}_t\phi_{q,t}=\mathbf{W}_s\phi_{q,t}-\sum_{j=s+1}^{t}\eta_j\left(\mathbf{W}_s\phi_j-v_j\right)\left(\phi_j^\top\phi_{q,t}\right), \tag*{(10)} o t = W t blk ϕ q , t = W s ϕ q , t − j = s + 1 ∑ t η j ( W s ϕ j − v j ) ( ϕ j ⊤ ϕ q , t ) , (10)
an attention-style readout with keys ϕ j \phi_j ϕ j , values the residuals W s ϕ j − v j \mathbf{W}_s\phi_j-v_j W s ϕ j − v j and queries ϕ q , t \phi_{q,t} ϕ q , t , so the readout sum can be evaluated with matrix multiplications once the features are given. The price is in the residuals: exact online descent would use the preceding readout weights and the current online features, so that the write of token j j j erases what earlier tokens of the block stored under similar keys (the delta rule), whereas the block form’s residuals see none of the block’s earlier writes and the writes simply add up, as in linear attention without a delta rule. The hidden-layer gradients are also evaluated at block-start weights. This freezes the features used to compute those gradients, but does not freeze the hidden weights or query features used for each token’s output. At b = 1 b=1 b = 1 the block form recovers online descent; at b = T b=T b = T all gradient evaluations share the initial weights. TTT-MLP’s default b = 16 b=16 b = 16 and LaCT’s larger blocks trade sequential depth against update frequency, and Figure 5 (a) measures the resulting error against the exact trajectory. ParaTTT instead targets the b = 1 b=1 b = 1 recurrence by iterating on the gradients.
Liu et al. [35 ] study LaCT-style and vision TTT layers. They report that adding inner steps only at inference can hurt downstream performance, that retraining with gradient ascent can preserve performance, and that replacing queries with keys can have little effect in their experiments. Their attention-style representation allows updated hidden features; it is not limited to a frozen-feature model. They also obtain parallel variants through further simplifications, including restricting adaptation to the final layer. These findings motivate testing which parts of nonlinear adaptation matter. Our experiments address this question for the online recurrence, but do not establish that the block approximation explains their observations.
Titans (Behrouz et al. (2025b)) adds momentum and weight decay to the inner optimizer and ATLAS (Behrouz et al. (2025a)) optimizes the memory over a window of past tokens rather than the current one; LaCT (Zhang et al. [65 ] ) makes the block very large so that the inner update is a dense matrix multiplication, and pairs it with a sliding-window attention branch (Section F.2). Two recent lines move the fast weights into the backbone. In-place TTT (Feng et al. [14 ] ) treats the down-projection of every MLP block as the fast weights, trains it with an objective aligned with next-token prediction, and updates it chunk-wise; end-to-end TTT (Tandon et al. [51 ] ) keeps a standard transformer with sliding-window attention and adapts selected MLP weights on the context by next-token prediction, meta-learned so that the test-time updates help. In its reported configuration, adaptation is restricted to the last quarter of the blocks. Both inherit the block structure: the weights are frozen inside a chunk or a window and updated between them. ParaTTT is orthogonal to these choices of what to adapt and with which loss; it changes how the inner trajectory of a nonlinear memory is computed, so that every token’s update is taken at the weights left by the previous token, and it does so with the parallel kernels of the linear recurrences above.
Sequential nonlinear RNNs. Nonlinear recurrent networks ([13 , 23 , 9 , 52 ] ) are trained by backpropagation through time (BPTT) ([45 , 58 ] ), which unrolls the recurrence over the sequence and runs the backward pass through every step in reverse. Their straightforward forward evaluation and BPTT are sequential over t t t , limiting sequence parallelism compared with attention ([54 ] ). The sLSTM cell of xLSTM ([4 ] ) keeps a sequential nonlinear transition next to a parallel linear one. M 2 M^2 M 2 RNN ([39 ] ) is the most direct recent attempt to make the sequential path competitive: it keeps a matrix-valued state, updated by a tanh of a linear map of the previous state plus an outer-product write with a learned forget gate, so that the state size scales as it does in linear attention, and it evaluates the recurrence sequentially in a Triton kernel that parallelizes over batch and heads while looping over time; the backward pass is full BPTT over the 4096-token training sequence, recomputing the forward states and running the reverse loop over every step. The nonlinear transition prevents direct evaluation by the linear scans used by SSMs. The paper reports perfect length generalization on its evaluated state-tracking tasks, with training throughput below its linear recurrent baselines. ParaTTT targets the same expressivity, a nonlinear transition on a large state, but it obtains a parallel evaluation of it, and its nonlinearity acts on the features that the memory reads and writes with rather than on the state transition directly.
Parallelizing nonlinear recurrences and its limits. DEER (Lim et al. [33 ] ) cast the evaluation of a nonlinear recurrence as a fixed-point problem over wholet trajectory and solved it with Newton’s method, each step a linear recurrence evaluated by a parallel scan; ELK and quasi-DEER (Gonzalez et al. [18 ] ) stabilize the Newton iteration through its connection to Kalman smoothing and make it scale by using diagonal Jacobians; ParaRNN (Danieli et al. [10 ] ) designs LSTM and GRU cells with structured Jacobians so that the Newton step is cheap and trains 7B-parameter nonlinear RNNs this way; Gonzalez et al. [19 ] show that the number of iterations these methods need is governed by the predictability of the dynamics, measured by the largest Lyapunov exponent, and is polylogarithmic in the sequence length for contractive systems. Our Theorem 1 is a statement of the same kind for our iteration: the contraction condition bounds the learning rate times the memory horizon, which is the quantity that controls how far a token’s update can propagate. A small iteration count improves parallelism, but its expressivity requires a separate analysis. An associative scan generally has logarithmic circuit depth; a fixed number of scans alone does not establish a T C 0 \mathrm{TC}^{0} TC 0 upper bound. The bounds of Merrill & Sabharwal [36 ] ; Merrill et al. [38 ] apply to the architectures and precision assumptions they analyze. For transformers, Merrill & Sabharwal [37 ] show that logarithmic depth suffices to recognize regular languages. We do not claim a circuit-complexity classification for ParaTTT. Our finite-step result instead shows that N ≥ T N \ge T N ≥ T recovers its sequential recurrence in the undamped case (Proposition 1). The practical question, which § 4.3 addresses, is how many iterations are needed for the tasks that matter.
Looped and implicit models. A second family iterates a block of layers in depth rather than a recurrence in time. Universal transformers (Dehghani et al., [12 ] ) apply the same layer repeatedly with adaptive halting; deep equilibrium models (Bai et al., [3 ] ) take the limit and define the output as the fixed point of a layer, found by a root solver and differentiated implicitly; looped transformers were shown to be programmable computers (Giannou et al., [17 ] ) and to match the reasoning ability of much deeper unlooped models (Saunshi et al., [46 ] ), and recent language models scale test-time compute by unrolling a recurrent block to a chosen depth (Geiping et al., [16 ] ; Zhu et al., [66 ] ). Fixed-point reasoners (Movahedi et al., [40 ] ) use convergence to a fixed point as the halting criterion of a looped transformer and add damping to remove oscillations of the iterates, and Labovich (2026) analyzes when the fixed points of a looped transformer are reachable, input-dependent and stable under backpropagation, and finds that recall connections with outer normalization improve stability and generalization in their studied architectures. Closest to our work, Schöne et al. (2025) iterate a state-space model over the whole sequence until convergence to a fixed point and show that the implicit model implements the nonlinear state transition of an RNN; like ParaTTT, they observe that approximate convergence suffices during training, and they scale the approach to 1.3B parameters. The essential difference is at decoding time. In recurrent-depth models, the chosen number of loops is part of the token computation; in fixed-point models, the output is defined by an equilibrium. Both require repeated computation per generated token, including in the sequential mode of Schöne et al. (2025). For ParaTTT the loop is only a method for computing a prefill; the object being computed is the exact sequential recurrence of Eq. (2) , which for one new token is a single gradient step from the cached weights. Decoding therefore requires no iteration at all (§ 3.2 ), and N N N affects the cost and the accuracy of the prefill only.
Parallelizing continuous chain of thought. A third family feeds a hidden state of one decoding step back into the input of the next, a nonlinear recurrence in time whose state is a full residual-stream vector rather than a token. Coconut (Hao et al., [21 ] ) feeds the last hidden state of a “continuous thought” position back as the next input embedding; because each thought depends on the previous one, n n n thoughts cost n + 1 n+1 n + 1 sequential forward passes at training time, which the authors note is the main obstacle to parallelism. Parallel continuous chain of thought (Wu et al., [61 ] ) removes the sequential dependence with the same tool we use: starting from dummy thought inputs, all thought positions are updated in parallel by one Jacobi iteration, each pass feeding the previous pass’s hidden states forward by one position, and the iteration is exact after as many passes as there are thoughts (their Theorem 1 is the finite-step exactness of our Proposition 1). They train and decode with a few extra passes (three, for 24 thoughts) and observe that the thought vectors do not in fact converge after training and that training with more than about six passes becomes unstable. Two recent language models apply the same construction to the whole sequence. T 2 T^2 T 2 MLR (Cai et al., [7 ] ) fuses a cache of a middle-layer representation from the previous token into an earlier layer of the current token, and the full-bandwidth transformer (Wang et al., [57 ] ) fuses the previous token’s top-layer state with the next token embedding through a gated linear unit. Both train by Jacobi iteration over the sequence, re-running the affected layers in parallel with the previous pass’s states shifted right by one position, and both decode sequentially with one pass per token, as ParaTTT does. They differ from each other in how the iteration is optimized: the full-bandwidth transformer uses at most three passes with gradients through all of them, on a schedule in which most batches use a single pass, whereas T 2 T^2 T 2 MLR runs sixteen passes but backpropagates only through the last four, detaching the earlier ones in the manner of truncated backpropagation through time, and reports that this approximation matches exact recurrent training to within 0.005 in validation loss at 135M to 1B parameters. The recurrences in these models run through many layers of a transformer, so each Jacobi pass is a forward pass of the network and the number of passes is kept small. In ParaTTT the recurrence is a gradient step on the weights of one fast-weight layer, the iteration acts on that layer’s pre-activation gradients, and each pass is a chunkwise scan inside one layer’s kernel, so the iteration is cheap enough that we backpropagate through all N N N passes and the analysis of § 3.3 applies to the converged object.
Proofs
Setting, restated. We analyze the iteration with L = 2 L = 2 L = 2 , the linear-attention hidden update Eq. (4) , and the uncentered readout. For this analysis only, replace the unit-ball projection used in the experiments with the soft clip n s o f t ( x ) = x / 1 + ∥ x ∥ 2 n_{\mathrm{soft}}(\mathbf{x}) = \mathbf{x}/\sqrt{1+\lVert\mathbf{x}\rVert^2} n soft ( x ) = x / 1 + ∥ x ∥ 2 . This keeps the features bounded and, under the activation assumptions below, makes their Jacobian globally Lipschitz. Write ψ = n s o f t ∘ σ \psi= n_{\mathrm{soft}} \circ\sigma ψ = n soft ∘ σ , where σ \sigma σ acts coordinatewise. The features of this smooth variant are
ϕ t = ψ ( z t ) , z t = W 0 ( 1 ) k t + b ( 1 ) + α t Δ W t − 1 ( 1 ) k t . \phi_t = \psi(\mathbf{z}_t), \qquad\mathbf{z}_t = \mathbf{W}^{(1)}_0\mathbf{k}_t + \mathbf{b}^{(1)} + \alpha_t \Delta\mathbf{W}^{(1)}_{t-1}\mathbf{k}_t. ϕ t = ψ ( z t ) , z t = W 0 ( 1 ) k t + b ( 1 ) + α t Δ W t − 1 ( 1 ) k t .
Step 3 uses the exact Jacobian H t = D ψ ( z t ) \mathbf{H}_t = D\psi(\mathbf{z}_t) H t = D ψ ( z t ) at the current pre-activation. Write ∥ D ∥ ∞ = max t ∥ δ z t ( 1 ) ∥ \lVert\mathbf{D}\rVert_\infty= \max_t\lVert\delta\mathbf{z}^{(1)}_t\rVert ∥ D ∥ ∞ = max t ∥ δ z t ( 1 ) ∥ and F F F for one iteration’s map D ↦ D ^ \mathbf{D} \mapsto\widehat{\mathbf{D}} D ↦ D .
Theorem (Theorem 1, restated). *For the soft-clipped variant defined above, assume ∥ k t ∥ ≤ 1 \lVert\mathbf{k}_t\rVert\le1 ∥ k t ∥ ≤ 1 , ∥ v t ∥ ≤ V \lVert\mathbf{v}_t\rVert\le V ∥ v t ∥ ≤ V , and ∥ W ( 2 ) ∥ o p ≤ w 0 \lVert\mathbf{W}^{(2)}\rVert_{\mathrm{op}} \le w_0 ∥ W ( 2 ) ∥ op ≤ w 0 ; shared decay 0 ≤ α t ≤ α ˉ < 1 0 \le\alpha_t \le\bar{\alpha} < 1 0 ≤ α t ≤ α ˉ < 1 and shared learning rates 0 < η t ( 1 ) = η t ( 2 ) = η t ≤ η ˉ ≤ 1 0 < \eta^{(1)}_t = \eta^{(2)}_t = \eta_t \le\bar{\eta} \le1 0 < η t ( 1 ) = η t ( 2 ) = η t ≤ η ˉ ≤ 1 ; and σ ∈ C 2 ( R ) \sigma\in C^2(\mathbb{R}) σ ∈ C 2 ( R ) with ∣ σ ′ ( x ) ∣ ≤ c σ \lvert\sigma'(x)\rvert\le c_\sigma ∣ σ ′ ( x )∣ ≤ c σ and ∣ σ ′ ′ ( x ) ∣ ≤ h σ \lvert\sigma''(x)\rvert\le h_\sigma ∣ σ ′′ ( x )∣ ≤ h σ for every x ∈ R x \in\mathbb{R} x ∈ R . Set ϵ = η ˉ / ( 1 − α ˉ ) \epsilon= \bar{\eta}/(1-\bar{\alpha}) ϵ = η ˉ / ( 1 − α ˉ ) and Q = V + w 0 Q = V + w_0 Q = V + w 0 . Then F F F is globally Lipschitz in ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ with constant
ρ = ϵ ( 1 + ϵ ) Q 2 [ c σ 2 ( 1 + ϵ ) ( 5 + 4 ϵ ) + h σ ( 2 + ϵ ) ] . \rho= \epsilon(1+\epsilon)Q^2\left[c_\sigma^2(1+\epsilon)(5+4\epsilon)+h_\sigma(2+\epsilon)\right]. ρ = ϵ ( 1 + ϵ ) Q 2 [ c σ 2 ( 1 + ϵ ) ( 5 + 4 ϵ ) + h σ ( 2 + ϵ ) ] . For ϵ ≤ 1 4 \epsilon\le\frac{1}{4} ϵ ≤ 4 1 , ρ ≤ ( 10 c σ 2 + 3 h σ ) ϵ Q 2 \rho\le(10c_\sigma^2+3h_\sigma)\epsilon Q^2 ρ ≤ ( 10 c σ 2 + 3 h σ ) ϵ Q 2 . If ρ < 1 \rho< 1 ρ < 1 , F F F has a unique fixed point D ⋆ \mathbf{D}^\star D ⋆ , the gradient sequence of the sequential recurrence Eq. (2) with n s o f t n_{\mathrm{soft}} n soft in place of the projection. For any initialization and damping λ ∈ ( 0 , 1 ] \lambda\in(0,1] λ ∈ ( 0 , 1 ] , Eq. (7) converges geometrically:
∥ D [ n ] − D ⋆ ∥ ∞ ≤ ( 1 − λ ( 1 − ρ ) ) n ∥ D [ 0 ] − D ⋆ ∥ ∞ . \lVert\mathbf{D}^{[n]}-\mathbf{D}^\star\rVert_\infty\le\left(1-\lambda(1-\rho)\right)^n\lVert\mathbf{D}^{[0]}-\mathbf{D}^\star\rVert_\infty. ∥ D [ n ] − D ⋆ ∥ ∞ ≤ ( 1 − λ ( 1 − ρ ) ) n ∥ D [ 0 ] − D ⋆ ∥ ∞ . The features, readout states and outputs converge to those of the sequential recurrence at the same rate, up to constant factors.*
Proof strategy. We compare two guesses of the hidden-gradient sequence and follow their differences through one iteration. First, the hidden scan turns the guesses into pre-activations and features. Next, these features determine the readout trajectory and its feature gradients. Finally, the backward pass multiplies each feature gradient by the activation Jacobian. Both factors in this last product change between the two guesses, so we must control both changes. The lemmas below give these bounds in the order they are needed.
Notation. We use the Euclidean norm for vectors and the operator norm for matrices. For a sequence, ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ is the maximum of the vector norms over tokens. There is one hidden layer, so we omit its layer index and write D = ( δ z t ) t ≤ T \mathbf{D} = (\delta\mathbf{z}_t)_{t\le T} D = ( δ z t ) t ≤ T . A prime denotes the quantity obtained from a second guess D ′ \mathbf{D}' D ′ , with the keys, values and model parameters held fixed.
Past writes are multiplied by the decay factors
γ j , t = ∏ i = j + 1 t − 1 α i . \gamma_{j,t} = \prod_{i=j+1}^{t-1}\alpha_i. γ j , t = i = j + 1 ∏ t − 1 α i .
The assumption α i ≤ α ˉ < 1 \alpha_i \le\bar{\alpha} < 1 α i ≤ α ˉ < 1 bounds their total contribution, independently of sequence length:
η ˉ ∑ j < t γ j , t ≤ η ˉ ∑ r ≥ 0 α ˉ r = η ˉ 1 − α ˉ = ϵ . \bar{\eta}\sum_{j<t}\gamma_{j,t} \le\bar{\eta}\sum_{r\ge0}\bar{\alpha}^r = \frac{\bar{\eta}}{1-\bar{\alpha}} = \epsilon. η ˉ j < t ∑ γ j , t ≤ η ˉ r ≥ 0 ∑ α ˉ r = 1 − α ˉ η ˉ = ϵ .
We will use this geometric-series bound for both the hidden and readout states. For brevity, write n = n s o f t n = n_{\mathrm{soft}} n = n soft throughout the proof.
Lemma 1 (Soft clip). For n ( x ) = x / 1 + ∥ x ∥ 2 n(\mathbf{x}) = \mathbf{x}/\sqrt{1+\lVert\mathbf{x}\rVert^2} n ( x ) = x / 1 + ∥ x ∥ 2 , the output and Jacobian satisfy
∥ n ( x ) ∥ ≤ 1 , ∥ D n ( x ) ∥ ≤ 1 , ∥ D n ( x ) − D n ( y ) ∥ ≤ 2 ∥ x − y ∥ . \lVert n(\mathbf{x})\rVert\le1, \qquad\lVert Dn(\mathbf{x})\rVert\le1, \qquad\lVert Dn(\mathbf{x})-Dn(\mathbf{y})\rVert\le2\lVert\mathbf{x}-\mathbf{y}\rVert. ∥ n ( x )∥ ≤ 1 , ∥ D n ( x )∥ ≤ 1 , ∥ D n ( x ) − D n ( y )∥ ≤ 2 ∥ x − y ∥ .
Proof. The output bound follows directly from the definition. To bound the derivatives, set s = 1 + ∥ x ∥ 2 s = \sqrt{1+\lVert\mathbf{x}\rVert^2} s = 1 + ∥ x ∥ 2 and a = n ( x ) \mathbf{a}=n(\mathbf{x}) a = n ( x ) . Differentiation gives
D n ( x ) = s − 1 I − s − 3 x x T = s − 1 ( I − a a T ) . Dn(\mathbf{x}) = s^{-1}\mathbf{I}-s^{-3}\mathbf{x}\mathbf{x}^{\mathsf{T}} = s^{-1}(\mathbf{I}-\mathbf{a}\mathbf{a}^{\mathsf{T}}). D n ( x ) = s − 1 I − s − 3 x x T = s − 1 ( I − a a T ) .
Along x x x , this matrix has eigenvalue s − 3 s^{-3} s − 3 ; in directions perpendicular to x x x , it has eigenvalue s − 1 s^{-1} s − 1 . At x = 0 x=0 x = 0 it is the identity. Thus ∥ D n ( x ) ∥ ≤ s − 1 ≤ 1 \lVert Dn(x)\rVert\le s^{-1} \le1 ∥ D n ( x )∥ ≤ s − 1 ≤ 1 , which also shows that n n n is nonexpansive.
It remains to bound the change in the Jacobian. In a direction h h h , the derivative of the scalar factor s − 1 s^{-1} s − 1 has magnitude at most ∥ x ∥ ∥ h ∥ / s 3 \lVert x\rVert\lVert h\rVert/s^3 ∥ x ∥ ∥ h ∥ / s 3 . The derivative of a a T aa^{\mathsf{T}} a a T has norm at most 2 ∥ a ∥ ∥ D n ( x ) h ∥ 2\lVert a\rVert\lVert Dn(x)h\rVert 2 ∥ a ∥ ∥ D n ( x ) h ∥ . Applying the product rule therefore gives
∥ D ( D n ) ( x ) [ h ] ∥ ≤ 3 ∥ x ∥ ( 1 + ∥ x ∥ 2 ) 3 / 2 ∥ h ∥ ≤ 2 3 ∥ h ∥ ≤ 2 ∥ h ∥ . \lVert D(Dn)(x)[h]\rVert\le\frac{3\lVert x\rVert}{(1+\lVert x\rVert^2)^{3/2}}\lVert h\rVert\le\frac{2}{\sqrt{3}}\lVert h\rVert\le2\lVert h\rVert. ∥ D ( D n ) ( x ) [ h ]∥ ≤ ( 1 + ∥ x ∥ 2 ) 3/2 3 ∥ x ∥ ∥ h ∥ ≤ 3 2 ∥ h ∥ ≤ 2 ∥ h ∥ .
Here we used max r ≥ 0 r / ( 1 + r 2 ) 3 / 2 = 2 / ( 3 3 ) \max_{r\ge0} r/(1+r^2)^{3/2}=2/(3\sqrt{3}) max r ≥ 0 r / ( 1 + r 2 ) 3/2 = 2/ ( 3 3 ) . Integrating this bound along the segment from x x x to y y y proves the final inequality.
Lemma 2 (Composite activation). The full activation ψ = n ∘ σ \psi=n\circ\sigma ψ = n ∘ σ satisfies
∥ D ψ ( z ) ∥ ≤ c σ , ∥ D ψ ( z ) − D ψ ( z ′ ) ∥ ≤ ( 2 c σ 2 + h σ ) ∥ z − z ′ ∥ . \lVert D\psi(\boldsymbol{z})\rVert\le c_{\sigma}, \qquad\lVert D\psi(\boldsymbol{z})-D\psi(\boldsymbol{z}')\rVert\le(2c_{\sigma}^{2}+h_{\sigma})\lVert\boldsymbol{z}-\boldsymbol{z}'\rVert. ∥ D ψ ( z )∥ ≤ c σ , ∥ D ψ ( z ) − D ψ ( z ′ )∥ ≤ ( 2 c σ 2 + h σ ) ∥ z − z ′ ∥ . Proof. The chain rule factors the Jacobian into the derivatives of the soft clip and the coordinatewise activation:
D ψ ( z ) = N ( z ) J ( z ) , N ( z ) = D n ( σ ( z ) ) , J ( z ) = diag σ ′ ( z ) . D\psi(\boldsymbol{z})=N(\boldsymbol{z})J(\boldsymbol{z}), \qquad N(\boldsymbol{z})=Dn(\sigma(\boldsymbol{z})), \qquad J(\boldsymbol{z})=\operatorname{diag}\sigma'(\boldsymbol{z}). D ψ ( z ) = N ( z ) J ( z ) , N ( z ) = D n ( σ ( z )) , J ( z ) = diag σ ′ ( z ) . Since ∥ N ∥ ≤ 1 \lVert N\rVert\le1 ∥ N ∥ ≤ 1 and ∥ J ∥ ≤ c σ \lVert J\rVert\le c_{\sigma} ∥ J ∥ ≤ c σ , their product has norm at most c σ c_{\sigma} c σ . For its variation, the activation assumptions and Lemma 1 give
∥ N ( z ) − N ( z ′ ) ∥ ≤ 2 ∥ σ ( z ) − σ ( z ′ ) ∥ ≤ 2 c σ ∥ z − z ′ ∥ , \lVert N(\boldsymbol{z})-N(\boldsymbol{z}')\rVert\le2\lVert\sigma(\boldsymbol{z})-\sigma(\boldsymbol{z}')\rVert\le2c_{\sigma}\lVert\boldsymbol{z}-\boldsymbol{z}'\rVert, ∥ N ( z ) − N ( z ′ )∥ ≤ 2 ∥ σ ( z ) − σ ( z ′ )∥ ≤ 2 c σ ∥ z − z ′ ∥ , ∥ J ( z ) − J ( z ′ ) ∥ ≤ h σ ∥ z − z ′ ∥ . \lVert J(\boldsymbol{z})-J(\boldsymbol{z}')\rVert\le h_{\sigma}\lVert\boldsymbol{z}-\boldsymbol{z}'\rVert. ∥ J ( z ) − J ( z ′ )∥ ≤ h σ ∥ z − z ′ ∥ . Now split the difference of the products as N J − N ′ J ′ = ( N − N ′ ) J + N ′ ( J − J ′ ) NJ-N'J'=(N-N')J+N'(J-J') N J − N ′ J ′ = ( N − N ′ ) J + N ′ ( J − J ′ ) . The first term contributes at most 2 c σ 2 ∥ z − z ′ ∥ 2c_{\sigma}^{2}\lVert\boldsymbol{z}-\boldsymbol{z}'\rVert 2 c σ 2 ∥ z − z ′ ∥ , and the second contributes at most h σ ∥ z − z ′ ∥ h_{\sigma}\lVert\boldsymbol{z}-\boldsymbol{z}'\rVert h σ ∥ z − z ′ ∥ . Adding them proves the claim.
Lemma 3 (Features). For any two gradient guesses, let δ = ∥ D − D ′ ∥ ∞ \delta=\lVert D-D'\rVert_{\infty} δ = ∥ D − D ′ ∥ ∞ . Then
∥ ϕ t ∥ ≤ 1 , ∥ z t − z t ′ ∥ ≤ ϵ δ , ∥ ϕ t − ϕ t ′ ∥ ≤ c σ ϵ δ . \lVert\phi_t\rVert\le1, \qquad\lVert\boldsymbol{z}_t-\boldsymbol{z}'_t\rVert\le\epsilon\delta, \qquad\lVert\phi_t-\phi'_t\rVert\le c_{\sigma}\epsilon\delta. ∥ ϕ t ∥ ≤ 1 , ∥ z t − z t ′ ∥ ≤ ϵδ , ∥ ϕ t − ϕ t ′ ∥ ≤ c σ ϵδ . Proof. Only the fast weights depend on the gradient guess. Their contribution to the pre-activation is, by Eq. (4) ,
u t = α t Δ W t − 1 ( 1 ) k t = − α t ∑ j < t γ j , t η j ( k j T k t ) δ z j . \boldsymbol{u}_t=\alpha_t\Delta W^{(1)}_{t-1}\boldsymbol{k}_t=-\alpha_t\sum_{j<t}\gamma_{j,t}\eta_j(\boldsymbol{k}_j^{\mathsf{T}}\boldsymbol{k}_t)\delta\boldsymbol{z}_j. u t = α t Δ W t − 1 ( 1 ) k t = − α t j < t ∑ γ j , t η j ( k j T k t ) δ z j . The normalized keys satisfy ∣ k j T k t ∣ ≤ 1 \lvert\boldsymbol{k}_j^{\mathsf{T}}\boldsymbol{k}_t\rvert\le1 ∣ k j T k t ∣ ≤ 1 . Subtracting the two sums and using α t ≤ 1 \alpha_t\le1 α t ≤ 1 gives
∥ z t − z t ′ ∥ = ∥ u t − u t ′ ∥ ≤ η ˉ ∑ j < t γ j , t ∥ δ z j − δ z j ′ ∥ ≤ ϵ δ . \lVert\boldsymbol{z}_t-\boldsymbol{z}'_t\rVert=\lVert\boldsymbol{u}_t-\boldsymbol{u}'_t\rVert\le\bar{\eta}\sum_{j<t}\gamma_{j,t}\lVert\delta\boldsymbol{z}_j-\delta\boldsymbol{z}'_j\rVert\le\epsilon\delta. ∥ z t − z t ′ ∥ = ∥ u t − u t ′ ∥ ≤ η ˉ j < t ∑ γ j , t ∥ δ z j − δ z j ′ ∥ ≤ ϵδ . Applying the c σ c_{\sigma} c σ -Lipschitz map ψ \psi ψ gives the feature-difference bound. The bound ∥ ϕ t ∥ ≤ 1 \lVert\phi_t\rVert\le1 ∥ ϕ t ∥ ≤ 1 follows from the soft clip.
Lemma 4 (Readout trajectory). Let S t = Δ W t ( 2 ) S_t=\Delta W^{(2)}_t S t = Δ W t ( 2 ) be the readout’s fast weights for features satisfying ∥ ϕ t ∥ ≤ 1 \lVert\phi_t\rVert\le1 ∥ ϕ t ∥ ≤ 1 . Then
∥ S t ∥ ≤ ϵ Q , ∥ W ‾ t − 1 ( 2 ) ∥ ≤ ( 1 + ϵ ) Q . \lVert S_t\rVert\le\epsilon Q, \qquad\lVert\overline{W}_{t-1}^{(2)}\rVert\le(1+\epsilon)Q. ∥ S t ∥ ≤ ϵ Q , ∥ W t − 1 ( 2 ) ∥ ≤ ( 1 + ϵ ) Q . For two such feature sequences, writing δ ϕ = ∥ ϕ − ϕ ′ ∥ ∞ \delta_{\phi}=\lVert\phi-\phi'\rVert_{\infty} δ ϕ = ∥ ϕ − ϕ ′ ∥ ∞ , we also have
∥ S t − S t ′ ∥ ≤ 2 ϵ ( 1 + ϵ ) Q δ ϕ . \lVert S_t-S'_t\rVert\le2\epsilon(1+\epsilon)Q\delta_{\phi}. ∥ S t − S t ′ ∥ ≤ 2 ϵ ( 1 + ϵ ) Q δ ϕ . Proof. Separate each readout update into the part carried from the previous state and the new write:
S t = α t S t − 1 ( I − η t ϕ t ϕ t T ) + η t ( v t − W 0 ( 2 ) ϕ t ) ϕ t T , S 0 = 0. (11) S_t=\alpha_tS_{t-1}(I-\eta_t\phi_t\phi_t^{\mathsf{T}})+\eta_t(\boldsymbol{v}_t-W_0^{(2)}\phi_t)\phi_t^{\mathsf{T}}, \qquad S_0=0. \tag*{(11)} S t = α t S t − 1 ( I − η t ϕ t ϕ t T ) + η t ( v t − W 0 ( 2 ) ϕ t ) ϕ t T , S 0 = 0. (11) We first bound the state itself, then compare the states generated by different features.
State size. Because η t ∥ ϕ t ∥ 2 ≤ 1 \eta_t\lVert\phi_t\rVert^2 \le1 η t ∥ ϕ t ∥ 2 ≤ 1 , the matrix I − η t ϕ t ϕ t ⊤ I-\eta_t\phi_t\phi_t^\top I − η t ϕ t ϕ t ⊤ has operator norm at most one. The new write has norm at most η ˉ ( V + w 0 ) = η ˉ Q \bar{\eta}(V+w_0)=\bar{\eta}Q η ˉ ( V + w 0 ) = η ˉ Q . Hence
∥ S t ∥ ≤ α ˉ ∥ S t − 1 ∥ + η ˉ Q . \lVert S_t\rVert\le\bar{\alpha}\lVert S_{t-1}\rVert+\bar{\eta}Q. ∥ S t ∥ ≤ α ˉ ∥ S t − 1 ∥ + η ˉ Q .
Starting from zero and summing the geometric series gives ∥ S t ∥ ≤ ϵ Q \lVert S_t\rVert\le\epsilon Q ∥ S t ∥ ≤ ϵ Q . Adding the slow weights then gives
∥ W ‾ t − 1 ( 2 ) ∥ = ∥ W 0 ( 2 ) + α t S t − 1 ∥ ≤ w 0 + ϵ Q ≤ ( 1 + ϵ ) Q . \lVert\overline{W}^{(2)}_{t-1}\rVert=\lVert W^{(2)}_0+\alpha_tS_{t-1}\rVert\le w_0+\epsilon Q\le(1+\epsilon)Q. ∥ W t − 1 ( 2 ) ∥ = ∥ W 0 ( 2 ) + α t S t − 1 ∥ ≤ w 0 + ϵ Q ≤ ( 1 + ϵ ) Q .
Sensitivity to the features. Write δ t = ∥ S t − S t ′ ∥ \delta_t=\lVert S_t-S'_t\rVert δ t = ∥ S t − S t ′ ∥ . Two elementary bounds control the feature-dependent terms in Eq. (11) :
∥ ϕ t ϕ t ⊤ − ϕ t ′ ϕ t ′ ⊤ ∥ ≤ 2 δ ϕ , \lVert\phi_t\phi_t^\top-\phi'_t\phi_t^{\prime\top}\rVert\le2\delta_\phi, ∥ ϕ t ϕ t ⊤ − ϕ t ′ ϕ t ′⊤ ∥ ≤ 2 δ ϕ ,
∥ ( v t − W 0 ( 2 ) ϕ t ) ϕ t ⊤ − ( v t − W 0 ( 2 ) ϕ t ′ ) ϕ t ′ ⊤ ∥ ≤ ( V + 2 w 0 ) δ ϕ . \lVert(v_t-W^{(2)}_0\phi_t)\phi_t^\top-(v_t-W^{(2)}_0\phi'_t)\phi_t^{\prime\top}\rVert\le(V+2w_0)\delta_\phi. ∥( v t − W 0 ( 2 ) ϕ t ) ϕ t ⊤ − ( v t − W 0 ( 2 ) ϕ t ′ ) ϕ t ′⊤ ∥ ≤ ( V + 2 w 0 ) δ ϕ .
The first follows by changing one factor of the outer product at a time. For the second, the value term contributes V δ ϕ V\delta_\phi V δ ϕ and the quadratic feature term contributes 2 w 0 δ ϕ 2w_0\delta_\phi 2 w 0 δ ϕ . Subtracting the two recurrences now gives
δ t ≤ α ˉ δ t − 1 + 2 η ˉ α ˉ ∥ S t − 1 ′ ∥ δ ϕ + η ˉ ( V + 2 w 0 ) δ ϕ ≤ α ˉ δ t − 1 + 2 η ˉ ( 1 + ϵ ) Q δ ϕ . \begin{aligned}
\delta_t\le\bar{\alpha}\delta_{t-1}+2\bar{\eta}\bar{\alpha}\lVert S'_{t-1}\rVert\delta_\phi+\bar{\eta}(V+2w_0)\delta_\phi\\
&\le\bar{\alpha}\delta_{t-1}+2\bar{\eta}(1+\epsilon)Q\delta_\phi.
\end{aligned} δ t ≤ α ˉ δ t − 1 + 2 η ˉ α ˉ ∥ S t − 1 ′ ∥ δ ϕ + η ˉ ( V + 2 w 0 ) δ ϕ ≤ α ˉ δ t − 1 + 2 η ˉ ( 1 + ϵ ) Q δ ϕ .
In the last line we used the state-size bound and V + 2 w 0 ≤ 2 Q V+2w_0\le2Q V + 2 w 0 ≤ 2 Q . Since δ 0 = 0 \delta_0=0 δ 0 = 0 , another geometric sum yields δ t ≤ 2 ϵ ( 1 + ϵ ) Q δ ϕ \delta_t\le2\epsilon(1+\epsilon)Q\delta_\phi δ t ≤ 2 ϵ ( 1 + ϵ ) Q δ ϕ .
Lemma 5 (Feature gradient). For features in the unit ball, let
g t = W ‾ t − 1 ( 2 ) ⊤ ( W ‾ t − 1 ( 2 ) ϕ t − v t ) . g_t=\overline{W}^{(2)\top}_{t-1}(\overline{W}^{(2)}_{t-1}\phi_t-v_t). g t = W t − 1 ( 2 ) ⊤ ( W t − 1 ( 2 ) ϕ t − v t ) . Its size and sensitivity to the feature sequence satisfy
∥ g t ∥ ≤ B , ∥ g t − g t ′ ∥ ≤ A ∥ ϕ − ϕ ′ ∥ ∞ , \lVert g_t\rVert\le B,\qquad\lVert g_t-g'_t\rVert\le A\lVert\phi-\phi'\rVert_\infty, ∥ g t ∥ ≤ B , ∥ g t − g t ′ ∥ ≤ A ∥ ϕ − ϕ ′ ∥ ∞ , where
B = ( 1 + ϵ ) ( 2 + ϵ ) Q 2 , A = ( 1 + ϵ ) ( 1 + 7 ϵ + 4 ϵ 2 ) Q 2 . B=(1+\epsilon)(2+\epsilon)Q^2,\qquad A=(1+\epsilon)(1+7\epsilon+4\epsilon^2)Q^2. B = ( 1 + ϵ ) ( 2 + ϵ ) Q 2 , A = ( 1 + ϵ ) ( 1 + 7 ϵ + 4 ϵ 2 ) Q 2 .
Proof. Fix a token and abbreviate its decayed readout by W ‾ \overline{W} W and its residual by r = W ‾ ϕ t − v t r=\overline{W}\phi_t-v_t r = W ϕ t − v t . Set M = ( 1 + ϵ ) Q M=(1+\epsilon)Q M = ( 1 + ϵ ) Q . By Lemma 4 ,
∥ W ‾ ∥ ≤ M , ∥ r ∥ ≤ M + V . \lVert\overline{W}\rVert\le M,\qquad\lVert r\rVert\le M+V. ∥ W ∥ ≤ M , ∥ r ∥ ≤ M + V .
Therefore ∥ g t ∥ ≤ M ( M + V ) ≤ B \lVert g_t\rVert\le M(M+V)\le B ∥ g t ∥ ≤ M ( M + V ) ≤ B , using V ≤ Q V\le Q V ≤ Q .
For the difference bound, there are two sources of change: the readout weights and the features at this token. Writing Δ W = ∥ W ‾ − W ‾ ′ ∥ \Delta_W=\lVert\overline{W}-\overline{W}'\rVert Δ W = ∥ W − W ′ ∥ and δ ϕ = ∥ ϕ − ϕ ′ ∥ ∞ \delta_\phi=\lVert\phi-\phi'\rVert_\infty δ ϕ = ∥ ϕ − ϕ ′ ∥ ∞ , we have
r − r ′ = ( W ‾ − W ‾ ′ ) ϕ t + W ‾ ′ ( ϕ t − ϕ t ′ ) , ∥ r − r ′ ∥ ≤ Δ W + M δ ϕ . r-r'=(\overline{W}-\overline{W}')\phi_t+\overline{W}'(\phi_t-\phi'_t),\qquad\lVert r-r'\rVert\le\Delta_W+M\delta_\phi. r − r ′ = ( W − W ′ ) ϕ t + W ′ ( ϕ t − ϕ t ′ ) , ∥ r − r ′ ∥ ≤ Δ W + M δ ϕ .
Splitting the gradient difference in the same way gives
∥ g t − g t ′ ∥ ≤ Δ W ∥ r ∥ + ∥ W ‾ ′ ∥ ∥ r − r ′ ∥ ≤ ( 2 M + V ) Δ W + M 2 δ ϕ . \begin{aligned}
\lVert g_t-g'_t\rVert\le\Delta_W\lVert r\rVert+\lVert\overline{W}'\rVert\lVert r-r'\rVert\\
&\le(2M+V)\Delta_W+M^2\delta_\phi.
\end{aligned} ∥ g t − g t ′ ∥ ≤ Δ W ∥ r ∥ + ∥ W ′ ∥ ∥ r − r ′ ∥ ≤ ( 2 M + V ) Δ W + M 2 δ ϕ .
The readout lemma supplies Δ W ≤ 2 ϵ ( 1 + ϵ ) Q δ ϕ \Delta_W\le2\epsilon(1+\epsilon)Q\delta_\phi Δ W ≤ 2 ϵ ( 1 + ϵ ) Q δ ϕ . Substituting this and 2 M + V ≤ ( 3 + 2 ϵ ) Q 2M+V\le(3+2\epsilon)Q 2 M + V ≤ ( 3 + 2 ϵ ) Q yields
∥ g t − g t ′ ∥ ≤ [ 2 ϵ ( 1 + ϵ ) ( 3 + 2 ϵ ) + ( 1 + ϵ ) 2 ] Q 2 δ ϕ = A δ ϕ . \lVert g_t-g'_t\rVert\le\left[2\epsilon(1+\epsilon)(3+2\epsilon)+(1+\epsilon)^2\right]Q^2\delta_\phi=A\delta_\phi. ∥ g t − g t ′ ∥ ≤ [ 2 ϵ ( 1 + ϵ ) ( 3 + 2 ϵ ) + ( 1 + ϵ ) 2 ] Q 2 δ ϕ = A δ ϕ .
Proof of Theorem 1. Bounding one iteration. Let δ = ∥ D − D ′ ∥ ∞ \delta=\lVert D-D'\rVert_\infty δ = ∥ D − D ′ ∥ ∞ . The backward pass returns F t ( D ) = H t ⊤ g t F_t(D)=H_t^\top g_t F t ( D ) = H t ⊤ g t , where H t = D ψ ( z t ) H_t=D\psi(z_t) H t = D ψ ( z t ) . The preceding lemmas give the four bounds needed to compare these products:
∥ H t ∥ ≤ c σ , ∥ H t − H t ′ ∥ ≤ ( 2 c σ 2 + h σ ) ϵ δ , \lVert H_t\rVert\le c_\sigma,\qquad\lVert H_t-H'_t\rVert\le(2c_\sigma^2+h_\sigma)\epsilon\delta, ∥ H t ∥ ≤ c σ , ∥ H t − H t ′ ∥ ≤ ( 2 c σ 2 + h σ ) ϵδ ,
∥ g t ′ ∥ ≤ B , ∥ g t − g t ′ ∥ ≤ A c σ ϵ δ . \lVert g'_t\rVert\le B,\qquad\lVert g_t-g'_t\rVert\le Ac_\sigma\epsilon\delta. ∥ g t ′ ∥ ≤ B , ∥ g t − g t ′ ∥ ≤ A c σ ϵδ .
Here A A A bounds the sensitivity of the feature gradient and B B B bounds its size, as defined in Lemma 5 . The change in the gradient contributes the first term below; the change in the Jacobian contributes the second:
∥ F t ( D ) − F t ( D ′ ) ∥ ≤ ∥ H t ∥ ∥ g t − g t ′ ∥ + ∥ H t − H t ′ ∥ ∥ g t ′ ∥ ≤ ϵ [ c σ 2 A + ( 2 c σ 2 + h σ ) B ] δ . \begin{aligned}
\lVert F_t(\mathbf{D})-F_t(\mathbf{D}')\rVert\le\lVert\mathbf{H}_t\rVert\lVert\mathbf{g}_t-\mathbf{g}'_t\rVert+\lVert\mathbf{H}_t-\mathbf{H}'_t\rVert\lVert\mathbf{g}'_t\rVert\\
&\le\epsilon\left[c_\sigma^2 A+\left(2c_\sigma^2+h_\sigma\right)B\right]\delta.
\end{aligned} ∥ F t ( D ) − F t ( D ′ )∥ ≤ ∥ H t ∥ ∥ g t − g t ′ ∥ + ∥ H t − H t ′ ∥ ∥ g t ′ ∥ ≤ ϵ [ c σ 2 A + ( 2 c σ 2 + h σ ) B ] δ .
Substituting A A A and B B B and collecting terms gives exactly the constant ρ \rho ρ in Eq. (8) . Taking the maximum over tokens proves ∥ F ( D ) − F ( D ′ ) ∥ ∞ ≤ ρ δ \lVert F(\mathbf{D})-F(\mathbf{D}')\rVert_\infty\le\rho\delta ∥ F ( D ) − F ( D ′ ) ∥ ∞ ≤ ρ δ . The simpler bound for ϵ ≤ 1 / 4 \epsilon\le1/4 ϵ ≤ 1/4 follows from
( 1 + ϵ ) 2 ( 5 + 4 ϵ ) ≤ 75 8 < 10 , ( 1 + ϵ ) ( 2 + ϵ ) ≤ 45 16 < 3. (1+\epsilon)^2(5+4\epsilon)\le\frac{75}{8}<10,\qquad(1+\epsilon)(2+\epsilon)\le\frac{45}{16}<3. ( 1 + ϵ ) 2 ( 5 + 4 ϵ ) ≤ 8 75 < 10 , ( 1 + ϵ ) ( 2 + ϵ ) ≤ 16 45 < 3.
Identifying the limit. If ρ < 1 \rho<1 ρ < 1 , the map is a contraction on the complete space ( R T × d , ∥ ⋅ ∥ ∞ ) (\mathbb{R}^{T\times d},\lVert\cdot\rVert_\infty) ( R T × d , ∥ ⋅ ∥ ∞ ) . Banach’s theorem therefore gives a unique fixed point D ⋆ \mathbf{D}^\star D ⋆ . At that point, every guessed gradient equals the actual gradient returned by the backward pass. This identifies the resulting trajectory with the sequential recurrence using the soft clip: both start from zero fast weights; if their states agree before a token, their features and gradients agree at that token, so their updates give the same next states. Induction over tokens proves agreement of the whole trajectory.
Damping and the convergence rate. The damped map is G ( D ) = ( 1 − λ ) D + λ F ( D ) G(\mathbf{D})=(1-\lambda)\mathbf{D}+\lambda F(\mathbf{D}) G ( D ) = ( 1 − λ ) D + λ F ( D ) . It has the same fixed point and satisfies
∥ G ( D ) − G ( D ′ ) ∥ ∞ ≤ ( 1 − λ + λ ρ ) ∥ D − D ′ ∥ ∞ . \lVert G(\mathbf{D})-G(\mathbf{D}')\rVert_\infty\le(1-\lambda+\lambda\rho)\lVert\mathbf{D}-\mathbf{D}'\rVert_\infty. ∥ G ( D ) − G ( D ′ ) ∥ ∞ ≤ ( 1 − λ + λ ρ ) ∥ D − D ′ ∥ ∞ .
Applying this inequality repeatedly with D ′ = D ⋆ \mathbf{D}'=\mathbf{D}^\star D ′ = D ⋆ gives the stated geometric rate.
Features, states and outputs. The feature and readout lemmas show that these quantities change by at most a constant times the error in D \mathbf{D} D . The same is true for query-side features: their unshifted hidden scan has gain at most ϵ max t ∥ q t ∥ \epsilon\max_t\lVert\mathbf{q}_t\rVert ϵ max t ∥ q t ∥ , followed by the c σ c_\sigma c σ -Lipschitz map ψ \psi ψ . Finally, the output is o t = W t ( 2 ) ϕ t q \mathbf{o}_t=\mathbf{W}_t^{(2)}\boldsymbol{\phi}_t^{q} o t = W t ( 2 ) ϕ t q , a product of bounded quantities that are Lipschitz in D \mathbf{D} D . Thus their errors all inherit the same geometric rate, up to constant factors. □
SiLU satisfies the assumptions. For σ ( x ) = x s ( x ) \sigma(x)=xs(x) σ ( x ) = x s ( x ) with s ( x ) = ( 1 + e − x ) − 1 s(x)=(1+e^{-x})^{-1} s ( x ) = ( 1 + e − x ) − 1 , σ ′ ( x ) = s ( x ) + x s ′ ( x ) \sigma'(x)=s(x)+xs'(x) σ ′ ( x ) = s ( x ) + x s ′ ( x ) and σ ′ ′ ( x ) = 2 s ′ ( x ) + x s ′ ( x ) ( 1 − 2 s ( x ) ) \sigma''(x)=2s'(x)+xs'(x)(1-2s(x)) σ ′′ ( x ) = 2 s ′ ( x ) + x s ′ ( x ) ( 1 − 2 s ( x )) . Since s ′ ( x ) ≤ min ( 1 / 4 , e − ∣ x ∣ ) s'(x)\le\min(1/4,e^{-|x|}) s ′ ( x ) ≤ min ( 1/4 , e − ∣ x ∣ ) and ∣ x ∣ e − ∣ x ∣ ≤ 1 / e |x|e^{-|x|}\le1/e ∣ x ∣ e − ∣ x ∣ ≤ 1/ e , valid global bounds are c σ = 1 + 1 / e c_\sigma=1+1/e c σ = 1 + 1/ e and h σ = 1 / 2 + 1 / e h_\sigma=1/2+1/e h σ = 1/2 + 1/ e .
Fixed point and finite-step exactness. This is the formal version of Proposition 1.
Proposition 2 (Finite-step exactness of the undamped iteration). For the iteration of § 3.2 with λ = 1 \lambda=1 λ = 1 and any initialization, the estimated hidden gradients at tokens 1 , … , min ( n , T ) 1,\ldots,\min(n,T) 1 , … , min ( n , T ) equal their sequential values after n n n iterations. Consequently, for N ≥ T N\ge T N ≥ T , the final evaluation reproduces the sequential recurrence exactly in exact arithmetic. The argument applies at any memory depth, with either projection or soft clipping, using the same derivative convention in the parallel and sequential evaluations, and requires no contraction condition. For 0 < λ < 1 0<\lambda<1 0 < λ < 1 , finite-step exactness fails in general: already at token 1, D 1 [ n ] − D 1 ⋆ = ( 1 − λ ) n ( D 1 [ 0 ] − D 1 ⋆ ) \mathbf{D}_1^{[n]}-\mathbf{D}_1^\star=(1-\lambda)^n(\mathbf{D}_1^{[0]}-\mathbf{D}_1^\star) D 1 [ n ] − D 1 ⋆ = ( 1 − λ ) n ( D 1 [ 0 ] − D 1 ⋆ ) .
Proof. Group all hidden-layer gradient estimates at token t t t into D t \mathbf{D}_t D t . The iteration is strictly causal: F t ( D ) F_t(\mathbf{D}) F t ( D ) depends only on D < t = D 1 , … , D t − 1 \mathbf{D}_{<t}=\mathbf{D}_1,\ldots,\mathbf{D}_{t-1} D < t = D 1 , … , D t − 1 . To see this at arbitrary depth, proceed layer by layer in the forward pass. At token s ≤ t s\le t s ≤ t , each hidden scan uses only writes j < s j<s j < s , and its layer inputs depend only on estimates at tokens preceding their own positions. Thus all key-side features through token t t t depend only on D < t \mathbf{D}_{<t} D < t . The readout before token t t t is determined by earlier features and updates. The backward pass uses these same hidden and readout states, so it introduces no dependence on D t \mathbf{D}_t D t or later estimates.
The sequential gradient sequence D ⋆ \mathbf{D}^\star D ⋆ satisfies F ( D ⋆ ) = D ⋆ F(\mathbf{D}^\star)=\mathbf{D}^\star F ( D ⋆ ) = D ⋆ : at D ⋆ \mathbf{D}^\star D ⋆ the shifted scans of Step 1 sum the writes of tokens j < t j<t j < t and reproduce the decayed weights W ~ t − 1 ( l ) \widetilde{\mathbf{W}}_{t-1}^{(l)} W t − 1 ( l ) of Eq. (2) at every token, Step 2 reproduces W ‾ t − 1 ( L ) \overline{\mathbf{W}}_{t-1}^{(L)} W t − 1 ( L ) , and Step 3 is the backpropagation of the sequential model, so δ z ~ = δ z \delta\widetilde{\mathbf{z}}=\delta\mathbf{z} δ z = δ z ; the parallel evaluation at D ⋆ \mathbf{D}^\star D ⋆ therefore coincides with the sequential recurrence, and damping changes the iterates but not this fixed point. Since F 1 F_1 F 1 has no dependence on D \mathbf{D} D , the first undamped iteration gives D 1 [ 1 ] = D 1 ⋆ \mathbf{D}_1^{[1]}=\mathbf{D}_1^\star D 1 [ 1 ] = D 1 ⋆ . Suppose the first n n n tokens are exact after n n n iterations. For every t ≤ min ( n + 1 , T ) t\le\min(n+1,T) t ≤ min ( n + 1 , T ) , all arguments of F t F_t F t then agree with those of F t ( D ⋆ ) F_t(\mathbf{D}^\star) F t ( D ⋆ ) , hence D t [ n + 1 ] = D t ⋆ \mathbf{D}_t^{[n+1]}=\mathbf{D}_t^\star D t [ n + 1 ] = D t ⋆ . Induction proves the prefix claim and gives D [ N ] = D ⋆ \mathbf{D}^{[N]}=\mathbf{D}^\star D [ N ] = D ⋆ whenever N ≥ T N\ge T N ≥ T . The final forward and readout passes therefore reproduce the sequential states and outputs. The argument uses only causality and identical local derivative evaluations, so it applies to either clipping choice; at a nondifferentiable point of the hard projection, both evaluations must use the same fixed derivative convention.
Convergence with the hidden-layer delta rule. Theorem 1 analyzes the linear-attention hidden update. We now prove the analogous statement for the hidden-layer delta rule of § 3.4 (Section C.2), which the reported models use, and then extend both theorems to the centered readout (Corollary 1). The setting is otherwise that of Theorem 1: L = 2 L=2 L = 2 , the uncentered readout and the soft clip. The notation and Lemmas 1, 2, 4 and 5 are reused unchanged; only the feature bound of Lemma 3 is replaced.
With one hidden layer, the keys of the hidden delta rule are the layer inputs k t k_t k t , which do not depend on the iterate. Write κ t = ∥ k t ∥ \kappa_t=\lVert k_t\rVert κ t = ∥ k t ∥ and k ^ t = k t / κ t \hat{k}_t=k_t/\kappa_t k ^ t = k t / κ t , and take the write rate β t = η t \beta_t=\eta_t β t = η t as in Section C.2. Substituting the target y t y_t y t into Eq. (16) , a forward pass with gradient guess D = ( δ z t ) t ≤ T \mathbf{D}=(\delta z_t)_{t\le T} D = ( δ z t ) t ≤ T and previous reads R ~ = ( r ~ t ) t ≤ T \widetilde{\mathbf{R}}=(\tilde{r}_t)_{t\le T} R = ( r ~ t ) t ≤ T evolves the hidden fast weights from Δ W 0 ( 1 ) = 0 \Delta W_0^{(1)}=0 Δ W 0 ( 1 ) = 0 by
Δ W t ( 1 ) = α t Δ W t − 1 ( 1 ) ( I − η t k ^ t k ^ t ⊤ ) + η t ( r ~ t − κ t δ z t ) k ^ t ⊤ , r t = α t Δ W t − 1 ( 1 ) k ^ t , (12) \Delta W_t^{(1)}=\alpha_t\Delta W_{t-1}^{(1)}\left(\mathbf{I}-\eta_t\hat{k}_t\hat{k}_t^\top\right)+\eta_t\left(\tilde{r}_t-\kappa_t\delta z_t\right)\hat{k}_t^\top,\qquad r_t=\alpha_t\Delta W_{t-1}^{(1)}\hat{k}_t, \tag*{(12)} Δ W t ( 1 ) = α t Δ W t − 1 ( 1 ) ( I − η t k ^ t k ^ t ⊤ ) + η t ( r ~ t − κ t δ z t ) k ^ t ⊤ , r t = α t Δ W t − 1 ( 1 ) k ^ t , (12)
and the pre-activations are z t = W 0 ( 1 ) k t + b ( 1 ) + κ t r t z_t=W_0^{(1)}k_t+b^{(1)}+\kappa_t r_t z t = W 0 ( 1 ) k t + b ( 1 ) + κ t r t . Steps 2 and 3 do not involve the hidden update rule, so the backward pass again returns D ^ t = H t ⊤ g t \hat{\mathbf{D}}_t=H_t^\top g_t D ^ t = H t ⊤ g t with H t = D ψ ( z t ) H_t=D\psi(z_t) H t = D ψ ( z t ) and g t g_t g t as in Lemma 5. The iteration now carries the reads as well as the gradients. Write Ψ \Psi Ψ for the map of one iteration, Ψ ( D , R ~ ) = ( D ^ , R ) \Psi(\mathbf{D},\widetilde{\mathbf{R}})=(\hat{\mathbf{D}},\mathbf{R}) Ψ ( D , R ) = ( D ^ , R ) with R = ( r t ) t ≤ T \mathbf{R}=(r_t)_{t\le T} R = ( r t ) t ≤ T . As in the algorithm, the gradients are damped by Eq. (7) and the reads are replaced:
D [ n ] = D [ n − 1 ] + λ ( D ^ − D [ n − 1 ] ) , R [ n ] = R , where ( D ^ , R ) = Ψ ( D [ n − 1 ] , R [ n − 1 ] ) , (13) \mathbf{D}^{[n]}=\mathbf{D}^{[n-1]}+\lambda\left(\hat{\mathbf{D}}-\mathbf{D}^{[n-1]}\right),\qquad\mathbf{R}^{[n]}=\mathbf{R},\qquad\text{where }(\hat{\mathbf{D}},\mathbf{R})=\Psi\left(\mathbf{D}^{[n-1]},\mathbf{R}^{[n-1]}\right), \tag*{(13)} D [ n ] = D [ n − 1 ] + λ ( D ^ − D [ n − 1 ] ) , R [ n ] = R , where ( D ^ , R ) = Ψ ( D [ n − 1 ] , R [ n − 1 ] ) , (13)
and the algorithm starts from D [ 0 ] = 0 \mathbf{D}^{[0]}=0 D [ 0 ] = 0 , R [ 0 ] = 0 \mathbf{R}^{[0]}=0 R [ 0 ] = 0 . For λ ∈ ( 0 , 1 ] \lambda\in(0,1] λ ∈ ( 0 , 1 ] we measure pairs in the norm ∥ ( D , R ) ∥ λ = ∥ D ∥ ∞ + λ ∥ R ∥ ∞ \lVert(\mathbf{D},\mathbf{R})\rVert_\lambda=\lVert\mathbf{D}\rVert_\infty+\lambda\lVert\mathbf{R}\rVert_\infty ∥( D , R ) ∥ λ = ∥ D ∥ ∞ + λ ∥ R ∥ ∞ .
Theorem 2 (Convergence of ParaTTT with the hidden-layer delta rule, L = 2 L=2 L = 2 ). Consider the soft-clipped variant with the hidden-layer delta rule Eq. (12) . Let the assumptions of Theorem 1 hold, with nonzero keys, 0 < ∥ k t ∥ ≤ 1 0<\lVert k_t\rVert\le1 0 < ∥ k t ∥ ≤ 1 , and let ϵ \epsilon ϵ , Q Q Q and ρ \rho ρ be as defined there. Then:
For any two pairs ( D , R ~ ) (\mathbf{D},\widetilde{\mathbf{R}}) ( D , R ) and ( D ′ , R ~ ′ ) (\mathbf{D}',\widetilde{\mathbf{R}}') ( D ′ , R ′ ) , with δ D = ∥ D − D ′ ∥ ∞ \delta_D=\lVert\mathbf{D}-\mathbf{D}'\rVert_\infty δ D = ∥ D − D ′ ∥ ∞ and δ R = ∥ R ~ − R ~ ′ ∥ ∞ \delta_R=\lVert\widetilde{\mathbf{R}}-\widetilde{\mathbf{R}}'\rVert_\infty δ R = ∥ R − R ′ ∥ ∞ , the outputs of one iteration satisfy
∥ D ^ − D ^ ′ ∥ ∞ ≤ ρ ( δ D + δ R ) , ∥ R − R ′ ∥ ∞ ≤ ϵ ( δ D + δ R ) . \lVert\hat{\mathbf{D}}-\hat{\mathbf{D}}'\rVert_\infty\le\rho\left(\delta_D+\delta_R\right),\qquad\lVert\mathbf{R}-\mathbf{R}'\rVert_\infty\le\epsilon\left(\delta_D+\delta_R\right). ∥ D ^ − D ^ ′ ∥ ∞ ≤ ρ ( δ D + δ R ) , ∥ R − R ′ ∥ ∞ ≤ ϵ ( δ D + δ R ) . In particular, Ψ \Psi Ψ is globally Lipschitz with constant ρ + ϵ \rho+\epsilon ρ + ϵ in the norm ∥ D ∥ ∞ + ∥ R ∥ ∞ \lVert\mathbf{D}\rVert_\infty+\lVert\mathbf{R}\rVert_\infty ∥ D ∥ ∞ + ∥ R ∥ ∞ .
If ρ + ϵ < 1 \rho+\epsilon<1 ρ + ϵ < 1 , Ψ \Psi Ψ has a unique fixed point ( D ⋆ , R ⋆ ) (\mathbf{D}^\star,\mathbf{R}^\star) ( D ⋆ , R ⋆ ) ; D ⋆ \mathbf{D}^\star D ⋆ is the gradient sequence of the sequential recurrence Eq. (2) with n s o f t n_{\mathrm{soft}} n soft in place of the projection, the same as in Theorem 1, and r t ⋆ = α t Δ W t − 1 ( 1 ) k ^ t r_t^\star=\alpha_t\Delta W_{t-1}^{(1)}\hat{k}_t r t ⋆ = α t Δ W t − 1 ( 1 ) k ^ t are the key reads of that recurrence. For any initialization and damping λ ∈ ( 0 , 1 ] \lambda\in(0,1] λ ∈ ( 0 , 1 ] , the iteration Eq. (13) converges geometrically:
∥ ( D [ n ] , R [ n ] ) − ( D ⋆ , R ⋆ ) ∥ λ ≤ ( 1 − λ ( 1 − ρ − ϵ ) ) n ∥ ( D [ 0 ] , R [ 0 ] ) − ( D ⋆ , R ⋆ ) ∥ λ . \left\lVert(\mathbf{D}^{[n]},\mathbf{R}^{[n]})-(\mathbf{D}^\star,\mathbf{R}^\star)\right\rVert_\lambda\le\left(1-\lambda(1-\rho-\epsilon)\right)^n\left\lVert(\mathbf{D}^{[0]},\mathbf{R}^{[0]})-(\mathbf{D}^\star,\mathbf{R}^\star)\right\rVert_\lambda. ( D [ n ] , R [ n ] ) − ( D ⋆ , R ⋆ ) λ ≤ ( 1 − λ ( 1 − ρ − ϵ ) ) n ( D [ 0 ] , R [ 0 ] ) − ( D ⋆ , R ⋆ ) λ . The features, readout states and outputs converge to those of the sequential recurrence at the same rate, up to constant factors.
The statement has the same form as Theorem 1, with ρ + ϵ \rho+\epsilon ρ + ϵ in place of ρ \rho ρ : the additional ϵ \epsilon ϵ is the gain of the reads that the delta rule carries from one iteration to the next. As before, the condition is sufficient, does not depend on the sequence length, and does not cover the hard projection; centering is treated in Corollary 1.
Lemma 6 (Hidden delta-rule trajectory). For two pairs ( D , R ~ ) (\mathbf{D},\widetilde{\mathbf{R}}) ( D , R ) and ( D ′ , R ~ ′ ) (\mathbf{D}',\widetilde{\mathbf{R}}') ( D ′ , R ′ ) , with δ D \delta_D δ D and δ R \delta_R δ R as in Theorem 2, the trajectories of Eq. (12) satisfy, for every token t t t ,
∥ Δ W t ( 1 ) − Δ W t ′ ( 1 ) ∥ ≤ ϵ ( δ D + δ R ) , ∥ r t − r t ′ ∥ ≤ ϵ ( δ D + δ R ) , ∥ z t − z t ′ ∥ ≤ ϵ ( δ D + δ R ) , \lVert\Delta W_t^{(1)}-\Delta W_t'^{(1)}\rVert\le\epsilon(\delta_D+\delta_R),\qquad\lVert r_t-r_t'\rVert\le\epsilon(\delta_D+\delta_R),\qquad\lVert z_t-z_t'\rVert\le\epsilon(\delta_D+\delta_R), ∥ Δ W t ( 1 ) − Δ W t ′ ( 1 ) ∥ ≤ ϵ ( δ D + δ R ) , ∥ r t − r t ′ ∥ ≤ ϵ ( δ D + δ R ) , ∥ z t − z t ′ ∥ ≤ ϵ ( δ D + δ R ) , and therefore ∥ ϕ t − ϕ t ′ ∥ ≤ c σ ϵ ( δ D + δ R ) \lVert\phi_t-\phi_t'\rVert\le c_\sigma\epsilon(\delta_D+\delta_R) ∥ ϕ t − ϕ t ′ ∥ ≤ c σ ϵ ( δ D + δ R ) , while ∥ ϕ t ∥ ≤ 1 \lVert\phi_t\rVert\le1 ∥ ϕ t ∥ ≤ 1 .
Proof. Let E t = Δ W t ( 1 ) − Δ W t ′ ( 1 ) \mathbf{E}_t = \Delta\mathbf{W}^{(1)}_t - \Delta\mathbf{W}'^{(1)}_t E t = Δ W t ( 1 ) − Δ W t ′ ( 1 ) . The keys, gates and learning rates are the same for both pairs, so subtracting the two copies of Eq. (12) gives a linear recurrence for the difference,
E t = α t E t − 1 ( I − η t k ^ t k ^ t T ) + η t [ ( r ~ t − r ~ t ′ ) − κ t ( δ z t − δ z t ′ ) ] k ^ t T , E 0 = 0. \mathbf{E}_t = \alpha_t \mathbf{E}_{t-1}\left(\mathbf{I} - \eta_t \hat{\mathbf{k}}_t\hat{\mathbf{k}}_t^{\mathsf{T}}\right) + \eta_t\left[(\tilde{\mathbf{r}}_t - \tilde{\mathbf{r}}'_t) - \kappa_t(\delta\mathbf{z}_t - \delta\mathbf{z}'_t)\right]\hat{\mathbf{k}}_t^{\mathsf{T}}, \qquad\mathbf{E}_0 = \mathbf{0}. E t = α t E t − 1 ( I − η t k ^ t k ^ t T ) + η t [ ( r ~ t − r ~ t ′ ) − κ t ( δ z t − δ z t ′ ) ] k ^ t T , E 0 = 0 .
Because k ^ t \hat{\mathbf{k}}_t k ^ t is a unit vector and 0 < η t ≤ 1 0 < \eta_t \le1 0 < η t ≤ 1 , the matrix I − η t k ^ t k ^ t T \mathbf{I} - \eta_t\hat{\mathbf{k}}_t\hat{\mathbf{k}}_t^{\mathsf{T}} I − η t k ^ t k ^ t T has eigenvalues 1 − η t 1-\eta_t 1 − η t and 1 1 1 , hence operator norm at most one. The second term is an outer product with a unit vector, so its norm is at most η ˉ ( δ R + κ t δ D ) ≤ η ˉ ( δ D + δ R ) \bar{\eta}(\delta_R+\kappa_t\delta_D) \le\bar{\eta}(\delta_D+\delta_R) η ˉ ( δ R + κ t δ D ) ≤ η ˉ ( δ D + δ R ) , using κ t ≤ 1 \kappa_t \le1 κ t ≤ 1 . Therefore
∥ E t ∥ ≤ α ˉ ∥ E t − 1 ∥ + η ˉ ( δ D + δ R ) , \lVert\mathbf{E}_t\rVert\le\bar{\alpha}\lVert\mathbf{E}_{t-1}\rVert+ \bar{\eta}(\delta_D+\delta_R), ∥ E t ∥ ≤ α ˉ ∥ E t − 1 ∥ + η ˉ ( δ D + δ R ) ,
and summing the geometric series from E 0 = 0 \mathbf{E}_0=\mathbf{0} E 0 = 0 gives ∥ E t ∥ ≤ ϵ ( δ D + δ R ) \lVert\mathbf{E}_t\rVert\le\epsilon(\delta_D+\delta_R) ∥ E t ∥ ≤ ϵ ( δ D + δ R ) for every t t t . The reads and pre-activations differ by r t − r t ′ = α t E t − 1 k ^ t \mathbf{r}_t-\mathbf{r}'_t=\alpha_t\mathbf{E}_{t-1}\hat{\mathbf{k}}_t r t − r t ′ = α t E t − 1 k ^ t and z t − z t ′ = κ t ( r t − r t ′ ) \mathbf{z}_t-\mathbf{z}'_t=\kappa_t(\mathbf{r}_t-\mathbf{r}'_t) z t − z t ′ = κ t ( r t − r t ′ ) , and α t , κ t ≤ 1 \alpha_t,\kappa_t\le1 α t , κ t ≤ 1 , which gives the next two bounds. The feature bound follows because ψ \psi ψ is c σ c_\sigma c σ -Lipschitz (Lemma 2), and ∥ ϕ t ∥ ≤ 1 \lVert\boldsymbol{\phi}_t\rVert\le1 ∥ ϕ t ∥ ≤ 1 by the soft clip.
Proof of Theorem 2. Bounding one iteration. The bound on the reads is part of Lemma 6 . For the gradients, Lemma 6 plays the role of Lemma 3 , with δ D + δ R \delta_D+\delta_R δ D + δ R in place of δ \delta δ ; the rest of the argument is that of Theorem 1. By Lemma 2 and Lemma 6 , ∥ H t ∥ ≤ c σ \lVert\mathbf{H}_t\rVert\le c_\sigma ∥ H t ∥ ≤ c σ and ∥ H t − H t ′ ∥ ≤ ( 2 c σ + h σ ) ϵ ( δ D + δ R ) \lVert\mathbf{H}_t-\mathbf{H}'_t\rVert\le(2c_\sigma+h_\sigma)\epsilon(\delta_D+\delta_R) ∥ H t − H t ′ ∥ ≤ ( 2 c σ + h σ ) ϵ ( δ D + δ R ) . Lemmas 4 and 5 hold for any feature sequences in the unit ball, so ∥ g t ′ ∥ ≤ B \lVert\mathbf{g}'_t\rVert\le B ∥ g t ′ ∥ ≤ B and ∥ g t − g t ′ ∥ ≤ A ∥ ϕ − ϕ ′ ∥ ∞ ≤ A c σ ϵ ( δ D + δ R ) \lVert\mathbf{g}_t-\mathbf{g}'_t\rVert\le A\lVert\boldsymbol{\phi}-\boldsymbol{\phi}'\rVert_\infty\le Ac_\sigma\epsilon(\delta_D+\delta_R) ∥ g t − g t ′ ∥ ≤ A ∥ ϕ − ϕ ′ ∥ ∞ ≤ A c σ ϵ ( δ D + δ R ) . Hence
∥ D ^ t − D ^ t ′ ∥ ≤ ∥ H t ∥ ∥ g t − g t ′ ∥ + ∥ H t − H t ′ ∥ ∥ g t ′ ∥ ≤ ϵ [ c σ 2 A + ( 2 c σ + h σ ) B ] ( δ D + δ R ) = ρ ( δ D + δ R ) , \begin{aligned}
\lVert\hat{\mathbf{D}}_t-\hat{\mathbf{D}}'_t\rVert\le\lVert\mathbf{H}_t\rVert\lVert\mathbf{g}_t-\mathbf{g}'_t\rVert+ \lVert\mathbf{H}_t-\mathbf{H}'_t\rVert\lVert\mathbf{g}'_t\rVert\\
&\le\epsilon\left[c_\sigma^2 A+(2c_\sigma+h_\sigma)B\right](\delta_D+\delta_R)=\rho(\delta_D+\delta_R),
\end{aligned} ∥ D ^ t − D ^ t ′ ∥ ≤ ∥ H t ∥ ∥ g t − g t ′ ∥ + ∥ H t − H t ′ ∥ ∥ g t ′ ∥ ≤ ϵ [ c σ 2 A + ( 2 c σ + h σ ) B ] ( δ D + δ R ) = ρ ( δ D + δ R ) ,
where the last equality is the computation in the proof of Theorem 1. Taking the maximum over tokens and adding the two bounds proves the first part.
Damping and the convergence rate. Fix λ ∈ ( 0 , 1 ] \lambda\in(0,1] λ ∈ ( 0 , 1 ] and let G λ ( D , R ~ ) = ( ( 1 − λ ) D + λ D ^ , R ) G_\lambda(\mathbf{D},\widetilde{\mathbf{R}})=((1-\lambda)\mathbf{D}+\lambda\hat{\mathbf{D}},\mathbf{R}) G λ ( D , R ) = (( 1 − λ ) D + λ D ^ , R ) be one step of Eq. (13) . By the first part,
∥ G λ ( D , R ~ ) − G λ ( D ′ , R ~ ′ ) ∥ λ ≤ ( 1 − λ ) δ D + λ ρ ( δ D + δ R ) + λ ϵ ( δ D + δ R ) = θ δ D + λ ( ρ + ϵ ) δ R , θ = 1 − λ ( 1 − ρ − ϵ ) . \begin{aligned}
\lVert G_\lambda(\mathbf{D},\widetilde{\mathbf{R}})-G_\lambda(\mathbf{D}',\widetilde{\mathbf{R}}')\rVert_\lambda\le(1-\lambda)\delta_D+\lambda\rho(\delta_D+\delta_R)+\lambda\epsilon(\delta_D+\delta_R) \\
&= \theta\delta_D+\lambda(\rho+\epsilon)\delta_R,\qquad\theta=1-\lambda(1-\rho-\epsilon).
\end{aligned} ∥ G λ ( D , R ) − G λ ( D ′ , R ′ ) ∥ λ ≤ ( 1 − λ ) δ D + λ ρ ( δ D + δ R ) + λ ϵ ( δ D + δ R ) = θ δ D + λ ( ρ + ϵ ) δ R , θ = 1 − λ ( 1 − ρ − ϵ ) .
Since θ − ( ρ + ϵ ) = ( 1 − λ ) ( 1 − ρ − ϵ ) ≥ 0 \theta-(\rho+\epsilon)=(1-\lambda)(1-\rho-\epsilon)\ge0 θ − ( ρ + ϵ ) = ( 1 − λ ) ( 1 − ρ − ϵ ) ≥ 0 when ρ + ϵ ≤ 1 \rho+\epsilon\le1 ρ + ϵ ≤ 1 , the right-hand side is at most θ ( δ D + λ δ R ) = θ ∥ ( D , R ~ ) − ( D ′ , R ~ ′ ) ∥ λ \theta(\delta_D+\lambda\delta_R)=\theta\lVert(\mathbf{D},\widetilde{\mathbf{R}})-(\mathbf{D}',\widetilde{\mathbf{R}}')\rVert_\lambda θ ( δ D + λ δ R ) = θ ∥( D , R ) − ( D ′ , R ′ ) ∥ λ . If ρ + ϵ < 1 \rho+\epsilon<1 ρ + ϵ < 1 , then θ < 1 \theta<1 θ < 1 and G λ G_\lambda G λ is a contraction on the complete space R T × d × R T × d \mathbb{R}^{T\times d}\times\mathbb{R}^{T\times d} R T × d × R T × d , ∥ ⋅ ∥ λ \lVert\cdot\rVert_\lambda ∥ ⋅ ∥ λ . Banach’s theorem gives a unique fixed point, and applying the inequality repeatedly with the second pair equal to the fixed point gives the stated rate. A pair is fixed by G λ G_\lambda G λ exactly when R = R ~ \mathbf{R}=\widetilde{\mathbf{R}} R = R and D ^ = D \hat{\mathbf{D}}=\mathbf{D} D ^ = D , that is, when it is fixed by Ψ \Psi Ψ , so the fixed point does not depend on λ \lambda λ .
Identifying the limit. At a fixed point, r ~ t = r t = α t Δ W t − 1 ( 1 ) k ^ t \tilde{\mathbf{r}}_t=\mathbf{r}_t=\alpha_t\Delta\mathbf{W}^{(1)}_{t-1}\hat{\mathbf{k}}_t r ~ t = r t = α t Δ W t − 1 ( 1 ) k ^ t for every t t t . Substituting this into Eq. (12) , the two terms containing the read cancel and, since κ t k ^ t = k t \kappa_t\hat{\mathbf{k}}_t=\mathbf{k}_t κ t k ^ t = k t ,
Δ W t ( 1 ) = α t Δ W t − 1 ( 1 ) − η t δ z t k t T , \Delta\mathbf{W}^{(1)}_t=\alpha_t\Delta\mathbf{W}^{(1)}_{t-1}-\eta_t\delta\mathbf{z}_t\mathbf{k}_t^{\mathsf{T}}, Δ W t ( 1 ) = α t Δ W t − 1 ( 1 ) − η t δ z t k t T ,
which is the hidden-layer recurrence Eq. (3) . Moreover D ^ = D \hat{\mathbf{D}}=\mathbf{D} D ^ = D , so every guessed gradient equals the gradient returned by the backward pass on this trajectory. The induction over tokens in the proof of Theorem 1 then identifies the trajectory with the sequential recurrence using the soft clip. Hence D = D ⋆ \mathbf{D}=\mathbf{D}^\star D = D ⋆ , and the fixed reads are the key reads r t ⋆ \mathbf{r}_t^\star r t ⋆ of that recurrence.
Features, states and outputs. By Lemma 6 , the hidden fast weights, pre-activations and features of the final forward pass differ from those of the sequential recurrence by at most a constant times ∥ D [ N ] − D ⋆ ∥ ∞ + ∥ R [ N ] − R ⋆ ∥ ∞ ≤ λ − 1 ∥ ( D [ N ] , R [ N ] ) − ( D ⋆ , R ⋆ ) ∥ λ \lVert\mathbf{D}^{[N]}-\mathbf{D}^\star\rVert_\infty+\lVert\mathbf{R}^{[N]}-\mathbf{R}^\star\rVert_\infty\le\lambda^{-1}\lVert(\mathbf{D}^{[N]},\mathbf{R}^{[N]})-(\mathbf{D}^\star,\mathbf{R}^\star)\rVert_\lambda ∥ D [ N ] − D ⋆ ∥ ∞ + ∥ R [ N ] − R ⋆ ∥ ∞ ≤ λ − 1 ∥( D [ N ] , R [ N ] ) − ( D ⋆ , R ⋆ ) ∥ λ , and Lemma 4 transfers this bound to the readout states. The query-side features read the updated weights Δ W t ( 1 ) \Delta\mathbf{W}^{(1)}_t Δ W t ( 1 ) , whose error is bounded by the same lemma, so their pre-activations differ by at most max t ∥ q t ∥ \max_t\lVert\mathbf{q}_t\rVert max t ∥ q t ∥ times the same quantity. The output is again a product of bounded quantities that are Lipschitz in ( D , R ) (\mathbf{D},\mathbf{R}) ( D , R ) , so all of them inherit the geometric rate, up to constant factors.
Remark 1 (Finite-step exactness with the delta rule). The map Ψ \Psi Ψ is strictly causal: by Eq. (12) , r t \mathbf{r}_t r t and z t \mathbf{z}_t z t depend only on δ z j \delta\mathbf{z}_j δ z j and r ~ j \tilde{\mathbf{r}}_j r ~ j with j < t j<t j < t , and so does D ^ t \hat{\mathbf{D}}_t D ^ t . The induction in the proof of Proposition 2 therefore applies to the pair ( D , R ) (\mathbf{D},\mathbf{R}) ( D , R ) : without damping, the gradients and reads at tokens 1 , … , min ( n , T ) 1,\ldots,\min(n,T) 1 , … , min ( n , T ) are exact after n n n iterations, with no contraction condition.
Centering. The centered readout of § 3.4 changes only the values that the readout fits, by a term that does not depend on the iterate, so both theorems carry over.
Corollary 1 (Centered readout). Theorems 1 and 2 hold for the centered readout Eq. (24) with Q Q Q replaced by Q c = V + 2 w 0 Q_{\mathrm{c}} = V + 2w_0 Q c = V + 2 w 0 in the constant ρ \rho ρ . The fixed point is then the gradient sequence, and for the delta rule the key reads, of the sequential recurrence with the centered readout and n s o f t n_{\mathrm{soft}} n soft in place of the projection.
Proof. Let ϕ 0 , t = ψ ( W 0 ( 1 ) k t + b ( 1 ) ) \phi_{0,t} = \psi(\mathbf{W}^{(1)}_0\mathbf{k}_t + \mathbf{b}^{(1)}) ϕ 0 , t = ψ ( W 0 ( 1 ) k t + b ( 1 ) ) be the features of the slow MLP. They are determined by the keys and the slow parameters, so they do not depend on the gradient guess or the reads, and ∥ ϕ 0 , t ∥ ≤ 1 \lVert\phi_{0,t}\rVert\le1 ∥ ϕ 0 , t ∥ ≤ 1 by the soft clip. By Eq. (24) , the centered inner loss of token t t t is
ℓ t = 1 2 ∥ W ~ t − 1 ( 2 ) ϕ t − W 0 ( 2 ) ϕ 0 , t − v t ∥ 2 = 1 2 ∥ W ~ t − 1 ( 2 ) ϕ t − v t c ∥ 2 , v t c = v t + W 0 ( 2 ) ϕ 0 , t , \ell_t = \frac{1}{2}\left\lVert\widetilde{\mathbf{W}}^{(2)}_{t-1}\phi_t - \mathbf{W}^{(2)}_0\phi_{0,t} - \mathbf{v}_t \right\rVert^2 = \frac{1}{2}\left\lVert\widetilde{\mathbf{W}}^{(2)}_{t-1}\phi_t - \mathbf{v}^{\mathrm{c}}_t \right\rVert^2,\qquad\mathbf{v}^{\mathrm{c}}_t = \mathbf{v}_t + \mathbf{W}^{(2)}_0\phi_{0,t}, ℓ t = 2 1 W t − 1 ( 2 ) ϕ t − W 0 ( 2 ) ϕ 0 , t − v t 2 = 2 1 W t − 1 ( 2 ) ϕ t − v t c 2 , v t c = v t + W 0 ( 2 ) ϕ 0 , t ,
which is the uncentered loss with the values v t \mathbf{v}_t v t replaced by the fixed sequence v t c \mathbf{v}^{\mathrm{c}}_t v t c . The centered sequential recurrence is therefore the uncentered recurrence Eq. (2) run on the values v t c \mathbf{v}^{\mathrm{c}}_t v t c . The same holds for every step of the iteration: the corrected values of Section C.5 are v t − W 0 ( 2 ) ( ϕ t − ϕ 0 , t ) = v t c − W 0 ( 2 ) ϕ t \mathbf{v}_t - \mathbf{W}^{(2)}_0(\phi_t-\phi_{0,t}) = \mathbf{v}^{\mathrm{c}}_t - \mathbf{W}^{(2)}_0\phi_t v t − W 0 ( 2 ) ( ϕ t − ϕ 0 , t ) = v t c − W 0 ( 2 ) ϕ t , those of the uncentered readout on v t c \mathbf{v}^{\mathrm{c}}_t v t c , so the readout trajectory, its feature gradient and the backward pass coincide with their uncentered counterparts, with either hidden update. The centered output differs from the uncentered output on v t c \mathbf{v}^{\mathrm{c}}_t v t c only by the term − W 0 ( 2 ) ϕ 0 , q , t -\mathbf{W}^{(2)}_0\phi_{0,q,t} − W 0 ( 2 ) ϕ 0 , q , t , which does not depend on the iterate. The values enter the proofs of Theorems 1 and 2 only through the bound ∥ v t ∥ ≤ V \lVert\mathbf{v}_t\rVert\le V ∥ v t ∥ ≤ V . Since ∥ v t c ∥ ≤ V + w 0 \lVert\mathbf{v}^{\mathrm{c}}_t\rVert\le V+w_0 ∥ v t c ∥ ≤ V + w 0 , both theorems apply with V + w 0 V+w_0 V + w 0 in place of V V V , that is, with Q Q Q replaced by V + 2 w 0 V+2w_0 V + 2 w 0 .
Algorithm details
Shifted and unshifted scans
Every scan in Algorithm 1 is one of two forms of the same recurrence, which differ only in whether token t t t reads the state before or after its own write. For a gated linear attention with keys k j \mathbf{k}_j k j , values v j \mathbf{v}_j v j and gates α j \alpha_j α j , the state is S t = α t S t − 1 + v t k t T \mathbf{S}_t = \alpha_t\mathbf{S}_{t-1} + \mathbf{v}_t\mathbf{k}_t^{\mathsf{T}} S t = α t S t − 1 + v t k t T , and the two reads of a query q t \mathbf{q}_t q t are
unshifted: o t = S t q t = ∑ j ≤ t ( ∏ i = j + 1 t α i ) v j ( k j T q t ) , shifted: o t = α t S t − 1 q t = ∑ j < t ( ∏ i = j + 1 t α i ) v j ( k j T q t ) . (14) \begin{aligned}
\text{unshifted:}\quad\mathbf{o}_t &= \mathbf{S}_t\mathbf{q}_t = \sum_{j\le t}\left(\prod_{i=j+1}^{t}\alpha_i\right)\mathbf{v}_j(\mathbf{k}_j^{\mathsf{T}}\mathbf{q}_t),\\
\text{shifted:}\quad\mathbf{o}_t &= \alpha_t\mathbf{S}_{t-1}\mathbf{q}_t = \sum_{j<t}\left(\prod_{i=j+1}^{t}\alpha_i\right)\mathbf{v}_j(\mathbf{k}_j^{\mathsf{T}}\mathbf{q}_t). \tag*{(14)}
\end{aligned} unshifted: o t shifted: o t = S t q t = j ≤ t ∑ ( i = j + 1 ∏ t α i ) v j ( k j T q t ) , = α t S t − 1 q t = j < t ∑ ( i = j + 1 ∏ t α i ) v j ( k j T q t ) . (14)
The unshifted read is the standard one, in which the sum includes the current token j = t j=t j = t ; the shifted read excludes it. We need both because the sequential recurrence Eq. (2) evaluates everything about the key of token t t t at the decayed weights W ‾ t − 1 \overline{\mathbf{W}}_{t-1} W t − 1 , before its own gradient step, and the query at W t \mathbf{W}_t W t , after it. The hidden-layer pre-activation of key t t t , Eq. (4) , is therefore a shifted read with q t = h t ( l − 1 ) \mathbf{q}_t = \mathbf{h}^{(l-1)}_t q t = h t ( l − 1 ) and v j = − η j ( l ) δ z j ( l ) \mathbf{v}_j = -\eta^{(l)}_j\delta\mathbf{z}^{(l)}_j v j = − η j ( l ) δ z j ( l ) ; the transposed read of Step 3, Eq. (6) , is a shifted read with the roles of keys and values exchanged; DELTARULEGRAD reads the readout state before token t t t ’s write, Eq. (23) , which is again shifted; and the final output pass reads the query-side features under the updated weights, which is unshifted. The shift is also what makes the fixed-point argument of § 3.3 work: with a shifted read no token’s update depends on its own write, so all T T T updates can be evaluated simultaneously from an estimate of D \mathbf{D} D ; with an unshifted read the update of token t t t would depend on δ z t \delta\mathbf{z}_t δ z t itself and the parallel evaluation would not reproduce the sequential recurrence. The use of shifted kernels for keys is consistent with standard formulation of online gradient descent used in GDN and TTT-MLP, where the gradient (on the keys) is evaluated after applying the decay (shifted), and the readout from the query is applied using the updated state (unshifted).
In the chunkwise form the difference is confined to the intra-chunk term. Writing the chunk’s queries, keys and values as matrices and Γ \boldsymbol{\Gamma} Γ for the matrix of decay ratios, the two reads are
unshifted: O = Q S c T + tril ( Q K T ⊙ Γ ) V , shifted: O = Γ Q S c T + tril ( Q K T ⊙ Γ , − 1 ) V , (15) \text{unshifted:}\quad\mathbf{O} = \mathbf{Q}\mathbf{S}_{\mathrm{c}}^{\mathsf{T}} + \operatorname{tril}(\mathbf{Q}\mathbf{K}^{\mathsf{T}}\odot\boldsymbol{\Gamma})\mathbf{V},\qquad
\text{shifted:}\quad\mathbf{O} = \boldsymbol{\Gamma}\mathbf{Q}\mathbf{S}_{\mathrm{c}}^{\mathsf{T}} + \operatorname{tril}(\mathbf{Q}\mathbf{K}^{\mathsf{T}}\odot\boldsymbol{\Gamma},-1)\mathbf{V}, \tag*{(15)} unshifted: O = Q S c T + tril ( Q K T ⊙ Γ ) V , shifted: O = Γ Q S c T + tril ( Q K T ⊙ Γ , − 1 ) V , (15)
Algorithm 1 ParaTTT with linear-attention hidden updates and an uncentered readout (one head; all operations act on the full sequence of T T T tokens). Practical modifications are in § 3.4 .
Require: keys k 1 : T \mathbf{k}_{1:T} k 1 : T , queries q 1 : T \mathbf{q}_{1:T} q 1 : T , values v 1 : T \mathbf{v}_{1:T} v 1 : T ; gates α 1 : T \alpha_{1:T} α 1 : T , learning rates η 1 : T ( 1 … L ) \eta_{1:T}^{(1\ldots L)} η 1 : T ( 1 … L ) ; weights W ( l ) \mathbf{W}^{(l)} W ( l ) , b ( l ) \mathbf{b}^{(l)} b ( l ) ; iterations N N N ; damping λ \lambda λ
δ z ^ 1 : T ( l ) ← 0 \delta\hat{\mathbf{z}}_{1:T}^{(l)} \leftarrow\mathbf{0} δ z ^ 1 : T ( l ) ← 0 for every hidden layer l < L l < L l < L ▷ initial estimate of the backward gradients
for n = 1 , … , N n = 1,\ldots,N n = 1 , … , N do
Step 1: parallel forward pass
h ( 0 ) ← k \mathbf{h}^{(0)} \leftarrow\mathbf{k} h ( 0 ) ← k
for l = 1 , … , L − 1 l = 1,\ldots,L-1 l = 1 , … , L − 1 do
a ( l ) ← W ( l ) h ( l − 1 ) + b ( l ) + SHIFTEDSCAN ( keys h ( l − 1 ) , values − η ( l ) δ z ( l ) , gates α ) \mathbf{a}^{(l)} \leftarrow\mathbf{W}^{(l)}\mathbf{h}^{(l-1)} + \mathbf{b}^{(l)} + \operatorname{SHIFTEDSCAN}(\text{keys }\mathbf{h}^{(l-1)},\text{ values }-\eta^{(l)}\delta\mathbf{z}^{(l)},\text{ gates }\alpha) a ( l ) ← W ( l ) h ( l − 1 ) + b ( l ) + SHIFTEDSCAN ( keys h ( l − 1 ) , values − η ( l ) δ z ( l ) , gates α )
h ( l ) ← σ ( a ( l ) ) \mathbf{h}^{(l)} \leftarrow\sigma(\mathbf{a}^{(l)}) h ( l ) ← σ ( a ( l ) ) ; store the Jacobian J ( l ) \mathbf{J}^{(l)} J ( l )
end for
Step 2: readout trajectory and its gradient
v ~ ← v − W 0 ( L ) h ( L − 1 ) \tilde{\mathbf{v}} \leftarrow\mathbf{v} - \mathbf{W}_{0}^{(L)}\mathbf{h}^{(L-1)} v ~ ← v − W 0 ( L ) h ( L − 1 )
e ← DELTARULEGRAD ( h ( L − 1 ) , v ~ , α , η ( L ) , W 0 ( L ) ) \mathbf{e} \leftarrow\operatorname{DELTARULEGRAD}(\mathbf{h}^{(L-1)},\tilde{\mathbf{v}},\alpha,\eta^{(L)},\mathbf{W}_{0}^{(L)}) e ← DELTARULEGRAD ( h ( L − 1 ) , v ~ , α , η ( L ) , W 0 ( L ) ) ▷ Eq. (5) ; includes slow offset
Step 3: parallel backward pass
g ← e / η ( L ) \mathbf{g} \leftarrow\mathbf{e}/\eta^{(L)} g ← e / η ( L )
for l = L − 1 , … , 1 l = L-1,\ldots,1 l = L − 1 , … , 1 do
δ z ^ ( l ) ← J ( l ) ⊤ g \delta\hat{\mathbf{z}}^{(l)} \leftarrow\mathbf{J}^{(l)\top}\mathbf{g} δ z ^ ( l ) ← J ( l ) ⊤ g
if l > 1 l > 1 l > 1 then
g ← W 0 ( l ) ⊤ δ z ^ ( l ) \mathbf{g} \leftarrow\mathbf{W}_{0}^{(l)\top}\delta\hat{\mathbf{z}}^{(l)} g ← W 0 ( l ) ⊤ δ z ^ ( l )
g ← g + TRANSPOSEDSCAN ( keys δ z ( l ) , values − η ( l ) h ( l − 1 ) , gates α ) δ z ^ ( l ) \mathbf{g} \leftarrow\mathbf{g} + \operatorname{TRANSPOSEDSCAN}(\text{keys }\delta\mathbf{z}^{(l)},\text{ values }-\eta^{(l)}\mathbf{h}^{(l-1)},\text{ gates }\alpha)\delta\hat{\mathbf{z}}^{(l)} g ← g + TRANSPOSEDSCAN ( keys δ z ( l ) , values − η ( l ) h ( l − 1 ) , gates α ) δ z ^ ( l )
end if
end for
δ z ( l ) ← δ z ( l ) + λ ( δ z ^ ( l ) − δ z ( l ) ) \delta\mathbf{z}^{(l)} \leftarrow\delta\mathbf{z}^{(l)} + \lambda(\delta\hat{\mathbf{z}}^{(l)}-\delta\mathbf{z}^{(l)}) δ z ( l ) ← δ z ( l ) + λ ( δ z ^ ( l ) − δ z ( l ) ) for every hidden layer l l l
end for
Output
ϕ k ← \phi_k \leftarrow ϕ k ← features of k \mathbf{k} k from a final Step 1 (shifted); ϕ q ← \phi_q \leftarrow ϕ q ← features of q \mathbf{q} q from the same scans, unshifted
v ~ ← v − W 0 ( L ) ϕ k \tilde{\mathbf{v}} \leftarrow\mathbf{v} - \mathbf{W}_{0}^{(L)}\phi_k v ~ ← v − W 0 ( L ) ϕ k
o f a s t ← DELTARULE ( queries ϕ q , keys ϕ k , values v ~ , gates α , step sizes η ( L ) ) \mathbf{o}_{\mathrm{fast}} \leftarrow\operatorname{DELTARULE}(\text{queries }\phi_q,\text{ keys }\phi_k,\text{ values }\tilde{\mathbf{v}},\text{ gates }\alpha,\text{ step sizes }\eta^{(L)}) o fast ← DELTARULE ( queries ϕ q , keys ϕ k , values v ~ , gates α , step sizes η ( L ) )
return o ← W 0 ( L ) ϕ q + o f a s t \mathbf{o} \leftarrow\mathbf{W}_{0}^{(L)}\phi_q + \mathbf{o}_{\mathrm{fast}} o ← W 0 ( L ) ϕ q + o fast
where tril ( ⋅ , − 1 ) \operatorname{tril}(\cdot,-1) tril ( ⋅ , − 1 ) is the strictly lower-triangular part. The boundary state S c \mathbf{S}_c S c contains only tokens of earlier chunks, so the inter-chunk term is identical in both forms; only the causal mask of the intra-chunk attention changes, from j ≤ t j \le t j ≤ t to j < t j < t j < t . This is how the shifted kernels are implemented: we copied FLA’s chunkwise gated-linear-attention and gated-delta-rule kernels and changed the intra-chunk mask in the forward output kernel and its two backward kernels (for the query/key gradients and for the value gradient) from a causal to a strictly causal comparison of the token indices, leaving the inter-chunk state recurrence, the WY solve and the chunk-boundary bookkeeping untouched. The same one-line change applied to the delta-rule kernel’s final read gives the shifted delta-rule scan used for the hidden layer in Section C.2 and, with the query and value operands exchanged, DELTARULEGRAD \operatorname{DELTARULEGRAD} DELTARULEGRAD . At decode time (T = 1 T = 1 T = 1 ) the shifted read is the read of the cached state before the token’s update, so no special case is needed.
Algorithm 1 is implemented with three chunkwise kernels of the FLA library and their transposes: the gated-linear-attention scan for the hidden layers (Eq. (4) and its transpose Eq. (6) ), the gated-delta-rule scan for the hidden layer in the delta-rule form of Eq. (16) , and the gated-delta-rule kernel with the query and value operands of its final read exchanged for DELTARULEGRAD \operatorname{DELTARULEGRAD} DELTARULEGRAD (Section C.3). The shift is implemented inside the chunk: the intra-chunk attention is strictly lower-triangular for the key-side read and lower-triangular including the diagonal for the query-side read, and the inter-chunk state is the same for both. Every iteration recomputes the hidden-layer scans from the current pseudo-labels. In the reported one-hidden-layer configuration, the slow hidden contribution W 0 ( 1 ) k t \mathbf{W}_{0}^{(1)}\mathbf{k}_t W 0 ( 1 ) k t and the iteration-0 features are computed once. For deeper memories, the inputs of subsequent hidden layers change with the iteration, so their slow contributions are recomputed. Training backpropagates through all N N N iterations with the kernels’ chunkwise backward passes; iteration n n n stores the pseudo-labels, the features and the Jacobians of σ \sigma σ , and the readout states at chunk boundaries are recomputed in the backward pass rather than stored.
Delta rule for the hidden layers
Step 1 evolves each hidden layer’s fast weights by the gated linear attention Eq. (4) , which sums the writes − η j ( l ) δ z j ( l ) h j ( l − 1 ) T -\eta_{j}^{(l)}\delta z_{j}^{(l)}h_{j}^{(l-1)\mathsf{T}} − η j ( l ) δ z j ( l ) h j ( l − 1 ) T without ever erasing. The same weights can be evolved by a gated delta rule. With the unit key k ^ t = h t ( l − 1 ) / ∥ h t ( l − 1 ) ∥ \hat{k}_{t}=h_{t}^{(l-1)}/\lVert h_{t}^{(l-1)}\rVert k ^ t = h t ( l − 1 ) / ∥ h t ( l − 1 ) ∥ and write rate β t = η t ( l ) \beta_{t}=\eta_{t}^{(l)} β t = η t ( l ) , the delta rule is
Δ W t ( l ) = α t Δ W t − 1 ( l ) + β t ( y t − r t ) k ^ t T , (16) \Delta\mathbf{W}_{t}^{(l)}=\alpha_{t}\Delta\mathbf{W}_{t-1}^{(l)}+\beta_{t}(\mathbf{y}_{t}-\mathbf{r}_{t})\hat{\mathbf{k}}_{t}^{\mathsf{T}}, \tag*{(16)} Δ W t ( l ) = α t Δ W t − 1 ( l ) + β t ( y t − r t ) k ^ t T , (16)
r t = α t Δ W t − 1 ( l ) k ^ t , y t = r ~ t − η t ( l ) ∥ h t ( l − 1 ) ∥ β t δ z t ( l ) , \mathbf{r}_{t}=\alpha_{t}\Delta\mathbf{W}_{t-1}^{(l)}\hat{\mathbf{k}}_{t},\qquad
\mathbf{y}_{t}=\tilde{\mathbf{r}}_{t}-\frac{\eta_{t}^{(l)}\lVert h_{t}^{(l-1)}\rVert}{\beta_{t}}\delta\mathbf{z}_{t}^{(l)}, r t = α t Δ W t − 1 ( l ) k ^ t , y t = r ~ t − β t η t ( l ) ∥ h t ( l − 1 ) ∥ δ z t ( l ) ,
where r t \mathbf{r}_{t} r t is the read of the key from the decayed weights and the target y t \mathbf{y}_{t} y t is built from r ~ t \tilde{\mathbf{r}}_{t} r ~ t , the same read taken on the trajectory of the previous iteration. Substituting y t \mathbf{y}_{t} y t ,
Δ W t ( l ) = α t Δ W t − 1 ( l ) − η t ( l ) δ z t ( l ) h t ( l − 1 ) T + β t ( r ~ t − r t ) k ^ t T , (17) \Delta\mathbf{W}_{t}^{(l)}=\alpha_{t}\Delta\mathbf{W}_{t-1}^{(l)}-\eta_{t}^{(l)}\delta\mathbf{z}_{t}^{(l)}h_{t}^{(l-1)\mathsf{T}}+\beta_{t}(\tilde{\mathbf{r}}_{t}-\mathbf{r}_{t})\hat{\mathbf{k}}_{t}^{\mathsf{T}}, \tag*{(17)} Δ W t ( l ) = α t Δ W t − 1 ( l ) − η t ( l ) δ z t ( l ) h t ( l − 1 ) T + β t ( r ~ t − r t ) k ^ t T , (17)
which is Eq. (3) plus a correction that erases the part of the previous iteration’s read that the current trajectory does not reproduce. At the fixed point r ~ t = r t \tilde{\mathbf{r}}_{t}=\mathbf{r}_{t} r ~ t = r t and the correction vanishes, so the two forms agree there; along the iterates the delta-rule form erases stale writes rather than accumulating them. It is computed with a chunkwise gated-delta-rule kernel; Table 1 compares the two.
The readout gradient
Following the notation of Yang et al. [63 ] ; Tumma et al. [53 ] , write S t = Δ W t ( L ) T \mathbf{S}_{t}=\Delta\mathbf{W}_{t}^{(L)\mathsf{T}} S t = Δ W t ( L ) T for the fast part of the readout, which decays as S ˉ t − 1 = α t S t − 1 \bar{\mathbf{S}}_{t-1}=\alpha_{t}\mathbf{S}_{t-1} S ˉ t − 1 = α t S t − 1 , and keep the slow readout W 0 ( L ) \mathbf{W}_{0}^{(L)} W 0 ( L ) as a fixed offset, so that the weight token t t t reads is S ˉ t − 1 + W 0 ( L ) T \bar{\mathbf{S}}_{t-1}+\mathbf{W}_{0}^{(L)\mathsf{T}} S ˉ t − 1 + W 0 ( L ) T . (Equivalently, the full state S t + W 0 ( L ) T \mathbf{S}_{t}+\mathbf{W}_{0}^{(L)\mathsf{T}} S t + W 0 ( L ) T decays toward W 0 ( L ) T \mathbf{W}_{0}^{(L)\mathsf{T}} W 0 ( L ) T : α t ( S t − 1 + W 0 ( L ) T ) + ( 1 − α t ) W 0 ( L ) T \alpha_{t}(\mathbf{S}_{t-1}+\mathbf{W}_{0}^{(L)\mathsf{T}})+(1-\alpha_{t})\mathbf{W}_{0}^{(L)\mathsf{T}} α t ( S t − 1 + W 0 ( L ) T ) + ( 1 − α t ) W 0 ( L ) T .) With the corrected values v ~ t = v t − W 0 ( L ) ϕ t \tilde{\mathbf{v}}_{t}=\mathbf{v}_{t}-\mathbf{W}_{0}^{(L)}\phi_{t} v ~ t = v t − W 0 ( L ) ϕ t (or their centered form, Section C.5) and u t = η t ( L ) ( v ~ t − S ˉ t − 1 T ϕ t ) \mathbf{u}_{t}=\eta_{t}^{(L)}(\tilde{\mathbf{v}}_{t}-\bar{\mathbf{S}}_{t-1}^{\mathsf{T}}\phi_{t}) u t = η t ( L ) ( v ~ t − S ˉ t − 1 T ϕ t ) the value that token t t t writes, the fast readout’s recurrence is S t = S ˉ t − 1 + ϕ t u t T \mathbf{S}_{t}=\bar{\mathbf{S}}_{t-1}+\phi_{t}\mathbf{u}_{t}^{\mathsf{T}} S t = S ˉ t − 1 + ϕ t u t T , and token t t t ’s inner loss and the gradient that Step 3 needs are
ℓ t = 1 2 ∥ S ˉ t − 1 T ϕ t − v ~ t ∥ 2 , e t = η t ( L ) ∂ ℓ t ∂ ϕ t = − ( S ˉ t − 1 + W 0 ( L ) T ) u t . (18) \ell_{t}=\frac{1}{2}\lVert\bar{\mathbf{S}}_{t-1}^{\mathsf{T}}\phi_{t}-\tilde{\mathbf{v}}_{t}\rVert^{2},\qquad
\mathbf{e}_{t}=\eta_{t}^{(L)}\frac{\partial\ell_{t}}{\partial\phi_{t}}=-(\bar{\mathbf{S}}_{t-1}+\mathbf{W}_{0}^{(L)\mathsf{T}})\mathbf{u}_{t}. \tag*{(18)} ℓ t = 2 1 ∥ S ˉ t − 1 T ϕ t − v ~ t ∥ 2 , e t = η t ( L ) ∂ ϕ t ∂ ℓ t = − ( S ˉ t − 1 + W 0 ( L ) T ) u t . (18)
The chunkwise gated-delta-rule kernel of Yang et al. [63 , 64 ] , as implemented in FLA, already forms every quantity on the right: the chunk-boundary states, the intra-chunk decays and the corrected values u j \mathbf{u}_{j} u j of a chunk, which it obtains jointly from the WY representation. DELTARULEGRAD therefore reuses that kernel and changes only its final read, exchanging the roles of the queries and the values; its cost equals one delta-rule pass. The chunkwise form follows.
Chunkwise computation of the readout gradient
We use the chunk notation of Yang et al. [63 , 64 ] ; Tumma et al. [53 ] . Partition the sequence into chunks of C C C tokens and, for a chunk, stack its features, values, step sizes and outputs as Φ ∈ R C × d \Phi\in\mathbb{R}^{C\times d} Φ ∈ R C × d , V ∈ R C × d v \mathbf{V}\in\mathbb{R}^{C\times d_{v}} V ∈ R C × d v , β = ( η t ) t ∈ C \boldsymbol{\beta}=(\eta_{t})_{t\in C} β = ( η t ) t ∈ C and O \mathbf{O} O , with S c \mathbf{S}_{c} S c the readout state at the chunk’s start and γ t = ∏ i < t α i \gamma_{t}=\prod_{i<t}\alpha_{i} γ t = ∏ i < t α i the cumulative decay within the chunk. Write γ t , j = γ t / γ j \boldsymbol{\gamma}_{t,j}=\gamma_{t}/\gamma_{j} γ t , j = γ t / γ j , Γ = diag ( γ t ) \mathbf{\Gamma}=\operatorname{diag}(\boldsymbol{\gamma}_{t}) Γ = diag ( γ t ) , Γ C = diag ( γ C / γ j ) \mathbf{\Gamma}_{C}=\operatorname{diag}(\boldsymbol{\gamma}_{C}/\boldsymbol{\gamma}_{j}) Γ C = diag ( γ C / γ j ) and L = diag ( β ) \mathbf{L}=\operatorname{diag}(\boldsymbol{\beta}) L = diag ( β ) . In this transposed convention the gated delta rule advances the state by S t = ( I − η t ϕ t ϕ t T ) S ˉ t − 1 + η t ϕ t v t T \mathbf{S}_{t}=(\mathbf{I}-\eta_{t}\phi_{t}\phi_{t}^{\mathsf{T}})\bar{\mathbf{S}}_{t-1}+\eta_{t}\phi_{t}\mathbf{v}_{t}^{\mathsf{T}} S t = ( I − η t ϕ t ϕ t T ) S ˉ t − 1 + η t ϕ t v t T , and the transitions of a chunk, applied in order from the left, have the WY form ( I − η C ϕ C ϕ C T ) ⋯ ( I − η 1 ϕ 1 ϕ 1 T ) = I − Φ T U (\mathbf{I}-\eta_{C}\phi_{C}\phi_{C}^{\mathsf{T}})\cdots(\mathbf{I}-\eta_{1}\phi_{1}\phi_{1}^{\mathsf{T}})=\mathbf{I}-\Phi^{\mathsf{T}}\mathbf{U} ( I − η C ϕ C ϕ C T ) ⋯ ( I − η 1 ϕ 1 ϕ 1 T ) = I − Φ T U with
U = T 0 L Φ , T 0 = ( I + tril ( L Φ Φ T , − 1 ) ) − 1 , (19) \mathbf{U}=\mathbf{T}_{0}\mathbf{L}\boldsymbol{\Phi},\qquad
\mathbf{T}_{0}=(\mathbf{I}+\operatorname{tril}(\mathbf{L}\boldsymbol{\Phi}\boldsymbol{\Phi}^{\mathsf{T}},-1))^{-1}, \tag*{(19)} U = T 0 L Φ , T 0 = ( I + tril ( L Φ Φ T , − 1 ) ) − 1 , (19)
a C × C C\times C C × C triangular solve against the Gram matrix of the chunk’s features. Including the decay gives the solve
T = Γ T 0 Γ − 1 = ( I + tril ( ( L Φ Φ T ) ⊙ Γ , − 1 ) ) − 1 . \mathbf{T}=\mathbf{\Gamma}\mathbf{T}_{0}\mathbf{\Gamma}^{-1}=(\mathbf{I}+\operatorname{tril}((\mathbf{L}\boldsymbol{\Phi}\boldsymbol{\Phi}^{\mathsf{T}})\odot\mathbf{\Gamma},-1))^{-1}. T = Γ T 0 Γ − 1 = ( I + tril (( L Φ Φ T ) ⊙ Γ , − 1 ) ) − 1 .
The rows u j \mathbf{u}_j u j of the corrected values
U v = T L ( V ~ − Γ Φ S c ) (20) \mathbf{U}_v = \mathbf{T}\mathbf{L}\left(\widetilde{\mathbf{V}}-\mathbf{\Gamma}\mathbf{\Phi}\mathbf{S}_c\right) \tag*{(20)} U v = TL ( V − ΓΦ S c ) (20)
are exactly the writes u j = η j ( v ~ j − S ‾ j − 1 ⊤ ϕ j ) \mathbf{u}_j=\eta_j\left(\widetilde{\mathbf{v}}_j-\overline{\mathbf{S}}_{j-1}^{\top}\boldsymbol{\phi}_j\right) u j = η j ( v j − S j − 1 ⊤ ϕ j ) of Section C.3, expressed without the token-by-token state, and the state entering token t t t of the chunk is
S ‾ t − 1 = γ t S c + ∑ j ∈ C j < t γ t , j ϕ j u j ⊤ . (21) \overline{\mathbf{S}}_{t-1}=\gamma_t\mathbf{S}_c+\sum_{\substack{j\in\mathcal{C}\\j<t}}\gamma_{t,j}\boldsymbol{\phi}_j\mathbf{u}_j^{\top}. \tag*{(21)} S t − 1 = γ t S c + j ∈ C j < t ∑ γ t , j ϕ j u j ⊤ . (21)
The kernel computes, per chunk, T \mathbf{T} T , U v \mathbf{U}_v U v , the next boundary state S c + 1 = γ C S c + Φ ⊤ Γ C U v \mathbf{S}_{c+1}=\gamma_C\mathbf{S}_c+\mathbf{\Phi}^{\top}\mathbf{\Gamma}_C\mathbf{U}_v S c + 1 = γ C S c + Φ ⊤ Γ C U v (the inter-chunk recurrence, a short scan over chunks), and the output
O = Γ Φ q S c + tril ( Φ q Φ ⊤ ⊙ Γ ) U v , (22) \mathbf{O}=\mathbf{\Gamma}\mathbf{\Phi}_q\mathbf{S}_c+\operatorname{tril}\left(\mathbf{\Phi}_q\mathbf{\Phi}^{\top}\odot\mathbf{\Gamma}\right)\mathbf{U}_v, \tag*{(22)} O = Γ Φ q S c + tril ( Φ q Φ ⊤ ⊙ Γ ) U v , (22)
where the tril \operatorname{tril} tril mask is causal; this is Eq. (21) substituted into o t = S t ⊤ ϕ q , t \mathbf{o}_t=\mathbf{S}_t^{\top}\boldsymbol{\phi}_{q,t} o t = S t ⊤ ϕ q , t (the slow term W 0 ( L ) ϕ q , t \mathbf{W}_0^{(L)}\boldsymbol{\phi}_{q,t} W 0 ( L ) ϕ q , t is added outside the kernel), a product with the boundary state plus a causal intra-chunk attention with scores ϕ q , t ⊤ ϕ j \boldsymbol{\phi}_{q,t}^{\top}\boldsymbol{\phi}_j ϕ q , t ⊤ ϕ j and values u j \mathbf{u}_j u j .
The feature gradient e t = − ( S ‾ t − 1 + W 0 ( L ) ⊤ ) u t \mathbf{e}_t=-\left(\overline{\mathbf{S}}_{t-1}+\mathbf{W}_0^{(L)\top}\right)\mathbf{u}_t e t = − ( S t − 1 + W 0 ( L ) ⊤ ) u t of Eq. (18) substitutes the same Eq. (21) and gives, per token and per chunk,
e t = − ( γ t S c + W 0 ( L ) ⊤ ) u t − ∑ j ∈ C j < t γ t , j ( u j ⊤ u t ) ϕ j , \mathbf{e}_t=-\left(\gamma_t\mathbf{S}_c+\mathbf{W}_0^{(L)\top}\right)\mathbf{u}_t-\sum_{\substack{j\in\mathcal{C}\\j<t}}\gamma_{t,j}\left(\mathbf{u}_j^{\top}\mathbf{u}_t\right)\boldsymbol{\phi}_j, e t = − ( γ t S c + W 0 ( L ) ⊤ ) u t − j ∈ C j < t ∑ γ t , j ( u j ⊤ u t ) ϕ j ,
E = − Γ U v S c ⊤ − U v W 0 ( L ) − tril ( U v U v ⊤ ⊙ Γ / , − 1 ) Φ , (23) \mathbf{E}=-\mathbf{\Gamma}\mathbf{U}_v\mathbf{S}_c^{\top}-\mathbf{U}_v\mathbf{W}_0^{(L)}-\operatorname{tril}\left(\mathbf{U}_v\mathbf{U}_v^{\top}\odot\mathbf{\Gamma}_{/,-1}\right)\mathbf{\Phi}, \tag*{(23)} E = − Γ U v S c ⊤ − U v W 0 ( L ) − tril ( U v U v ⊤ ⊙ Γ / , − 1 ) Φ , (23)
a product with the boundary state plus a causal intra-chunk attention in which the corrected values act as both queries and keys and the features as values. Comparing Eqs. (22) and (23) : the two expressions share S c \mathbf{S}_c S c , Γ \mathbf{\Gamma} Γ , T \mathbf{T} T and U v \mathbf{U}_v U v , and differ only in which operands enter the final read, ( Φ q , Φ , U v ) (\mathbf{\Phi}_q,\mathbf{\Phi},\mathbf{U}_v) ( Φ q , Φ , U v ) for the output against ( U v , U v , Φ ) (\mathbf{U}_v,\mathbf{U}_v,\mathbf{\Phi}) ( U v , U v , Φ ) for the gradient, and in the strict lower-triangular mask, which excludes j = t j=t j = t and implements the shifted arrangement. DELTARULEGRAD is therefore the gated-delta-rule kernel with the query and value operands of its final read exchanged, with the offset W 0 ( L ) ⊤ \mathbf{W}_0^{(L)\top} W 0 ( L ) ⊤ entering its final read only; its cost equals that of one delta-rule pass. Because T \mathbf{T} T depends on the features, which change with every iteration, it is recomputed in each of the N N N iterations and in the final scans. In the reported one-hidden-layer configuration, the hidden delta-rule solve (Section C.2) is computed once because its keys are the fixed inputs k t \mathbf{k}_t k t . In deeper memories, this reuse applies only to the first hidden layer; subsequent hidden-layer inputs change with the iteration. The transposed reads of Step 3, Eq. (6) , have the same structure with plain linear attention: values − η j ( l ) h j ( l − 1 ) -\eta_j^{(l)}\mathbf{h}_j^{(l-1)} − η j ( l ) h j ( l − 1 ) in place of U v \mathbf{U}_v U v , and no triangular solve.
Centering
The centered readout of § 3.4 is
f W ( k t ) = W ( L ) ϕ t − W 0 ( L ) ϕ 0 , t = Δ W ( L ) ϕ t + W ( L ) ( ϕ t − ϕ 0 , t ) , (24) f_{\mathbf{W}}(\mathbf{k}_t)=\mathbf{W}^{(L)}\boldsymbol{\phi}_t-\mathbf{W}_0^{(L)}\boldsymbol{\phi}_{0,t}=\Delta\mathbf{W}^{(L)}\boldsymbol{\phi}_t+\mathbf{W}^{(L)}\left(\boldsymbol{\phi}_t-\boldsymbol{\phi}_{0,t}\right), \tag*{(24)} f W ( k t ) = W ( L ) ϕ t − W 0 ( L ) ϕ 0 , t = Δ W ( L ) ϕ t + W ( L ) ( ϕ t − ϕ 0 , t ) , (24)
the fast readout applied to the features plus the slow readout applied to their drift from the slow features. With it, the inner loss of token t t t is ℓ t = 1 2 ∥ Δ W ‾ t − 1 ( L ) ϕ t + W ( L ) ( ϕ t − ϕ 0 , t ) − v t ∥ 2 \ell_t=\frac{1}{2}\left\lVert\Delta\overline{\mathbf{W}}_{t-1}^{(L)}\boldsymbol{\phi}_t+\mathbf{W}^{(L)}\left(\boldsymbol{\phi}_t-\boldsymbol{\phi}_{0,t}\right)-\mathbf{v}_t\right\rVert^2 ℓ t = 2 1 Δ W t − 1 ( L ) ϕ t + W ( L ) ( ϕ t − ϕ 0 , t ) − v t 2 . Writing v ~ t = v t − W ( L ) ( ϕ t − ϕ 0 , t ) \widetilde{\mathbf{v}}_t=\mathbf{v}_t-\mathbf{W}^{(L)}\left(\boldsymbol{\phi}_t-\boldsymbol{\phi}_{0,t}\right) v t = v t − W ( L ) ( ϕ t − ϕ 0 , t ) for the corrected values, this is the uncentered loss with v t \mathbf{v}_t v t replaced by v ~ t \widetilde{\mathbf{v}}_t v t , so the fast readout Δ W ( L ) \Delta\mathbf{W}^{(L)} Δ W ( L ) follows the same gated delta rule as before on the corrected values, and its trajectory is computed by the same kernel. The feature gradient of Eq. (5) becomes
e t = η t ( L ) ( Δ W ‾ t − 1 ( L ) + W 0 ( L ) ) ⊤ ( Δ W ‾ t − 1 ( L ) ϕ t − v ~ t ) , (25) \mathbf{e}_t=\eta_t^{(L)}\left(\Delta\overline{\mathbf{W}}_{t-1}^{(L)}+\mathbf{W}_0^{(L)}\right)^{\top}\left(\Delta\overline{\mathbf{W}}_{t-1}^{(L)}\boldsymbol{\phi}_t-\widetilde{\mathbf{v}}_t\right), \tag*{(25)} e t = η t ( L ) ( Δ W t − 1 ( L ) + W 0 ( L ) ) ⊤ ( Δ W t − 1 ( L ) ϕ t − v t ) , (25)
which DELTARULEGRAD returns when its readout weight is offset by W 0 ( L ) \mathbf{W}_0^{(L)} W 0 ( L ) ; the offset enters the kernel’s final read only. The output is o t = Δ W t ( L ) ϕ q , t + W 0 ( L ) ( ϕ q , t − ϕ 0 , q , t ) \mathbf{o}_t=\Delta\mathbf{W}_t^{(L)}\boldsymbol{\phi}_{q,t}+\mathbf{W}_0^{(L)}\left(\boldsymbol{\phi}_{q,t}-\boldsymbol{\phi}_{0,q,t}\right) o t = Δ W t ( L ) ϕ q , t + W 0 ( L ) ( ϕ q , t − ϕ 0 , q , t ) , one gated-delta-rule pass on the corrected values plus a matrix product. The slow features ϕ 0 \boldsymbol{\phi}_0 ϕ 0 of the keys are exactly those produced by iteration 0 0 0 , where D = 0 \mathbf{D}=\mathbf{0} D = 0 , so they cost nothing extra; the slow features of the queries need one additional pass through the slow MLP. The fixed point of § 3.3 is unchanged in form: it is the gradient sequence of the sequential recurrence with the centered readout. W 0 ( L ) \mathbf{W}_0^{(L)} W 0 ( L ) is trained by backpropagation like every other slow parameter; because the forward pass depends on it through ϕ t − ϕ 0 , t \boldsymbol{\phi}_t-\boldsymbol{\phi}_{0,t} ϕ t − ϕ 0 , t , its gradient is a true gradient and no stop-gradient is involved.
Model configurations and training recipe
The reported experiments use the hard unit-ball projection x / max ( 1 , ∥ x ∥ ) \mathbf{x}/\max(1,\lVert\mathbf{x}\rVert) x / max ( 1 , ∥ x ∥) specified below. The soft clip is used only for the theoretical analysis in Sections B and 3.3.
All models share one backbone: a stack of pre-norm residual blocks, each a token-mixing layer followed by a SwiGLU MLP, with RMSNorm and tied embeddings, 24 layers at both scales, d = 1024 d = 1024 d = 1024 at 340M and d = 2048 d = 2048 d = 2048 at 1.3B. The default head layouts are 8 heads of dimension 128 at 340M and 16 heads of dimension 128 at 1.3B; TTT-MLP and LaCT use the separate layouts below. The token-mixing layer is ParaTTT, Gated DeltaNet, Mamba-2, softmax attention with RoPE, or TTT-MLP; the recurrent layers apply a short causal convolution of width 4 to the query, key and value projections and an output gate. Mamba-2 uses expansion 1 and 4 groups, TTT-MLP its published layout of 16 heads of dimension 64 with a block of 16 tokens, and LaCT its published recipe with a chunk of 2048 tokens, without the sliding-window branch, so that it is a pure fast-weight model. ParaTTT uses the memory of § 3.1 with one hidden layer of width d h e a d = 128 d_{\mathrm{head}} = 128 d head = 128 and SiLU, the delta rule for the hidden layer and centering (§ 3.4 ), a per-token learning rate that is the sigmoid of a linear function of the input, the decay gate of Gated DeltaNet shared by both layers, damping λ = 0.8 \lambda= 0.8 λ = 0.8 , and N = 2 N = 2 N = 2 iterations during training unless stated. Every ParaTTT model starts from the initialization of the Gated DeltaNet model of the same size, so that the two differ only in the memory. Table 2 lists the models of Table 1 . All models are trained with FLAME on SlimPajama (parquet shards of the 100B-token subset) with the Llama tokenizer, sequence length 4096, 8 sequences per GPU with two accumulation steps on 8 GPUs (0.5M tokens per step), AdamW (β = ( 0.9 , 0.95 ) \beta= (0.9, 0.95) β = ( 0.9 , 0.95 ) , ϵ = 10 − 8 \epsilon= 10^{-8} ϵ = 1 0 − 8 , weight decay 0.01), peak learning rate 4 × 10 − 4 4 \times10^{-4} 4 × 1 0 − 4 with 1024 warmup steps and cosine decay to 4 × 10 − 5 4 \times10^{-5} 4 × 1 0 − 5 , gradient clipping at norm 1, bf16, seed 42 and the same data order. The 340M models train for 30k steps and the 1.3B models for 190k steps; the 1.3B ParaTTT model trains on two nodes with a sharding that reproduces the one-node data order. Every ParaTTT model is initialized from the Gated DeltaNet initialization of the same size (the slow weights of the memory are new parameters, initialized with the standard scheme and zero fast weights), so that at initialization the two models differ only in the memory.
Table 2. Models of Table 1 . Parameter counts include embeddings. The ParaTTT memory has a hidden layer of width d h e a d d_{\mathrm{head}} d head with SiLU, a bias, the unit-ball projection on the features, and a learned per-head gain on the fast pre-activations; its learning rates are η t = sigmoid ( w η T x t + b η ) \eta_t = \operatorname{sigmoid}(\mathbf{w}_\eta^{\mathsf{T}}\mathbf{x}_t + b_\eta) η t = sigmoid ( w η T x t + b η ) per head and layer, and its decay is the Gated DeltaNet gate shared by both layers.
Evaluation protocol
Perplexity and zero-shot. WikiText-103 word perplexity and LAMBADA (OpenAI) perplexity, and accuracy on PIQA, HellaSwag (normalized), ARC-easy, ARC-challenge (normalized), Wino-Grande, BoolQ and SciQ, all zero-shot with lm-evaluation-harness [15 ] . ParaTTT models use the chunkwise prefill at their training N N N unless stated otherwise. Both new 1.3B training variants use N = 2 N = 2 N = 2 for the main task evaluations, including recall, NIAH and LongBench; Table 4 separately varies the evaluation-time iteration count.
Recall. SWDE, FDA, TriviaQA (the cloze version of Arora et al. [1 ] ) and DROP are generation tasks: the model greedily decodes an answer after the prompt and the score is whether the gold answer appears in the generation, case-insensitively, on the full evaluation sets. ParaTTT prefills the prompt with the chunkwise iteration at the evaluation-time N N N specified above and decodes token by token with the exact recurrence; TTT-MLP decodes with a block-faithful cache at batch 1; the other models use their standard cached decoding.
RULER. The eight needle-in-a-haystack tasks of RULER [24 ] (single 1–3, multi-key 1–3, multiquery, multivalue) at 4K, 8K, 16K and 32K tokens with 100 samples per task; the table reports the average. Attention models use YaRN [44 ] with a scale of the evaluation length over 4096.
Synthetic tasks. Two-layer models of width 256 (head dimension 128) with the same backbone as the language models, trained from scratch for 4k steps with AdamW, batch 128, over learning rates { 3 × 10 − 4 , 10 − 3 , 3 × 10 − 3 } \{3 \times10^{-4}, 10^{-3}, 3 \times10^{-3}\} { 3 × 1 0 − 4 , 1 0 − 3 , 3 × 1 0 − 3 } and three seeds; the long-horizon settings (parity at 512 and 1024, MQAR at 512 with 32 pairs) are also trained for 8k steps at { 10 − 4 , 3 × 10 − 4 } \{10^{-4}, 3 \times10^{-4}\} { 1 0 − 4 , 3 × 1 0 − 4 } , and MQAR with 64 pairs at width 512. Figure 6 reports, per model and setting, the learning rate (and budget) with the best mean final accuracy over the three seeds. ParaTTT models train and evaluate at N = 2 N = 2 N = 2 (or N = 5 N = 5 N = 5 ). The group word problems of Table 3 use the same models and learning rates with batch 64, for 4k steps or, where stated, 16k steps; ParaTTT is trained and evaluated at each listed N N N , and every entry uses the learning rate with the best mean final accuracy over the three seeds.
Figure 6. Synthetic nonlinear-state tasks (two layers, width 256 except width 512 for MQAR with 64 pairs; mean over three seeds at the best learning rate). (a) Cumulative parity against sequence length; the dotted line is chance. (b) k k k -hop pointer following at T = 256 T = 256 T = 256 . (c) Multi-query associative recall for increasing sequence length and number of key-value pairs.
Table 3. Group word problems: accuracy (%) of the running product at every position, mean over three seeds with the range of the seeds below. Two-layer models of width 256; ParaTTT is trained and evaluated at the given N N N .
Per-position perplexity. 698 held-out SlimPajama documents (validation and test splits) of at least 4097 tokens, each contributing its first 4096 tokens, so that every position is the same number of tokens into a document; the mean NLL per position is b
Additional results
Synthetic nonlinear-state tasks
Three synthetic tasks isolate what a nonlinear memory adds, each learned from scratch by two-layer models of width 256 (width 512 for MQAR with 64 pairs) with the backbone of Section D (protocol in Section E; Figure 6 ). Cumulative parity asks, at every position, for the parity of the number of ones seen so far; k k k -hop asks for the token reached by following k k k pointer links back through the sequence; multi-query associative recall (MQAR) asks for the values of previously stored keys.
Parity probes whether a memory can maintain a state that flips with selected inputs. It is not a universal separation between nonlinear and linear memories: negative transition eigenvalues allow linear RNNs to solve it. The relevant limitation concerns finite-precision linear RNNs restricted to nonnegative real eigenvalues at arbitrary sequence lengths [20 ] . Our finite-length experiments test the architectures and training settings used here. In Figure 6 , Gated DeltaNet and Mamba-2 fall from 74–77% at length 128 to 63% at 256 and 52–55% at 1024, approaching chance, and the two-layer attention model stays below 63% at every length, whereas ParaTTT reaches 96–100% at lengths up to 256, 79–90% at 512 and 78–80% at 1024, with the model trained at N = 5 N = 5 N = 5 best at every length. Turning the loop off (N = 0 N = 0 N = 0 ) drops ParaTTT to chance, indicating that the iteration is important in this setup. TTT-MLP sits between the two groups (87% at length 128, 58% at 1024). Its block form evaluates gradients at the weights entering each 16-token block, although each token’s readout incorporates preceding within-block updates [50 ] . The frozen gradient evaluations differ from the online recurrence that ParaTTT targets; these results alone do not establish that this difference causes the parity gap.
The other two tasks are solvable by a linear memory and serve as controls. On k k k -hop (Figure 6 b) the recurrent models with an erasing update, ParaTTT and Gated DeltaNet, solve every k k k , Mamba-2 breaks at k = 4 k = 4 k = 4 , and the two-layer attention model cannot compose hops at this depth. MQAR (Figure 6 c) is a capacity test: Gated DeltaNet and ParaTTT solve it up to 32 pairs, with ParaTTT more seed-sensitive at length 1024 and the N = 5 N = 5 N = 5 model the more reliable of the two, and only attention solves 64 pairs, which exceeds the empirical capacity of the recurrent models in this setting. ParaTTT thus keeps the linear recurrences’ recall while achieving stronger state-tracking performance in these experiments.
State tracking in groups. To test the nonlinear memory on tasks that require successive state updates, we add group word problems ([34 , 38 ] ): the tokens are uniformly random elements of a finite group, and the target at every position is the running product of the elements seen so far. For Z 2 \mathbb{Z}_2 Z 2 this is the cumulative parity task above; S 3 S_3 S 3 is the smallest non-abelian group. We train ParaTTT at N ∈ { 1 , 2 , 5 , 8 , 10 } N \in\{1, 2, 5, 8, 10\} N ∈ { 1 , 2 , 5 , 8 , 10 } and evaluate each model at its training N N N , with Gated DeltaNet and attention as baselines, all with the two-layer models of width 256 used above (protocol in Section E). Table 3 reports the mean and the range over three seeds.
On parity, ParaTTT with two iterations is already well above the linear baseline, and models trained with more iterations solve the task. The shorter S 3 S_3 S 3 task gives a similar separation: every ParaTTT seed at N = 2 N = 2 N = 2 outperforms every Gated DeltaNet seed in the standard training budget. Giving Gated DeltaNet four times as many steps does not close this gap, while the longer-trained ParaTTT models with more iterations approach perfect accuracy. More iterations often help, although the shorter runs vary substantially across seeds and their mean accuracy is not monotonic in N N N . The longer S 3 S_3 S 3 task remains unsolved by all models in the tested budget.
Evaluation at other iteration counts. Table 4 gives the full sweeps behind Figure 3 (c,Figure 3 d), including the 1.3B model, on WikiText-103, LAMBADA and SWDE. The last two blocks are the two 1.3B variants of § 4.1 , which change only the number of iterations during training.
LongBench. Table 5 reports LongBench at four context lengths for every model in Table 1 .
Comparisons with sliding-window attention
The published TTT architectures that scale beyond the original TTT-Linear and TTT-MLP do not evaluate their memory alone. LaCT [65 ] pairs its large-chunk fast-weight update with a sliding-window attention branch in every layer and reports only the combination; Titans (Behrouz et al., 2025b) reports its long-context results for the hybrids that add attention to the memory (MAC, MAG and MAL). A window of 2048 tokens covers half of a 4096-token training context, so the attention branch alone already accounts for local modeling, and the fast-weight memory only has to contribute beyond the window.
The LaCT configuration without SWA in Table 1 performs poorly on perplexity and recall. Its large-chunk update provides one reason to expect local attention to matter: the gradients for a chunk are evaluated at its incoming weights, so the update does not follow the token-by-token recurrence within that chunk. However, this row alone does not isolate the contribution of SWA; its configuration is documented separately in Section D. We therefore compare all three recurrences with the same sliding-window branch below. The ParaTTT models in Table 1 remain pure recurrences, with an update at every token and no attention branch.
Evaluating all recurrences with SWA. We give ParaTTT and Gated DeltaNet the same kind of SWA branch as LaCT: a 2048-token sliding-window attention whose output is summed with the recurrent output. The runs share the training recipe, data and seed of Table 1 , but are not matched in training compute. We also train each model with per-head RMS normalization of the attention queries, keys and values (“+ norm”). Table 6 reports the results.
With a normalized window, ParaTTT is ahead of both LaCT variants on perplexity, most recall tasks, LongBench and NIAH; DROP is the exception. The benefit over LaCT therefore remains when both models have access to local attention.
The comparison also shows that Gated DeltaNet and ParaTTT benefit more from SWA on SWDE and FDA when the branch is normalized. Their short convolutions are initialized with small weights (standard deviation 0.02), which can leave the attention queries and keys small and the attention close to uniform. This suggests that calibration of the attention branch contributes to the different gains from SWA. The published LaCT hybrid has no short convolution; adding one in our experiment made training unstable. These observations do not isolate an inherent synergy between LaCT and SWA.
Normalization also reduces long-range NIAH accuracy for Gated DeltaNet and ParaTTT. One possible explanation is that the models learn to rely more on the local window, which cannot retrieve a needle beyond its receptive field. We leave this as a hypothesis.
Centering ablation
Table 7 isolates the readout parameterization. All three models share the recipe, initialization and data order of Section D and differ only in how the readout layer W ( L ) W^{(L)} W ( L ) is parameterized:
Zero-initialized readout. W 0 ( L ) = 0 W^{(L)}_0 = 0 W 0 ( L ) = 0 and f W ( k t ) = Δ W ( L ) ϕ t f_W(\mathbf{k}_t) = \Delta W^{(L)}\phi_t f W ( k t ) = Δ W ( L ) ϕ t , the standard delta-rule state that starts at zero and decays toward zero (the “zero-init readout” row of Table 1 ). By Eq. (5) the hidden layers receive no gradient until the fast readout has grown.
Learned initial readout, uncentered. A slow W 0 ( L ) W^{(L)}_0 W 0 ( L ) is learned and f W ( k t ) = ( Δ W ( L ) + W 0 ( L ) ) ϕ t f_W(\mathbf{k}_t) = (\Delta W^{(L)} + W^{(L)}_0)\phi_t f W ( k t ) = ( Δ W ( L ) + W 0 ( L ) ) ϕ t : the full weights decay toward W 0 ( L ) W^{(L)}_0 W 0 ( L ) instead of zero, so the hidden layers receive gradient from the first token, but the output carries the bias W 0 ( L ) ϕ t W^{(L)}_0\phi_t W 0 ( L ) ϕ t before any test-time learning has taken place. This model has exactly the parameters of the centered one.
Centered readout. Eq. (24) , the ParaTTT default.
Read against the Gated DeltaNet reference, the three rows form a ladder. The zero-initialized loop already improves recall over the linear recurrence (SWDE +4, NIAH at 4K +3) but costs 3 points of LAMBADA perplexity. The learned initial readout repairs LAMBADA to the Gated DeltaNet level and leaves recall essentially unchanged, showing that giving the hidden layers a gradient at the start of a sequence matters. Centering buys the rest with the same parameters: 0.7 WikiText-103 and 2.1 LAMBADA perplexity, 4 points of SWDE and 2.5 of TriviaQA, and the largest NIAH gain at 32K. The commonsense tasks move within noise across the ladder, and each row is a single seed. The gain of centering over the uncentered learned readout is therefore not explained by the extra slow readout: it comes from removing the bias W 0 ( L ) ϕ t W^{(L)}_0\phi_t W 0 ( L ) ϕ t so that the readout at the start of a sequence is zero while its Jacobian is not.
Centering for TTT-MLP. We also trained TTT-MLP with the centered readout of Eq. (24) . It does not require centering: the results moved by about two points in either direction (SWDE 33.1 → \rightarrow → 35.7, FDA 16.2 → \rightarrow → 18.2, perplexity unchanged within noise), so we report the original recipe.
Prefill–decode mismatch on long sequences
Prompts are processed by the chunkwise prefill with N N N refinement iterations, whereas generated tokens are processed by the exact recurrence, which needs no iteration (§ 3.2 ). The two agree only in the limit of the iteration, so a model trained at a finite N N N decodes with a slightly different memory than the one it was trained with. Figure 5 (b,Figure 5 c) compares the two on 512-token sequences. Here we measure whole documents and the effect on decoding, using the held-out SlimPajama documents of Figure 3 (a,Figure 3 b) truncated to 4K, 16K or 32K tokens (16 documents at 4K, 8 at 16K and at 32K). Besides the two ParaTTT models of Table 1 , we include two variants of the 1.3B model that differ only in the iteration count during training: one switches to N = 5 N = 5 N = 5 for the last 11B of its 100B training tokens, and one samples N N N per layer from a Poisson distribution with parameter 2, resampling zero and capping the result at 5, throughout training. All models are trained at 4K tokens.
Decoding after a prefill. We prefill the first part of each document at N = 2 N = 2 N = 2 and process the remaining tokens one at a time with the exact recurrence. Both evaluations use the reference tokens, so the sequence stays fixed while the recurrent computation changes. This teacher-forced evaluation uses the decoding update, but does not measure free-running generation quality. We compare perplexity on the continuation with its perplexity when the whole document is prefilled at N = 2 N = 2 N = 2 (Table 8 ).
Table 8. Prefill–decode mismatch: increase in perplexity (%) relative to the N = 2 N = 2 N = 2 prefill. The whole-document column processes and scores all 4K tokens with the exact recurrence. The remaining columns prefill a prompt at N = 2 N = 2 N = 2 , then process and score only the continuation with the exact recurrence. Both paths use the reference tokens.
The mismatch has a measurable cost for the models trained at a fixed N = 2 N = 2 N = 2 , and the penalty grows with continuation length for the submitted 1.3B model. Both training changes substantially reduce it. With randomized N N N , the increase stays below 2% even when the continuation is four times the training length, and remains small throughout that span.
To test exact prompt processing as well, we run the exact recurrence from the first token of each 4K document. The whole-document column in Table 8 shows the same pattern: the submitted checkpoints incur a perplexity penalty, which the new training recipes reduce.
Agreement over whole documents. Table 9 gives the top-1 agreement between the prefill of a whole document and the exact recurrence run from its first token. Agreement is lower than on 512-token sequences and decreases with the document length. Beyond the training length the prefill and the exact recurrence drift apart gradually, and additional iterations close only part of the gap. For every tested model and length, the N = 2 N = 2 N = 2 prefill has a lower perplexity than the exact recurrence. As a reference for numerical differences between implementations, the chunkwise and token-by-token evaluations of Gated DeltaNet agree on 99.0–99.3% of tokens in bf16, despite computing the same function. This reference does not establish how much of ParaTTT’s gap comes from numerical precision. The exact recurrence is the fixed point of the iteration, but a model trained at a fixed small N N N is not trained to operate at that fixed point over long sequences; randomizing N N N during training reduces the effect on decoding (Table 8 ) but does not remove the drift at 32K (Table 9 ).
Table 9. Top-1 agreement (%) between the prefill at N N N iterations and the exact recurrence over whole documents of 4K, 16K and 32K tokens, for the 1.3B models.
Learned decay and learning rates
The conditions of Theorems 1 and 2 are stated in terms of ϵ = η ˉ / ( 1 − α ˉ ) \epsilon= \bar{\eta}/(1 - \bar{\alpha}) ϵ = η ˉ / ( 1 − α ˉ ) . Table 10 reports the decay α t \alpha_t α t and the learning rates η t ( l ) \eta_t^{(l)} η t ( l ) that the trained models produce on held-out SlimPajama text. In most layers the decay is close to one, so the ratio η t / ( 1 − α t ) \eta_t/(1 - \alpha_t) η t / ( 1 − α t ) has a median of about 4 for the readout and 7 to 9 for the hidden layer, with layer medians up to 100. The conditions require ϵ \epsilon ϵ well below one, so the trained models are far outside them. Their agreement with the exact recurrence is measured in § 4.3 and F.4.
Table 10. Learned decay and learning rates of the trained ParaTTT models on held-out text. Each layer is summarized by its median over tokens and heads; the table gives the median and the range of these 24 layer medians.
Kernel implementation and efficiency details
The ParaTTT layer is built from the chunkwise Triton kernels of FLA with a chunk size of 64: the gated-linear-attention and gated-delta-rule forward and backward kernels, the transposed reads of Eq. (6) (the same kernel with the roles of keys and values exchanged) and DELTARULEGRAD, which is the gated-delta-rule kernel with the final read transposed; the pointwise epilogues (nonlinearity, Jacobian, projection, damping) are fused into two Triton kernels described in Section G.2. Per iteration and hidden layer the layer stores the pseudo-labels, the features and the Jacobian, each T × d h e a d T \times d_{\mathrm{head}} T × d head per head; the readout states at chunk boundaries are recomputed in the backward pass. The layer and full-model measurements are described separately below.
Layer benchmarks. For Figure 4 (a,Figure 4 b), we time the forward and backward passes of one token-mixing layer on one MI325X, in bf16 at batch size 8, over 20 iterations after 3 warm-up iterations. TTT-MLP uses its fused Triton kernel at head dimension 64 and block size 16. The larger-head points in (b) use a torch.compiled version of its PyTorch scan; at the supported configuration, this scan is 2.2× slower than the fused kernel in the same forward-and-backward setting. The batch-1 matched-error experiment in Figure 5 (a) instead uses whichever implementation is faster at each block size (§ 15 ).
Full-model training. For Figure 4 (c), all 340M models use the recipe and data loader of Section D: 4096-token sequences, eight sequences per GPU, two accumulation steps and eight MI325X GPUs, in bf16. This gives 0.5M tokens per optimizer step. We average throughput over 50 steps after warm-up and report peak memory alongside it in Table 11 . ParaTTT stores activations across all refinement iterations, so its peak memory grows with N N N . TTT-MLP uses its supported fused configuration (16 heads of dimension 64, block size 16) and checkpoints segments of its sequential scan. These different checkpointing strategies are part of the reported time–memory trade-off.
Table 11. Training throughput and peak memory of the full 340M models on eight MI325X GPUs at T = 4096 T=4096 T = 4096 .
Training cost of the two 1.3B variants. We compare the training logs of the three 1.3B runs, which use the same hardware and batch size. Switching to N = 5 N=5 N = 5 for the final 11B of 100B tokens increases total training time by about 7%; randomizing N N N increases it by about 5%. The former uses more expensive steps only near the end of training, while the latter averages about 2.3 iterations after zero draws are resampled and draws above 5 are capped. Activation storage also grows with N N N : peak memory is 164 GiB per GPU at N = 2 N = 2 N = 2 , 205 GiB with randomized N N N and 247 GiB at N = 5 N = 5 N = 5 .
Decode latency
Figure 7 shows the time to decode one token as a function of the context length, for the trained models of Table 1 . Each model processes a prompt of 1K to 128K tokens and then generates one token at a time, at batch size 1 in bf16 on one MI325X. We drop the first 32 generated tokens and take the median time of the next 224 (2048 for LaCT, which covers one of its chunks). All models of a size run back to back on the same GPU, five times; we report the median of the five passes and their range.
Figure 7. Decode time per token against context length at batch size 1 on one MI325X, for the trained models of Table 1 . Lines: median over five passes; bands: their range. TTT-MLP and LaCT are trained at 340M only.
The recurrent models decode in constant time per token at every context length. ParaTTT decodes with the exact recurrence, without iterations, and the single-token step of a layer is implemented as one fused kernel. Its decode time is close to that of Gated DeltaNet and Mamba-2, about 10% above Gated DeltaNet, and less than half that of TTT-MLP and LaCT, the two TTT baselines. The two model sizes are close because, at batch size 1, the time is dominated by the fixed cost of launching the kernels of each layer. Attention with a key–value cache is the fastest model at short contexts, but its time grows linearly with the context, and beyond 8K tokens it is slower than ParaTTT. The decoding state of ParaTTT is the fast weights of the hidden layer and of the readout, two 128 × 128 128 \times128 128 × 128 matrices per head against one for Gated DeltaNet, and does not grow with the context.
The fused epilogue kernels
With one hidden layer (L = 2 L = 2 L = 2 ) the only operations of an iteration that are not scans are pointwise in the token: the nonlinearity and projection that turn the hidden pre-activation into the features (Step 1), and their transposed Jacobians, which are all that Step 3 needs, because the transposed fast weight of Eq. (6) is only required to reach a layer below the first. Both are implemented as one Triton kernel each, with one program per (sequence, token, head) that holds the d h e a d d_{\mathrm{head}} d head -vector of its row in fp32 registers; every reduction is over this vector, so the kernels are memory-bound and read and write each activation once. We write s ( x ) = 1 / ( 1 + e − x ) s(x) = 1/(1 + e^{-x}) s ( x ) = 1/ ( 1 + e − x ) for the sigmoid, σ ( x ) = x s ( x ) \sigma(x) = x\,s(x) σ ( x ) = x s ( x ) for SiLU and n ( x ) = x min ( 1 , 1 / ∥ x ∥ ) n(x) = x\min(1,1/\lVert x\rVert) n ( x ) = x min ( 1 , 1/ ∥ x ∥) for the unit-ball projection.
Forward epilogue. The hidden pre-activation of token t t t is the sum of a slow part and a fast part,
z t = W 0 ( 1 ) k t + b ( 1 ) ⏟ one matmul per forward + κ α t Δ W t − 1 ( 1 ) k t ⏟ shifted scan, xref‘Eq. (4)‘ , h t = σ ( z t ) , ϕ t = n ( h t ) , (26) z_t = \underbrace{\mathbf{W}^{(1)}_0 \mathbf{k}_t + \mathbf{b}^{(1)}}_{\text{one matmul per forward}} + \underbrace{\kappa\,\alpha_t\Delta\mathbf{W}^{(1)}_{t-1}\mathbf{k}_t}_{\text{shifted scan, {xref}`Eq. (4)`}}, \qquad h_t = \sigma(z_t), \qquad\phi_t = n(h_t), \tag*{(26)} z t = one matmul per forward W 0 ( 1 ) k t + b ( 1 ) + shifted scan, xref ‘Eq. (4)‘ κ α t Δ W t − 1 ( 1 ) k t , h t = σ ( z t ) , ϕ t = n ( h t ) , (26)
where κ \kappa κ is the learned per-head gain of § 3.4 . The slow part does not depend on the pseudo-labels, so it is computed once per layer forward and shared by all iterations; the scan output is read in bf16 straight from the layout the scan writes, without the copy that separating the two arms of a SwiGLU would need (the value arm is the constant 1 in our models). The kernel forms z t z_t z t , h t h_t h t and ϕ t \phi_t ϕ t in fp32, stores ϕ t \phi_t ϕ t in bf16 for the readout kernels, and saves z t z_t z t and h t h_t h t for the backward epilogue; ∥ h t ∥ \lVert h_t\rVert ∥ h t ∥ is recomputed there from h t h_t h t .
Backward epilogue. DeltaRuleGrad returns e t = η t ( L ) ∂ ℓ t / ∂ ϕ t e_t = \eta_t^{(L)} \, \partial\ell_t/\partial\phi_t e t = η t ( L ) ∂ ℓ t / ∂ ϕ t Eq. (5) . The backward epilogue turns it into the value the hidden scan consumes in the next iteration, in five register-resident steps:
g t = e t / η t ( L ) , g t ′ = { ( g t − h ^ t h ^ t ⊤ g t ) / ∥ h t ∥ ∥ h t ∥ ≥ 1 g t ∥ h t ∥ < 1 , δ z ^ t = σ ′ ( z t ) ⊙ g t ′ (27) g_t = e_t/\eta_t^{(L)}, \qquad
g'_t =
\begin{cases}
\left(g_t-\hat{h}_t\hat{h}_t^\top g_t\right)/\lVert h_t\rVert& \lVert h_t\rVert\ge1 \\
g_t & \lVert h_t\rVert< 1
\end{cases},
\qquad
\delta\hat{z}_t = \sigma'(z_t)\odot g'_t \tag*{(27)} g t = e t / η t ( L ) , g t ′ = { ( g t − h ^ t h ^ t ⊤ g t ) / ∥ h t ∥ g t ∥ h t ∥ ≥ 1 ∥ h t ∥ < 1 , δ z ^ t = σ ′ ( z t ) ⊙ g t ′ (27)
with h ^ t = h t / ∥ h t ∥ \hat{h}_t = h_t/\lVert h_t\rVert h ^ t = h t / ∥ h t ∥ and σ ′ ( x ) = s ( x ) ( 1 + x ( 1 − s ( x ) ) ) \sigma'(x) = s(x)(1+x(1-s(x))) σ ′ ( x ) = s ( x ) ( 1 + x ( 1 − s ( x ))) . The second step is the transposed Jacobian of the projection, ( I − h ^ h ^ ⊤ ) / ∥ h ∥ (I-\hat{h}\hat{h}^{\top})/\lVert h\rVert ( I − h ^ h ^ ⊤ ) / ∥ h ∥ outside the unit ball and the identity inside it, and the third that of SiLU; together they are the transposed Jacobian H t ⊤ H_t^\top H t ⊤ of § 3.3 , with the hard projection in place of the soft clip. The value the scan receives, v ^ t = − η t ( 1 ) δ z ^ t \hat{v}_t=-\eta_t^{(1)}\delta\hat{z}_t v ^ t = − η t ( 1 ) δ z ^ t , attaches the hidden learning rate and the sign with which the pseudo-label enters as a value. The damped Picard step Eq. (7) is fused into the same store, v t [ n ] = v t [ n − 1 ] + λ ( v ^ t − v t [ n − 1 ] ) v_t^{[n]}=v_t^{[n-1]}+\lambda(\hat{v}_t-v_t^{[n-1]}) v t [ n ] = v t [ n − 1 ] + λ ( v ^ t − v t [ n − 1 ] ) , with v t [ 0 ] = 0 v_t^{[0]}=0 v t [ 0 ] = 0 handled without a read, and the result is written directly in the ( B , T , H , d h e a d ) (B,T,H,d_{\mathrm{head}}) ( B , T , H , d head ) layout the scan reads, so an iteration is four kernel launches per layer: the hidden scan, the forward epilogue, DeltaRuleGrad and the backward epilogue. The kernels also implement a pre-activation RMSNorm, a per-token norm clip of δ z ^ t \delta\hat{z}_t δ z ^ t (hard or soft), a skip connection and a second SwiGLU arm, none of which the reported models use.
Training through the iteration. End-to-end training backpropagates through all N N N iterations, so each epilogue is an autograd function whose backward is itself a fused kernel: the vector-Jacobian product of Eq. (26) (through σ ′ \sigma' σ ′ and the projection Jacobian) and that of Eq. (27) , which differentiates the SiLU Jacobian a second time (σ ′ ′ \sigma'' σ ′′ ) and the learning rates. For decoding, the hidden scan, the forward epilogue and the DeltaRuleGrad step of one token are fused into a single kernel per (sequence, head) with the states resident in registers, since at that length the launch latency of the chunkwise kernels dominates.
Cost of the hidden-layer update and of centering
Table 12 measures what the two design choices of § 3.4 cost, in the two settings of Figure 4 : one token-mixing layer in isolation and the full 340M model in training.
Table 12. Cost of the hidden-layer update and of the readout parameterization at 340M, N = 2 N = 2 N = 2 . Left: forward+backward time of one token-mixing layer (bf16, batch 8, d = 1024 d = 1024 d = 1024 , 8 heads of dimension 128, one MI325X), the setting of Figure 4 (a). Right: training throughput and peak memory of the full model on one node of eight MI325X GPUs with the recipe of Section D. The readout rows use the optimized kernels of Section G; the two bodies are timed in the reference implementation, which supports both updates, so the body rows are comparable with each other but slower than the readout rows.
Readout parameterization. The three readouts of Table 7 run on the same kernels and differ only in the readout offset and in the extra terms of Eq. (24) . Relative to the zero-initialized readout, the uncentered learned readout adds the offset W 0 ( L ) W_0^{(L)} W 0 ( L ) to the final read of DeltaRuleGrad and costs 12% of the layer time at 4K and 9% of training throughput. Centering adds to this one more pass of the slow MLP over the queries (the features ϕ 0 , q , t \phi_{0,q,t} ϕ 0 , q , t of Section C.5) and the corrected values, which is a further 1–2% of layer time and throughput and 9 GiB of activation memory. Centering therefore costs about 11% of training throughput in total, against the gains of Table 7 .
Hidden-layer update. The delta-rule body replaces the gated-linear-attention scan of the hidden layer with a gated-delta-rule scan of the same shape, plus the transposed reads of Eq. (5) that the delta rule’s gradient needs (Section C.2); the readout side of the layer is unchanged. In the reference implementation, which supports both updates, the delta-rule body costs 17% of the layer time at every length (35.6 against 30.5 ms at 4K) and 1.7 GiB more activation memory per layer for the extra chunk states; in training this is 13% of throughput (32.5k against 37.4k tokens/s) and 30 GiB of peak memory per GPU. Together with Table 1 (SWDE 28.4 → 41.9, NIAH at 4K 7.2 → 32.6), this is the more cost-effective of the two design choices.
Convergence versus compute at other sequence lengths
Figure 8 shows the comparison of Figure 5 (a) at every sequence length from 256 to 4096 with the same inner MLP, step size and decay. The picture is the same at every length: TTT-MLP’s error is set by its block size and its time by the number of sequential blocks T / b T / b T / b , whereas ParaTTT’s error is set by the iteration count and its time, being a fixed number of chunkwise-parallel scans, is nearly independent of T T T at batch 1, so the gap widens with the sequence length.
Table 13 lists the points of Figure 5 (a) at T = 4096 T = 4096 T = 4096 . At its default block size of 16, TTT-MLP has 2.4 × 2.4\times 2.4 × the error of ParaTTT at N = 2 N = 2 N = 2 and takes 9.3 × 9.3\times 9.3 × as long. The 32 × 32\times 32 × of § 4.3 compares the two at equal error (1.3%), which TTT-MLP reaches at b ≈ 5 b \approx5 b ≈ 5 by interpolation between b = 4 b = 4 b = 4 and b = 8 b = 8 b = 8 . TTT-MLP is timed with the faster of its fused kernel and a compiled version of its scan; the fused kernel pads blocks to 16 tokens, so the compiled scan is the faster one at every block size in the table. A bf16 variant of the fused kernel that we wrote for b = 16 b = 16 b = 16 takes 19.5 ms; it is not part of the baseline implementation and is not used in the comparison.
Table 13. Error against exact online gradient descent and time to compute the recurrence at T = 4096 T = 4096 T = 4096 , for the setting of Figure 5 (a): one random inner MLP, d = 64 d = 64 d = 64 , eight heads, step size 0.2 and decay 0.98.
Figure 8. Relative error against exact online GD versus the time to compute the forward recurrence of one 340M layer (batch 1), at five sequence lengths; same setting and legend as Figure 5 (a).