betaFeedback

WorksComputer Science

ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories

Abstract

Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration. Training and prefill approximate this recurrence with a finite number of iterations, while decoding applies the exact token-by-token update without refinement. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory 32×32\times faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains 4.9×4.9\times faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.

!GitHub

Introduction

Softmax attention scales poorly to long contexts: its compute grows quadratically with sequence length, while its key–value cache grows linearly. Recurrent language models address both problems by processing the sequence in linear time and carrying a fixed-size state while decoding. Moreover, modern recurrent architectures such as gated linear attention, Gated DeltaNet, and Mamba-2 [62, 64, 11] admit an efficiently parallelizable structure commonly referred to as the chunkwise-parallel form, in which tokens within each chunk are processed in parallel using dense matrix multiplications, while only the recurrent state is propagated sequentially across chunks. Given this scalable structure, the central question for this class of models is how to make the recurrent memory more expressive without sacrificing parallel evaluation.

Test-time training (TTT) addresses the expressivity side of this question by treating the recurrent state as the weights of a small model that learns from the sequence through online gradient descent [50]. With a linear memory and a squared reconstruction loss, this recovers the delta rule of [59], the update used by DeltaNet [47, 63], whereas a nonlinear memory can adapt its features as well as its readout. TTT-MLP, Titans, and

LaCT (Sun et al., [50]; Behrouz et al., 2025b; Zhang et al., [65]) explore such neural memories, but their gradients depend nonlinearly on the current weights: each token changes the weights at which the next token’s gradient should be evaluated, creating a sequential dependency. To recover parallelism, existing methods group gradient evaluations into blocks and evaluate every gradient in a block at the weights entering it. This creates a fidelity–speed tradeoff: larger blocks reduce the sequential depth but more coarsely approximate token-by-token learning, while smaller blocks are more faithful but require more sequential steps.

In this paper, we introduce ParaTTT (PTTT), an algorithm that evaluates an MLP-based TTT memory in parallel (Figure 1). The difficulty in parallelizing the recurrence is the inner weight trajectory: the gradient step at each token is taken at weights that depend on every earlier step, so the trajectory cannot be computed without stepping through it sequentially. PTTT instead refines an estimate of this trajectory, represented by the gradients of the inner loss with respect to the hidden-layer pre-activations. Each refinement consists of three parallel passes: a forward pass that constructs the features implied by the current gradient estimate, a readout pass that computes each token’s loss gradient with respect to its features, and a backward pass that propagates these gradients through the hidden layers to form a new estimate. Crucially, each refinement step is a composition of modified instances of existing parallel recurrences such as Gated DeltaNet (Yang et al., [63]; Yang et al., [64]), Mamba-2 (Dao & Gu, [11]), or gated linear attention (Yang et al., [62]). The exact nonlinear sequential recurrence is a fixed point of the iteration. Training and prefill use a finite number of these iterations, while decoding applies the exact token-by-token update.

Parallelism and approximation in nonlinear TTT

Figure 1. Parallelism and approximation in nonlinear TTT. Each cell represents a hidden pre-activation gradient δzt(W)=∂ℓt∂zt∣W\delta z_t(W) = \left.\frac{\partial\ell_t}{\partial z_t}\right|_W, with the layer index omitted. Columns index tokens; downward steps mark computational dependencies, not approximation error or elapsed time. (a) Exact gradients δzt∗\delta z_t^* use the latest online weights Wˉt−1\bar{W}_{t-1}, including decay. (b) Block-wise TTT, as used by TTT-MLP (Sun et al., [50]) and Titans (Behrouz et al., 2025b), evaluates gradients at shared block-entry weights WsblkW_s^{\mathrm{blk}} for s<t≤s+bs < t \le s + b, omitting earlier within-block updates. (c) ParaTTT refines the full gradient trajectory using three chunkwise-parallel passes per round. The block size bb and refinement count NN control the respective approximations; the exact trajectory is a fixed point of the ParaTTT iteration.

Our technical contributions can be summarized as follows:

  • We introduce ParaTTT (PTTT), an algorithm for parallelizing MLP-based recurrent memories of arbitrary depth, and prove geometric convergence for a two-layer variant under sufficient conditions (§ 3).

  • We demonstrate PTTT’s superior speed–accuracy trade-off over the block-based evaluation of TTT-MLP: at 4K context, two PTTT refinements match the output error of TTT-MLP’s block approximation while running 32× faster, and full 340M models train 4.9× faster than TTT-MLP (§ 4.2 and § 4.3).

  • At 340M and 1.3B parameters, PTTT attains lower perplexity and stronger recall on SWDE and FDA than Gated DeltaNet and Mamba-2, showing that the proposed nonlinear memory improves over the evaluated linear recurrent baselines § 4.1.

Background

We write x1:Tx_{1:T} for the input sequence of a layer and qtq_t, ktk_t, vtv_t for the per-token query, key, and value projections; the fast-weight state is a matrix StS_t or, for a deep memory, a set of weights WtW_t; CC is the chunk size of the parallel kernels. Heads are independent, and we suppress the head index. Section A extends this section.

Linear recurrences

Linear recurrences retain the associative-memory view of attention while replacing its growing key–value cache with a fixed-size state. Linear attention ([29, 47]) does this by dropping the softmax: it maintains a matrix state updated by an outer product, St=St−1+vtktTS_t = S_{t-1} + v_t k_t^{\mathsf{T}}, and reads it with the query, ot=Stqto_t = S_t q_t. Because this state only accumulates, practical variants introduce forgetting. Gated linear attention ([62]) decays the state before the write, St=αtSt−1+vtktTS_t = \alpha_t S_{t-1} + v_t k_t^{\mathsf{T}}, using a diagonal gate, while Mamba-2 ([11]) is the scalar-gated case. DeltaNet ([47, 63]) updates the state by replacing the additive write with one step of gradient descent on 12∥Skt−vt∥2\frac{1}{2}\lVert S k_t - v_t\rVert^2, St=St−1(I−βtktktT)+βtvtktTS_t = S_{t-1}(I-\beta_t k_t k_t^{\mathsf{T}})+\beta_t v_t k_t^{\mathsf{T}}, which erases the value stored along ktk_t before writing the new one. Gated DeltaNet ([64]) combines this delta-rule update with decay. All of these updates are linear in the recurrent state. Consequently, the state at the end of a chunk of CC tokens is an affine function of the state entering the chunk. This allows tokens within each chunk to be evaluated in parallel with dense matrix multiplications, while only the chunk-boundary states are propagated sequentially with a short scan. This exact evaluation is commonly referred to as the chunkwise-parallel form.

Test-time training

Test-time training ([50]) makes the memory more expressive than a single matrix by interpreting the recurrent state as the parameters WtW_t of a small model fWf_W trained online. At token tt, the model takes a gradient step on the inner loss ℓt(W)=12∥fW(kt)−vt∥2\ell_t(W) = \frac{1}{2}\lVert f_W(k_t) - v_t\rVert^2 and produces the layer output fWt(qt)f_{W_t}(q_t). The projections, gates, and learning rates that define this inner problem are trained end-to-end by backpropagating through the inner updates.

TTT-linear. The simplest choice of inner model recovers the delta-rule recurrence described above. With fW(k)=Wkf_W(k) = Wk, one gradient step on the squared reconstruction loss is exactly the DeltaNet update, while adding a decay term gives Gated DeltaNet. Because this update remains linear in the recurrent state, it retains the exact chunkwise-parallel form described above. We exploit this identity for the linear readout of a nonlinear memory.

TTT-nonlinear. A nonlinear inner model can adapt the features it reads and writes with, rather than only the readout; this additional expressivity motivates deep memories. TTT-MLP ([50], for example, uses a two-layer MLP as fWf_W. Exact online gradient descent evaluates each token’s gradient at the weights produced by all preceding updates, which for a nonlinear memory prevents the chunkwise-parallel rearrangement of the linear case, so TTT-MLP uses a mini-batch (dual) approximation: for a block of bb tokens starting at position ss, every gradient is evaluated at the weights WsW_s entering the block,

exact: Wj=Wj−1−ηj∇ℓj(Wj−1),block: Wtblk=Ws−∑j=s+1tηj∇ℓj(Ws),s<t≤s+b.\text{exact: } W_j = W_{j-1} - \eta_j \nabla\ell_j(W_{j-1}), \qquad\text{block: } W_t^{\mathrm{blk}} = W_s - \sum_{j=s+1}^{t} \eta_j \nabla\ell_j(W_s), \qquad s < t \le s+b.

The causal partial sum lets tokens within the block be evaluated in parallel, but the gradients are stale: updates from earlier tokens in the block do not change the weights at which later gradients are evaluated. Blocks remain sequential, requiring T/bT/b steps, so larger blocks reduce sequential depth at the cost of a coarser approximation to token-by-token learning. Titans ((Behrouz et al., 2025b)) adds momentum and weight decay under the same block form, while LaCT ([65]) uses blocks of thousands of tokens together with a sliding-window attention branch for local context.

Other nonlinear recurrent approaches

Sequential nonlinear RNNs. Classic recurrent networks [13, 23, 9] carry a vector state through a nonlinear map. Their standard evaluation is sequential over tokens, limiting parallelism across the sequence. This motivates attention [54] and parallel linear recurrences, but the choice of transition also affects expressivity. Under their precision and architectural assumptions, Merrill & Sabharwal [36]; Merrill et al. [38] place transformers and certain SSMs in TC0\mathrm{TC}^{0}. This rules out general permutation composition if TC0≠NC1\mathrm{TC}^{0} \ne\mathrm{NC}^{1}. The xLSTM family [4] includes both a sequential sLSTM cell and a parallelizable mLSTM cell. More recently, M2^{2}RNN [39] uses a nonlinear transition with a matrix-valued state, but pays for it with sequential forward and backward passes.

Parallelizing nonlinear recurrences. DEER [33] and ParaRNN [10] cast a nonlinear recurrence ht=g(ht−1,xt)h_t = g(h_{t-1}, x_t) as a fixed-point problem over the state trajectory and solve it with Newton iterations, where each iteration reduces to a linear recurrence evaluated by a parallel scan; deep equilibrium models [3] share the fixed-point perspective. A generic Newton step uses a transition Jacobian whose size is quadratic in the state dimension; ParaRNN makes this tractable with structured Jacobians. ParaTTT instead iterates on the pre-activation gradient at each token and hidden layer. For hidden width dd, each gradient is a dd-dimensional vector, compared with the O(d2)O(d^2) fast-weight state. Conditioned on these gradients, the fast-weight trajectory becomes a linear recurrence, so each refinement can be evaluated with chunkwise linear recurrent kernels without ever forming the full weight trajectory or its transition Jacobian.

Method

Test-time training with a deep memory

We consider a single head and omit the head index. As in Gated DeltaNet, the input token xtx_t is mapped to a key, a query and a value, kt,qt∈Rdk_t, q_t \in\mathbb{R}^{d} and vt∈Rdvv_t \in\mathbb{R}^{d_v}, by linear projections followed by a short causal convolution; keys and queries are RMS-normalized, ∥kt∥=∥qt∥=d\lVert k_t\rVert= \lVert q_t\rVert= \sqrt{d}. The token also produces a decay gate αt∈(0,1]\alpha_t \in(0, 1] and, for each layer ll of the memory defined below, a learning rate ηt(l)∈(0,1)\eta_t^{(l)} \in(0, 1). The gate follows Yang et al. [64], each ηt(l)\eta_t^{(l)} is the sigmoid of a learned linear function of xtx_t, and all layers of the memory share the decay.

Memory. The recurrent state is the set of weights W=(W(1),…,W(L))\mathbf{W} = (\mathbf{W}^{(1)}, \ldots, \mathbf{W}^{(L)}) of an LL-layer MLP. For given weights W\mathbf{W}, with pre-activations z(l)\mathbf{z}^{(l)} and activations h(l)\mathbf{h}^{(l)}, the memory computes

z(l)=W(l)h(l−1)+b(l),h(l)=σ(z(l))(1≤l<L),h(0)=k,fW(k)=W(L)h(L−1),(1)\mathbf{z}^{(l)} = \mathbf{W}^{(l)}\mathbf{h}^{(l-1)} + \mathbf{b}^{(l)}, \qquad\mathbf{h}^{(l)} = \sigma(\mathbf{z}^{(l)}) \quad(1 \le l < L), \qquad\mathbf{h}^{(0)} = \mathbf{k}, \qquad f_{\mathbf{W}}(\mathbf{k}) = \mathbf{W}^{(L)}\mathbf{h}^{(L-1)}, \tag*{(1)}

where σ\sigma is a pointwise nonlinearity and b(l)\mathbf{b}^{(l)} are learned biases. The final linear layer is the readout, and its input ϕt=ht(L−1)\boldsymbol{\phi}_t = \mathbf{h}_t^{(L-1)} defines the features. The features are additionally projected onto the unit ball, n(x)=x/max⁡(1,∥x∥)\mathfrak{n}(\mathbf{x}) = \mathbf{x}/\max(1,\lVert\mathbf{x}\rVert), which we absorb into the last σ\sigma. This projection is used in all experiments. Our experiments use L=2L = 2, as in Sun et al. [50], but the method applies to any LL. Each weight is decomposed as Wt(l)=W0(l)+ΔWt(l)\mathbf{W}_t^{(l)} = \mathbf{W}_0^{(l)} + \Delta\mathbf{W}_t^{(l)}, where W0(l)\mathbf{W}_0^{(l)} is a parameter trained by backpropagation and ΔWt(l)\Delta\mathbf{W}_t^{(l)} is the recurrent state, initialized to zero at the start of every sequence.

Sequential recurrence. Token tt first decays the state and then takes one gradient step on the reconstruction loss ℓt(W)=12∥fW(kt)−vt∥2\ell_t(\mathbf{W}) = \frac{1}{2}\lVert f_{\mathbf{W}}(\mathbf{k}_t) - \mathbf{v}_t\rVert^2, with the layer’s learning rate as the step size. For each layer ll,

Wˉt−1(l)=W0(l)+αtΔWt−1(l)(decay),Wt(l)=Wˉt−1(l)−ηt(l)∂ℓt∂W(l)∣Wˉt−1(gradient step).(2)\bar{\mathbf{W}}_{t-1}^{(l)} = \mathbf{W}_0^{(l)} + \alpha_t \Delta\mathbf{W}_{t-1}^{(l)} \quad\text{(decay)}, \qquad\mathbf{W}_t^{(l)} = \bar{\mathbf{W}}_{t-1}^{(l)} - \eta_t^{(l)} \left.\frac{\partial\ell_t}{\partial\mathbf{W}^{(l)}}\right|_{\bar{\mathbf{W}}_{t-1}} \quad\text{(gradient step)}. \tag*{(2)}

The layer output is ot=fW(qt)\mathbf{o}_t = f_{\mathbf{W}}(\mathbf{q}_t): the key is evaluated under the decayed weights, before its own gradient step, and the query after it, exactly as in Gated DeltaNet, whose memory is the linear case L=1L = 1 of Eq. (1).

Equation (2) is the exact token-by-token recurrence that we seek to evaluate. Computing it directly requires TT sequential gradient steps because the gradient at each token tt depends on all preceding updates. The block approximation used by TTT-MLP, described in § 2.2, reduces this sequential depth by evaluating multiple gradients at shared weights, but changes the recurrence. Our goal is instead to parallelize the trajectory while retaining the exact online recurrence as the fixed point.

ParaTTT: Iterated refinement of the inner-loop weight trajectory

We first describe the ParaTTT iteration, which uses a linear attention update for the hidden layers and an uncentered readout. The practical design choices used by our models are introduced in § 3.4.

References

  1. [1]Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. In International Conference on Machine Learning, 2024a. URL https://proceedings.mlr.press/v235/arora24a.html.proceedings.mlr.press/v235/arora24a.html
  2. [3]Bai, S., Kolter, J. Z., & Koltun, V. (2019). Deep Equilibrium Models. Advances in Neural Information Processing Systems. https://proceedings.neurips.cc/paper/2019/hash/01386bd6d8e091c2ab4c7c7de644d37b-Abstract.htmlproceedings.neurips.cc/paper/2019/hash/01386bd6d8e091c2ab4c7c7de644d37b-Abstract.html
  3. [4]Beck, M., Poppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024). xLSTM: Extended Long Short-Term Memory. Advances in Neural Information Processing Systems, 37. https://proceedings.neurips.cc/paper_files/paper/2024/hash/c2ce2f2701c10a2b2f2ea0bfa43cfaa3-Abstract-Conference.htmlproceedings.neurips.cc/paper_files/paper/2024/hash/c2ce2f2701c10a2b2f2ea0bfa43cfaa3-Abstract-Conference.html
  4. [7]Cai, Z., Zhu, X., Dong, Y., He, Y., & Arora, S. (2026). T^2MLR: Transformer with Temporal Middle-Layer Recurrence. In arXiv preprint arXiv:2607.15178. https://arxiv.org/abs/2607.15178
  5. [8]Chizat, L., Oyallon, E., & Bach, F. (2019). On Lazy Training in Differentiable Programming. Advances in Neural Information Processing Systems. https://proceedings.neurips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.htmlproceedings.neurips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.html
  6. [9]Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1724–1734. https://doi.org/10.3115/v1/D14-1179
  7. [10]Federico Danieli, Pau Rodríguez, Miguel Sarabia, Xavier Suau, and Luca Zappella. ParaRNN: Unlocking parallel training of nonlinear RNNs for large language models. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=mX8b64iUaa.openreview.net/forum?id=mX8b64iUaa
  8. [11]Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning, 2024. URL https://proceedings.mlr.press/v235/dao24a.html.proceedings.mlr.press/v235/dao24a.html
  9. [12]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1807.03819.
  10. [13]Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179–211, 1990. URL https://doi.org/10.1207/s15516709cog1402_1.
  11. [14]Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, and Tianle Cai. In-place test-time training. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/ba6f5879d7f382241316f7dda7e6b8ac-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2026/hash/ba6f5879d7f382241316f7dda7e6b8ac-Abstract-Conference.html
  12. [15]Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., … Zou, A. (2023). A framework for few-shot language model evaluation. https://zenodo.org/records/10256836zenodo.org/records/10256836
  13. [16]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In Advances in Neural Information Processing Systems, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/3b01972cf31e6fa0fe29e4b8b5c2a0a1-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/3b01972cf31e6fa0fe29e4b8b5c2a0a1-Abstract-Conference.html
  14. [17]Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning, 2023. URL https://proceedings.mlr.press/v202/giannou23a.html.proceedings.mlr.press/v202/giannou23a.html
  15. [18]Xavier Gonzalez, Andrew Warrington, Jimmy T. H. Smith, and Scott W. Linderman. Towards scalable and stable parallelization of nonlinear RNNs. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2407.19115.
  16. [19]Gonzalez, X., Kozachkov, L., Zoltowski, D. M., Clarkson, K. L., & Linderman, S. W. (2025). Predictability enables parallelization of nonlinear state space models. Advances in Neural Information Processing Systems. https://proceedings.neurips.cc/paper_files/paper/2025/hash/1b7a239766f041abcd4286ec5548037-Abstract-Conference.htmlproceedings.neurips.cc/paper_files/paper/2025/hash/1b7a239766f041abcd4286ec5548037-Abstract-Conference.html
  17. [20]Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K. H. Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear RNNs through negative eigenvalues. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/5a0ce3abb720b740419e193c87afd080-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2025/hash/5a0ce3abb720b740419e193c87afd080-Abstract-Conference.html
  18. [21]Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. In Second Conference on Language Modeling (COLM), 2025. URL https://arxiv.org/abs/2412.06769.
  19. [22]Ali Hatamizadeh, Yejin Choi, and Jan Kautz. Gated DeltaNet-2: Decoupling erase and write in linear attention. arXiv preprint arXiv:2605.22791, 2026. URL https://arxiv.org/abs/2605.22791.
  20. [23]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.
  21. [24]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling (COLM), 2024. URL https://openreview.net/forum?id=kIoBbc76Sy.openreview.net/forum?id=kIoBbc76Sy
  22. [25]Jiaxi Hu, Yongqi Pan, Jusen Du, Disen Lan, Xiaqiang Tang, Qingsong Wen, Yuxuan Liang, and Weigao Sun. Improving bilinear RNNs with closed-loop control. In Advances in Neural Information Processing Systems, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/9a439efaa34fe37177eba00737624824-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/9a439efaa34fe37177eba00737624824-Abstract-Conference.html
  23. [26]Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=Hke3gyHYwH.openreview.net/forum?id=Hke3gyHYwH
  24. [27]Yulong Huang, Xiang Liu, Hongxiang Huang, Xiaopeng Lin, Zunchang Liu, Xiaowen Chu, Zeke Xie, and Bojun Cheng. MDN: Parallelizing stepwise momentum for delta linear attention. In International Conference on Machine Learning, 2026. URL https://icml.cc/virtual/2026/poster/63901.icml.cc/virtual/2026/poster/63901
  25. [28]Mahdi Karami, Razvan Pascanu, and Vahab Mirrokni. Lattice: Learning to efficiently compress the memory. arXiv preprint arXiv:2504.05646, 2025. URL https://arxiv.org/abs/2504.05646.
  26. [29]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html.proceedings.mlr.press/v119/katharopoulos20a.html
  27. [30]Team, K. (2025). Kimi Linear: An expressive, efficient attention architecture. In arXiv preprint arXiv:2510.26692. 2510.26692
  28. [32]Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, and Albert Gu. Mamba-3: Improved sequence modeling using state space principles. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/8abd2043b71a074278d5f687947bff9c-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2026/hash/8abd2043b71a074278d5f687947bff9c-Abstract-Conference.html
  29. [33]Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing nonlinear sequential models over the sequence length. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=E34AlVLN0v.openreview.net/forum?id=E34AlVLN0v
  30. [34]Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=De4FYqjFueZ.openreview.net/forum?id=De4FYqjFueZ
  31. [35]Liu, J., Elfein, S., Litany, O., Gojcic, Z., & Li, R. (2026). Test-time training with KV binding is secretly linear attention. International Conference on Machine Learning. https://icml.cc/virtual/2026/poster/64516icml.cc/virtual/2026/poster/64516
  32. [36]William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics, 11:531–545, 2023. URL https://aclanthology.org/2023.tacl-1.31/.aclanthology.org/2023.tacl-1.31
  33. [37]William Merrill and Ashish Sabharwal. A little depth goes a long way: The expressive power of log-depth transformers. In Advances in Neural Information Processing Systems, 2025. URL https://proceedings.neurips.cc/paper/2025/hash/88dd7a6979e352fda7c4952ca8eac59-Abstract-Conference.html.proceedings.neurips.cc/paper/2025/hash/88dd7a6979e352fda7c4952ca8eac59-Abstract-Conference.html
  34. [38]William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In International Conference on Machine Learning, 2024. URL https://proceedings.mlr.press/v235/merrill24a.html.proceedings.mlr.press/v235/merrill24a.html
  35. [39]Mishra, M., Tan, S., Stoica, I., Gonzalez, J. E., & Dao, T. (2026). M²RNN: Nonlinear RNNs with matrix-valued states for scalable language modeling. In arXiv preprint. https://arxiv.org/abs/2603.14360
  36. [40]Sajad Movahedi, Vera Milovanovic, Shlomo Libo Feigin, Alexander Theus, Thomas Hofmann, Valentina Boeva, T. Konstantin Rusch, and Antonio Orvieto. Fixed-point reasoners: Stable and adaptive deep looped transformers. arXiv preprint arXiv:2606.18206, 2026. URL https://arxiv.org/abs/2606.18206.
  37. [41]Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, and Stefano Massaroli. Quantifying memory utilization with effective state-size, 2025. URL https://arxiv.org/abs/2504.19561.
  38. [42]Peng, B., Goldstein, D., Anthony, Q., Albalak, A., Alcaide, E., Biderman, S., Cheah, E., Ferdinan, T., Hou, H., GV, K. K., Krishna, S., McClelland, R. J., Muennighoff, N., Obeid, F., Saito, A., Song, G., Tu, H., Zhang, R., Zhao, B., … Zhu, R.-J. (2024). Eagle and Finch: RWKV with matrix-valued states and dynamic recurrence. First Conference on Language Modeling (COLM). https://openreview.net/pdf/9f3d829735b231725665ff226a4f732293ad4aa8.pdfopenreview.net/pdf/9f3d829735b231725665ff226a4f732293ad4aa8.pdf
  39. [43]Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. RWKV-7 “Goose” with expressive dynamic state evolution. In Second Conference on Language Modeling (COLM), 2025. URL https://openreview.net/forum?id=ayB1PACN5j.openreview.net/forum?id=ayB1PACN5j
  40. [44]Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=wHBfxhZu1u.openreview.net/forum?id=wHBfxhZu1u
  41. [45]Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by backpropagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0
  42. [46]Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/2676109d49d1eb26d6bc584a8f556305-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2025/hash/2676109d49d1eb26d6bc584a8f556305-Abstract-Conference.html
  43. [47]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, 2021. URL https://proceedings.mlr.press/v139/schlag21a.html.proceedings.mlr.press/v139/schlag21a.html
  44. [49]Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. DeltaProduct: Improving state-tracking in linear RNNs via Householder products. In Advances in Neural Information Processing Systems, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/e1ea2520fbf9cd1600b287dde67e0a3c-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/e1ea2520fbf9cd1600b287dde67e0a3c-Abstract-Conference.html
  45. [50]Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): RNNs with expressive hidden states. In International Conference on Machine Learning, 2025. URL https://proceedings.mlr.press/v267/sun25h.html.proceedings.mlr.press/v267/sun25h.html
  46. [51]Tandon, A., Dalal, K., Li, X., Koceja, D., Rød, M., Buchanan, S., Wang, X., Leskovec, J., Koyejo, S., Hashimoto, T., Guestrin, C., McCaleb, J., Choi, Y., & Sun, Y. (2025). End-to-end test-time training for long context. In arXiv preprint. https://arxiv.org/abs/2512.23675
  47. [52]Neehal Tumma, Mathias Lechner, Noel Loo, Ramin Hasani, and Daniela Rus. Leveraging low-rank and sparse recurrent connectivity for robust closed-loop control, 2023. URL https://arxiv.org/abs/2310.03915.
  48. [53]Neehal Tumma, Noel Loo, and Daniela Rus. Preconditioned DeltaNet: Curvature-aware sequence modeling for linear recurrences. In International Conference on Machine Learning, 2026. URL https://icml.cc/virtual/2026/poster/63733.icml.cc/virtual/2026/poster/63733
  49. [54]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  50. [55]Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Aguera y Arcas, and João Sacramento. MesaNet: Sequence modeling by locally optimal test-time training. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/2c3d5a4eff20d24390a445762d61bfbb-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2026/hash/2c3d5a4eff20d24390a445762d61bfbb-Abstract-Conference.html
  51. [56]Ke Alexander Wang, Jiaxin Shi, and Emily B. Fox. Test-time regression: a unifying framework for designing sequence models with associative memory. arXiv preprint arXiv:2501.12352, 2025. URL https://arxiv.org/abs/2501.12352.
  52. [57]Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, and John Langford. Full-bandwidth transformer. arXiv preprint arXiv:2608.08888, 2026. URL https://arxiv.org/abs/2608.08888.
  53. [58]Paul J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560, 1990. URL https://doi.org/10.1109/5.58337.
  54. [59]Bernard Widrow and Marcian E. Hoff, Jr. Adaptive switching circuits. In 1960 IRE WESCON Convention Record, Part 4, pp. 96–104, 1960. URL https://isl.stanford.edu/~widrow/papers/c1960adaptiveswitching.pdf.DOI
  55. [60]Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., & Srebro, N. (2020). Kernel and rich regimes in overparametrized models. Conference on Learning Theory. https://proceedings.mlr.press/v125/woodworth20a.htmlproceedings.mlr.press/v125/woodworth20a.html
  56. [61]Haoyi Wu, Zhihao Teng, and Kewei Tu. Parallel continuous chain-of-thought with Jacobi iteration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 914–926, 2025. doi: 10.18653/v1/2025.emnlp-main.47. URL https://aclanthology.org/2025.emnlp-main.47/.DOI
  57. [62]Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In International Conference on Machine Learning, 2024a. URL https://proceedings.mlr.press/v235/yang24ab.html.proceedings.mlr.press/v235/yang24ab.html
  58. [63]Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In Advances in Neural Information Processing Systems, 2024b. URL https://arxiv.org/abs/2406.06484.
  59. [64]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=r8H7xhYPwz.openreview.net/forum?id=r8H7xhYPwz
  60. [65]Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T Freeman, and Hao Tan. Test-time training done right. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/ffbfff81fb78e4bb558273b91e12e318-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2026/hash/ffbfff81fb78e4bb558273b91e12e318-Abstract-Conference.html
  61. [66]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025. URL https://arxiv.org/abs/2510.25741.

Paper details

vlnc:W_p92ymntk
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents