betaFeedback

WorksComputer Science

Kronecker Delta Rule for Long Context Linear Attention

Abstract

We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations while modifying the memory only along a single direction in matrix space. This minimally invasive memory revision reduces interference with previously stored information compared with delta-rule updates, making it particularly well suited for long-context language modeling. The resulting memory dynamics can be mathematically described as a Markov chain that converges to a steady-state distribution, while the expected norm of the memory matrix remains uniformly bounded throughout the entire input sequence. Hence, KronLSTM ensures stable memory dynamics without requiring additional normalization and stabilization mechanisms for the memory state. Experiments show that KronLSTM is competitive with, and often outperforms, existing linear attention architectures, including models of the (Gated) DeltaNet family, with particularly strong performance on long context benchmarks. These results establish the Kronecker Delta Rule as an effective and analytically tractable foundation for stable associative memory in recurrent linear attention architectures. Our code (https://github.com/NX-AI/KronLSTM) and models (https://huggingface.co/collections/NX-AI/kronlstm) are open-source.

Figure 1. Representation of the memory matrix state revision for mLSTM, Gated DeltaNet, and KronLSTM. Interference with stored values: mLSTM affects all values, Gated DeltaNet affects those with overlapping keys, and KronLSTM affects only those with overlapping key and value pairs.

Introduction

Large language models (LLMs) predominantly use the Transformer architecture ([41, 54, 56] (Qwen Team 2023)), whose softmax attention mechanism has computational costs that grow quadratically with context length, while its key–value cache grows linearly. Consequently, inference becomes increasingly costly as contexts grow. To reduce these costs, recent work has explored replacing softmax attention with linear attention mechanisms ([31, 47]), which can be traced back to the idea of fast weight programmers Schmidhuber, [48]. Linear attention compresses the context into a fixed-size memory matrix that is updated as new tokens arrive, yielding linear computational complexity and a constant memory footprint. Recently, recurrent architectures based on associative memories Anderson, [3]; Kohonen [33], including xLSTM Beck et al., [5], as well as related state-space and memory-based approaches such as Mamba Dao and Gu, [18]; Gu and Dao [22]; Lahoti et al. [34], DeltaNet Hatamizadeh et al., [24]; Kimi Team et al. [32]; Yang et al. [61, 62], RWKV-5 and 6 Peng et al., [43], and other recent architectures Behrouz et al., [9]; Liu et al. [35]; Sun et al. [53], have emerged as competitive alternatives to Transformers.

A crucial challenge for linear attention is controlling memory growth. If the memory of recurrent architectures grows without being bounded, its readout can become arbitrarily large, leading to numerical instabilities, particularly during learning. The original LSTM Hochreiter, [25]; Hochreiter and Schmidhuber [26] uses a bounded squashing function to control the exposed cell state, while its forget gate regulates the accumulation of information in memory. RWKV-5 and RWKV-6 Peng et al., [43] similarly control memory growth through a forget gate. Mamba architectures Dao and Gu, [18]; Gu and Dao [22]; Lahoti et al. [34] also restrain memory growth through a forget mechanism derived from a stable state-space ordinary differential equation. However, forget-based memory control becomes increasingly restrictive for long contexts. To retain information over long time spans, forget gates must remain close to one; otherwise, information from the beginning of the context is progressively attenuated. xLSTM Beck et al., [5] addresses this limitation by allowing forget gates close to one and introducing normalizer and stabilizer states to control the scale of its matrix memory and readout. However, normalization based on ratios of potentially large quantities can itself lead to numerical difficulties and learning instabilities, particularly in larger xLSTM networks Beck et al., [7]. mLSTM’s numerical instabilities are illustrated by the stress test in Figure 2.

Figure 2. Stress test with high learning rates for the same mLSTM and KronLSTM architectures on language modeling. Learning becomes unstable for mLSTM as the gradient norm spikes show, while KronLSTM yields stable learning.

The DeltaNet family Hatamizadeh et al., [24]; Kimi Team et al. [32]; Yang et al. [61, 62] controls memory growth through Delta Rule updates, whose dynamics converge to a steady-state distribution with bounded memory ensuring stable learning dynamics. However, revising a stored value can perturb previously stored values, leading to unwanted memory interference for long sequences. We introduce KronLSTM, which, like (Gated) DeltaNet, provides bounded memory through a delta-style update, but uses the Kronecker Delta Rule. The Kronecker Delta Rule is less invasive: it modifies the memory only along a single direction in matrix space, reducing interference with previously stored associations. This targeted memory revision results in a less noisy gradient flow over time.

Our contributions are:

  • (i) Kronecker Delta Rule is introduced as a new update rule for associative matrix memories, which corrects the scalar associations while having minimal interference with previously stored values.

  • (ii) KronLSTM is introduced as a novel linear attention architecture based on the Kronecker Delta Rule equipped with a vector forget gate, a β\beta gate, and a target gate.

  • (iii) Stability properties for the Kronecker Delta Rule are mathematically proved via bounds on the expected memory norm. These bounds ensure stable learning and proper gradient flow.

  • (iv) Experiments empirically validate the strong performance of KronLSTM at long-context language modeling and retrieval tasks. KronLSTM performs strongly across language modeling, recall, and state tracking benchmarks.

KronLSTM

In Subsection 2.1, we introduce the Kronecker Delta Rule and analyze its memory dynamics, and in Subsection 2.2, the Kronecker Delta Rule is embedded in the KronLSTM sequence-mixer block.

Kronecker Delta Rule

Linear attention update rules for matrix memory. Linear attention [31] compresses the context into a matrix memory SS that is updated per token and read out per query,

St+1=St+vtktT,ht=Stqt=∑i<tvi(kiTqt),(1)S_{t+1} = S_t + v_t k_t^{\mathsf{T}}, \qquad h_t = S_t q_t = \sum_{i<t} v_i \left(k_i^{\mathsf{T}} q_t\right), \tag*{(1)}

with St∈Rdv×dkS_t \in\mathbb{R}^{d_v \times d_k}, keys ktk_t and queries qtq_t in Rdk\mathbb{R}^{d_k}, and values vtv_t in Rdv\mathbb{R}^{d_v}. Therefore, SS is an associative memory [3, 33] with a Hebbian rule, i.e., an outer product update rule. Modern linear attention architectures like xLSTM [5], Mamba [18, 22, 34], RWKV-5 and 6 [43] rely on these associative memories equipped with gates and transformations. The memory growth is controlled by a forget gate, which stabilizes the memory dynamics.

In contrast, the Delta Rule [47, 59] corrects the memory with the difference of the retrieval to the current value when writing into the memory

St+1=St−βt(Stkt−vt)ktT=St(I−βtktktT)+βtvtktT,(2)S_{t+1} = S_t - \beta_t \left(S_t k_t - v_t\right) k_t^{\mathsf{T}} = S_t \left(I - \beta_t k_t k_t^{\mathsf{T}}\right) + \beta_t v_t k_t^{\mathsf{T}}, \tag*{(2)}

with step size βt\beta_t. Each step is gradient descent on 12∥Stkt−vt∥2\frac{1}{2}\lVert S_t k_t - v_t\rVert^2, and the transition I−βtktktTI - \beta_t k_t k_t^{\mathsf{T}} is a generalized Householder matrix, which admits chunkwise-parallel training [61].

The two rules differ in how an association, once stored, is changed. The Hebbian write of (1) is additive, so the memory grows with the sequence and an earlier value can be buried by later writes but never revised. The delta rule (2) revises: it removes the value currently stored at ktk_t before writing vtv_t. The correction, however, proceeds along (Stkt−vt)ktT\left(S_t k_t - v_t\right) k_t^{\mathsf{T}}, a direction determined by the current memory content, so values stored at keys that overlap with ktk_t are affected as well. We aim at an error-correcting write that acts solely on the pair (kt,vt)(k_t, v_t).

Kronecker Delta Rule. Let St∈Rdv×dkS_t \in\mathbb{R}^{d_v \times d_k} be the memory matrix, kt∈Rdkk_t \in\mathbb{R}^{d_k} and vt∈Rdvv_t \in\mathbb{R}^{d_v} the key and value at step tt, ct∈Rc_t \in\mathbb{R} a target and βt\beta_t a step size. The scalar vtTStktv_t^{\mathsf{T}} S_t k_t measures how strongly the memory associates ktk_t with vtv_t. The Kronecker Delta Rule corrects this scalar towards the target along the outer product of the pair as follows

St+1=St−βt(vtTStkt−ct)vtktT=St−βt(vtTStkt)vtktT⏟erase+βtctvtktT⏟write.(3)S_{t+1} = S_t - \beta_t \left(v_t^{\mathsf{T}} S_t k_t - c_t\right) v_t k_t^{\mathsf{T}} = S_t - \underbrace{\beta_t \left(v_t^{\mathsf{T}} S_t k_t\right) v_t k_t^{\mathsf{T}}}_{\text{erase}} + \underbrace{\beta_t c_t v_t k_t^{\mathsf{T}}}_{\text{write}}. \tag*{(3)}

In place of the dvd_v-dimensional residual Stkt−vtS_t k_t - v_t of the delta rule, a scalar is corrected, and the direction vtktTv_t k_t^{\mathsf{T}} of the correction is determined by the input pair alone without utilizing the memory content. A target ct≡1c_t \equiv1 stores each association with unit strength, while ct=0c_t = 0 erases it. The target ctc_t may depend on the input, i.e., serves as a gate, so that what is written becomes input-dependent. The right-hand side of (3) separates the update into an erase and a write term, both scaled by βt\beta_t. Choosing the target c~t=ct/βt\tilde{c}_t = c_t / \beta_t removes the step size from the write, so that βt\beta_t alone determines how much of the stored association is erased and ctc_t alone how much is written. KronLSTM uses this form, with both quantities computed from the input as gates (§ 2.2).

Properties of the Kronecker Delta Rule. (i) Error correction (see Appendix Property A2 and Property A6). An update is a step of gradient descent with step size βt\beta_t on 12(vtTSkt−ct)2\frac{1}{2}\left(v_t^{\mathsf{T}} S k_t - c_t\right)^2. Re-presenting the same pair gives vtTSt+1kt−ct=(vtTStkt−ct)(1−βt∥at∥2)v_t^{\mathsf{T}} S_{t+1} k_t - c_t = \left(v_t^{\mathsf{T}} S_t k_t - c_t\right)\left(1 - \beta_t \lVert a_t\rVert^2\right), so it contracts the error iff 0<βt<2/∥at∥20 < \beta_t < 2/\lVert a_t\rVert^2, and the Newton step βt=1/∥at∥2\beta_t = 1/\lVert a_t\rVert^2 meets the target in a single update; for ct=0c_t = 0 this deletes the association. *(ii) Delta Rule in vectorized form (see Appendix Property Property A7). With the Kronecker product at=kt⊗vt=vec⁡(vtkt⊤)a_t = k_t \otimes v_t = \operatorname{vec}(v_t k_t^\top) and the vectorized memory s=vec⁡(S)s = \operatorname{vec}(S), we have vt⊤Skt=at⊤sv_t^\top S k_t = a_t^\top s and ∥at∥2=∥kt∥2∥vt∥2\lVert a_t\rVert^2 = \lVert k_t\rVert^2 \lVert v_t\rVert^2, and (3) becomes

st+1=(I−βtatat⊤)st+βtctat,(4)s_{t+1} = \left(I - \beta_t a_t a_t^\top\right)s_t + \beta_t c_t a_t, \tag*{(4)}

References

  1. [2]B. Alkin, M. Beck, K. Pöppel, S. Hochreiter, and J. Brandstetter. Vision-LSTM: xLSTM as Generic Vision Backbone. In International Conference on Learning Representations, volume 13, Singapore, Singapore, 2025. OpenReview. URL https://openreview.net/forum?id=SiH7DwN KZZ.arxiv.org/abs/2406.04303
  2. [3]J. A. Anderson. A simple neural network generating an interactive memory. Mathematical Biosciences, 14(3-4): 197–220, 1972.DOI
  3. [4]S. Arora, S. Eyuboglu, M. Zhang, A. Tımalsina, S. Alberti, D. Zinsley, J. Zou, A. Rudra, and C. Ré. Simple linear attention language models balance the recall-throughput tradeoff. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 1763–1840, Vienna, Austria, 2024. PMLR. URL https://proceedings.mlr.press/v235/arora24a.html.arxiv.org/abs/2402.18668
  4. [5]M. Beck, K. Pöppel, M. Spannring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. xLSTM: Extended Long Short-Term Memory. In Advances in Neural Information Processing Systems, volume 37, pages 107547–107603, Vancouver, BC, Canada, 2024. Curran Associates, Inc. doi: 10.52202/079017-3417. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/c2cef27O1c10a2b2f2ea0a43c3aa3-Abstract-Conference.html.DOI
  5. [6]M. Beck, K. Pöppel, P. Lippe, and S. Hochreiter. Tiled flash linear attention: More efficient linear RNN and xLSTM kernels. In Advances in Neural Information Processing Systems (NeurIPS), volume 38, San Diego, CA, USA, 2025a. Curran Associates, Inc. URL https://openreview.net/forum?id=b6H64u6TqI.arxiv.org/abs/2503.14376
  6. [7]M. Beck, K. Pöppel, P. Lippe, R. Kurle, P. M. Blies, G. Klambauer, S. Böck, and S. Hochreiter. xLSTM 7B: A recurrent LLM for fast and efficient inference. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 3335–3357, Vancouver, BC, Canada, 2025b. PMLR. URL https://proceedings.mlr.press/v267/beck25b.html.arxiv.org/abs/2503.13427
  7. [8]M. Beck, K. Schweighofer, S. Böck, S. Lehner, and S. Hochreiter. xLSTM scaling laws: Competitive performance with linear time-complexity. In International Conference on Learning Representations. OpenReview, 2026. doi: 10.48550/arXiv.2510.02228. URL https://arxiv.org/abs/2510.02228. arXiv:2510.02228 [cs].
  8. [9]A. Behrouz, P. Zhong, and V. Mirrokni. Titans: Learning to memorize at test time. In Advances in Neural Information Processing Systems, 2025. arXiv:2501.00663.arxiv.org/abs/2501.00663
  9. [10]A. Behrouz, M. Razaviyayn, P. Zhong, and V. Mirrokni. It’s all connected: A journey through test-time memorization, attentional bias, retention, and online optimization. International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2504.13173.
  10. [11]C. H. Bischof and C. F. VanLoan. The WY representation for products of householder matrices. SIAM Journal on Scientific and Statistical Computing, 8(1):s2–s13, 1987. doi: 10.1137/0908009.DOI
  11. [12]Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6239.arxiv.org/abs/1911.11641
  12. [13]R. Bogacz and M. W. Brown. An anti-Hebbian model of familiarity discrimination in the perirhinal cortex. Neurocomputing, 52–54:1–6, 2003. doi: 10.1016/S0925-2312(02)00738-5.DOI
  13. [14]L. Cabannes, M. Beck, M. Lomeli, G. Szilvasy, M. Douze, J. Copet, P.-E. Mazare, G. Synnaeve, and H. Jégou. Short window attention enables long-term memorization. In International Conference on Learning Representations, volume 14, pages 79759–79774, 2026. URL https://openreview.net/forum?id=btgVfhudI1.arxiv.org/abs/2509.24552
  14. [15]A. Cegielski. Iterative Methods for Fixed Point Problems in Hilbert Spaces, volume 2057 of Lecture Notes in Mathematics. Springer, 2012.DOI
  15. [16]X. Chen. The Kaczmarz algorithm, row action methods, and statistical learning algorithms. In Contemporary Mathematics, volume 706, pages 115–127, 2018. ISBN 9781470436193. doi: 10.1090/conm/706/14216.DOI
  16. [17]P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803.05457.
  17. [18]T. Dao and A. Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, volume 235, pages 10041–10071, Vienna, Austria, 2024. PMLR. URL https://openreview.net/forum?id=ztn8FCR1td.arxiv.org/abs/2405.21060
  18. [20]T. Gao, A. Wettig, H. Yen, and D. Chen. How to train long-context language models (effectively). ArXiv:2410.02660, 2024. URL https://arxiv.org/abs/2410.02660.
  19. [21]R. Grazzi, J. Siems, A. Zela, J. K. H. Franke, F. Hutter, and M. Pontil. Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. In International Conference on Learning Representations, volume 13, Singapore, Singapore, 2025. URL https://openreview.net/forum?id=UvTo3tVBk2.arxiv.org/abs/2411.12537
  20. [22]A. Gu and T. Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In Conference on Language Modeling, volume 1, Philadelphia, PA, USA, 2024. OpenReview. URL https://openreview.net/forum?id=tEYskw1VV2.arxiv.org/abs/2312.00752
  21. [23]H. Guo, S. Yang, T. Goel, E. P. Xing, T. Dao, and Y. Kim. Log-Linear Attention. In International Conference on Learning Representations. OpenReview, 2026. URL https://openreview.net/forum?id=m0JgZWkKXW.arxiv.org/abs/2506.04761
  22. [24]A. Hatamizadeh, Y. Choi, and J. Kautz. Gated DeltaNet-2: Decoupling erase and write in linear attention, 2026. URL https://arxiv.org/abs/2605.22791.
  23. [25]S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München, 1991.
  24. [26]S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, 1997. ISSN 0899-7667, 1530-888X. doi: 10.1162/neco.1997.9.8.1735. URL https://www.mitpressjournals.org/doi/abs/10.1162/neco.1997.9.8.1735.DOI
  25. [27]S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In J. Kolen and S. Kremer, editors, A Field Guide to Dynamical Recurrent Networks. IEEE Press, 2000.
  26. [28]P.J. Hoedt, F. Kratzert, D. Klotz, C. Halmi​​ch, M. Holzleitner, G. Nearing, S. Hochreiter, and G. Klambauer. Mc-lstm: Mass-conserving lstm. In International Conference on Machine Learning, pages 4275–4286. PMLR, 2021.arxiv.org/abs/2101.05186
  27. [29]C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. RULER: What’s the Real Context Size of Your Long-Context Language Models? In Conference on Language Modeling, volume 1, Philadelphia, PA, USA, 2024. OpenReview. URL https://openreview.net/pdf?id=kIoBbc76Sy.arxiv.org/abs/2404.06654
  28. [30]S. Kaczmarz. Angenäherte Auflösung von Systemen linearer Gleichungen. Bulletin International de l’Académie Polonaise des Sciences et des Lettres, Classe des Sciences Mathématiques et Naturelles, Série A, 35:355–357, 1937.
  29. [31]A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In E. H. Daumé III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5156–5165, virtual, 2020. PMLR. URL http://proceedings.mlr.press/v119/katharopoulos20a.html.arxiv.org/abs/2006.16236
  30. [32]Kimi Team, Y. Zhang, Z. Lin, X. Yao, J. Hu, F. Meng, C. Liu, X. Men, S. Yang, Z. Li, W. Li, E. Lu, W. Liu, Y. Chen, W. Xu, L. Yu, Y. Wang, Y. Fan, L. Zhong, E. Yuan, D. Zhang, Y. Zhang, T. Y. Liu, H. Wang, S. Fang, W. He, S. Liu, Y. Li, J. Su, J. Qiu, B. Pang, J. Yan, Z. Jiang, W. Huang, B. Yin, J. You, C. Wei, Z. Wang, C. Hong, Y. Chen, G. Chen, Y. Wang, H. Zheng, F. Wang, Y. Liu, M. Dong, Z. Zhang, S. Pan, W. Wu, Y. Wu, L. Guan, J. Tao, G. Fu, X. Xu, Y. Wang, G. Lai, Y. Wu, X. Zhou, Z. Yang, and Y. Du. Kimi linear: An expressive, efficient attention architecture, 2025. URL https://arxiv.org/abs/2510.26692. arXiv:2510.26692 [cs].
  31. [33]T. Kohonen. Correlation matrix memories. IEEE Transactions on Computers, C-21(4):353–359, 1972.
  32. [34]A. Lahoti, K. Y. Li, B. Chen, C. Wang, A. Bick, J. Z. Kolter, T. Dao, and A. Gu. Mamba-3: Improved sequence modeling using state space principles. In The Fourteenth International Conference on Learning Representations, volume 14, pages 86173–86202, Rio de Janeiro, Brazil, 2026. OpenReview. URL https://openreview.net/forum?id=HwCvaJOiCj.
  33. [35]B. Liu, R. Wang, L. Wu, Y. Feng, P. Stone, and Q. Liu. Longhorn: State space models are amortized online learners. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2407.14207.
  34. [36]J. Liu, S. Elflein, O. Litany, Z. Gojcic, and R. Li. Test-time training with KV binding is secretly linear attention. Forty-third International Conference on Machine Learning (ICML), 2026. URL https://arxiv.org/abs/2602.21204.
  35. [37]W. Merrill, Y. Li, T. Romero, A. Svete, C. Costello, P. Dasigi, D. Groeneveld, D. Heineman, B. Kuehl, N. Lambert, C. Li, K. Lo, S. Malik, D. J. Matusz, B. Minixhofer, J. Morrison, L. Soldaini, F. Timbers, P. Walsh, N. A. Smith, H. Hajishirzi, and A. Sabharwal. Olmo Hybrid: From Theory to Practice and Back, 2026. URL http://arxiv.org/abs/2604.03444. arXiv:2604.03444 [cs].
  36. [39]M. Mishra. LM Engine: A Hyper-Optimized Library for Pretraining and Finetuning, 2024. URL https://github.com/open-lm-engine/lm-engine.
  37. [40]NVIDIA. Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning, 2025. URL http://arxiv.org/abs/2512.20848. arXiv:2512.20848 [cs].
  38. [41]OpenAI. gpt-oss-120b & gpt-oss-20b Model Card, 2025. URL https://arxiv.org/abs/2508.10925. arXiv:2508.10925 [cs].
  39. [42]D. Paperno, G. Kruszewski, A. Lazaridou, N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, 2016. doi: 10.18653/v1/P16-1144.DOI
  40. [43]B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, P. Kazienko, K. K. GJV, J. Kocoń, B. Koptyra, S. Krishna, R. McClelland Jr., J. Lin, N. Muennighoff, F. Obeid, A. Saito, G. Song, H. Tu, C. Wirawan, S. Wozniak, R. Zhang, B. Zhao, Q. Zhao, P. Zhou, J. Zhu, and R.-J. Zhu. Eagle and Finch: RWKV with matrix-valued states and dynamic recurrence, 2024. URL https://arxiv.org/abs/2404.05892.
  41. [44]B. Peng, R. Zhang, D. Goldstein, E. Alcaide, X. Du, H. Hou, J. Lin, J. Liu, J. Lu, W. Merrill, G. Song, K. Tan, S. Utpala, N. Wilce, J. S. Wind, T. Wu, D. Wuttke, and C. Zhou-Zheng. RWKV-7 Goose with expressive dynamic state evolution. arXiv:2503.14456, 2025. URL https://arxiv.org/abs/2503.14456.
  42. [45]D. Prados and S. Kak. Neural network capacity using delta rule. Electronics Letters, 25(3):197–199, 1989.DOI
  43. [46]K. Sakaguchi, R. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
  44. [47]I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139, pages 9355–9366, virtual, 2021. PMLR. URL https://proceedings.mlr.press/v139/schlag21a.html.arxiv.org/abs/2102.11174
  45. [48]J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992.DOI
  46. [49]N. Schmidinger, L. Schneckenreiter, P. Seidl, J. Schimunek, P.-J. Hoedt, J. Brandstetter, A. Mayr, S. Luukkonen, S. Hochreiter, and G. Klambauer. Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=Ij bXZd ugdj.arxiv.org/abs/2411.04165
  47. [50]T. Schmied, T. Adler, V. P. Patil, M. Beck, K. Pöppel, J. Brandstetter, G. Klambauer, R. Pascanu, and S. Hochreiter. A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 53343–53387, Vancouver, BC, Canada, 2025. PMLR. URL https://proceedings.mlr.press/v267/schmied25a.html.arxiv.org/abs/2410.22391
  48. [51]J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, and R. Grazzi. DeltaProduct: Improving state-tracking in linear RNNs via Householder products. In Advances in Neural Information Processing Systems (NeurIPS), volume 38, San Diego, CA, USA, 2025. Curran Associates, Inc. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/e1ea2520fb9cd1600b287dde67e0a3c0-Abstract-Conference.html.arxiv.org/abs/2502.10297
  49. [52]T. Strohmer and R. Vershynin. A randomized Kaczmarz algorithm with exponential convergence. Journal of Fourier Analysis and Applications, 15(2):262–278, 2009. URL https://link.springer.com/article/10.1007/s00041-008-9030-4.arxiv.org/abs/math/0702226
  50. [53]Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, and C. Guestrin. Learning to (learn at test time): RNNs with expressive hidden states. arXiv:2407.04620, 2024. URL https://arxiv.org/abs/2407.04620.
  51. [54]H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971.
  52. [55]H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https://arxiv.org/abs/2307.09288.
  53. [56]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, Aidan N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 5998–6008, Long Beach, CA, USA, 2017. Curran Associates, Inc. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.arxiv.org/abs/1706.03762
  54. [57]C. Villani. Optimal Transport: Old and New, volume 338 of Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2009. ISBN 978-3-540-71049-3. doi: 10.1007/978-3-540-71050-9.
  55. [58]J. von Oswald, M. Schlegel, A. Meulemans, S. Kobayashi, E. Niklasson, N. Zuchet, N. Scherrer, N. Miller, M. Sandler, B. Agüera y Arcas, M. Vladymyrov, R. Pascanu, and J. Sacramentu. Uncovering mesa-optimization algorithms in transformers. arXiv:2309.05858, 2023. URL https://arxiv.org/abs/2309.05858.
  56. [59]B. Widrow and M. E. Hoff. Adaptive switching circuits. In 1960 IRE WESCON Convention Record, Part 4, pages 96–104, 1960.DOI
  57. [60]S. Yang and Y. Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism, 2026. URL https://github.com/fla-org/flash-linear-attention.
  58. [61]S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim. Parallelizing linear transformers with the delta rule over sequence length. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 115491–115522, Vancouver, BC, Canada, 2024. Curran Associates, Inc. doi: 10.52202/079017-3668. URL https://arxiv.org/abs/2406.06484.
  59. [62]S. Yang, J. Kautz, and A. Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/ forum?id=r8H7xhYPwz.arxiv.org/abs/2412.06464
  60. [63]R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019. URL https://aclanthology.org/P19-1472/.arxiv.org/abs/1905.07830

Paper details

vlnc:W_a3r5jvw2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Opus 5 (bare)

View statement
Looking for related papers…
Contents