betaFeedback

WorksComputer Science

Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs

Abstract

Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core tiles and fast vector registers make per-channel decay rescaling and sequential in-block solves cheap. On Google TPUs, which concentrate compute in wide 256×256256 \times 256 systolic matrix units backed by a slower vector unit, porting these layers directly leaves half of each matrix tile empty and bottlenecks on vector operations. We present Systolic Linear Attention (SLA), which reshapes gated delta attention around the matrix unit by widening keys to 256256 with grouped value heads, grouping decay gates into four 6464-channel tiles, and computing the chunk solve entirely with block matrix products in a single fused kernel. On TPU v7x, a full SLA-R4 layer runs 1.6×1.6\times–2.5×2.5\times faster than the open-source Tokamax KDA kernel (1.4×1.4\times–2.4×2.4\times with exact chunk inversion), driven primarily by head sharing and kernel fusion. Across a 2,0482{,}048-chip TPU v7x pre-training study of sparse Mixture-of-Experts (MoE) models from 6.5B6.5\text{B} to 63.1B63.1\text{B} parameters, SLA trains 11%11\%–24%24\% faster than KDA at similar perplexity, and reaches slightly lower perplexity than KDA in smaller matched-seed runs.

Introduction

Linear attention removes the quadratic compute and key-value (KV) cache of softmax attentions [15, 63] by storing the context in a fixed-size state matrix St∈Rdk×dvS_t \in\mathbb{R}^{d_k \times d_v} (for key width dkd_k and value width dvd_v), updated at each token tt with its key kt∈Rdkk_t \in\mathbb{R}^{d_k} and value vt∈Rdvv_t \in\mathbb{R}^{d_v} as [33]:

St=St−1+ktvt⊤.S_t = S_{t-1} + k_t v_t^\top.

Because this sum grows without bound, the state saturates on long sequences. Gated models such as Mamba-2 [14] and GLA [66] add a data-dependent forget (decay) gate Ft∈(0,1]dk×dkF_t \in(0, 1]^{d_k \times d_k},

St=FtSt−1+ktvt⊤,S_t = F_t S_{t-1} + k_t v_t^\top,

but decay fades all stored keys together and hurts associative recall [5].

To overwrite individual keys selectively, DeltaNet [50, 67] introduced the delta rule with unit-norm key ktk_t (∥kt∥2=1\lVert k_t\rVert_2 = 1), scalar write gate βt\beta_t, and dk×dkd_k \times d_k identity IdkI_{d_k}:

St=(Idk−βtktkt⊤)St−1+βtktvt⊤.S_t = (I_{d_k} - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top.

Viewing such updates as online gradient descent, prior work also added heavy-ball momentum so that a single noisy update cannot overwrite established memories [9, 44]. Combining decay with the delta rule produced Gated DeltaNet (GDN) [68], adopted in Qwen3-Next [48]. Kimi Delta Attention (KDA) [71] then expanded GDN’s scalar forget gate into a per-channel diagonal gate Ft=diag⁡(ft)F_t = \operatorname{diag}(f_t)

Preprint. with channel-wise decay vector ft∈(0,1]dkf_t \in(0, 1]^{d_k}, and GDN-2 [25] decoupled per-channel erase and write gates. § 5 surveys further variants.

However, these architectures and their chunkwise WY training algorithms [10, 66, 68] were developed and tuned on NVIDIA GPUs (fla [65]). They map less well onto 2D systolic-array accelerators such as Google TPUs [18, 31, 32], for three reasons:

Small tiles and unpacked heads. GPU Tensor Cores operate on small fragments, so dk=dv=128d_k = d_v = 128 with one key head per value head already reaches high occupancy. The TPU v6e/v7x Matrix Multiply Unit (MXU) is a 256 × 256 systolic array: a 128-wide state pads half of every tile, and projections, convolutions, and chunk solves are repeated for every head.

Per-channel decay gates. In KDA and GDN-2, each key channel decays at its own rate. Kernels must rescale keys elementwise in 16-token sub-blocks to avoid float32 overflow. GPUs absorb this vector work cheaply; on TPUs, where matrix throughput exceeds vector throughput by roughly 128×, it stalls the kernel.

Sequential triangular solves. The chunkwise WY algorithm [10, 68] partitions training sequences into chunks of length CC and solves a C×CC \times C unit lower-triangular system per head and chunk. Existing kernels [19, 65] substitute sequentially within diagonal blocks; on TPUs this runs on the Vector Processing Unit (VPU) and takes about half of the core kernel time in our measurements.

Kernel tuning alone does not close this gap: fusing a GPU-shaped KDA layer into on-chip memory speeds it up by only 1.03× on TPU v7x (see Table 5), because the time goes to VPU solves and padded MXU tiles rather than High-Bandwidth Memory (HBM) traffic. We therefore change the layer itself. Systolic Linear Attention (SLA) applies a block-diagonal, tile-grouped forget gate Ft∈Rdk×dkF_t \in\mathbb{R}^{d_k \times d_k} to St−1S_{t-1} and then performs the delta update with key kt∈Rdkk_t \in\mathbb{R}^{d_k}, packed value vt∈RRdvv_t \in\mathbb{R}^{Rd_v} (stacking R∈{2,4}R \in\{2, 4\} value heads per key head), and a write gate βt∈(0,2)\beta_t \in(0, 2) shared across the packed value heads:

Table 5: Seven-operator forward audit (ms) on TPU v7x (D=2048D = 2048, T=8,192T = 8{,}192). Unfused columns report isolated per-stage forward medians; fused brackets report single-kernel Ops 2–7a latency.

Stage Latency (D=2048D = 2048, T=8KT = 8\mathrm{K})KDA (Unf.)KDA (Fus.)SLA-R4 (Unf.)SLA-R4 (Fused)
Op 1: QKV + Gate Proj (MXU)0.2510.2510.1680.168
Op 2: Causal Conv4 + QK Norm0.562Fused0.341Fused
Op 3–5: Gram + WY Inversion1.184Fused0.418Fused
Op 6: Chunk State Scan (ScS_c)1.2483.1040.6141.112
Op 7a: Intra + State Readout0.2190.219
Op 7b–7c: Out Norm + Out Proj0.2550.2550.2450.245
Measured Full Layer Fwd (ms)3.7193.6102.0051.525
Analytical Roofline Fwd (ms)2.8902.7401.3800.867
Measured Full Layer F+B (ms)7.1086.9123.9822.992
F+B Speedup vs. KDA (Unf.)1.00×1.03×1.79×2.38×

Table 5. Seven-operator forward audit (ms) on TPU v7x.

St=FtSt−1+βtkt(vt−(FtSt−1)Tkt)T.(1)S_t = F_t S_{t-1} + \beta_t k_t \left(v_t - (F_t S_{t-1})^\mathsf{T} k_t\right)^\mathsf{T}. \tag*{(1)}

SLA also smooths chunk updates with a chunk-boundary momentum term with coefficient μ=0.25\mu= 0.25 that stays in on-chip SRAM (VMEM). SLA combines three changes (§ 2): Wide keys shared across value heads. SLA widens keys to dk=256d_k = 256 and shares each key head and gate βt\beta_t across R∈{2,4}R \in\{2,4\} value heads (SLA-R2, SLA-R4), following grouped value attention (GVA) [47]. The packed state fills whole MXU tiles, and key-query projections, convolutions, and chunk solves are computed R×R\times less often.

Tile-grouped forget gate. SLA splits the 256 key channels into 4 tiles of 64 channels, each with one scalar decay rate. Intra-chunk scores then become four 64-wide MXU matmuls scaled by decay masks, with no VPU rescaling. This is a middle point between GDN’s single rate per head and KDA’s rate per channel.

Chunk inversion on the MXU. Because βt\beta_t is shared, SLA needs one C×CC \times C solve per key head. It computes the inverse only with block matrix products on the MXU: an exact recursive block-doubling inverse by default, or an optional 1-step Neumann approximation for faster solves, fused with the rest of the layer in VMEM.

We evaluate SLA on TPU v7x at the kernel, layer, serving, and context-parallel level, and across a 2,048-chip TPU v7x pre-training study of sparse Mixture-of-Experts (MoE) models [16, 52] from 6.5B to 63.1B parameters on Dolma [54, 60] (§ 3). We find that most of SLA’s speedup comes from head sharing and fusion, which the tile-grouped gate and MXU chunk inversion make possible. On matched seeds, SLA improves on standard KDA and stays within about 0.1 perplexity of a wide-key per-channel KDA control while avoiding per-channel long-context drift; the optional 1-step Neumann inverse trades about 0.2 perplexity for speed. SLA is therefore best understood as a hardware-efficient operating point rather than a more expressive update rule; it is complementary to modeling advances such as GDN-2, which reaches lower perplexity per token (§ 3.5).

In the following sections, § 2 reviews gated delta attention, its TPU bottlenecks, and SLA’s design; § 3 evaluates SLA from kernels and serving to MoE pre-training; Section 4 details optional low-precision key whitening; and § 5 covers related work.

Hardware Bottlenecks and the SLA Design

To motivate SLA, we first review gated delta recurrences and their chunkwise WY training algorithm (§ 2.1), identify three hardware bottlenecks that GPU-centric designs hit on TPUs (§ 2.2), and then present SLA’s mathematical formulation and chunk-boundary momentum (§ 2.3), its three algorithmic pillars and mathematical derivations (§ 2.4), and its single-kernel VMEM backward pass (§ 2.5), followed by empirical evaluation in § 3.

Preliminaries: Gated Delta Recurrences and Chunkwise WY Parallelism

Given an input sequence X=(x1,…,xT)∈RT×DX=(x_1,\ldots,x_T)\in\mathbb{R}^{T\times D} of length TT and model width DD, a gated-delta linear attention layer with HvH_v value heads and HkH_k query-key heads maps each token xt∈RDx_t\in\mathbb{R}^{D} (t∈{1,…,T}t\in\{1,\ldots,T\}) to L2L_2-normalized queries and keys qt,kt∈Rdkq_t,k_t\in\mathbb{R}^{d_k} (∥qt∥2=∥kt∥2=1\lVert q_t\rVert_2=\lVert k_t\rVert_2=1), values vt∈Rdvv_t\in\mathbb{R}^{d_v}, and head output ot∈Rdvo_t\in\mathbb{R}^{d_v} (with Hk=HvH_k=H_v in unpacked baselines). Rather than caching tt past tokens, each head maintains a fixed-size associative state matrix St∈Rdk×dvS_t\in\mathbb{R}^{d_k\times d_v} updated by applying a diagonal forget gate Ft∈(0,1]dk×dkF_t\in(0,1]^{d_k\times d_k} followed by a rank-one delta update [9, 50, 59, 67] with scalar step size βt\beta_t:

Prior Work: Gated Delta Recurrence in GDN and KDA

St=FtSt−1+βtkt(vt−(FtSt−1)⊤kt)⊤=(Idk−βtktkt⊤)FtSt−1+βtktvt⊤,ot=St⊤qt.(2)\begin{aligned} S_t &= F_tS_{t-1}+\beta_t k_t\left(v_t-\left(F_tS_{t-1}\right)^\top k_t\right)^\top\\ &= \left(I_{d_k}-\beta_t k_tk_t^\top\right)F_tS_{t-1}+\beta_t k_tv_t^\top, \\ o_t &= S_t^\top q_t. \tag*{(2)} \end{aligned}

References

  1. [1]AI-Hypercomputer Contributors. MaxText: A high performance, highly scalable, open-source LLM library written in pure Python/JAX. https://github.com/AI-Hypercomputer/maxtext, 2026.
  2. [2]J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2305.13245.arxiv.org/abs/2305.13245
  3. [3]N. Amsel, D. Persson, C. Musco, and R. M. Gower. The Polar Express: Optimal matrix sign methods and their application to the Muon algorithm. arXiv preprint arXiv:2505.16932, 2025.arxiv.org/abs/2505.16932
  4. [4]R. Anil, V. Gupta, T. Koren, K. Regan, and Y. Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020.arxiv.org/abs/2002.09018
  5. [5]S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. Zoology: Measuring and improving recall in efficient language models. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2312.04927
  6. [6]J. Austin, S. Douglas, R. Fros­tig, A. Levskaya, C. Chen, S. Vikram, F. Lebron, P. Choy, V. Ramasesh, A. Webson, and R. Pope. How to scale your model. https://jax-ml.github.io/scaling-book/, 2025.
  7. [7]M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. xLSTM: Extended long short-term memory. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2405.04517
  8. [8]A. Behrouz, Z. Li, P. Kacham, M. Daliri, Y. Deng, P. Zhong, M. Razaviyayn, and V. Mirrokni. Atlas: Learning to optimally memorize the context at test time. arXiv preprint arXiv:2505.23735, 2025.arxiv.org/abs/2505.23735
  9. [9]A. Behrouz, P. Zhong, and V. Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2025.arxiv.org/abs/2501.00663
  10. [10]C. Bischof and C. Van Loan. The WY representation for products of Householder matrices. SIAM Journal on Scientific and Statistical Computing, 8(1):s2–s13, 1987.DOI
  11. [11]G. E. Blelloch. Prefix sums and their applications. Technical Report CMU-CS-90-190, Carnegie Mellon University, 1990.
  12. [12]K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller. Rethinking attention with performers. In International Conference on Learning Representations (ICLR), 2021.arxiv.org/abs/2009.14794
  13. [13]T. Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2307.08691
  14. [14]T. Dao and A. Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2405.21060
  15. [15]T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2205.14135
  16. [16]W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research (JMLR), 23(120):1–39, 2022. arXiv:2101.03961.arxiv.org/abs/2101.03961
  17. [17]L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. L. Noac’h, H. Li, K. McDonnell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowr​​on, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation. Zenodo, https://doi.org/10.5281/zenodo.10256836, 2023.
  18. [18]Google Cloud. TPU7x (Ironwood) architecture and performance optimizations. https://docs.cloud.google.com/tpu/docs/ironwood-performance, 2025.
  19. [19]Google DeepMind. Tokamax: Open-source accelerator kernel library for JAX. https://github.com/google-deepmind/tokamax, 2026.
  20. [20]R. Grazzi, J. Siems, J. K. H. Franke, A. Zela, F. Hutter, and M. Pontil. Unlocking state-tracking in linear RNNs through negative eigenvalues. In International Conference on Learning Representations (ICLR), 2025. arXiv:2411.12537.arxiv.org/abs/2411.12537
  21. [21]E. Grishina, M. Smirnov, and M. Rakhuba. Accelerating Newton-Schulz iteration for orthogonalization via Chebyshev-type polynomials. arXiv preprint arXiv:2506.10935, 2025.arxiv.org/abs/2506.10935
  22. [22]A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.arxiv.org/abs/2312.00752
  23. [23]H. Guo, S. Yang, T. Goel, E. P. Xing, T. Dao, and Y. Kim. Log-linear attention. arXiv preprint arXiv:2506.04761, 2025.arxiv.org/abs/2506.04761
  24. [24]V. Gupta, T. Koren, and Y. Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning (ICML), 2018.arxiv.org/abs/1802.09568
  25. [25]A. Hatamizadeh, Y. Choi, and J. Kautz. Gated DeltaNet-2: Decoupling erase and write in linear attention. arXiv preprint arXiv:2605.22791, 2026.arxiv.org/abs/2605.22791
  26. [26]N. J. Higham. Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics (SIAM), 2008.
  27. [27]J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennig, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optimal large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2203.15556.arxiv.org/abs/2203.15556
  28. [28]C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2404.06654
  29. [29]S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, S. L. Song, S. Rajbhandari, and Y. He. DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509, 2023.arxiv.org/abs/2309.14509
  30. [30]S. Jelassi, D. Brandfonbrener, S. M. Kakade, and E. Malach. Repeat after me: Transformers are better than state space models at copying. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2402.01032
  31. [31]N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P.-L. Cantin, C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V. Ghaemmaghami, R. Gotipati, W. G. Gulland, R. Hagmann, C. R. Ho, D. Ho, D. Hoerger, J. Hu, R. Hundt, D. Hurt, J. Ibarr, A. Jaffey, A. Jaworski, A. Kaplan, H. Khaitan, A. Koch, N. Kumar, S. Lacy, J. Laudon, J. Law, D. Le, C. Leary, Z. Liu, K. Lucke, A. Lundin, G. MacKean, A. Maggiore, M. Mahony, K. Miller, R. Nagarajan, R. Narayanaswami, R. Ni, K. Nix, T. Norrie, M. Omernick, N. Penukonda, A. Phelps, J. Ross, M. Ross, R. Ross, A. Salek, E. Samadiani, C. Severn, G. Sizikov, M. Snelham, J. Souter, D. Steinberg, A. Swing, M. Tan, G. Thorson, B. Tian, H. Toma, E. Tuttle, V. Vasudevan, R. Walter, W. Wang, E. Wilcox, and D. H. Yoon. In-datacenter performance analysis of a tensor processing unit. In International Symposium on Computer Architecture (ISCA), 2017.
  32. [32]N. P. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles, C. Young, X. Zhou, Z. Zhou, and D. Patterson. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In International Symposium on Computer Architecture (ISCA), 2023.arxiv.org/abs/2304.01433
  33. [33]A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning (ICML), 2020.arxiv.org/abs/2006.16236
  34. [34]A. Lahoti, K. Y. Li, B. Chen, C. Wang, A. Bick, J. Z. Kolter, T. Dao, and A. Gu. Mamba-3: Improved sequence modeling using state space principles. In International Conference on Learning Representations (ICLR), 2026. arXiv:2603.15569.arxiv.org/abs/2603.15569
  35. [35]Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (ICML), 2023. arXiv:2211.17192.arxiv.org/abs/2211.17192
  36. [36]H. Liu, M. Zaharia, and P. Abbeel. Ring attention with blockwise transformers for near-infinite context. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.01889
  37. [37]J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, Y. Chen, H. Zheng, Y. Liu, S. Liu, B. Yin, W. He, H. Zhu, Y. Wang, J. Wang, M. Dong, Z. Zhang, Y. Kang, H. Zhang, X. Xu, Y. Zhang, Y. Wu, X. Zhou, and Z. Yang. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
  38. [38]I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101.arxiv.org/abs/1711.05101
  39. [39]E. Martin and C. Cundy. Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1709.04057
  40. [40]K. M. Nakanishi. Scalable-softmax is superior for attention. arXiv preprint arXiv:2501.19399, 2025.arxiv.org/abs/2501.19399
  41. [41]T. M. Nguyen, R. G. Baraniuk, R. M. Kirby, S. J. Osher, and B. Wang. Momentum transformer: Closing the performance gap between self-attention and its linearization. arXiv preprint arXiv:2208.00579, 2022.
  42. [42]B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, P. Kazienko, K. K. GV, J. Kocoń, B. Koptyra, S. Krishna, R. McClelland, Jr., J. Lin, N. Muennighoff, F. Obeid, A. Saito, G. Song, H. Tu, C. Wirawan, S. Woźniak, R. Zhang, B. Zhao, Q. Zhao, P. Zhou, J. Zhu, and R.-J. Zhu. Eagle and finch: RWKV with matrix-valued states and dynamic recurrence. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2404.05892
  43. [43]B. Peng, R. Zhang, D. Goldstein, E. Alcaide, X. Du, H. Hou, J. Lin, J. Liu, J. Lu, W. Merrill, G. Song, K. Tan, S. Utpala, N. Wilce, J. S. Wind, T. Wu, D. Wuttke, and C. Zhou-Zheng. RWKV-7 “goose” with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456, 2025.arxiv.org/abs/2503.14456
  44. [44]B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964.DOI
  45. [45]Z. Qin, W. Sun, D. Li, X. Shen, W. Sun, and Y. Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. In International Conference on Machine Learning (ICML), 2024. arXiv:2405.17381.arxiv.org/abs/2405.17381
  46. [46]Z. Qin, S. Yang, W. Sun, X. Shen, D. Li, W. Sun, and Y. Zhong. HGRN2: Gated linear RNNs with state expansion. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2404.07904
  47. [47]Z. Qiu, Z. Wang, B. Zheng, Z. Huang, K. Wen, S. Yang, R. Men, L. Yu, F. Huang, S. Huang, D. Liu, J. Zhou, and J. Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2505.06708.arxiv.org/abs/2505.06708
  48. [48]Qwen Team. Qwen3-Next: Towards ultimate training and inference efficiency. https://qwenlm.github.io/blog/qwen3-next/, 2025.
  49. [49]P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), 2016. arXiv:1606.05250.arxiv.org/abs/1606.05250
  50. [50]I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2102.11174
  51. [51]N. Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.arxiv.org/abs/2002.05202
  52. [52]N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017. arXiv:1701.03538.
  53. [53]J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, and R. Grazzi. DeltaProduct: Improving state-tracking in linear RNNs via Householder products. In International Conference on Learning Representations (ICLR), 2026. arXiv:2502.10297.arxiv.org/abs/2502.10297
  54. [54]L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Arthur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, V. Hofmann, A. H. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, A. Ravichander, K. Richardson, Z. Shen, E. Strubell, N. Subramani, O. Tafjord, P. Walsh, L. Zettlemoyer, N. A. Smith, H. Hajishirzi, I. Beltagy, D. Groeneveld, J. Dodge, and K. Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Association for Computational Linguistics (ACL), 2024.
  55. [55]J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. arXiv:2104.09864.arxiv.org/abs/2104.09864
  56. [56]W. Sun, Z. Qin, D. Li, X. Shen, Y. Qiao, and Y. Zhong. Linear attention sequence parallelism. arXiv preprint arXiv:2404.02882, 2024.arxiv.org/abs/2404.02882
  57. [57]W. Sun, Z. Qin, D. Li, X. Shen, Y. Qiao, and Y. Zhong. LASP-2: Rethinking sequence parallelism for linear attention and its hybrid. arXiv preprint arXiv:2502.07563, 2025.arxiv.org/abs/2502.07563
  58. [58]Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.arxiv.org/abs/2307.08621
  59. [59]Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, and C. Guestrin. Learning to (learn at test time): RNNs with expressive hidden states. In International Conference on Machine Learning (ICML), 2024. arXiv:2407.04620.arxiv.org/abs/2407.04620
  60. [60]Team Olmo. Olmo 3. arXiv preprint arXiv:2512.13961, 2025.
  61. [61]Team OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, N. Lambert, D. Schwenk, O. Tafjord, T. Anderson, D. Atkinson, F. Brahman, C. Clark, P. Dasigi, N. Dziri, A. Ettinger, M. Guerquin, D. Heineman, H. Ivison, P. W. Koh, J. Liu, S. Malik, W. Merrill, L. J. V. Miranda, J. Morrison, T. Murray, C. Nam, J. Poznanski, Y. Pyatkin, A. Rangapur, M. Schmitz, S. Skjonsberg, D. Wadden, C. Wilhelm, M. Wilson, L. Zettlemoyer, A. Farhadi, N. A. Smith, and H. Hajishirzi. 2 OLMo 2 furious. arXiv preprint arXiv:2501.00656, 2025.arxiv.org/abs/2501.00656
  62. [62]N. Tumma, N. Loo, and D. Rus. Preconditioned DeltaNet: Curvature-aware sequence modeling for linear recurrences. arXiv preprint arXiv:2604.21100, 2026.arxiv.org/abs/2604.21100
  63. [63]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.arxiv.org/abs/1706.03762
  64. [64]J. von Oswald, N. Scherrer, S. Kobayashi, L. Versari, S. Yang, S. Mittal, M. Schlegel, K. Maile, Y. Schimpf, O. Sieberling, A. Meulemans, R. A. Saurous, G. Lajoie, C. Frenkel, R. Pascanu, B. A. y Arcas, and J. Sacramento. MesaNet: Sequence modeling by locally optimal test-time training. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2506.05233.arxiv.org/abs/2506.05233
  65. [65]S. Yang and Y. Zhang. FLA: A Triton-based library for hardware-efficient implementations of linear attention mechanism. https://github.com/fla-org/flash-linear-attention, 2024.
  66. [66]S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated linear attention transformers with hardware-efficient training. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2312.06635
  67. [67]S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim. Parallelizing linear transformers with the delta rule over sequence length. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2406.06484
  68. [68]S. Yang, J. Kautz, and A. Hatam izadeh. Gated Delta networks: Improving Mamba2 with Delta rule. In International Conference on Learning Representations (ICLR), 2025. arXiv:2412.06464.arxiv.org/abs/2412.06464
  69. [69]Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Empirical Methods in Natural Language Processing (EMNLP), 2018. arXiv:1809.09600.arxiv.org/abs/1809.09600
  70. [70]T. Zadouri, M. Hoehn erb ach, J. Shah, T. Liu, V. Thakkar, and T. Dao. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. arXiv preprint arXiv:2603.05451, 2026.arxiv.org/abs/2603.05451
  71. [71]Y. Zhang, Z. Lin, X. Yao, J. Hu, F. Meng, C. Liu, X. Men, S. Yang, Z. Li, Z. Yang, Y. Du, and Kimi Team. Kimi Linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025.arxiv.org/abs/2510.26692

Paper details

vlnc:W_g56jqnhm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents