Introduction
Linear attention removes the quadratic compute and key-value (KV) cache of softmax attentions [15 , 63 ] by storing the context in a fixed-size state matrix S t ∈ R d k × d v S_t \in\mathbb{R}^{d_k \times d_v} S t ∈ R d k × d v (for key width d k d_k d k and value width d v d_v d v ), updated at each token t t t with its key k t ∈ R d k k_t \in\mathbb{R}^{d_k} k t ∈ R d k and value v t ∈ R d v v_t \in\mathbb{R}^{d_v} v t ∈ R d v as [33 ] :
S t = S t − 1 + k t v t ⊤ . S_t = S_{t-1} + k_t v_t^\top. S t = S t − 1 + k t v t ⊤ .
Because this sum grows without bound, the state saturates on long sequences. Gated models such as Mamba-2 [14 ] and GLA [66 ] add a data-dependent forget (decay) gate F t ∈ ( 0 , 1 ] d k × d k F_t \in(0, 1]^{d_k \times d_k} F t ∈ ( 0 , 1 ] d k × d k ,
S t = F t S t − 1 + k t v t ⊤ , S_t = F_t S_{t-1} + k_t v_t^\top, S t = F t S t − 1 + k t v t ⊤ ,
but decay fades all stored keys together and hurts associative recall [5 ] .
To overwrite individual keys selectively, DeltaNet [50 , 67 ] introduced the delta rule with unit-norm key k t k_t k t (∥ k t ∥ 2 = 1 \lVert k_t\rVert_2 = 1 ∥ k t ∥ 2 = 1 ), scalar write gate β t \beta_t β t , and d k × d k d_k \times d_k d k × d k identity I d k I_{d_k} I d k :
S t = ( I d k − β t k t k t ⊤ ) S t − 1 + β t k t v t ⊤ . S_t = (I_{d_k} - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top. S t = ( I d k − β t k t k t ⊤ ) S t − 1 + β t k t v t ⊤ .
Viewing such updates as online gradient descent, prior work also added heavy-ball momentum so that a single noisy update cannot overwrite established memories [9 , 44 ] . Combining decay with the delta rule produced Gated DeltaNet (GDN) [68 ] , adopted in Qwen3-Next [48 ] . Kimi Delta Attention (KDA) [71 ] then expanded GDN’s scalar forget gate into a per-channel diagonal gate F t = diag ( f t ) F_t = \operatorname{diag}(f_t) F t = diag ( f t )
Preprint. with channel-wise decay vector f t ∈ ( 0 , 1 ] d k f_t \in(0, 1]^{d_k} f t ∈ ( 0 , 1 ] d k , and GDN-2 [25 ] decoupled per-channel erase and write gates. § 5 surveys further variants.
However, these architectures and their chunkwise WY training algorithms [10 , 66 , 68 ] were developed and tuned on NVIDIA GPUs (fla [65 ] ). They map less well onto 2D systolic-array accelerators such as Google TPUs [18 , 31 , 32 ] , for three reasons:
Small tiles and unpacked heads. GPU Tensor Cores operate on small fragments, so d k = d v = 128 d_k = d_v = 128 d k = d v = 128 with one key head per value head already reaches high occupancy. The TPU v6e/v7x Matrix Multiply Unit (MXU) is a 256 × 256 systolic array: a 128-wide state pads half of every tile, and projections, convolutions, and chunk solves are repeated for every head.
Per-channel decay gates. In KDA and GDN-2, each key channel decays at its own rate. Kernels must rescale keys elementwise in 16-token sub-blocks to avoid float32 overflow. GPUs absorb this vector work cheaply; on TPUs, where matrix throughput exceeds vector throughput by roughly 128×, it stalls the kernel.
Sequential triangular solves. The chunkwise WY algorithm [10 , 68 ] partitions training sequences into chunks of length C C C and solves a C × C C \times C C × C unit lower-triangular system per head and chunk. Existing kernels [19 , 65 ] substitute sequentially within diagonal blocks; on TPUs this runs on the Vector Processing Unit (VPU) and takes about half of the core kernel time in our measurements.
Kernel tuning alone does not close this gap: fusing a GPU-shaped KDA layer into on-chip memory speeds it up by only 1.03× on TPU v7x (see Table 5 ), because the time goes to VPU solves and padded MXU tiles rather than High-Bandwidth Memory (HBM) traffic. We therefore change the layer itself. Systolic Linear Attention (SLA) applies a block-diagonal, tile-grouped forget gate F t ∈ R d k × d k F_t \in\mathbb{R}^{d_k \times d_k} F t ∈ R d k × d k to S t − 1 S_{t-1} S t − 1 and then performs the delta update with key k t ∈ R d k k_t \in\mathbb{R}^{d_k} k t ∈ R d k , packed value v t ∈ R R d v v_t \in\mathbb{R}^{Rd_v} v t ∈ R R d v (stacking R ∈ { 2 , 4 } R \in\{2, 4\} R ∈ { 2 , 4 } value heads per key head), and a write gate β t ∈ ( 0 , 2 ) \beta_t \in(0, 2) β t ∈ ( 0 , 2 ) shared across the packed value heads:
Table 5 : Seven-operator forward audit (ms) on TPU v7x (D = 2048 D = 2048 D = 2048 , T = 8,192 T = 8{,}192 T = 8 , 192 ). Unfused columns report isolated per-stage forward medians; fused brackets report single-kernel Ops 2–7a latency.
Table 5. Seven-operator forward audit (ms) on TPU v7x.
S t = F t S t − 1 + β t k t ( v t − ( F t S t − 1 ) T k t ) T . (1) S_t = F_t S_{t-1} + \beta_t k_t \left(v_t - (F_t S_{t-1})^\mathsf{T} k_t\right)^\mathsf{T}. \tag*{(1)} S t = F t S t − 1 + β t k t ( v t − ( F t S t − 1 ) T k t ) T . (1)
SLA also smooths chunk updates with a chunk-boundary momentum term with coefficient μ = 0.25 \mu= 0.25 μ = 0.25 that stays in on-chip SRAM (VMEM). SLA combines three changes (§ 2 ): Wide keys shared across value heads. SLA widens keys to d k = 256 d_k = 256 d k = 256 and shares each key head and gate β t \beta_t β t across R ∈ { 2 , 4 } R \in\{2,4\} R ∈ { 2 , 4 } value heads (SLA-R2, SLA-R4), following grouped value attention (GVA) [47 ] . The packed state fills whole MXU tiles, and key-query projections, convolutions, and chunk solves are computed R × R\times R × less often.
Tile-grouped forget gate. SLA splits the 256 key channels into 4 tiles of 64 channels, each with one scalar decay rate. Intra-chunk scores then become four 64-wide MXU matmuls scaled by decay masks, with no VPU rescaling. This is a middle point between GDN’s single rate per head and KDA’s rate per channel.
Chunk inversion on the MXU. Because β t \beta_t β t is shared, SLA needs one C × C C \times C C × C solve per key head. It computes the inverse only with block matrix products on the MXU: an exact recursive block-doubling inverse by default, or an optional 1-step Neumann approximation for faster solves, fused with the rest of the layer in VMEM.
We evaluate SLA on TPU v7x at the kernel, layer, serving, and context-parallel level, and across a 2,048-chip TPU v7x pre-training study of sparse Mixture-of-Experts (MoE) models [16 , 52 ] from 6.5B to 63.1B parameters on Dolma [54 , 60 ] (§ 3 ). We find that most of SLA’s speedup comes from head sharing and fusion, which the tile-grouped gate and MXU chunk inversion make possible. On matched seeds, SLA improves on standard KDA and stays within about 0.1 perplexity of a wide-key per-channel KDA control while avoiding per-channel long-context drift; the optional 1-step Neumann inverse trades about 0.2 perplexity for speed. SLA is therefore best understood as a hardware-efficient operating point rather than a more expressive update rule; it is complementary to modeling advances such as GDN-2, which reaches lower perplexity per token (§ 3.5 ).
In the following sections, § 2 reviews gated delta attention, its TPU bottlenecks, and SLA’s design; § 3 evaluates SLA from kernels and serving to MoE pre-training; Section 4 details optional low-precision key whitening; and § 5 covers related work.
Hardware Bottlenecks and the SLA Design
To motivate SLA, we first review gated delta recurrences and their chunkwise WY training algorithm (§ 2.1 ), identify three hardware bottlenecks that GPU-centric designs hit on TPUs (§ 2.2 ), and then present SLA’s mathematical formulation and chunk-boundary momentum (§ 2.3 ), its three algorithmic pillars and mathematical derivations (§ 2.4 ), and its single-kernel VMEM backward pass (§ 2.5 ), followed by empirical evaluation in § 3 .
Preliminaries: Gated Delta Recurrences and Chunkwise WY Parallelism
Given an input sequence X = ( x 1 , … , x T ) ∈ R T × D X=(x_1,\ldots,x_T)\in\mathbb{R}^{T\times D} X = ( x 1 , … , x T ) ∈ R T × D of length T T T and model width D D D , a gated-delta linear attention layer with H v H_v H v value heads and H k H_k H k query-key heads maps each token x t ∈ R D x_t\in\mathbb{R}^{D} x t ∈ R D (t ∈ { 1 , … , T } t\in\{1,\ldots,T\} t ∈ { 1 , … , T } ) to L 2 L_2 L 2 -normalized queries and keys q t , k t ∈ R d k q_t,k_t\in\mathbb{R}^{d_k} q t , k t ∈ R d k (∥ q t ∥ 2 = ∥ k t ∥ 2 = 1 \lVert q_t\rVert_2=\lVert k_t\rVert_2=1 ∥ q t ∥ 2 = ∥ k t ∥ 2 = 1 ), values v t ∈ R d v v_t\in\mathbb{R}^{d_v} v t ∈ R d v , and head output o t ∈ R d v o_t\in\mathbb{R}^{d_v} o t ∈ R d v (with H k = H v H_k=H_v H k = H v in unpacked baselines). Rather than caching t t t past tokens, each head maintains a fixed-size associative state matrix S t ∈ R d k × d v S_t\in\mathbb{R}^{d_k\times d_v} S t ∈ R d k × d v updated by applying a diagonal forget gate F t ∈ ( 0 , 1 ] d k × d k F_t\in(0,1]^{d_k\times d_k} F t ∈ ( 0 , 1 ] d k × d k followed by a rank-one delta update [9 , 50 , 59 , 67 ] with scalar step size β t \beta_t β t :
Prior Work: Gated Delta Recurrence in GDN and KDA
S t = F t S t − 1 + β t k t ( v t − ( F t S t − 1 ) ⊤ k t ) ⊤ = ( I d k − β t k t k t ⊤ ) F t S t − 1 + β t k t v t ⊤ , o t = S t ⊤ q t . (2) \begin{aligned}
S_t &= F_tS_{t-1}+\beta_t k_t\left(v_t-\left(F_tS_{t-1}\right)^\top k_t\right)^\top\\
&= \left(I_{d_k}-\beta_t k_tk_t^\top\right)F_tS_{t-1}+\beta_t k_tv_t^\top, \\
o_t &= S_t^\top q_t. \tag*{(2)}
\end{aligned} S t o t = F t S t − 1 + β t k t ( v t − ( F t S t − 1 ) ⊤ k t ) ⊤ = ( I d k − β t k t k t ⊤ ) F t S t − 1 + β t k t v t ⊤ , = S t ⊤ q t . (2)
Here ( F t S t − 1 ) ⊤ k t \left(F_tS_{t-1}\right)^\top k_t ( F t S t − 1 ) ⊤ k t is the value currently retrieved for key k t k_t k t , and β t \beta_t β t sets how far it is moved toward v t v_t v t . Existing linear attentions differ only in ( F t , β t ) (F_t,\beta_t) ( F t , β t ) in Eq. (2) , where we characterize F t F_t F t by its number of decay groups G G G (distinct decay rates across the d k d_k d k key channels of a head). Unweighted Linear Attention [33 ] sets F t = I d k F_t = I_{d_k} F t = I d k without erasure (S t = S t − 1 + k t v t ⊤ S_t = S_{t-1} + k_t v_t^\top S t = S t − 1 + k t v t ⊤ ), while DeltaNet [50 , 67 ] and test-time learning [59 , 64 ] retain F t = I d k F_t = I_{d_k} F t = I d k with the rank-one delta update. Recent gated-delta architectures instantiate ( F t , β t ) (F_t, \beta_t) ( F t , β t ) using the logistic sigmoid σ ( ⋅ ) \sigma(\cdot) σ ( ⋅ ) on scalar write-gate logit β ~ t ∈ R \tilde{\beta}_t \in\mathbb{R} β ~ t ∈ R and softplus( u ) = ln ( 1 + e u ) (u) = \ln(1 + e^u) ( u ) = ln ( 1 + e u ) on forget-gate logits f ~ t ∈ R G \tilde{f}_t \in\mathbb{R}^{G} f ~ t ∈ R G :
Gated DeltaNet (GDN). GDN [68 ] uses one scalar decay γ t ∈ ( 0 , 1 ] \gamma_t \in(0,1] γ t ∈ ( 0 , 1 ] per head (G = 1 G = 1 G = 1 ) together with a unit-interval scalar step size:
F t = γ t I d k , β t = σ ( β ~ t ) ∈ ( 0 , 1 ) . F_t = \gamma_t I_{d_k}, \qquad\beta_t = \sigma(\tilde{\beta}_t) \in(0,1). F t = γ t I d k , β t = σ ( β ~ t ) ∈ ( 0 , 1 ) .
Kimi Delta Attention (KDA). KDA [71 ] assigns an independent decay rate to every key channel (G = d k G = d_k G = d k , f t = exp ( − softplus ( f ~ t ) ) f_t = \exp(-\operatorname{softplus}(\tilde{f}_t)) f t = exp ( − softplus ( f ~ t )) ):
F t = diag ( f t ) , f t ∈ ( 0 , 1 ] d k . F_t = \operatorname{diag}(f_t), \qquad f_t \in(0,1]^{d_k}. F t = diag ( f t ) , f t ∈ ( 0 , 1 ] d k .
While standard GDN and KDA use β t = σ ( β ~ t ) ∈ ( 0 , 1 ) \beta_t = \sigma(\tilde{\beta}_t) \in(0,1) β t = σ ( β ~ t ) ∈ ( 0 , 1 ) , negative-eigenvalue extensions [20 ] widen the step-size range to
β t = 2 σ ( β ~ t ) ∈ ( 0 , 2 ) , \beta_t = 2\sigma(\tilde{\beta}_t) \in(0,2), β t = 2 σ ( β ~ t ) ∈ ( 0 , 2 ) ,
so that I d k − β t k t k t ⊤ I_{d_k} - \beta_t k_t k_t^\top I d k − β t k t k t ⊤ has eigenvalue 1 − β t ∈ ( − 1 , 1 ) 1 - \beta_t \in(-1,1) 1 − β t ∈ ( − 1 , 1 ) along k t k_t k t (adopted by SLA in § 2.3 ).
GDN-2. GDN-2 [25 ] retains per-channel decay (G = d k G = d_k G = d k ) and further replaces the scalar β t \beta_t β t with separate channel-wise erase and write gates.
Chunkwise WY parallelization. To saturate matrix units during training, chunkwise WY parallelization [10 , 67 , 68 ] partitions the sequence into T / C T/C T / C contiguous chunks of length C C C (Algorithm 1, written unified across sharing ratio R = H v / H k R = H_v/H_k R = H v / H k and gate scale s β ∈ { 1 , 2 } s_\beta\in\{1,2\} s β ∈ { 1 , 2 } , where unpacked GDN/KDA set R = 1 R = 1 R = 1 , s β = 1 s_\beta= 1 s β = 1 ). Op 1 projects X X X to pre-convolution activations Q ~ , K ~ , V ~ \tilde{Q}, \tilde{K}, \tilde{V} Q ~ , K ~ , V ~ , gate logits ( β ~ t , f ~ t ) t = 1 T (\tilde{\beta}_t,\tilde{f}_t)_{t=1}^{T} ( β ~ t , f ~ t ) t = 1 T , and output-gate logits ( z t ) t = 1 T (z_t)_{t=1}^{T} ( z t ) t = 1 T (z t ∈ R R d v z_t \in\mathbb{R}^{R d_v} z t ∈ R R d v per key head), followed by causal short convolutions, L 2 L_2 L 2 normalization into Q Q Q , K K K , and gate activations (Op 2). Within chunk c ∈ { 1 , … , T / C } c \in\{1,\ldots,T/C\} c ∈ { 1 , … , T / C } starting from boundary state S c − 1 ∈ R d k × R d v S_{c-1} \in\mathbb{R}^{d_k \times R d_v} S c − 1 ∈ R d k × R d v (S 0 = 0 S_0 = 0 S 0 = 0 ), let local token indices satisfy $1 \le j \le t \le C$, let Q c , K c ∈ R C × d k Q_c, K_c \in\mathbb{R}^{C \times d_k} Q c , K c ∈ R C × d k and V c , O c ∈ R C × R d v V_c, O_c \in\mathbb{R}^{C \times R d_v} V c , O c ∈ R C × R d v stack the chunk’s queries, keys, values, and readouts as rows, let β c = ( β 1 , … , β C ) ⊤ ∈ R C \beta_c = (\beta_1,\ldots,\beta_C)^\top\in\mathbb{R}^{C} β c = ( β 1 , … , β C ) ⊤ ∈ R C stack its write gates, and let I C I_C I C denote the C × C C \times C C × C identity. Define
P t ← j = ∏ s = j + 1 t F s ( P t ← t = I d k ) , F c = P C ← 0 = ∏ s = 1 C F s P_{t\leftarrow j} = \prod_{s=j+1}^{t} F_s \quad(P_{t\leftarrow t} = I_{d_k}), \qquad F_c = P_{C\leftarrow0} = \prod_{s=1}^{C} F_s P t ← j = s = j + 1 ∏ t F s ( P t ← t = I d k ) , F c = P C ← 0 = s = 1 ∏ C F s
as the cumulative diagonal decay from local token j j j to t t t and the total chunk decay, and let K ˉ c , Q ˉ c , K ^ c ∈ R C × d k \bar{K}_c,\bar{Q}_c,\hat{K}_c \in\mathbb{R}^{C \times d_k} K ˉ c , Q ˉ c , K ^ c ∈ R C × d k denote the decayed key and query matrices whose t t t -th rows are ( P t ← 0 k t ) ⊤ (P_{t\leftarrow0}k_t)^\top ( P t ← 0 k t ) ⊤ , ( P t ← 0 q t ) ⊤ (P_{t\leftarrow0}q_t)^\top ( P t ← 0 q t ) ⊤ , and ( P C ← t k t ) ⊤ (P_{C\leftarrow t}k_t)^\top ( P C ← t k t ) ⊤ . Within a chunk, each token’s write depends on what earlier tokens in the same chunk have already written, so the C C C writes cannot be computed independently. Collect them as rows of E c ∈ R C × R d v E_c \in\mathbb{R}^{C \times R d_v} E c ∈ R C × R d v . Each row is what the token would write on its own, diag ( β c ) ( V c − K ˉ c S c − 1 ) \operatorname{diag}(\beta_c)(V_c - \bar{K}_c S_{c-1}) diag ( β c ) ( V c − K ˉ c S c − 1 ) , minus the earlier tokens’ writes, weighted by how strongly their decayed keys overlap with its own key. Using indicator 1 [ ⋅ ] ∈ { 0 , 1 } 1[\cdot] \in\{0,1\} 1 [ ⋅ ] ∈ { 0 , 1 } , these overlaps define the strictly lower-triangular Gram matrix M c ∈ R C × C M_c \in\mathbb{R}^{C \times C} M c ∈ R C × C (Op 3a, alongside lower-triangular intra-chunk query–key score matrix L c ∈ R C × C L_c \in\mathbb{R}^{C \times C} L c ∈ R C × C , Op 3b), so the chunk writes satisfy (derived in § 2.4.2 ):
E c = diag ( β c ) ( I C + M c ) − 1 ( V c − K ˉ c S c − 1 ) . E_c = \operatorname{diag}(\beta_c)(I_C + M_c)^{-1}(V_c - \bar{K}_c S_{c-1}). E c = diag ( β c ) ( I C + M c ) − 1 ( V c − K ˉ c S c − 1 ) .
Inverting A c = ( I C + M c ) − 1 A_c = (I_C + M_c)^{-1} A c = ( I C + M c ) − 1 (Op 4) and forming the WY factors U c , W c U_c,W_c U c , W c (Op 5) runs in parallel across chunks, leaving a sequential scan over the T / C T/C T / C chunk boundaries for S c S_c S c and optional velocity buffer Z c ∈ R d k × R d v Z_c \in\mathbb{R}^{d_k \times R d_v} Z c ∈ R d k × R d v (Z 0 = 0 Z_0 = 0 Z 0 = 0 , momentum μ ≥ 0 \mu\ge0 μ ≥ 0 ; Op 6), parallel chunk readouts O c O_c O c (rows o t o_t o t ; Op 7a), and gated root-mean-square normalization (RMSNorm, with elementwise product ⊙ \odot ⊙ ) followed by the output projection to Y ∈ R T × D Y \in\mathbb{R}^{T \times D} Y ∈ R T × D (Op 7b–c).
Three Hardware Bottlenecks on TPUs
TPUs pair a wide systolic MXU with a much slower VPU. On TPU v7x the MXU tile is 256 × 256 256 \times256 256 × 256 and the MXU:VPU throughput ratio is about 128 × 128\times 128 × , compared with about 15 × 15\times 15 × on H100 (Table 1 ). This skew creates three bottlenecks for Algorithm 1:
Table 1. Documented hardware specifications per chip for NVIDIA H100 SXM and four Google TPU generations [6 , 18 , 32 ] . MXU:VPU is the ratio of peak BF16 matrix throughput to FP32 vector throughput; Ridge is the BF16 arithmetic-intensity ridge point (FLOPs/byte). * 8-bit peak is FP8 on H100 and TPU v7x and INT8 on TPU v4–v6e.
Algorithm 1 Chunkwise WY Parallel Forward Pass
Require: Sequence X ∈ R T × D X \in\mathbb{R}^{T \times D} X ∈ R T × D , chunk size C C C , head ratio R R R , gate scale s β s_{\beta} s β , momentum μ \mu μ (( R , s β , μ ) = ( 1 , 1 , 0 ) (R, s_{\beta}, \mu) = (1, 1, 0) ( R , s β , μ ) = ( 1 , 1 , 0 ) for GDN/KDA; ( R , 2 , 0.25 ) (R, 2, 0.25) ( R , 2 , 0.25 ) with R ∈ { 2 , 4 } R \in\{2,4\} R ∈ { 2 , 4 } for SLA)
Ensure: Output Y ∈ R T × D Y \in\mathbb{R}^{T \times D} Y ∈ R T × D , final state S T / C ∈ R d k × R d v S_{T/C} \in\mathbb{R}^{d_k \times R d_v} S T / C ∈ R d k × R d v
Q ~ , K ~ , V ~ , ( β ~ t , f ~ t , z t ) t = 1 T ← LinearProj ( X ) \tilde{Q}, \tilde{K}, \tilde{V}, (\tilde{\beta}_t, \tilde{f}_t, z_t)_{t=1}^{T} \leftarrow\operatorname{LinearProj}(X) Q ~ , K ~ , V ~ , ( β ~ t , f ~ t , z t ) t = 1 T ← LinearProj ( X ) ▷ Op 1 (MXU)
Q , K ← L2Norm ( Conv1D ( Q ~ , K ~ ) ) Q, K \leftarrow\operatorname{L2Norm}(\operatorname{Conv1D}(\tilde{Q}, \tilde{K})) Q , K ← L2Norm ( Conv1D ( Q ~ , K ~ )) ▷ Op 2a (VPU)
V ← SiLU ( Conv1D ( V ~ ) ) V \leftarrow\operatorname{SiLU}(\operatorname{Conv1D}(\tilde{V})) V ← SiLU ( Conv1D ( V ~ ))
β t ← s β σ ( β ~ t ) ; F t ← diag ( e − softplus ( f ~ t ) ) \beta_t \leftarrow s_{\beta}\sigma(\tilde{\beta}_t); \ F_t \leftarrow\operatorname{diag}(e^{-\operatorname{softplus}(\tilde{f}_t)}) β t ← s β σ ( β ~ t ) ; F t ← diag ( e − softplus ( f ~ t ) ) ▷ Op 2b (VPU)
Partition Q , K , V , β , F Q, K, V, \beta, F Q , K , V , β , F into T / C T/C T / C chunks; S 0 ← 0 S_0 \leftarrow0 S 0 ← 0 , Z 0 ← 0 ∈ R d k × R d v Z_0 \leftarrow0 \in\mathbb{R}^{d_k \times R d_v} Z 0 ← 0 ∈ R d k × R d v
for all c ∈ { 1 , … , T / C } c \in\{1,\ldots,T/C\} c ∈ { 1 , … , T / C } in parallel do
P t ← j ← ∏ s = j + 1 t F s ∀ 0 ≤ j ≤ t ≤ C ( P t ← t = I d k ) P_{t\leftarrow j} \leftarrow\prod_{s=j+1}^{t} F_s \quad\forall\ 0 \leq j \leq t \leq C \quad(P_{t\leftarrow t}=I_{d_k}) P t ← j ← ∏ s = j + 1 t F s ∀ 0 ≤ j ≤ t ≤ C ( P t ← t = I d k )
( K ˉ c , Q ˉ c , K ^ c ) t ← P t ← 0 k t , P t ← 0 q t , P C ← t k t (\bar{K}_c,\bar{Q}_c,\hat{K}_c)_t \leftarrow P_{t\leftarrow0}k_t, P_{t\leftarrow0}q_t, P_{C\leftarrow t}k_t ( K ˉ c , Q ˉ c , K ^ c ) t ← P t ← 0 k t , P t ← 0 q t , P C ← t k t ▷ each row t t t
( M c ) t , j ← 1 [ t > j ] β j k t T P t ← j k j (M_c)_{t,j} \leftarrow\mathbf{1}[t>j]\beta_j k_t^{\mathsf{T}}P_{t\leftarrow j}k_j ( M c ) t , j ← 1 [ t > j ] β j k t T P t ← j k j ▷ Op 3a: Gram
( L c ) t , j ← 1 [ t ≥ j ] q t T P t ← j k j (L_c)_{t,j} \leftarrow\mathbf{1}[t\geq j]q_t^{\mathsf{T}}P_{t\leftarrow j}k_j ( L c ) t , j ← 1 [ t ≥ j ] q t T P t ← j k j ▷ Op 3b: Score
A c ← ( I C + M c ) − 1 ∈ R C × C A_c \leftarrow(I_C+M_c)^{-1} \in\mathbb{R}^{C \times C} A c ← ( I C + M c ) − 1 ∈ R C × C ▷ Op 4: Solve
U c ← diag ( β c ) A c V c ∈ R C × R d v U_c \leftarrow\operatorname{diag}(\beta_c)A_cV_c \in\mathbb{R}^{C \times R d_v} U c ← diag ( β c ) A c V c ∈ R C × R d v ▷ Op 5a (MXU)
W c ← diag ( β c ) A c K ˉ c ∈ R C × d k W_c \leftarrow\operatorname{diag}(\beta_c)A_c\bar{K}_c \in\mathbb{R}^{C \times d_k} W c ← diag ( β c ) A c K ˉ c ∈ R C × d k ▷ Op 5b (MXU)
end for
for c = 1 c=1 c = 1 to T / C T/C T / C do
E c ← U c − W c S c − 1 E_c \leftarrow U_c-W_cS_{c-1} E c ← U c − W c S c − 1 ▷ Op 6a (MXU)
Z c ← μ Z c − 1 + K ^ c T E c Z_c \leftarrow\mu Z_{c-1}+\hat{K}_c^{\mathsf{T}}E_c Z c ← μ Z c − 1 + K ^ c T E c ▷ Op 6b (MXU)
S c ← F c S c − 1 + Z c S_c \leftarrow F_cS_{c-1}+Z_c S c ← F c S c − 1 + Z c
end for
for all c ∈ { 1 , … , T / C } c \in\{1,\ldots,T/C\} c ∈ { 1 , … , T / C } in parallel do
O c ← Q ˉ c S c − 1 + L c E c O_c \leftarrow\bar{Q}_cS_{c-1}+L_cE_c O c ← Q ˉ c S c − 1 + L c E c ▷ Op 7a (MXU)
end for
Y ← OutProj ( ( σ ( z t ) ⊙ RMSNorm ( o t ) ) t = 1 T ) Y \leftarrow\operatorname{OutProj}((\sigma(z_t)\odot\operatorname{RMSNorm}(o_t))_{t=1}^{T}) Y ← OutProj (( σ ( z t ) ⊙ RMSNorm ( o t ) ) t = 1 T ) ▷ Op 7b–c
return Y , S T / C Y, S_{T/C} Y , S T / C
Bottleneck 1: Narrow states and a fixed per-step scan cost. Op 6 carries the state across T / C T/C T / C chunk boundaries sequentially. On TPU v7x, each scan step costs a nearly fixed ≈ 12 μ s \approx12\ \mu\mathrm{s} ≈ 12 μ s regardless of chunk size: 8 × 8\times 8 × more FLOPs per step adds under 1% time (Figure 2 ). Scan cost is thus set by the number of steps and heads, not by matmul size. A 256-wide tile also processes a d k = 128 d_k=128 d k = 128 state at the same cost as a d k = 256 d_k=256 d k = 256 state, so 128-wide states leave half of each tile idle while paying the full per-step cost for every head.
Figure 2. Sequential state-scan cost on TPU v7x. A 64-step scan of ( C , 256 ) × ( 256 , 256 ) (C, 256) \times(256, 256) ( C , 256 ) × ( 256 , 256 ) matmuls takes nearly constant time for C ∈ { 32 , … , 256 } C \in\{32,\ldots,256\} C ∈ { 32 , … , 256 } , so a fixed per-step overhead, not matmul FLOPs, dominates the scan.
Bottleneck 2: Per-channel decay. KDA [71 ] and GDN-2 [25 ] give every key channel its own forget gate ( F t = diag ( f t ) ) (F_t=\operatorname{diag}(f_t)) ( F t = diag ( f t )) . Let Λ t , i = ∑ s = 1 t log f s , i \Lambda_{t,i}=\sum_{s=1}^{t}\log f_{s,i} Λ t , i = ∑ s = 1 t log f s , i be the cumulative decay of channel i i i within a chunk. Each Gram entry then weights channels differently:
( M c ) t , j = 1 [ t > j ] β j ∑ i e Λ t , i − Λ j , i k t , i k j , i . (M_c)_{t,j}=\mathbf{1}[t>j]\beta_j\sum_i e^{\Lambda_{t,i}-\Lambda_{j,i}}k_{t,i}k_{j,i}. ( M c ) t , j = 1 [ t > j ] β j i ∑ e Λ t , i − Λ j , i k t , i k j , i .
To use the MXU, kernels fold e Λ t , i e^{\Lambda_{t,i}} e Λ t , i and e − Λ j , i e^{-\Lambda_{j,i}} e − Λ j , i into the keys, but e − Λ j , i e^{-\Lambda_{j,i}} e − Λ j , i overflows FP32 for long chunks. KDA kernels therefore split each chunk into 16-token sub-blocks and re-center the keys on the VPU before every sub-block matmul, which stalls the kernel. In our KDA runs, about 1% of channels also learn near-unit forget gates, which hurts length extrapolation unless the gates are clamped (§ 3.5 ). GDN’s scalar gate (G = 1 G = 1 G = 1 ) avoids both issues, at the cost of a single memory horizon per head.
Bottleneck 3: Sequential triangular solves. Existing kernels [19 , 65 ] compute A c = ( I C + M c ) − 1 A_c = (I_C + M_c)^{-1} A c = ( I C + M c ) − 1 (Op 4) by sequential substitution within small diagonal blocks, which runs on the VPU. Repeated for every chunk and each of H k = H v = 8 H_k = H_v = 8 H k = H v = 8 heads, together with the Gram matrices (Ops 3–5) it accounts for about 45% of the core intra-chunk and scan time in our measurements (Table 5 ), keeping the best chunk size near 64–128.
SLA: Addressing the Three Bottlenecks
Guided by the three systolic bottlenecks above, we now formulate SLA, which reshapes the state geometry, decay structure, and chunk inversion around the 256 × 256 256 \times256 256 × 256 systolic tile:
Definition 2.1 (SLA Recurrence). Consider a linear attention layer with H v H_v H v value heads of width d v d_v d v and H k = H v / R H_k = H_v/R H k = H v / R query-key heads of width d k = 256 d_k = 256 d k = 256 indexed by h k ∈ { 1 , … , H k } h_k \in\{1, \ldots, H_k\} h k ∈ { 1 , … , H k } , where each query-key head is shared across R ∈ { 2 , 4 } R \in\{2, 4\} R ∈ { 2 , 4 } value heads. At token t t t , each query-key head computes:
Shared query, key, and write gate: L 2 L_2 L 2 -normalized q t , k t ∈ R 256 q_t, k_t \in\mathbb{R}^{256} q t , k t ∈ R 256 with ∥ q t ∥ 2 = ∥ k t ∥ 2 = 1 \lVert q_t\rVert_2 = \lVert k_t\rVert_2 = 1 ∥ q t ∥ 2 = ∥ k t ∥ 2 = 1 , and a scalar step size β t = 2 σ ( β ~ t ) ∈ ( 0 , 2 ) \beta_t = 2\sigma(\widetilde{\beta}_t) \in(0, 2) β t = 2 σ ( β t ) ∈ ( 0 , 2 ) (s β = 2 s_\beta= 2 s β = 2 ) that doubles the ( 0 , 1 ) (0, 1) ( 0 , 1 ) range of standard GDN/KDA;
Packed values and output gates: R R R value vectors v t , 1 , … , v t , R ∈ R d v v_{t,1}, \ldots, v_{t,R} \in\mathbb{R}^{d_v} v t , 1 , … , v t , R ∈ R d v stacked as v t = [ v t , 1 ; … ; v t , R ] ∈ R R d v v_t = [v_{t,1}; \ldots; v_{t,R}] \in\mathbb{R}^{Rd_v} v t = [ v t , 1 ; … ; v t , R ] ∈ R R d v , and output-gate vector z t = [ z t , 1 ; … ; z t , R ] ∈ R R d v z_t = [z_{t,1}; \ldots; z_{t,R}] \in\mathbb{R}^{Rd_v} z t = [ z t , 1 ; … ; z t , R ] ∈ R R d v with per-value-head slices z t , m ∈ R d v z_{t,m} \in\mathbb{R}^{d_v} z t , m ∈ R d v (m ∈ { 1 , … , R } m \in\{1, \ldots, R\} m ∈ { 1 , … , R } );
Four-tile forget gate: a block-diagonal gate F t ∈ ( 0 , 1 ] 256 × 256 F_t \in(0,1]^{256 \times256} F t ∈ ( 0 , 1 ] 256 × 256 that splits the 256 key channels into G = 4 G = 4 G = 4 contiguous tiles of b = 64 b = 64 b = 64 channels, with forget-gate logit f ~ t = ( f ~ t , 1 , … , f ~ t , 4 ) ∈ R 4 \widetilde{f}_t = (\widetilde{f}_{t,1}, \ldots, \widetilde{f}_{t,4}) \in\mathbb{R}^4 f t = ( f t , 1 , … , f t , 4 ) ∈ R 4 , all-ones vector 1 64 ∈ R 64 1_{64} \in\mathbb{R}^{64} 1 64 ∈ R 64 , and one scalar decay γ t , g = exp ( − softplus ( f ~ t , g ) ) ∈ ( 0 , 1 ) \gamma_{t,g} = \exp(-\operatorname{softplus}(\widetilde{f}_{t,g})) \in(0,1) γ t , g = exp ( − softplus ( f t , g )) ∈ ( 0 , 1 ) on each tile g ∈ { 1 , 2 , 3 , 4 } g \in\{1, 2, 3, 4\} g ∈ { 1 , 2 , 3 , 4 } :
F t = diag ( γ t , 1 1 64 , γ t , 2 1 64 , γ t , 3 1 64 , γ t , 4 1 64 ) . (3) F_t = \operatorname{diag}(\gamma_{t,1}1_{64}, \gamma_{t,2}1_{64}, \gamma_{t,3}1_{64}, \gamma_{t,4}1_{64}). \tag*{(3)} F t = diag ( γ t , 1 1 64 , γ t , 2 1 64 , γ t , 3 1 64 , γ t , 4 1 64 ) . (3)
(i) Intra-chunk token recurrence. Maintaining a packed state S t = [ S t , 1 , … , S t , R ] ∈ R 256 × R d v S_t = [S_{t,1}, \ldots, S_{t,R}] \in\mathbb{R}^{256 \times Rd_v} S t = [ S t , 1 , … , S t , R ] ∈ R 256 × R d v initialized at S 0 = 0 S_0 = 0 S 0 = 0 , with slice S t , m ∈ R 256 × d v S_{t,m} \in\mathbb{R}^{256 \times d_v} S t , m ∈ R 256 × d v and output slice o t , m ∈ R d v o_{t,m} \in\mathbb{R}^{d_v} o t , m ∈ R d v (o t = [ o t , 1 ; … ; o t , R ] o_t = [o_{t,1}; \ldots; o_{t,R}] o t = [ o t , 1 ; … ; o t , R ] ) for value head m ∈ { 1 , … , R } m \in\{1, \ldots, R\} m ∈ { 1 , … , R } , SLA decays the state, writes the prediction error, and normalizes each value head:
Figure 4. SLA Token-Level Recurrence (d k = 256 d_k = 256 d k = 256 , G = 4 G = 4 G = 4 , b = 64 b = 64 b = 64 ).
Here ( F t S t − 1 ) ⊤ k t (F_tS_{t-1})^\top k_t ( F t S t − 1 ) ⊤ k t is the value currently stored for k t k_t k t , so Eq. (4) writes only the prediction error, and one rank-one write updates all R R R value heads.
(ii) Chunk-boundary momentum. Let S ~ c \widetilde{S}_c S c be the state after unrolling Eq. (4) over the C C C tokens of chunk c c c from S c − 1 S_{c-1} S c − 1 , and let F c = ∏ t ∈ c F t F_c = \prod_{t \in c} F_t F c = ∏ t ∈ c F t . SLA updates the chunk-boundary state S c S_c S c via a velocity buffer Z c ∈ R 256 × R d v Z_c \in\mathbb{R}^{256 \times Rd_v} Z c ∈ R 256 × R d v reset to zero at document boundaries:
Z c = μ Z c − 1 + ( S ~ c − F c S c − 1 ) , S c = F c S c − 1 + Z c , (4) Z_c = \mu Z_{c-1} + (\widetilde{S}_c - F_c S_{c-1}), \qquad S_c = F_c S_{c-1} + Z_c, \tag*{(4)} Z c = μ Z c − 1 + ( S c − F c S c − 1 ) , S c = F c S c − 1 + Z c , (4)
with μ = 0.25 \mu= 0.25 μ = 0.25 by default, while μ = 0 \mu= 0 μ = 0 recovers S c = S ~ c S_c = \widetilde{S}_c S c = S c .
Notation and operator taxonomy. Unless redefined locally, symbols follow Algorithm 1 and Definition 2.1 ; Table 2 collects them for quick reference (symbols used only inside one section, such as the roofline constants in § 3.1 , are defined where they first appear). Table 3 contrasts the algebraic operators and systolic hardware mappings of GDN [68 ] , KDA [71 ] , GDN-2 [25 ] , SLA (SLA-R2 / SLA-R4), and SPD-SLA-W, an optional extension of SLA that preconditions chunks with Σ c \Sigma_c Σ c to form whitened keys K ~ c \widetilde{K}_c K c (rows k ~ t ⊤ \widetilde{k}_t^\top k t ⊤ ) under step-size margin ε β > 0 \varepsilon_\beta> 0 ε β > 0 (Section 4); SPD-SLA-W is not used in any core SLA result except the middle bar of Figure 7 a and the conditioning analysis of Section 4.
Table 2. Summary of notation used across the paper.
Table 3. Side-by-side algebraic and systolic hardware comparison across prior GPU gated-delta recurrences and our SLA family.
Figure 7. (a) Per-step XProf breakdown for 63.1B MoE on 64 TPU v7x chips across KDA Base, SPD-SLA-W (R2) (SLA-R2 with optional in-chunk key whitening, Section 4), and SLA-R4 (Table 13); most of SLA’s saving is in linear attention ops. (b) Held-out Dolma tail perplexity from 4K to 32K for 12.5B MoEs (Table 11a; seed s0). (c) Zero-shot 8K → 64K RULER HotpotQA for KDA (raw and clamped) and SLA (60.4B-token curves: seed s0; 15.1B-token curves: mean of s0 and s1).
Why chunk-level momentum, implementation, and caveats. Unlike token-level momentum [9 ] , applying heavy-ball momentum [44 ] once per chunk boundary (Op 6b of Algorithm 1) leaves the intra-chunk WY solve unchanged. Specifically, in Op 6b, the net chunk state increment in Eq. (4) is
Δ S c = S ~ c − F c S c − 1 = K ^ c ⊤ E c . (5) \Delta S_c = \widetilde{S}_c - F_cS_{c-1} = \widehat{K}_c^\top E_c. \tag*{(5)} Δ S c = S c − F c S c − 1 = K c ⊤ E c . (5)
The velocity buffer Z c ∈ R d k × D v , p a c k Z_c \in\mathbb{R}^{d_k \times D_{v,\mathrm{pack}}} Z c ∈ R d k × D v , pack has the same shape as S c S_c S c and stays in VMEM inside the fused kernel, adding zero extra HBM state traffic. On packed multi-document streams, intra-chunk Gram and score matrices apply an exact segment mask 1 [ s e g t = s e g j ] \mathbf{1}[\mathrm{seg}_t = \mathrm{seg}_j] 1 [ seg t = seg j ] , only the final document segment writes to Δ S c \Delta S_c Δ S c , and any document reset anywhere inside chunk c c c zeros both incoming state carry F c F_c F c and κ c = 0 \kappa_c = 0 κ c = 0 , preventing mid-chunk velocity leakage. Moreover, for any eigenmode with chunk decay f ∈ ( 0 , 1 ) f \in(0,1) f ∈ ( 0 , 1 ) and generalized WY survival j ∈ [ − f , f ] j \in[-f,f] j ∈ [ − f , f ] (including β t ∈ ( 0 , 2 ) \beta_t \in(0,2) β t ∈ ( 0 , 2 ) ), the lifted 2 × 2 2 \times2 2 × 2 companion block [ j μ j − f μ ] \begin{bmatrix} j & \mu\\ j-f & \mu\end{bmatrix} [ j j − f μ μ ] has characteristic polynomial p ( λ ) = λ 2 − ( j + μ ) λ + μ f p(\lambda) = \lambda^2 - (j+\mu)\lambda+ \mu f p ( λ ) = λ 2 − ( j + μ ) λ + μ f (j μ j\mu j μ cancels in det = μ f \det= \mu f det = μ f ), satisfying Jury’s Schur stability conditions ∣ det ∣ = μ f < 1 |\det| = \mu f < 1 ∣ det ∣ = μ f < 1 , p ( 1 ) ≥ ( 1 − f ) ( 1 − μ ) > 0 p(1) \ge(1-f)(1-\mu) > 0 p ( 1 ) ≥ ( 1 − f ) ( 1 − μ ) > 0 , and p ( − 1 ) = 1 + j + μ ( 1 + f ) > 0 p(-1) = 1+j+\mu(1+f) > 0 p ( − 1 ) = 1 + j + μ ( 1 + f ) > 0 unconditionally for all μ ∈ [ 0 , 1 ) \mu\in[0,1) μ ∈ [ 0 , 1 ) . Because Z c Z_c Z c changes only at chunk boundaries, autoregressive decoding reproduces training exactly by updating Z c Z_c Z c once every C C C tokens. Under context parallelism, momentum couples ( S c , Z c ) (S_c,Z_c) ( S c , Z c ) into a 2 d k × 2 d k 2d_k \times2d_k 2 d k × 2 d k affine map (Eq. (12) ); Table 8 therefore reports both the uncoupled (μ = 0 \mu=0 μ = 0 , 256 2 256^2 25 6 2 ) and coupled (μ = 0.25 \mu=0.25 μ = 0.25 , 512 2 512^2 51 2 2 ) tree scans.
Table 8. Single-chip 32-layer prefill latency (a), 32-layer attention-layer serving throughput with speculative verification of T v e r ∈ { 8 , 16 } T_{\mathrm{ver}} \in\{8, 16\} T ver ∈ { 8 , 16 } drafted tokens (b; OOM > 190 > 190 > 190 GiB/chip), and context-parallel F+B scan latency across P ∈ { 32 , 64 , 128 } P \in\{32, 64, 128\} P ∈ { 32 , 64 , 128 } TensorCores (c) on TPU v7x (D = 2048 D = 2048 D = 2048 ).
Table 4 compares μ = 0 \mu=0 μ = 0 and μ = 0.25 \mu=0.25 μ = 0.25 on a 12.5B-total (0.79B-active) MoE trained on 15.1B Dolma 3.5 tokens (7,200 steps, 2.1M tokens/step, 128 TPU v4 chips, seed 42). With 4:1 key sharing (R = 4 R=4 R = 4 ), momentum lowers loss by 0.019–0.023 nats and matches KDA, recovering the capacity lost by sharing each key head across four value heads; with 2:1 sharing (R = 2 R=2 R = 2 ), it changes loss by at most +0.004 nats. An input-dependent μ \mu μ obtained by reusing the β \beta β projection degrades loss by +0.026 nats, so we keep μ \mu μ fixed. We did not sweep μ \mu μ beyond { 0 , 0.25 } \{0,0.25\} { 0 , 0.25 } . In the TPU v7x ablation ladder (Table 11 b), momentum is introduced together with VMEM fusion in A5, so its effect is not isolated there. Moreover, the ablation evidence is from a single seed, and momentum has costs: the recurrence now depends on the chunk size C C C , decoding must track chunk boundaries, and context parallelism must carry a 2 × 2\times 2 × larger transition state (§ 3.3 ).
Table 11.
The Three Algorithmic Pillars of SLA
Each pillar targets one of the bottlenecks in § 2.2 (Figure 1 c–Figure 1 e; detailed diagrams in Figure 3 ).
Figure 1. Overview of SLA. (a) Direct ports of GPU-tuned gated-delta layers (GDN [68 ] , KDA [71 ] , GDN-2 [25 ] ) run as an unfused pipeline on TPUs: intermediates round-trip through HBM, per-channel decay rescaling in KDA and GDN-2 and in-block triangular substitution run on the VPU, and 128-wide states leave at least half of each 256 × 256 MXU tile padded. (b) SLA fuses stages 2–5 into one VMEM kernel. (c) A head with key and value widths d k = d v = 128 d_k = d_v = 128 d k = d v = 128 fills a quarter of an MXU tile, while SLA’s wide packed state fills whole tiles. (d) Decay gates are grouped into four 64-channel tiles, so intra-chunk scores are four 64-wide MXU matmuls with no VPU rescaling. (e) The chunk solve uses only block matrix products on the MXU.
Figure 3. Detailed architectural diagram of the three mathematical pillars of SLA. (a) Pillar 1: Widening the key dimension to d k = 256 d_k = 256 d k = 256 and sharing each key head h k h_k h k and write gate β t ∈ ( 0 , 2 ) \beta_t \in(0, 2) β t ∈ ( 0 , 2 ) across R ∈ { 2 , 4 } R \in\{2, 4\} R ∈ { 2 , 4 } value heads forms a dense 256 × D v , p a c k 256 \times D_{v,\mathrm{pack}} 256 × D v , pack state that fills whole 256 × 256 256 \times256 256 × 256 MXU tiles without padding while cutting QK projections, short convolutions, and chunk solves by R × R \times R × . (b) Pillar 2: Partitioning the d k = 256 d_k = 256 d k = 256 key channels into G = 4 G = 4 G = 4 contiguous tiles of b = 64 b = 64 b = 64 channels decomposes the intra-chunk coupling matrix M c M_c M c into four 64-wide MXU matrix multiplications weighted by scalar decay masks Γ g \Gamma_g Γ g , eliminating VPU sub-block rescaling. (c) Pillar 3: The default Exact Inv. computes ( I C + M c ) − 1 (I_C + M_c)^{-1} ( I C + M c ) − 1 by recursive block doubling on the MXU; the optional 1-step Neumann inverse keeps only the first off-diagonal term of I C + M c = D b l k ( I C + N ) I_C + M_c = D_{\mathrm{blk}}(I_C + N) I C + M c = D blk ( I C + N ) with 16 × 16 16 \times16 16 × 16 diagonal blocks, where N 8 = 0 N^8 = 0 N 8 = 0 at C = 128 C = 128 C = 128 . Both avoid sequential VPU substitutions and use dense MXU matrix multiplications (see § 2.4.3 ).
Table 4. Chunk-boundary momentum ablation: 20-step rolling training cross-entropy at step 7,199 (single pass over Dolma 3.5, 12.5B MoE, 128 TPU v4 chips, seed 42; lower is better). † \dagger † Trained on an 8,200-step cosine schedule and evaluated at step 7,199.
Pillar 1: Wide Keys with Grouped Value Heads
A key dimension below 256 still occupies a full 256-wide tile on TPU v6e/v7x, so SLA widens keys to d k = 256 d_k = 256 d k = 256 , which costs no extra cycles in the inter-chunk scan. To keep parameters and non-matmul work in check, SLA adopts linear grouped-value head sharing (GVA) [48 , 65 , 71 ] together with gated output normalization [47 ] : each query-key head and gate β t \beta_t β t is shared across R = H v / H k ∈ { 2 , 4 } R = H_v/H_k \in\{2, 4\} R = H v / H k ∈ { 2 , 4 } value heads. Packing these value heads side by side gives a state S t ∈ R d k × R d v S_t \in\mathbb{R}^{d_k \times R d_v} S t ∈ R d k × R d v that maps onto whole MXU tiles without padding. Query and key projections, short convolutions, M c M_c M c , L c L_c L c , and the solve A c A_c A c are computed once per key head, cutting their cost by R × R \times R × . Each value head’s output is normalized separately (eq:5 ). Head sharing itself is prior work; our contribution is choosing d k d_k d k and R R R to match the systolic tile.
Derivation of GVA systolic invariance. Fix total value width D v = H v d v D_v = H_vd_v D v = H v d v and set H k = H v / R H_k = H_v/R H k = H v / R with D v , p a c k = R d v D_{v,\mathrm{pack}} = Rd_v D v , pack = R d v . Each key head h k ∈ { 1 , … , H k } h_k \in\{1, \ldots, H_k\} h k ∈ { 1 , … , H k } maintains state S ( h k ) ∈ R d k × D v , p a c k S^{(h_k)} \in\mathbb{R}^{d_k \times D_{v,\mathrm{pack}}} S ( h k ) ∈ R d k × D v , pack , so total state elements are invariant to R R R :
H k d k ( R d v ) = H v d k d v . H_kd_k(Rd_v) = H_vd_kd_v. H k d k ( R d v ) = H v d k d v .
Inter-chunk state scan FLOPs per token are likewise invariant to R R R :
H k ( 2 d k D v , p a c k [ e r a s e ] + 2 d k D v , p a c k [ w r i t e ] + 2 d k D v , p a c k [ r e a d o u t ] ) = 6 H v d k d v . H_k(2d_kD_{v,\mathrm{pack}}\ [\mathrm{erase}] + 2d_kD_{v,\mathrm{pack}}\ [\mathrm{write}] + 2d_kD_{v,\mathrm{pack}}\ [\mathrm{readout}]) = 6H_vd_kd_v. H k ( 2 d k D v , pack [ erase ] + 2 d k D v , pack [ write ] + 2 d k D v , pack [ readout ]) = 6 H v d k d v .
Because β t \beta_t β t is scalar per key head h k h_k h k , QK projections (2 D H d k 2D_Hd_k 2 D H d k parameters), QK causal Conv1D (for kernel width K c o n v = 4 K_{\mathrm{conv}} = 4 K conv = 4 , 2 K c o n v H k d k 2K_{\mathrm{conv}}H_kd_k 2 K conv H k d k weights), chunk Gram matmuls (4 C d k 4Cd_k 4 C d k FLOPs/token), and [ C , C ] [C,C] [ C , C ] WY/Neumann solves (H k H_k H k systems per chunk) scale linearly with H k = H v / R H_k = H_v/R H k = H v / R and shrink by R × R \times R × . Reshaping S ~ t T q t ∈ R R d v \widetilde{S}_t^{\mathsf{T}}q_t \in\mathbb{R}^{Rd_v} S t T q t ∈ R R d v into R R R slices of width d v d_v d v and applying independent RMSNorm to all H v = H k R H_v = H_kR H v = H k R value heads prior to the sigmoid output gate σ ( z t ) ∈ ( 0 , 1 ) R d v \sigma(z_t) \in(0,1)^{Rd_v} σ ( z t ) ∈ ( 0 , 1 ) R d v (slices σ ( z t , m ) \sigma(z_{t,m}) σ ( z t , m ) in Eq. (5) retains H v H_v H v independent head-scale degrees of freedom while keeping [ 256 , D v , p a c k ] [256, D_{v,\mathrm{pack}}] [ 256 , D v , pack ] MXU tiles contiguous.
Pillar 2: Tile-Grouped Forget Gate and Packed WY Equivalence
Bottleneck 2 arises because the decay cannot be pulled out of key–key inner products when every channel has its own rate. SLA splits each query and key into the same G = 4 G=4 G = 4 groups of b = 64 b=64 b = 64 channels as the forget gate in Eq. (3) , q t = [ q t , 1 ; … ; q t , 4 ] q_t=[q_{t,1};\ldots;q_{t,4}] q t = [ q t , 1 ; … ; q t , 4 ] and k t = [ k t , 1 ; … ; k t , 4 ] k_t=[k_{t,1};\ldots;k_{t,4}] k t = [ k t , 1 ; … ; k t , 4 ] . Between tokens j j j and t t t , group g g g decays by a single number ∏ s = j + 1 t γ s , g ≤ 1 \prod_{s=j+1}^{t}\gamma_{s,g}\leq1 ∏ s = j + 1 t γ s , g ≤ 1 , which moves outside the group’s inner product. Stacking group-g g g queries and keys as Q c , g , K c , g ∈ R C × 64 Q_{c,g},K_{c,g}\in\mathbb{R}^{C\times64} Q c , g , K c , g ∈ R C × 64 and defining causal masks ( Ψ g ) t , j = 1 [ t ≥ j ] ∏ s = j + 1 t γ s , g (\Psi_g)_{t,j}=\mathbf{1}[t\geq j]\prod_{s=j+1}^{t}\gamma_{s,g} ( Ψ g ) t , j = 1 [ t ≥ j ] ∏ s = j + 1 t γ s , g and ( Γ g ) t , j = 1 [ t > j ] β j ( Ψ g ) t , j (\Gamma_g)_{t,j}=\mathbf{1}[t>j]\beta_j(\Psi_g)_{t,j} ( Γ g ) t , j = 1 [ t > j ] β j ( Ψ g ) t , j , both intra-chunk matrices factor across tiles as
M c = ∑ g = 1 4 Γ g ⊙ ( K c , g K c , g ⊤ ) , L c = ∑ g = 1 4 Ψ g ⊙ ( Q c , g K c , g ⊤ ) , M_c=\sum_{g=1}^{4}\Gamma_g\odot(K_{c,g}K_{c,g}^{\top}),\qquad L_c=\sum_{g=1}^{4}\Psi_g\odot(Q_{c,g}K_{c,g}^{\top}), M c = g = 1 ∑ 4 Γ g ⊙ ( K c , g K c , g ⊤ ) , L c = g = 1 ∑ 4 Ψ g ⊙ ( Q c , g K c , g ⊤ ) ,
where each term is a [ C , 64 ] × [ 64 , C ] [C,64]\times[64,C] [ C , 64 ] × [ 64 , C ] MXU matmul followed by an elementwise scaling by factors no larger than 1 1 1 , so there is no overflow and no need for VPU sub-blocks. The trade-off relative to GDN’s single rate (G = 1 G=1 G = 1 ) is modest: in our sweep (§ 3.5 ), G = 4 G=4 G = 4 lowers in-domain perplexity slightly across all three paired seeds but extrapolates somewhat worse than G = 1 G=1 G = 1 ; architecturally, unpacked GDN (H k = 8 H_k=8 H k = 8 , G = 1 G=1 G = 1 ) provides H k G = 8 H_kG=8 H k G = 8 decay timescales per layer, whereas sharing key heads by R = 4 R=4 R = 4 (H k = 2 H_k=2 H k = 2 ) with G = 1 G=1 G = 1 collapses the layer to only 2 rates; setting G = 4 G=4 G = 4 (b = 64 b=64 b = 64 ) restores H k G = 8 H_kG=8 H k G = 8 (SLA-R4) or 16 (SLA-R2) independent timescales per layer at zero latency overhead over G = 1 G=1 G = 1 .
Proof of tile-grouped packed WY equivalence. Consider chunk c ∈ { 1 , … , T / C } c\in\{1,\ldots,T/C\} c ∈ { 1 , … , T / C } for a single key head h k ∈ { 1 , … , H k } h_k\in\{1,\ldots,H_k\} h k ∈ { 1 , … , H k } serving R = H v / H k R=H_v/H_k R = H v / H k packed value heads of total width D v , p a c k = R d v D_{v,\mathrm{pack}}=Rd_v D v , pack = R d v , starting from incoming state S ~ 0 = S c − 1 ∈ R d k × D v , p a c k \widetilde{S}_0=S_{c-1}\in\mathbb{R}^{d_k\times D_{v,\mathrm{pack}}} S 0 = S c − 1 ∈ R d k × D v , pack . Each token t ∈ { 1 , … , C } t\in\{1,\ldots,C\} t ∈ { 1 , … , C } has G = 4 G=4 G = 4 tile-grouped diagonal decay F t = ⨁ g = 1 4 γ t , g I 64 F_t=\bigoplus_{g=1}^{4}\gamma_{t,g}I_{64} F t = ⨁ g = 1 4 γ t , g I 64 (diagonal entries in ( 0 , 1 ] (0,1] ( 0 , 1 ] ), query and key q t , k t ∈ R d k q_t,k_t\in\mathbb{R}^{d_k} q t , k t ∈ R d k , packed value v t ∈ R D v , p a c k v_t\in\mathbb{R}^{D_{v,\mathrm{pack}}} v t ∈ R D v , pack , and scalar step size β t ∈ ( 0 , 2 ) \beta_t\in(0,2) β t ∈ ( 0 , 2 ) shared across the R R R packed value heads of key head h k h_k h k .
Let P t ← j = ∏ s = j + 1 t F s P_{t\leftarrow j}=\prod_{s=j+1}^{t}F_s P t ← j = ∏ s = j + 1 t F s denote the cumulative diagonal transition from step j j j to step t t t (P t ← t = I d k P_{t\leftarrow t}=I_{d_k} P t ← t = I d k ). Unrolling Eq. (4) (S ˉ t = F t S ~ t − 1 \bar{S}_t=F_t\widetilde{S}_{t-1} S ˉ t = F t S t − 1 , v ^ t = S ˉ t ⊤ k t \hat{v}_t=\bar{S}_t^{\top}k_t v ^ t = S ˉ t ⊤ k t , S ~ t = S ˉ t + β t k t ( v t − v ^ t ) ⊤ \widetilde{S}_t=\bar{S}_t+\beta_tk_t(v_t-\hat{v}_t)^{\top} S t = S ˉ t + β t k t ( v t − v ^ t ) ⊤ ) from S ~ 0 = S c − 1 \widetilde{S}_0=S_{c-1} S 0 = S c − 1 gives
S ~ t = P t ← 0 S c − 1 + ∑ j = 1 t β j ( P t ← j k j ) u j ⊤ , (6) \widetilde{S}_t=P_{t\leftarrow0}S_{c-1}+\sum_{j=1}^{t}\beta_j(P_{t\leftarrow j}k_j)u_j^{\top}, \tag*{(6)} S t = P t ← 0 S c − 1 + j = 1 ∑ t β j ( P t ← j k j ) u j ⊤ , (6)
where u t = v t − v ^ t ∈ R D v , p a c k u_t=v_t-\hat{v}_t\in\mathbb{R}^{D_{v,\mathrm{pack}}} u t = v t − v ^ t ∈ R D v , pack is the post-forget delta residual at local step t t t . Multiplying S ˉ t = F t S ~ t − 1 \bar{S}_t=F_t\widetilde{S}_{t-1} S ˉ t = F t S t − 1 on the left by k t ⊤ k_t^{\top} k t ⊤ substitutes F t P t − 1 ← j = P t ← j F_tP_{t-1\leftarrow j}=P_{t\leftarrow j} F t P t − 1 ← j = P t ← j into every prior term:
u t ⊤ = v t ⊤ − k t ⊤ P t ← 0 S c − 1 − ∑ j = 1 t − 1 β j ( k t ⊤ P t ← j k j ) u j ⊤ . (7) u_t^{\top}=v_t^{\top}-k_t^{\top}P_{t\leftarrow0}S_{c-1}-\sum_{j=1}^{t-1}\beta_j(k_t^{\top}P_{t\leftarrow j}k_j)u_j^{\top}. \tag*{(7)} u t ⊤ = v t ⊤ − k t ⊤ P t ← 0 S c − 1 − j = 1 ∑ t − 1 β j ( k t ⊤ P t ← j k j ) u j ⊤ . (7)
Whereas per-channel decay (G = d k G=d_k G = d k in KDA) overflows the naive factorization e Λ t , i ⋅ e − Λ j , i e^{\Lambda_{t,i}\cdot e^{-\Lambda_{j,i}}} e Λ t , i ⋅ e − Λ j , i in FP32 once ∣ Λ j , i ∣ > 88.7 |\Lambda_{j,i}|>88.7 ∣ Λ j , i ∣ > 88.7 and forces 16-row sub-block VPU pivots, grouping F t F_t F t onto G = 4 G=4 G = 4 scalar 64-lane tiles gives
k t ⊤ P t ← j k j = ∑ g = 1 4 exp ( Λ t , g − Λ j , g ) ⟨ k t , g , k j , g ⟩ , Λ t , g = ∑ s = 1 t log γ s , g , k_t^{\top}P_{t\leftarrow j}k_j=\sum_{g=1}^{4}\exp(\Lambda_{t,g}-\Lambda_{j,g})\langle k_{t,g},k_{j,g}\rangle,\qquad\Lambda_{t,g}=\sum_{s=1}^{t}\log\gamma_{s,g}, k t ⊤ P t ← j k j = g = 1 ∑ 4 exp ( Λ t , g − Λ j , g ) ⟨ k t , g , k j , g ⟩ , Λ t , g = s = 1 ∑ t log γ s , g ,
where Λ t , g \Lambda_{t,g} Λ t , g is the tile’s cumulative log-decay and q t , g , k t , g ∈ R 64 q_{t,g},k_{t,g}\in\mathbb{R}^{64} q t , g , k t , g ∈ R 64 are the tile-g g g slices of q t , k t q_t,k_t q t , k t . Because exp ( Λ t , g − Λ j , g ) ≤ 1 \exp(\Lambda_{t,g}-\Lambda_{j,g})\leq1 exp ( Λ t , g − Λ j , g ) ≤ 1 is formed directly as a scalar mask, this reduces to four direct [ C , 64 ] × [ 64 , C ] [C,64]\times[64,C] [ C , 64 ] × [ 64 , C ] MXU matmuls multiplied by G = 4 G=4 G = 4 causal scalar decay masks. Because β j \beta_j β j is scalar per key head h k h_k h k , the strictly lower-triangular coupling matrix M c ∈ R C × C M_c\in\mathbb{R}^{C\times C} M c ∈ R C × C defined by
( M c ) t , j = 1 [ t > j ] β j ( k t ⊤ P t ← j k j ) (8) (M_c)_{t,j}=\mathbf{1}[t>j]\beta_j(k_t^{\top}P_{t\leftarrow j}k_j) \tag*{(8)} ( M c ) t , j = 1 [ t > j ] β j ( k t ⊤ P t ← j k j ) (8)
is shared across all D v , p a c k D_{v,\mathrm{pack}} D v , pack value columns and requires inverting a single C × C C \times C C × C unit lower-triangular system A c = ( I C + M c ) − 1 A_c = (I_C + M_c)^{-1} A c = ( I C + M c ) − 1 per key head.
Stacking chunk values V c ∈ R C × D v , p a c k V_c \in\mathbb{R}^{C \times D_{v,\mathrm{pack}}} V c ∈ R C × D v , pack (row t t t equal to v t ⊤ v_t^\top v t ⊤ ), decayed keys K ˉ c ∈ R C × d k \bar{K}_c \in\mathbb{R}^{C \times d_k} K ˉ c ∈ R C × d k (row t t t equal to ( P t ← 0 k t ) ⊤ (P_{t \gets0}k_t)^\top ( P t ← 0 k t ) ⊤ ), and residuals u 1 : C ∈ R C × D v , p a c k u_{1:C} \in\mathbb{R}^{C \times D_{v,\mathrm{pack}}} u 1 : C ∈ R C × D v , pack (row t t t equal to u t ⊤ u_t^\top u t ⊤ ) yields β \beta β -scaled WY factors and chunk writes:
U c = diag ( β c ) A c V c ∈ R C × D v , p a c k , W c = diag ( β c ) A c K ˉ c ∈ R C × d k , U_c = \operatorname{diag}(\beta_c) A_c V_c \in\mathbb{R}^{C \times D_{v,\mathrm{pack}}}, \qquad W_c = \operatorname{diag}(\beta_c) A_c \bar{K}_c \in\mathbb{R}^{C \times d_k}, U c = diag ( β c ) A c V c ∈ R C × D v , pack , W c = diag ( β c ) A c K ˉ c ∈ R C × d k ,
E c = diag ( β c ) u 1 : C = U c − W c S c − 1 . E_c = \operatorname{diag}(\beta_c) u_{1:C} = U_c - W_c S_{c-1}. E c = diag ( β c ) u 1 : C = U c − W c S c − 1 .
Letting the carried keys K ^ c ∈ R C × d k \hat{K}_c \in\mathbb{R}^{C \times d_k} K ^ c ∈ R C × d k have row j j j equal to ( P C ← j k j ) ⊤ (P_{C \gets j}k_j)^\top ( P C ← j k j ) ⊤ and F c = P C ← 0 F_c = P_{C \gets0} F c = P C ← 0 , the end-of-chunk state and chunk increment satisfy
S ~ C = F c S c − 1 + K ^ c ⊤ ( U c − W c S c − 1 ) , Δ S c = S ~ C − F c S c − 1 = K ^ c ⊤ E c ∈ R d k × D v , p a c k . \widetilde{S}_C = F_c S_{c-1} + \hat{K}_c^\top(U_c - W_c S_{c-1}), \qquad\Delta S_c = \widetilde{S}_C - F_c S_{c-1} = \hat{K}_c^\top E_c \in\mathbb{R}^{d_k \times D_{v,\mathrm{pack}}}. S C = F c S c − 1 + K ^ c ⊤ ( U c − W c S c − 1 ) , Δ S c = S C − F c S c − 1 = K ^ c ⊤ E c ∈ R d k × D v , pack .
(When optional oblique key whitening k ~ j \tilde{k}_j k ~ j is enabled in SPD-SLA-W, Section 4, replacing the write key k j k_j k j with k ~ j \tilde{k}_j k ~ j in M c M_c M c and K ^ c \hat{K}_c K ^ c yields the identical WY dual equations.) Finally, the chunk-level velocity Z c = μ κ c Z c − 1 + Δ S c Z_c = \mu\kappa_c Z_{c-1} + \Delta S_c Z c = μ κ c Z c − 1 + Δ S c and S c = F c S c − 1 + Z c S_c = F_c S_{c-1} + Z_c S c = F c S c − 1 + Z c (Eq. (4) ), with reset indicator κ c ∈ { 0 , 1 } \kappa_c \in\{0,1\} κ c ∈ { 0 , 1 } equal to 0 iff chunk c c c starts a new document) act solely at chunk boundaries, preserving timestep key-value pairing inside every chunk and matching token-by-token decoding that accumulates Δ S c \Delta S_c Δ S c over C C C steps before applying Eq. (4) .
Intra-tile conditioning and orthogonal rotation absorption. Because each group’s decay is a scalar multiple of the identity, it preserves conditioning within each tile and commutes with rotary rotations inside the group:
Observation 2.2 (Intra-Tile Conditioning and Orthogonal Rotation Absorption). Within each group g g g , the decay block γ t , g I d k / G \gamma_{t,g} I_{d_k/G} γ t , g I d k / G has unit condition number κ 2 ( γ t , g I d k / G ) = 1 \kappa_2(\gamma_{t,g} I_{d_k/G}) = 1 κ 2 ( γ t , g I d k / G ) = 1 (preserving intra-tile cosine similarity) and commutes with any rotation acting within each tile, Rot ∈ SO ( d k / G ) G \operatorname{Rot} \in\operatorname{SO}(d_k/G)^G Rot ∈ SO ( d k / G ) G , such as Givens or rotary rotations, leaving the grouped Gram computation of Pillar 2 unchanged.
Proof. For a per-channel diagonal decay F t = diag ( f t ) F_t = \operatorname{diag}(f_t) F t = diag ( f t ) with f t , i ∈ ( 0 , 1 ) f_{t,i} \in(0,1) f t , i ∈ ( 0 , 1 ) and cumulative log-decay Λ t , i = ∑ s = 1 t log f s , i \Lambda_{t,i} = \sum_{s=1}^{t} \log f_{s,i} Λ t , i = ∑ s = 1 t log f s , i , the chunk decay F c = P C ← 0 = diag ( ( exp ( Λ C , i ) ) i = 1 d k ) F_c = P_{C \gets0} = \operatorname{diag}((\exp(\Lambda_{C,i}))_{i=1}^{d_k}) F c = P C ← 0 = diag (( exp ( Λ C , i ) ) i = 1 d k ) has singular values exp ( Λ C , i ) \exp(\Lambda_{C,i}) exp ( Λ C , i ) and condition number κ 2 ( F c ) = exp ( max i Λ C , i − min i Λ C , i ) \kappa_2(F_c) = \exp(\max_i \Lambda_{C,i} - \min_i \Lambda_{C,i}) κ 2 ( F c ) = exp ( max i Λ C , i − min i Λ C , i ) ; write Δ Λ C = max i Λ C , i − min i Λ C , i \Delta\Lambda_C = \max_i \Lambda_{C,i} - \min_i \Lambda_{C,i} Δ Λ C = max i Λ C , i − min i Λ C , i . In BF16 (7 mantissa bits, machine epsilon 2 − 7 ≈ 7.8 × 10 − 3 2^{-7} \approx7.8 \times10^{-3} 2 − 7 ≈ 7.8 × 1 0 − 3 , unit roundoff 2 − 8 ≈ 3.9 × 10 − 3 2^{-8} \approx3.9 \times10^{-3} 2 − 8 ≈ 3.9 × 1 0 − 3 , exponent range ≈ [ 2 − 126 , 2 127 ] \approx[2^{-126}, 2^{127}] ≈ [ 2 − 126 , 2 127 ] ), relative channel scaling across P t ← 0 P j ← 0 − 1 P_{t \gets0} P_{j \gets0}^{-1} P t ← 0 P j ← 0 − 1 loses all mantissa significance across fast vs. slow channels once Δ Λ C ≳ 5.5 \Delta\Lambda_C \gtrsim5.5 Δ Λ C ≳ 5.5 in BF16 (κ 2 ≈ 2 8 \kappa_2 \approx2^8 κ 2 ≈ 2 8 ) or Δ Λ C ≳ 16 \Delta\Lambda_C \gtrsim16 Δ Λ C ≳ 16 in FP32 accumulators (κ 2 ≈ 8.9 × 10 6 ≈ 2 23 \kappa_2 \approx8.9 \times10^6 \approx2^{23} κ 2 ≈ 8.9 × 1 0 6 ≈ 2 23 ), and exp ( ± Λ j , i ) \exp(\pm\Lambda_{j,i}) exp ( ± Λ j , i ) overflows or underflows FP32 when ∣ Λ j , i ∣ > 88.7 |\Lambda_{j,i}| > 88.7 ∣ Λ j , i ∣ > 88.7 . Under tile-grouped decay F t = ⨁ g = 1 4 γ t , g I 64 F_t = \bigoplus_{g=1}^{4} \gamma_{t,g} I_{64} F t = ⨁ g = 1 4 γ t , g I 64 , each 64-channel tile block is a scalar multiple of I 64 I_{64} I 64 with κ 2 ( γ t , g I 64 ) = 1 \kappa_2(\gamma_{t,g} I_{64}) = 1 κ 2 ( γ t , g I 64 ) = 1 , preserving exact intra-tile cosine similarity.
Furthermore, for any 2 × 2 2 \times2 2 × 2 block-diagonal (Givens/rotary) rotation with per-pair angles θ t = ( θ t , 1 , … , θ t , d k / 2 ) ∈ R d k / 2 \theta_t = (\theta_{t,1}, \ldots, \theta_{t,d_k/2}) \in\mathbb{R}^{d_k/2} θ t = ( θ t , 1 , … , θ t , d k /2 ) ∈ R d k /2 indexed by pair l ∈ { 1 , … , d k / 2 } l \in\{1,\ldots,d_k/2\} l ∈ { 1 , … , d k /2 } ,
Rot ( θ t ) = ⨁ l = 1 d k / 2 ( cos θ t , l − sin θ t , l sin θ t , l cos θ t , l ) ∈ SO ( 2 ) d k / 2 , \operatorname{Rot}(\theta_t) = \bigoplus_{l=1}^{d_k/2}
\begin{pmatrix}
\cos\theta_{t,l} & -\sin\theta_{t,l} \\
\sin\theta_{t,l} & \cos\theta_{t,l}
\end{pmatrix}
\in\operatorname{SO}(2)^{d_k/2}, Rot ( θ t ) = l = 1 ⨁ d k /2 ( cos θ t , l sin θ t , l − sin θ t , l cos θ t , l ) ∈ SO ( 2 ) d k /2 ,
the cumulative rotation ∏ t = 1 C Rot ( θ t ) = Rot ( Θ C ) \prod_{t=1}^{C} \operatorname{Rot}(\theta_t) = \operatorname{Rot}(\Theta_C) ∏ t = 1 C Rot ( θ t ) = Rot ( Θ C ) with Θ t = ∑ s = 1 t θ s \Theta_t = \sum_{s=1}^{t} \theta_s Θ t = ∑ s = 1 t θ s has all singular values equal to 1 (κ 2 = 1 \kappa_2 = 1 κ 2 = 1 ). Since Rot ( θ s ) \operatorname{Rot}(\theta_s) Rot ( θ s ) commutes with G = 4 G = 4 G = 4 scalar tile multiples (64 channels = 32 contiguous 2 × 2 2 \times2 2 × 2 pairs per tile), for tile-grouped F s F_s F s we have
k t ⊤ ( ∏ s = j + 1 t F s Rot ( θ s ) ) k j = ( Rot ( Θ t ) ⊤ k t ) ⊤ P t ← j ( Rot ( Θ j ) ⊤ k j ) . k_t^\top\left(\prod_{s=j+1}^{t} F_s \operatorname{Rot}(\theta_s)\right) k_j = \left(\operatorname{Rot}(\Theta_t)^\top k_t\right)^\top P_{t \gets j} \left(\operatorname{Rot}(\Theta_j)^\top k_j\right). k t ⊤ ( s = j + 1 ∏ t F s Rot ( θ s ) ) k j = ( Rot ( Θ t ) ⊤ k t ) ⊤ P t ← j ( Rot ( Θ j ) ⊤ k j ) .
Replacing ( q t , k t ) (q_t,k_t) ( q t , k t ) with the rotated queries and keys ( Rot ( Θ t ) q t , Rot ( Θ t ) ⊤ k t ) (\operatorname{Rot}(\Theta_t) q_t,\operatorname{Rot}(\Theta_t)^\top k_t) ( Rot ( Θ t ) q t , Rot ( Θ t ) ⊤ k t ) before the chunk kernel reduces the intra-chunk transition to pure G = 4 G = 4 G = 4 real tile decay, preserving the unchanged systolic WY kernel up to a final boundary unrotation Rot ( Θ C ) \operatorname{Rot}(\Theta_C) Rot ( Θ C ) on outgoing state.
Pillar 3: Chunk Inversion on the MXU
Existing solvers substitute sequentially inside diagonal blocks on the VPU. SLA instead inverts I C + M c I_C + M_c I C + M c by recursive block-doubling down to 2 × 2 2 \times2 2 × 2 blocks, so the whole inverse uses only matrix products. Because the matrix is lower-triangular, the inverse of two neighboring diagonal blocks can be assembled from the inverse of each block plus one new off-diagonal block, which costs two small matmuls. SLA starts from 2 × 2 2 \times2 2 × 2 blocks and doubles the block size six times until it covers the whole 128 × 128 128 \times128 128 × 128 chunk, so every step runs on the MXU (Figure 1 e). The result is the exact inverse in FP32, which we call Exact Inv.. SLA also offers a cheaper approximation, 1-Step Neumann, which keeps only the first correction term between blocks; the ablation runs and prefill benchmarks use a matmul-free variant of it. This approximation changes the learned model: in the pre-training ablation, training with it instead of Exact Inv. costs 0.22 perplexity, comparing rows A2 and A3 in § 3.5 , even though swapping Exact Inv. in at test time shifts 64K QA by at most 0.01 points. All multi-scale MoE pre-training and context-parallel runs therefore use Exact Inv., while single-layer and serving benchmarks report both.
Block-doubling and block-Neumann factorization
Let D b l k D_{\mathrm{blk}} D blk be the block-diagonal part of I C + M c I_C + M_c I C + M c (its C / 16 C/16 C /16 unit lower-triangular 16 × 16 16 \times16 16 × 16 diagonal blocks) and let N = D b l k − 1 ( I C + M c − D b l k ) N = D_{\mathrm{blk}}^{-1}(I_C + M_c - D_{\mathrm{blk}}) N = D blk − 1 ( I C + M c − D blk ) (local to Pillar 3), so that I C + M c = D b l k ( I C + N ) I_C + M_c = D_{\mathrm{blk}}(I_C + N) I C + M c = D blk ( I C + N ) with N N N strictly block-lower-triangular and N C / 16 = 0 N^{C/16} = 0 N C /16 = 0 . Algebraically, at C = 128 C = 128 C = 128 , the exact inverse and its 1-step block-Neumann truncation A N 1 ∈ R C × C A_{\mathrm{N1}} \in\mathbb{R}^{C \times C} A N1 ∈ R C × C take the form
A N 1 = ( I C − N ) D b l k − 1 , A_{\mathrm{N1}} = (I_C - N)D_{\mathrm{blk}}^{-1}, A N1 = ( I C − N ) D blk − 1 ,
( I C + M c ) − 1 = ∑ s = 0 7 ( − N ) s D b l k − 1 = ( I C + N 4 ) ( I C + N 2 ) ( I C − N ) D b l k − 1 . (9) (I_C + M_c)^{-1} = \sum_{s=0}^{7}(-N)^sD_{\mathrm{blk}}^{-1} = (I_C + N^4)(I_C + N^2)(I_C - N)D_{\mathrm{blk}}^{-1}. \tag*{(9)} ( I C + M c ) − 1 = s = 0 ∑ 7 ( − N ) s D blk − 1 = ( I C + N 4 ) ( I C + N 2 ) ( I C − N ) D blk − 1 . (9)
Rather than evaluating the expanded polynomial product ( I C + N 4 ) ( I C + N 2 ) ( I C − N ) D b l k − 1 (I_C + N^4)(I_C + N^2)(I_C - N)D_{\mathrm{blk}}^{-1} ( I C + N 4 ) ( I C + N 2 ) ( I C − N ) D blk − 1 , which can amplify powers of N N N in finite precision, our Pallas kernel (Exact Inv.) computes ( I C + M c ) − 1 (I_C + M_c)^{-1} ( I C + M c ) − 1 via the numerically stable recursive Schur/block-doubling recurrence in FP32 (Precision.HIGHEST):
X 2 s = X s − X s L s X s ( s = 2 , 4 , 8 , 16 , 32 , 64 ) , X_{2s} = X_s - X_sL_sX_s \qquad(s = 2, 4, 8, 16, 32, 64), X 2 s = X s − X s L s X s ( s = 2 , 4 , 8 , 16 , 32 , 64 ) ,
where X s X_s X s is the block-diagonal matrix of already-inverted s × s s \times s s × s diagonal blocks of I C + M c I_C + M_c I C + M c (X 128 = A c X_{128} = A_c X 128 = A c ) and L s L_s L s keeps only the strictly lower-left s × s s \times s s × s coupling block inside each 2 s × 2 s 2s \times2s 2 s × 2 s diagonal block (zero elsewhere), so L s X s L s = 0 L_sX_sL_s = 0 L s X s L s = 0 and the update is exact. Concretely, at each step s → 2 s s \to2s s → 2 s , given the inverse of the two s × s s \times s s × s diagonal halves (X t o p , X b o t X_{\mathrm{top}}, X_{\mathrm{bot}} X top , X bot ) and their strictly lower-left s × s s \times s s × s coupling block L o f f L_{\mathrm{off}} L off , the lower-left block of X 2 s X_{2s} X 2 s is computed via two MXU matmuls as − X b o t L o f f X t o p -X_{\mathrm{bot}}L_{\mathrm{off}}X_{\mathrm{top}} − X bot L off X top .
Variants of 1-Step Neumann and truncation properties
Two different first-order approximations appear under the name 1-Step Neumann, and we keep them distinct:
the block truncation A N 1 = ( I C − N ) D b l k − 1 A_{\mathrm{N1}} = (I_C - N)D_{\mathrm{blk}}^{-1} A N1 = ( I C − N ) D blk − 1 with block width B b l k = 16 B_{\mathrm{blk}} = 16 B blk = 16 , analyzed below and evaluated in our standalone lag/skew benchmark; and
the matmul-free row-scaled approximation A N 1 r o w ∈ R C × C A_{\mathrm{N1}}^{\mathrm{row}} \in\mathbb{R}^{C \times C} A N1 row ∈ R C × C , with row-scaling vector r = ( r 1 , … , r C ) T ∈ R C r = (r_1,\ldots,r_C)^\mathsf{T} \in\mathbb{R}^C r = ( r 1 , … , r C ) T ∈ R C , used by the pre-training ablation arms A3 and A5 (Table 11 b) and the prefill benchmarks
A N 1 r o w = I C − diag ( r ) − 1 M c , r i = max ( 1 , 1.25 ∥ ( M c ) i , : ∥ 1 ) . A_{\mathrm{N1}}^{\mathrm{row}} = I_C - \operatorname{diag}(r)^{-1}M_c,\qquad r_i = \max(1, 1.25\lVert(M_c)_{i,:}\rVert_1). A N1 row = I C − diag ( r ) − 1 M c , r i = max ( 1 , 1.25 ∥( M c ) i , : ∥ 1 ) .
The exactness property below (zero error for lags t − j ≤ 16 t-j \le16 t − j ≤ 16 ) holds for (1) but not for (2). Partition unit lower-triangular I C + M c = D b l k + M o f f = D b l k ( I C + N ) I_C + M_c = D_{\mathrm{blk}} + M_{\mathrm{off}} = D_{\mathrm{blk}}(I_C + N) I C + M c = D blk + M off = D blk ( I C + N ) , where D b l k = diag ( D 1 , … , D m ) D_{\mathrm{blk}} = \operatorname{diag}(D_1,\ldots,D_m) D blk = diag ( D 1 , … , D m ) consists of m = C / B b l k m = C/B_{\mathrm{blk}} m = C / B blk unit lower-triangular diagonal blocks D a ∈ R B b l k × B b l k D_a \in\mathbb{R}^{B_{\mathrm{blk}} \times B_{\mathrm{blk}}} D a ∈ R B blk × B blk (a ∈ { 1 , … , m } a \in\{1,\ldots,m\} a ∈ { 1 , … , m } ) and M o f f M_{\mathrm{off}} M off is zero on and above the block diagonal. Then N = D b l k − 1 M o f f N = D_{\mathrm{blk}}^{-1}M_{\mathrm{off}} N = D blk − 1 M off is strictly block-lower-triangular (its ( a , b ) (a,b) ( a , b ) block satisfies N a , b = 0 N_{a,b} = 0 N a , b = 0 for a < b a < b a < b , hence N a = 0 N^a = 0 N a = 0 for all a − b ≤ s a-b \le s a − b ≤ s and $N^m = 0). Comparing ( I C + M c ) − 1 = ∑ s = 0 m − 1 ( − N ) s D b l k − 1 (I_C + M_c)^{-1} = \sum_{s=0}^{m-1}(-N)^sD_{\mathrm{blk}}^{-1} ( I C + M c ) − 1 = ∑ s = 0 m − 1 ( − N ) s D blk − 1 against A N 1 = ( I C − N ) D b l k − 1 = D b l k − 1 − D b l k − 1 M o f f D b l k − 1 A_{\mathrm{N1}} = (I_C - N)D_{\mathrm{blk}}^{-1} = D_{\mathrm{blk}}^{-1} - D_{\mathrm{blk}}^{-1}M_{\mathrm{off}}D_{\mathrm{blk}}^{-1} A N1 = ( I C − N ) D blk − 1 = D blk − 1 − D blk − 1 M off D blk − 1 , the residual
( I C + M c ) − 1 − A N 1 = ( ∑ s = 2 m − 1 ( − 1 ) s N s ) D b l k − 1 (I_C + M_c)^{-1} - A_{\mathrm{N1}} = \left(\sum_{s=2}^{m-1}(-1)^sN^s\right)D_{\mathrm{blk}}^{-1} ( I C + M c ) − 1 − A N1 = ( s = 2 ∑ m − 1 ( − 1 ) s N s ) D blk − 1
vanishes identically on all block entries ( a , b ) (a,b) ( a , b ) with a − b ≤ 1 a-b \le1 a − b ≤ 1 , namely every diagonal block ( a , a ) (a,a) ( a , a ) and every first sub-diagonal block ( a + 1 , a ) (a+1,a) ( a + 1 , a ) . Any two token indices $1 \le j \le t \le C$ with t − j ≤ B b l k t-j \le B_{\mathrm{blk}} t − j ≤ B blk satisfy ⌈ t / B b l k ⌉ − ⌈ j / B b l k ⌉ ∈ { 0 , 1 } \lceil t/B_{\mathrm{blk}}\rceil-\lceil j/B_{\mathrm{blk}}\rceil\in\{0,1\} ⌈ t / B blk ⌉ − ⌈ j / B blk ⌉ ∈ { 0 , 1 } and therefore lie in block ( a , a ) (a,a) ( a , a ) or ( a + 1 , a ) (a+1,a) ( a + 1 , a ) , incurring zero truncation error.
In finite-precision BF16/FP32 arithmetic, exact WY training vs. stateful step-by-step decoding achieves measured output KL divergence < 10 − 6 <10^{-6} < 1 0 − 6 per layer (3.4 × 10 − 5 3.4 \times10^{-5} 3.4 × 1 0 − 5 across 4-layer BF16 stacks when chunk momentum syncs every C C C steps), while under 1-Step Neumann (B b l k = 16 B_{\mathrm{blk}}=16 B blk = 16 , C = 128 C=128 C = 128 ), truncation at lags t − j > 16 t-j>16 t − j > 16 exhibits relative Frobenius error 1.6 × 10 − 5 1.6 \times10^{-5} 1.6 × 1 0 − 5 in short-horizon 4L:1A layers up to 4.9 × 10 − 2 4.9 \times10^{-2} 4.9 × 1 0 − 2 in long-horizon 15L:1A layers, yielding single-layer output KL < 10 − 5 <10^{-5} < 1 0 − 5 against exact recurrent decoding (and zero discrepancy if chunked prefill uses 1-Step Neumann). Training with the row-scaled approximation (2) nonetheless changes the learned model: in the pre-training ablation (Table 11 b, A2→ \to → A3) it raises 11-set perplexity by 0.22.
Single-Kernel VMEM Execution and Custom-VJP Backward Pass
Figure 4 (PDF p. 13). Single-kernel VMEM execution and custom VJP backward pass in SLA. Top (Forward): Streamed Q Q Q , K K K , V V V and gates enter the 64 MiB VMEM scratchpad once; a 3-token ring buffer executes causal Conv1D and L 2 L_2 L 2 normalization (Op 2), followed in-place by 4-tile MXU Gram contractions (Op 3), MXU chunk inversion and WY factors (Ops 4–5), persistent [ 256 , D v , p a c k ] [256,D_{v,\mathrm{pack}}] [ 256 , D v , pack ] state updates (Op 6), and chunk readout (Op 7a) before writing O O O to HBM. Bottom (Backward): During the reverse state scan, the custom VJP kernel recomputes intra-chunk matrices M c M_c M c , A c A_c A c on the MXU directly inside VMEM, eliminating [ B , H k , T / C , C , C ] [B,H_k,T/C,C,C] [ B , H k , T / C , C , C ] intermediate HBM writes for batch size B B B .
Algorithm 2 details the custom-VJP backward pass of fused SLA (illustrated in Figure 4, bottom), where for a scalar training loss L \mathcal{L} L we write d M c = ∂ L / ∂ M c dM_c=\partial\mathcal{L}/\partial M_c d M c = ∂ L / ∂ M c (and analogously d Q c , d K c , d V c , … dQ_c,dK_c,dV_c,\ldots d Q c , d K c , d V c , … ) for the loss gradient (adjoint) with respect to each tensor. Let γ c ∈ ( 0 , 1 ) C × C \gamma_c \in(0,1)^{C \times C} γ c ∈ ( 0 , 1 ) C × C stack chunk c c c ’s tile decays γ t , g \gamma_{t,g} γ t , g (with tile column γ c , g ∈ ( 0 , 1 ] C \gamma_{c,g} \in(0,1]^C γ c , g ∈ ( 0 , 1 ] C ), and let d O c = ∂ L / ∂ O c ∈ R C × R d v dO_c=\partial\mathcal{L}/\partial O_c \in\mathbb{R}^{C \times R d_v} d O c = ∂ L / ∂ O c ∈ R C × R d v denote the upstream gradient of chunk c c c ’s output after backpropagating through the per-value-head RMSNorm and output gate in Eq. (5). For batch size B B B , a standard autodiff tape stores the intra-chunk matrices M c M_c M c , L c L_c L c , A c ∈ R C × C A_c \in\mathbb{R}^{C \times C} A c ∈ R C × C from the forward pass in HBM, consuming 3 B H k ( T / C ) C 2 3BH_k(T/C)C^2 3 B H k ( T / C ) C 2 words. Because SLA’s Pillar 2 and Pillar 3 evaluate both the G = 4 G=4 G = 4 Gram matrices and the chunk inverse A c = ( I C + M c ) − 1 A_c=(I_C+M_c)^{-1} A c = ( I C + M c ) − 1 as MXU matmuls, our backward kernel instead saves only the chunk-boundary states S c − 1 ∈ R d k × R d v S_{c-1}\in\mathbb{R}^{d_k \times R d_v} S c − 1 ∈ R d k × R d v and recomputes M c M_c M c , L c L_c L c , A c A_c A c , and the unscaled writes E ~ c = A c ( V c − K ˉ c S c − 1 ) ∈ R C × R d v \widetilde{E}_c=A_c(V_c-\bar{K}_cS_{c-1})\in\mathbb{R}^{C \times R d_v} E c = A c ( V c − K ˉ c S c − 1 ) ∈ R C × R d v on-the-fly inside VMEM during the reverse scan over c = T / C , … , 1 c=T/C,\ldots,1 c = T / C , … , 1 .
Crucially, whereas unpacked baselines must execute a second sequential VPU triangular solve for the transposed system ( I C + M c ) − T (I_C+M_c)^{-\mathsf{T}} ( I C + M c ) − T on every head in the backward pass, SLA already holds the dense inverse A c A_c A c in VMEM. Differentiating E c = diag ( β c ) A c ( V c − K ˉ c S c − 1 ) = diag ( β c ) E ~ c E_c=\operatorname{diag}(\beta_c)A_c(V_c-\bar{K}_cS_{c-1})=\operatorname{diag}(\beta_c)\widetilde{E}_c E c = diag ( β c ) A c ( V c − K ˉ c S c − 1 ) = diag ( β c ) E c through A c = ( I C + M c ) − 1 A_c=(I_C+M_c)^{-1} A c = ( I C + M c ) − 1 reduces the inverse-solve backpropagation to two MXU matmuls for the scaled adjoint D c ∈ R C × R d v D_c\in\mathbb{R}^{C\times R d_v} D c ∈ R C × R d v and Gram adjoint d M c ∈ R C × C dM_c\in\mathbb{R}^{C\times C} d M c ∈ R C × C , shared across all R R R packed value heads:
D c = A c T diag ( β c ) d E c ∈ R C × R d v , d M c = − tril − 1 ( D c E ~ c T ) ∈ R C × C (10) D_c=A_c^{\mathsf{T}}\operatorname{diag}(\beta_c)dE_c\in\mathbb{R}^{C\times R d_v},\qquad dM_c=-\operatorname{tril}_{-1}(D_c\widetilde{E}_c^{\mathsf{T}})\in\mathbb{R}^{C\times C} \tag*{(10)} D c = A c T diag ( β c ) d E c ∈ R C × R d v , d M c = − tril − 1 ( D c E c T ) ∈ R C × C (10)
where tril − 1 ( ⋅ ) \operatorname{tril}_{-1}(\cdot) tril − 1 ( ⋅ ) and tril 0 ( ⋅ ) \operatorname{tril}_{0}(\cdot) tril 0 ( ⋅ ) mask to strictly ( t > j ) (t>j) ( t > j ) and weakly ( t ≥ j ) (t\ge j) ( t ≥ j ) lower-triangular entries, rowsum ( ⋅ ) \operatorname{rowsum}(\cdot) rowsum ( ⋅ ) and colsum ( ⋅ ) \operatorname{colsum}(\cdot) colsum ( ⋅ ) sum matrix rows and columns, and p c , g i n , p c , g o u t ∈ ( 0 , 1 ] C p^{\mathrm{in}}_{c,g},p^{\mathrm{out}}_{c,g}\in(0,1]^C p c , g in , p c , g out ∈ ( 0 , 1 ] C denote the tile-g g g boundary-decay vectors (p c , g i n ) t = ∏ s = 1 t − 1 γ s , g p^{\mathrm{in}}_{c,g})_t=\prod_{s=1}^{t-1}\gamma_{s,g} p c , g in ) t = ∏ s = 1 t − 1 γ s , g and ( p c , g o u t ) t = ∏ s = t + 1 C γ s , g (p^{\mathrm{out}}_{c,g})_t=\prod_{s=t+1}^{C}\gamma_{s,g} ( p c , g out ) t = ∏ s = t + 1 C γ s , g . From D c D_c D c , the value and decayed-key gradients follow immediately as d V c = D c dV_c=D_c d V c = D c and d K ˉ c = − D c S c − 1 T d\bar{K}_c=-D_cS_{c-1}^{\mathsf{T}} d K ˉ c = − D c S c − 1 T .
Algorithm 2 Fused SLA Custom-VJP Backward Pass (Per Key Head in VMEM)
Require: Saved boundary states { S c − 1 } c = 1 T / C \{S_{c-1}\}_{c=1}^{T/C} { S c − 1 } c = 1 T / C , chunk inputs { Q c , K c , V c , β c , γ c } c = 1 T / C \{Q_c,K_c,V_c,\beta_c,\gamma_c\}_{c=1}^{T/C} { Q c , K c , V c , β c , γ c } c = 1 T / C , upstream readout gradients { d O c } c = 1 T / C \{dO_c\}_{c=1}^{T/C} { d O c } c = 1 T / C
Ensure: Input gradients { d Q c , d K c , d V c , d β c , d γ c } c = 1 T / C \{dQ_c,dK_c,dV_c,d\beta_c,d\gamma_c\}_{c=1}^{T/C} { d Q c , d K c , d V c , d β c , d γ c } c = 1 T / C
Initialize reverse carry d S T / C ← 0 dS_{T/C} \leftarrow0 d S T / C ← 0 , d Z T / C ← 0 ∈ R d k × R d v dZ_{T/C} \leftarrow0 \in\mathbb{R}^{d_k \times R_{dv}} d Z T / C ← 0 ∈ R d k × R d v in VMEM
for c = T / C c=T/C c = T / C down to 1 do
// 1. On-chip VMEM recomputation (Ops 3–5 on MXU; zero [ C , C ] [C,C] [ C , C ] HBM reads)
Recompute p c , g i n p_{c,g}^{\mathrm{in}} p c , g in , p c , g o u t p_{c,g}^{\mathrm{out}} p c , g out , decayed ( K ˉ c , Q ˉ c , K ^ c ) (\bar{K}_c,\bar{Q}_c,\hat{K}_c) ( K ˉ c , Q ˉ c , K ^ c ) , G = 4 G=4 G = 4 masks ( Γ g , Ψ g ) (\Gamma_g,\Psi_g) ( Γ g , Ψ g ) , and M c , L c M_c,L_c M c , L c via 64-wide MXU matmuls
Recompute A c ← ( I C + M c ) − 1 A_c \leftarrow(I_C+M_c)^{-1} A c ← ( I C + M c ) − 1 on MXU; form E ~ c ← A c ( V c − K ˉ c S c − 1 ) \tilde{E}_c \leftarrow A_c(V_c-\bar{K}_cS_{c-1}) E ~ c ← A c ( V c − K ˉ c S c − 1 ) and E c ← diag ( β c ) E ~ c E_c \leftarrow\operatorname{diag}(\beta_c)\tilde{E}_c E c ← diag ( β c ) E ~ c
// 2. Reverse state and momentum scan adjoints (Ops 6–7a)
d Z ~ c ← d Z c + d S c d\tilde{Z}_c \leftarrow dZ_c+dS_c d Z ~ c ← d Z c + d S c ; d Z c − 1 ← μ κ c d Z ~ c dZ_{c-1}\leftarrow\mu\kappa_c d\tilde{Z}_c d Z c − 1 ← μ κ c d Z ~ c ▷ velocity carry (κ c = 0 \kappa_c=0 κ c = 0 at doc reset)
d E c ← K ^ c d Z ~ c + L c T d O c ∈ R C × R d v dE_c \leftarrow\hat{K}_c d\tilde{Z}_c+L_c^\mathsf{T}dO_c \in\mathbb{R}^{C\times R_{dv}} d E c ← K ^ c d Z ~ c + L c T d O c ∈ R C × R d v ; d K ^ c ← E c d Z ~ c T ∈ R C × d k d\hat{K}_c\leftarrow E_c d\tilde{Z}_c^\mathsf{T}\in\mathbb{R}^{C\times d_k} d K ^ c ← E c d Z ~ c T ∈ R C × d k ▷ MXU
d L c ← tril 0 ( d O c E c T ) ∈ R C × C dL_c\leftarrow\operatorname{tril}_0(dO_cE_c^\mathsf{T})\in\mathbb{R}^{C\times C} d L c ← tril 0 ( d O c E c T ) ∈ R C × C ; d Q ˉ c ← d O c S c − 1 T ∈ R C × d k d\bar{Q}_c\leftarrow dO_cS_{c-1}^\mathsf{T}\in\mathbb{R}^{C\times d_k} d Q ˉ c ← d O c S c − 1 T ∈ R C × d k ▷ MXU
// 3. Transposed WY inverse and value/key adjoints on MXU (Ops 4–5)
D c ← A c T diag ( β c ) d E c ∈ R C × R d v D_c\leftarrow A_c^\mathsf{T}\operatorname{diag}(\beta_c)dE_c\in\mathbb{R}^{C\times R_{dv}} D c ← A c T diag ( β c ) d E c ∈ R C × R d v ; d V c ← D c dV_c\leftarrow D_c d V c ← D c ; d K ˉ c ← − D c S c − 1 T d\bar{K}_c\leftarrow-D_cS_{c-1}^\mathsf{T} d K ˉ c ← − D c S c − 1 T ▷ MXU
d M c ← − tril − 1 ( D c E ~ c T ) ∈ R C × C dM_c\leftarrow-\operatorname{tril}_{-1}(D_c\tilde{E}_c^\mathsf{T})\in\mathbb{R}^{C\times C} d M c ← − tril − 1 ( D c E ~ c T ) ∈ R C × C ▷ contracts R d v R_{dv} R d v packed value cols on MXU
d S c − 1 ← F c T d S c − K ˉ c T D c + Q ˉ c T d O c ∈ R d k × R d v dS_{c-1}\leftarrow F_c^\mathsf{T}dS_c-\bar{K}_c^\mathsf{T}D_c+\bar{Q}_c^\mathsf{T}dO_c\in\mathbb{R}^{d_k\times R_{dv}} d S c − 1 ← F c T d S c − K ˉ c T D c + Q ˉ c T d O c ∈ R d k × R d v ▷ prev. chunk state carry (K ˉ c T D c = W c T d E c \bar{K}_c^\mathsf{T}D_c=W_c^\mathsf{T}dE_c K ˉ c T D c = W c T d E c )
// 4. Tile-grouped Q Q Q , K K K and gate adjoints (G = 4 G=4 G = 4 tiles of b = 64 b=64 b = 64 channels)
for g ∈ { 1 , 2 , 3 , 4 } g\in\{1,2,3,4\} g ∈ { 1 , 2 , 3 , 4 } do
d M ~ c , g ← d M c ⊙ Γ g d\tilde{M}_{c,g}\leftarrow dM_c\odot\Gamma_g d M ~ c , g ← d M c ⊙ Γ g ; d L ~ c , g ← d L c ⊙ Ψ g d\tilde{L}_{c,g}\leftarrow dL_c\odot\Psi_g d L ~ c , g ← d L c ⊙ Ψ g
d Q c , g ← d L ~ c , g K c , g + diag ( p c , g i n ) ( d Q ˉ c ) : , g dQ_{c,g}\leftarrow d\tilde{L}_{c,g}K_{c,g}+\operatorname{diag}(p_{c,g}^{\mathrm{in}})(d\bar{Q}_c)_{:,g} d Q c , g ← d L ~ c , g K c , g + diag ( p c , g in ) ( d Q ˉ c ) : , g ▷ 64-wide MXU matmul; ( ⋅ ) : , g (\cdot)_{:,g} ( ⋅ ) : , g is tile-g g g slice
d K c , g ← ( d M ~ c , g + d M ~ c , g T ) K c , g + d L ~ c , g T Q c , g + diag ( p c , g i n ) ( d K ˉ c ) : , g + diag ( p c , g o u t ) ( d K ^ c ) : , g dK_{c,g}\leftarrow(d\tilde{M}_{c,g}+d\tilde{M}_{c,g}^{\mathsf{T}})K_{c,g}+d\tilde{L}_{c,g}^{\mathsf{T}}Q_{c,g}+\operatorname{diag}(p_{c,g}^{\mathrm{in}})(d\bar{K}_c)_{:,g}+\operatorname{diag}(p_{c,g}^{\mathrm{out}})(d\hat{K}_c)_{:,g} d K c , g ← ( d M ~ c , g + d M ~ c , g T ) K c , g + d L ~ c , g T Q c , g + diag ( p c , g in ) ( d K ˉ c ) : , g + diag ( p c , g out ) ( d K ^ c ) : , g
end for ▷ concatenate d Q c = [ d Q c , 1 , … , d Q c , 4 ] dQ_c=[dQ_{c,1},\ldots,dQ_{c,4}] d Q c = [ d Q c , 1 , … , d Q c , 4 ] , d K c = [ d K c , 1 , … , d K c , 4 ] dK_c=[dK_{c,1},\ldots,dK_{c,4}] d K c = [ d K c , 1 , … , d K c , 4 ]
d β c ← rowsum ( d E c ⊙ E ~ c ) + colsum d M c ⊙ ∑ g = 1 4 Ψ g ⊙ ( K c , g K c , g T ) d\beta_c\leftarrow\operatorname{rowsum}(dE_c\odot\tilde{E}_c)+\operatorname{colsum}dM_c\odot\sum_{g=1}^{4}\Psi_g\odot(K_{c,g}K_{c,g}^{\mathsf{T}}) d β c ← rowsum ( d E c ⊙ E ~ c ) + colsum d M c ⊙ ∑ g = 1 4 Ψ g ⊙ ( K c , g K c , g T )
Accumulate columns d γ c , g d\gamma_{c,g} d γ c , g of d γ c d\gamma_c d γ c via reverse prefix sum over d M c , d L c , d K ˉ c , d Q ˉ c , d K ^ c dM_c,dL_c,d\bar{K}_c,d\bar{Q}_c,d\hat{K}_c d M c , d L c , d K ˉ c , d Q ˉ c , d K ^ c , and d S c S c − 1 T dS_cS_{c-1}^{\mathsf{T}} d S c S c − 1 T
end for
return { d Q c , d K c , d V c , d β c , d γ c } c = 1 T / C \{dQ_c,dK_c,dV_c,d\beta_c,d\gamma_c\}_{c=1}^{T/C} { d Q c , d K c , d V c , d β c , d γ c } c = 1 T / C
Empirical Results
We evaluate SLA on TPU v7x (Ironwood [6 , 18 ] ) at four levels: single-layer kernels and multi-generation rooflines (§ 3.1 ), long-context serving (§ 3.2 ), context-parallel training and end-to-end 12.5B MoE cube serving (§ 3.3 ), and multi-scale MoE pre-training throughput and quality (§ 3.4 –§ 3.5 ). We use two configurations with d k = 256 d_k=256 d k = 256 , G = 4 G=4 G = 4 decay tiles, chunk size C = 128 C=128 C = 128 , and H v = 8 H_v=8 H v = 8 value heads: SLA-R2 shares each query-key head across 2 value heads (H q = H k = 4 H_q=H_k=4 H q = H k = 4 , where H q H_q H q is the query-head count) and SLA-R4 across 4 (H q = H k = 2 H_q=H_k=2 H q = H k = 2 ); layer shapes are written H k × d k × R d v H_k\times d_k\times R_{dv} H k × d k × R d v . xL:1A denotes x linear attention layers per full softmax attention layer, so 0L:1A is a pure full softmax attention Transformer. SLA keeps the baseline’s value width, d v = 256 d_v=256 d v = 256 at model width D = 2048 D=2048 D = 2048 and 512 at D = 1792 D=1792 D = 1792 , so SLA’s BF16 state has twice the entries of FP32 KDA with d k = 128 d_k=128 d k = 128 in the same memory, and the same entries as the 63.1B KDA with d k = 256 d_k=256 d k = 256 in half the memory. SLA Iso-State instead halves d v d_v d v to match KDA’s element count in half the bytes. Part of the measured speedup therefore comes from precision; a BF16-state KDA is only slightly faster than FP32 KDA (Table 6 , footnote). All SLA pre-training runs except ablation arms A3 and A5 (Table 11 b), and SLA Tree-CP (Table 8 c), use Exact Inv.; 32-layer prefill and tree verification (Table 8 a–Table 8 b) use 1-Step Neumann, and single-layer benchmarks report both (Table 5 –Table 6 ).
Table 6: Measured TPU v7x (batch size B = 1 B = 1 B = 1 ) kernel and full-layer latency (ms), 32K temporary HBM (8K is ∼0.26×), and F+B speedups normalized to Open-Source Tokamax Mosaic KDA Base (1.00×) in each panel across (a) core kernels at D = 1792 D = 1792 D = 1792 , (b) iso-controlled core kernels at D = 2048 D = 2048 D = 2048 , and (c1–c2) full linear attention layers (Ops 1–7c, unified harness; Tokamax supports C ≤ 64 C \le64 C ≤ 64 and d k ≤ 256 d_k \le256 d k ≤ 256 ); bold marks the SLA-R4 rows.
Table 6. Measured TPU v7x kernel and full-layer latency.
Kernel Performance and Multi-Generation TPU Rooflines
Fusion and stage breakdown. Because every intra-chunk stage of SLA runs on the MXU, the fused kernel keeps Ops 2–7a in the 64 MiB VMEM scratchpad and recomputes M c M_c M c and A c A_c A c in the backward pass instead of storing them in HBM (§ 2.5 and Figure 4). Table 5 shows where the time goes. Fusing GPU-shaped KDA gains only 1.03×. Reshaping the layer without fusion (SLA-R4) gives 1.79×, mostly from the Gram/inversion stages and the state scan. Fusion then brings the total to 2.38×.
Comparison with open-source TPU baselines. Table 6 compares SLA with MaxText Pallas KDA [1 ] , Tokamax Mosaic KDA [19 ] (the fastest baseline; it supports only C ≤ 64 C\le64 C ≤ 64 ), and unpacked scalar GDN. The iso-controls (b) show that the speedup depends on head sharing: without 4:1 sharing,
Stateless functional harness (1-Step Neumann for SLA-R4). Fused: one kernel for Ops 2–7a; full NNX module latencies in Table 6 c.
d k = 256 d_k = 256 d k = 256 SLA variants run at 0.67×–0.77× of Tokamax, and only SLA-R4 is faster (1.09×–1.17×). For full layers (c1–c2), fused SLA-R4 is 1.6×–2.5× faster than Tokamax and needs 52%–61% less temporary HBM; with the exact inverse it is 1.4×–2.4× faster. These comparisons change the model (fewer key heads, BF16 state, grouped decay) as well as the kernel. We did not build a fused kernel for KDA or GDN with the same d k = 256 d_k = 256 d k = 256 , R = 4 R = 4 R = 4 head sharing; that control would separate the contribution of the tile-grouped gate from that of head sharing and fusion.
Multi-generation TPU rooflines and FP8 skew. On an identical portable kernel stack, SLA-R4’s forward speedup grows with the MXU:VPU ratio from 4.5× on TPU v4 to 6.7× on TPU v7x (Figure 5 and Table 7 ). Specifically, Table 7 reports measured single-core forward core-kernel latencies (B = 1 B = 1 B = 1 , T = 8,192 T = 8{,}192 T = 8 , 192 , D = 2048 D = 2048 D = 2048 ) and out-of-fold leave-one-out (LOO) roofline speedup predictions across TPU v4, v5p, v6e, and v7x. To keep the software implementation identical across all four hardware generations (since Tokamax Mosaic targets v6e/v7x at C ≤ 64 C \le64 C ≤ 64 ), Table 7 compares SLA-R4 against the uniform portable JAX/Pallas KDA (C = 64 C = 64 C = 64 , FP32; 6.99 ms on v7x) baseline on every chip, whereas Table 6 compares against optimized open-source MaxText Pallas (C = 128 C = 128 C = 128 , 2.67 ms fwd) and Tokamax Mosaic KDA (C = 64 C = 64 C = 64 , 1.07 ms fwd) on TPU v7x.
Figure 5. Forward speedup of SLA-R4 across TPU generations over a portable FP32 KDA baseline, with leave-one-out roofline predictions (3.1% error). Not comparable to the Tokamax-normalized speedups in Table 6 .
Table 7.
On a single TensorCore with systolic tile width W ∈ { 128 , 256 } W \in\{128, 256\} W ∈ { 128 , 256 } and single-core MXU and VPU peaks P M X U , P V P U P_{\mathrm{MXU}}, P_{\mathrm{VPU}} P MXU , P VPU (per-chip values in Table 1 divided by Cores), let ( d ~ k , D ~ v , p a c k ) (\tilde{d}_k,\tilde{D}_{v,\mathrm{pack}}) ( d ~ k , D ~ v , pack ) denote ( d k , D v , p a c k ) (d_k,D_{v,\mathrm{pack}}) ( d k , D v , pack ) padded to multiples of W W W , F s t e p = 2 max ( C , W ) d ~ k D ~ v , p a c k F_{\mathrm{step}} = 2\max(C,W)\tilde{d}_k\tilde{D}_{v,\mathrm{pack}} F step = 2 max ( C , W ) d ~ k D ~ v , pack the FLOP count of one per-step scan matmul, τ p i p e ≈ 1.5 \tau_{\mathrm{pipe}} \approx1.5 τ pipe ≈ 1.5 –2.5 μ s 2.5\,\mu\mathrm{s} 2.5 μ s the per-head in-VMEM Pallas scan-pipeline fill/drain overhead (smaller than the top-level XLA lax.scan HBM-barrier cost τ s y n c ≈ 12 μ s \tau_{\mathrm{sync}} \approx12\,\mu\mathrm{s} τ sync ≈ 12 μ s per chunk step in Figure 2 ), η s y s ≈ 8.6 % \eta_{\mathrm{sys}} \approx8.6\% η sys ≈ 8.6% the effective single-core scan MXU utilization, γ i n v \gamma_{\mathrm{inv}} γ inv the VPU operation count per entry of the triangular solve (unrelated to the decay rates γ t , g \gamma_{t,g} γ t , g ), and Φ M X U ( C ) = O ( H k T C d k + H v T C d v ) \Phi_{\mathrm{MXU}}(C) = O(H_kT C d_k + H_vT C d_v) Φ MXU ( C ) = O ( H k T C d k + H v T C d v ) the intra-chunk matmul FLOPs. We model single-core forward latency T f w d ( C ) T_{\mathrm{fwd}}(C) T fwd ( C ) at chunk size C C C as
T f w d ( C ) ≈ H k τ C ( τ p i p e + F s t e p η s y s P M X U ) + γ i n v H k T C P V P U + Φ M X U ( C ) P M X U . (11) T_{\mathrm{fwd}}(C) \approx H_k \frac{\tau}{C}\left(\tau_{\mathrm{pipe}} + \frac{F_{\mathrm{step}}}{\eta_{\mathrm{sys}}P_{\mathrm{MXU}}}\right) + \frac{\gamma_{\mathrm{inv}}H_kTC}{P_{\mathrm{VPU}}} + \frac{\Phi_{\mathrm{MXU}}(C)}{P_{\mathrm{MXU}}}. \tag*{(11)} T fwd ( C ) ≈ H k C τ ( τ pipe + η sys P MXU F step ) + P VPU γ inv H k T C + P MXU Φ MXU ( C ) . (11)
Across the four hardware generations in Table 7 , parameter-free two-cluster comparison directly isolates the 256-tile transition: on 128 × 128 128 \times128 128 × 128 -tile accelerators (0% KDA key-tile padding, α m v ≈ 85 × \alpha_{\mathrm{mv}} \approx85\times α mv ≈ 85 × ), SLA-R4 already achieves 4.47 × 4.47\times 4.47 × –4.64 × 4.64\times 4.64 × speedup (TPU v4 and v5p), and stepping to 256 × 256 256 \times256 256 × 256 -tile accelerators (50% KDA key-tile padding at d k = 128 d_k = 128 d k = 128 , α m v ≈ 128 × \alpha_{\mathrm{mv}} \approx128\times α mv ≈ 128 × ) lifts the speedup by a further 1.3 × 1.3\times 1.3 × –1.46 × 1.46\times 1.46 × to 5.86 × 5.86\times 5.86 × –6.65 × 6.65\times 6.65 × (TPU v6e and v7x; reduced-form linearized LOO fit shown for reference in Table 7 ). Separately, comparing SLA-R4 against the standalone JAX/Pallas C = 128 C = 128 C = 128 KDA core kernel on the isolated FP8 MXU microbenchmark harness (1.975 ms BF16, 2.168 ms E4M3 FP8, excluding MaxText’s segment-reset and L 2 L_2 L 2 -norm wrapper included in the 2.668 ms row of Table 6 b) yields a 1.69× forward speedup in native E4M3 FP8 MXU mode (1.280 vs. 2.168 ms at α m v = 256 × \alpha_{\mathrm{mv}} = 256\times α mv = 256 × ), matching BF16’s 1.67× (1.181 vs. 1.975 ms) on the same harness.
Table 7 : Measured forward core-kernel latency (ms, B = 1 , T = 8,192 , D = 2048 B = 1, T = 8{,}192, D = 2048 B = 1 , T = 8 , 192 , D = 2048 ) vs. portable JAX/Pallas KDA (C = 64 C = 64 C = 64 ) and out-of-fold LOO Roofline predictions across four TPU generations.
Long-Context Serving
Prefill latency and prefix caching. In 32-layer hybrid stacks with SwiGLU FFNs [51 ] (Table 8 a), 15L:1A SLA-R4 (one softmax layer per 16) reaches a 131K time-to-first-token (TTFT) of 1.67 s, compared with 2.66 s for the same hybrid layout with KDA and 7.14 s for full grouped-query attention (GQA) [2 ] . Most of the gain over GQA comes from the hybrid layout itself; the gain attributable to
Figure 6. Context parallelism and serving on TPU v7x. (a) Forward-backward latency of context-parallel scans across 8–128 TensorCores (§ 3.3 ). (b) Single-chip 32-layer serving throughput and 131K prefill latency (Table 8 ).
SLA over KDA is 1.6 × 1.6\times 1.6 × . A cached SLA state is also much smaller than a KV cache (0.5–1 MiB per layer versus 64 MiB at 32K), which makes multi-turn prefix caching cheap.
Speculative verification. Single-token decoding underuses the MXU, so we verify T v e r ∈ { 8 , 16 } T_{\mathrm{ver}} \in\{8, 16\} T ver ∈ { 8 , 16 } drafted tokens per step [35 ] with the same block inversion (Table 8 b). At equal state size (SLA Iso-State), SLA is 1.03 × 1.03\times 1.03 × –1.22 × 1.22\times 1.22 × faster than KDA on the attention layers, and 2%–7% faster once FFNs are included. SLA-R4, which holds twice KDA’s state elements, is slower than KDA at short context (84.3K vs. 117.7K tok/s at 8K), as expected from its larger state reads. Table 9 c reports a 12.5B MoE speculative-decoding setting in which SLA achieves longer greedy acceptance lengths than unclamped KDA on out-of-distribution 32K/64K HotpotQA (1.17×–1.33× vs. clamped KDA; 1.06×–1.10× under stochastic sampling, whereas KDA is slightly higher in-domain at 8K).
Table 9. Measured 64-chip TPU v7x (4×4×4, 128 TensorCores) end-to-end 16-layer 12.5B sparse MoE (0.79B active, 256 experts k = 8 k = 8 k = 8 ) (a) fixed-CP=EP=128 F+B latency and tok/s/chip across T = 128 T = 128 T = 128 K–1.05M (T l o c = T / 128 T_{\mathrm{loc}} = T/128 T loc = T /128 ; at T = 512 T = 512 T = 512 K, T l o c = 4,096 T_{\mathrm{loc}} = 4{,}096 T loc = 4 , 096 selects a faster XLA EP=128 MoE ragged-all-to-all tile schedule across all linear hybrids relative to T = 256 T = 256 T = 256 K), (b) EP=128 cube verification throughput (B c h i p = 64 B_{\mathrm{chip}} = 64 B chip = 64 , comparing Tree-N1 [= Tree-Neumann-1, T v e r = 16 T_{\mathrm{ver}} = 16 T ver = 16 ] and FP8Fused seq_scan_t16), and (c) real-checkpoint lossless speculative acceptance τ ˉ \bar{\tau} τ ˉ (K = 16 K = 16 K = 16 , N = 64 N = 64 N = 64 , greedy [stochastic]).
Context-Parallel Training and End-to-End 12.5B MoE Cube Serving
Context parallelism with a prefix-tree scan. Head-sharded Ulysses [29 ] needs at least as many key heads as devices, which SLA’s H k ∈ { 2 , 4 } H_k \in\{2, 4\} H k ∈ { 2 , 4 } do not provide, and ring-based LASP [56 ] passes the state through P − 1 P - 1 P − 1 sequential hops across P P P devices. For any gated delta rule with scalar β t \beta_t β t , a chunk’s state update is an affine map S ↦ J S + B S \mapsto JS + B S ↦ J S + B with a d k × d k d_k \times d_k d k × d k transition J ∈ R d k × d k J \in\mathbb{R}^{d_k \times d_k} J ∈ R d k × d k and offset B ∈ R d k × D v , p a c k B \in\mathbb{R}^{d_k \times D_{v,\mathrm{pack}}} B ∈ R d k × D v , pack , so P P P devices can reduce their local chunks independently and combine them in ⌈ log 2 P ⌉ \lceil\log_2 P \rceil ⌈ log 2 P ⌉ prefix-tree steps [11 , 39 ] . This construction is not specific to SLA, and our KDA baseline uses the same tree. SLA’s only advantage here is that one 256 × 256 256 \times256 256 × 256 transition serves all R R R value heads of a key head, so fewer transitions are exchanged; chunk momentum (μ = 0.25 \mu= 0.25 μ = 0.25 ) doubles the carried state to 512 × 512 512 \times512 512 × 512 . Table 8 c reports both settings on 32–128 TensorCores. With momentum, as trained, the tree scan is roughly on par with an AllGather-based LASP-2 baseline [57 ] (0.85×–1.37×); without momentum it is 1.3×–1.8× faster. Both are much faster than ring-based LASP at large P P P . In end-to-end 12.5B MoE training at 1M context (Table 9 a), 15L:1A SLA-R2 steps 13% faster than KDA using the same tree scan, consistent with its layer-level speedup.
Affine monoid derivation for uncoupled and coupled SLA Tree-CP. Shard p ∈ { 1 , … , P } p \in\{1, \ldots, P\} p ∈ { 1 , … , P } holds L l o c L_{\mathrm{loc}} L loc consecutive chunks; quantities of its c c c -th local chunk (c ∈ { 1 , … , L l o c } c \in\{1, \ldots, L_{\mathrm{loc}}\} c ∈ { 1 , … , L loc } ) carry the subscript ( p , c ) (p,c) ( p , c ) , with shard input state S p , 0 S_{p,0} S p , 0 and shard output state S p , L l o c S_{p,L_{\mathrm{loc}}} S p , L loc . Substituting E p , c = U p , c − W p , c S p , c − 1 E_{p,c} = U_{p,c} - W_{p,c}S_{p,c-1} E p , c = U p , c − W p , c S p , c − 1 (§ 2.4.2 ) into S ~ C = F p , c S p , c − 1 + K ^ p , c ⊤ E p , c \widetilde{S}_C = F_{p,c}S_{p,c-1} + \widehat{K}_{p,c}^{\top}E_{p,c} S C = F p , c S p , c − 1 + K p , c ⊤ E p , c gives per-chunk transition J p , c s t e p ∈ R d k × d k J_{p,c}^{\mathrm{step}} \in\mathbb{R}^{d_k \times d_k} J p , c step ∈ R d k × d k and offset B p , c ∈ R d k × D v , p a c k B_{p,c} \in\mathbb{R}^{d_k \times D_{v,\mathrm{pack}}} B p , c ∈ R d k × D v , pack such that, without momentum, each chunk is the affine map
S p , c = J p , c s t e p S p , c − 1 + B p , c , S_{p,c} = J_{p,c}^{\mathrm{step}} S_{p,c-1} + B_{p,c}, S p , c = J p , c step S p , c − 1 + B p , c ,
J p , c s t e p = F p , c − K ^ p , c ⊤ W p , c , B p , c = K ^ p , c ⊤ U p , c ; J_{p,c}^{\mathrm{step}} = F_{p,c} - \widehat{K}_{p,c}^{\top}W_{p,c}, \qquad B_{p,c} = \widehat{K}_{p,c}^{\top}U_{p,c}; J p , c step = F p , c − K p , c ⊤ W p , c , B p , c = K p , c ⊤ U p , c ;
J p , c s t e p J_{p,c}^{\mathrm{step}} J p , c step does not depend on the values, so one transition serves all D v , p a c k D_{v,\mathrm{pack}} D v , pack value columns of a key head (this holds for any gated delta rule, since the transition only involves keys and gates). With μ = 0 \mu= 0 μ = 0 , the L l o c L_{\mathrm{loc}} L loc local chunks compose into a single shard map ( J p , B p ) (J_p, B_p) ( J p , B p ) satisfying S p , L l o c = J p S p , 0 + B p S_{p,L_{\mathrm{loc}}} = J_pS_{p,0} + B_p S p , L loc = J p S p , 0 + B p , where J p = ∏ c = 1 L l o c J p , c s t e p J_p = \prod_{c=1}^{L_{\mathrm{loc}}} J_{p,c}^{\mathrm{step}} J p = ∏ c = 1 L loc J p , c step (ordered from c = 1 c = 1 c = 1 on the right to c = L l o c c = L_{\mathrm{loc}} c = L loc on the left) and B p B_p B p is the shard output S p , L l o c S_{p,L_{\mathrm{loc}}} S p , L loc obtained from S p , 0 = 0 S_{p,0} = 0 S p , 0 = 0 . (With μ > 0 \mu> 0 μ > 0 , the same construction applies to the lifted maps below.) Writing shard maps as pairs ( J a , B a ) (J_a, B_a) ( J a , B a ) and ( J b , B b ) (J_b, B_b) ( J b , B b ) for shards a a a and b b b with composition (apply a a a , then b b b )
( J b , B b ) ∘ ( J a , B a ) = ( J b J a , J b B a + B b ) , (J_b, B_b) \circ(J_a, B_a) = (J_bJ_a, J_bB_a + B_b), ( J b , B b ) ∘ ( J a , B a ) = ( J b J a , J b B a + B b ) ,
for any third shard d d d with map ( J d , B d ) (J_d, B_d) ( J d , B d ) , matrix associativity and left-distributivity yield
( ( J d , B d ) ∘ ( J b , B b ) ) ∘ ( J a , B a ) = ( J d J b J a , J d J b B a + J d B b + B d ) = ( J d , B d ) ∘ ( ( J b , B b ) ∘ ( J a , B a ) ) , \begin{aligned}
((J_d, B_d) \circ(J_b, B_b)) \circ(J_a, B_a) = (J_dJ_bJ_a, J_dJ_bB_a + J_dB_b + B_d) \\
&= (J_d, B_d) \circ((J_b, B_b) \circ(J_a, B_a)),
\end{aligned} (( J d , B d ) ∘ ( J b , B b )) ∘ ( J a , B a ) = ( J d J b J a , J d J b B a + J d B b + B d ) = ( J d , B d ) ∘ (( J b , B b ) ∘ ( J a , B a )) ,
so the P P P shard maps can be combined by any parallel prefix scan in ⌈ log 2 P ⌉ \lceil\log_2 P \rceil ⌈ log 2 P ⌉ rounds.
When the velocity Z c ∈ R d k × D v , p a c k Z_c \in\mathbb{R}^{d_k \times D_{v,\mathrm{pack}}} Z c ∈ R d k × D v , pack is carried across shard boundaries (μ > 0 \mu> 0 μ > 0 , SLA Tree-CP+Mom; shard subscript p p p dropped, with per-chunk transition J c s t e p J_c^{\mathrm{step}} J c step and offset B c B_c B c ), we have Δ S c = S ~ C − F c S c − 1 = ( J c s t e p − F c ) S c − 1 + B c \Delta S_c = \widetilde{S}_C - F_cS_{c-1} = (J_c^{\mathrm{step}} - F_c)S_{c-1} + B_c Δ S c = S C − F c S c − 1 = ( J c step − F c ) S c − 1 + B c . Substituting into Eq. (4) (Z c = μ κ c Z c − 1 + Δ S c Z_c = \mu\kappa_c Z_{c-1} + \Delta S_c Z c = μ κ c Z c − 1 + Δ S c , S c = F c S c − 1 + Z c S_c = F_cS_{c-1} + Z_c S c = F c S c − 1 + Z c ) defines a lifted 2 d k × 2 d k 2d_k \times2d_k 2 d k × 2 d k transition J ~ c ∈ R 2 d k × 2 d k \widetilde{J}_c \in\mathbb{R}^{2d_k \times2d_k} J c ∈ R 2 d k × 2 d k and offset B ~ c ∈ R 2 d k × D v , p a c k \widetilde{B}_c \in\mathbb{R}^{2d_k \times D_{v,\mathrm{pack}}} B c ∈ R 2 d k × D v , pack updating the stacked carry [ S c ; Z c ] ∈ R 2 d k × D v , p a c k [S_c; Z_c] \in\mathbb{R}^{2d_k \times D_{v,\mathrm{pack}}} [ S c ; Z c ] ∈ R 2 d k × D v , pack via the exact block-affine map
\begin{bmatrix} S_c \\ Z_c \end{bmatrix}
=
\underbrace{\begin{bmatrix}
J_c^{\mathrm{step}} & \mu\kappa_c I_{d_k} \\
J_c^{\mathrm{step}} - F_c & \mu\kappa_c I_{d_k}
\end{bmatrix}}_{\widetilde{J}_c}
\begin{bmatrix} S_{c-1} \\ Z_{c-1} \end{bmatrix}
+
\underbrace{\begin{bmatrix} B_c \\ B_c \end{bmatrix}}_{\widetilde{B}_c}. \tag*{(12)}\end{bmatrix}
For consecutive shards a a a and b b b , the shard-reduced lifted maps ( J ~ a , B ~ a ) (\widetilde{J}_a,\widetilde{B}_a) ( J a , B a ) and ( J ~ b , B ~ b ) (\widetilde{J}_b,\widetilde{B}_b) ( J b , B b ) obey the identical associative composition law ( J ~ b , B ~ b ) ∘ ( J ~ a , B ~ a ) = ( J ~ b J ~ a , J ~ b B ~ a + B ~ b ) (\widetilde{J}_b,\widetilde{B}_b) \circ(\widetilde{J}_a,\widetilde{B}_a) = (\widetilde{J}_b\widetilde{J}_a,\widetilde{J}_b\widetilde{B}_a + \widetilde{B}_b) ( J b , B b ) ∘ ( J a , B a ) = ( J b J a , J b B a + B b ) , with [ 512 , 512 ] [512,512] [ 512 , 512 ] transitions for d k = 256 d_k = 256 d k = 256 . □
End-to-end 12.5B sparse MoE 1M-token SLA Tree-CP, cube serving, and speculative acceptance. To complement the isolated 128-core SLA Tree-CP kernel and single-chip 32-layer dense-FFN serving benchmarks in Table 8 , Table 9 benchmarks a full 16-layer 12.5B sparse MoE (0.79B active, E = 256 E = 256 E = 256 routed experts with k = 8 k = 8 k = 8 active per token plus one shared expert, D = 1024 D = 1024 D = 1024 , D v = 2048 D_v = 2048 D v = 2048 , vocabulary size V = 100,352 V = 100{,}352 V = 100 , 352 ) end-to-end across all 64 chips (128 TensorCores) of a TPU v7x 4x4x4 cube across three workloads:
End-to-End CP=EP=128 Training Steps up to 1.05M Tokens (Table 9 a): With vocab-chunked cross-entropy across all 24 grid cells, uncoupled 15L:1A SLA Tree-CP R2 (2×-state 4 × 256 2 256^2 25 6 2 ”】【
Jacobians) completes T = 1,048,576 T = 1{,}048{,}576 T = 1 , 048 , 576 (per-core local sequence length T l o c = 8,192 T_{\mathrm{loc}} = 8{,}192 T loc = 8 , 192 ) in 4,737.6 ms (3,458 tok/s/chip; 4,904.0 ms and 3,341 tok/s/chip w/ +Mom), running +12.7% faster than our affine-tree port to 1×-state KDA (15L:1A KDA Tree, 8 × 128², 5,337.6 ms, 3,070 tok/s/chip, 2,760 MiB), 7.75× faster than 0L:1A Ring-CP (36,725.0 ms, 446 tok/s/chip), and cutting mixer ICI by −83% vs. 15L:1A Ring-LASP (5,204.6 ms, 2,992 vs. 17,392 MiB/rank). At T = 524,288 T = 524{,}288 T = 524 , 288 (T l o c = 4,096 T_{\mathrm{loc}} = 4{,}096 T loc = 4 , 096 /core), 15L:1A SLA Tree-CP R2 reaches 4,170 tok/s/chip (1,965 ms uncoupled, 5.45× vs. 0L:1A Ring-CP 10,712.1 ms; 2,003.1 ms +Mom, 5.35×).
Full 64-Chip EP=128 MoE Cube Serving and FP8Fused Scan (Table 9 b): Across per-chip batch size B c h i p = 64 B_{\mathrm{chip}} = 64 B chip = 64 (4,096 sequences/cube), our fused Pallas E4M3 kernel (FP8Fused, keeping row-scaled E4M3 DMA, dequantization, 16-token delta scan, and requantization inside VMEM) lifts 16-step sequential scan (seq_scan_t16) at 8K to 1,091.9K tok/s/cube (60.02 ms) on 15L SLA Iso-State (+18.8% vs. 15L KDA FP32 919.4K, +14.5% vs. iso-precision JAX 15L KDA FP8 953.6K, and +14.6% vs. pure-JAX SLA FP8 952.9K) and 1,047.2K–1,052.3K tok/s/cube (+10.3% vs. JAX KDA FP8 and +28.1% vs. pure-JAX SLA FP8 817.6K) on 15L SLA-R2; applying the same FP8Fused recurrent scan kernel to KDA reaches 1,083.0K (within 0.8% of SLA Iso-State, since token-level recurrent decode involves no chunk WQV solve). With parallel Tree-Neumann-1 (T v e r = 16 T_{\mathrm{ver}} = 16 T ver = 16 ), raw verification reaches 1.121M–1.158M tok/s/cube at 8K and sustains 1.037M–1.042M tok/s/cube in 16.24–16.47 GiB/chip (−93.6% vs. 0L:1A OOM) at 131K.
Real-Checkpoint Lossless Speculative Acceptance incl. Doubly-Clamped KDA (Table 9 c): Under exact lossless rejection [35 ] (draft length K = 16 K = 16 K = 16 , N = 64 N = 64 N = 64 prompts/site), zero-clamp 0.35B SLA → 12.5B 7L SLA-R4 (iso-param) (60.4B) lifts greedy acceptance τ ˉ \bar{\tau} τ ˉ (mean accepted tokens per draft step) on 32K/64K HoptoqQA from 2.63/2.14 (unclamped KDA) to 5.86/4.80 (2.23 × /2.23 × vs. 5.68/4.66 in FP8 PTO), and still beats doubly-clamped KDA (0.35B KDA TrainMaxTau=8K → 12.5B 7L KDA with retention-horizon cap τ max = 8 \tau_{\max} = 8 τ m a x = 8 K, § 3.5 : 4.42/4.09 greedy, 1.72/1.65 stochastic) by 1.33 × /1.17× greedy (1.06 × /1.10 × stochastic) on OOD 32K/64K HoptoqQA (379.8K vs. 287.6K greedy [1.32×] and 118.0K vs. 111.9K stochastic accepted tok/s/cube at 32K, while KDA edges in-distribution 8K Dolma 4.84–5.30 vs. 4.56).
MoE Pre-Training Setup, Architecture, and Throughput
All pre-training runs within each cohort share identical optimizer, tokenization, MoE routing, and batch-ordering configurations so that the attention layer design is the sole independent variable:
Optimizer and Schedule: AdamW [38 ] (moment coefficients ( 0.9 , 0.95 ) (0.9, 0.95) ( 0.9 , 0.95 ) , numerical stability constant ϵ = 10 − 8 \epsilon= 10^{-8} ϵ = 1 0 − 8 , weight decay 0.1 0.1 0.1 with embedding/norm exclusion), global gradient norm clip 1.0 1.0 1.0 , router load-balancing weight 0.01 0.01 0.01 , z-loss 10 − 5 10^{-5} 1 0 − 5 , peak learning rate 3.0 × 10 − 4 3.0 \times10^{-4} 3.0 × 1 0 − 4 with 5 % 5\% 5% linear warmup and cosine decay to 3.0 × 10 − 5 3.0 \times10^{-5} 3.0 × 1 0 − 5 (plus tuned LR = 4.5 × 10 − 4 = 4.5 \times10^{-4} = 4.5 × 1 0 − 4 with RoPE base frequency θ max = 5 × 10 5 \theta_{\max} = 5 \times10^{5} θ m a x = 5 × 1 0 5 [55 ] for fair 0L:1A GQA+RoPE baselines across all scales). +MS denotes multi-scale depth-staggered decay bias initialization across linear layers.
Corpus, FP8 State PTQ, and QA Scoring Protocol: Real Dolma 3.5 [54 , 60 , 61 ] streams (training sequence length T t r a i n = 8,192 T_{\mathrm{train}} = 8{,}192 T train = 8 , 192 ) with deterministic OlmoCoreOrderSampler batch parity and document-boundary EOS resets (eos_id = 100257 = 100257 = 100257 ). In-loop validation tracks geometric-mean PPL across all 11 OLMo v3 validation sets. Post-training evaluation covers: (1) a disjoint 16.78M-token (512 × 32 512 \times32 512 × 32 K) Dolma 3 split (4K, 8K, 16K, 32K full and Tail-4K PPL; true row-scaled E4M3 FP8 recurrent-state PTQ shifts 8K and 32K NLL by only + 0.00001 +0.00001 + 0.00001 and + 0.00006 +0.00006 + 0.00006 nats with 0.00 % 0.00\% 0.00% NIAH loss across SLA checkpoints, whereas an earlier post-layer activation-quantization probe altered 32K Tail PPL by − 0.07 -0.07 − 0.07 to − 0.54 -0.54 − 0.54 and is excluded from state-PTQ claims), (2) multi-needle Passkey (MK-NIAH 4/8-Needle, N = 256 N = 256 N = 256 –512 512 512 evaluation items), (3) 9-task zero-shot lm-eval [17 ] across 11,793 items (1 SE ≈ 0.65 \approx0.65 ≈ 0.65 pts), and (4) real-text 4-candidate RULER QA [28 ] on SQuAD [49 ] and HotpotQA [69 ] at 8K, 32K, 64K (25% chance): unless labeled N = 512 N = 512 N = 512 (1 SE ≈ 2.0 \approx2.0 ≈ 2.0 pts), tables report raw log-likelihood cand4_acc at N = 128 N = 128 N = 128 (1 SE ≈ 4.0 \approx4.0 ≈ 4.0 pts; byte-length-normalized cand4_acc_norm agrees within ± 2 \pm2 ± 2 –5 5 5 pts, e.g. 82.0/76.6% vs. 76.6/82.0% on clamped 63.1B KDA).
Four Sparse MoE Scales (10.7× \times × Active Span; 20 tok/active param ≈ 1 × \approx1\times ≈ 1 × Chinchilla [27 ] ): Across two MXU-aligned backbone width tiers (D = 1024 D = 1024 D = 1024 and D = 1792 D = 1792 D = 1792 ), we evaluate (1) 0.35B active / 6.5B total (decoder depth L = 10 L = 10 L = 10 , D = 1024 D = 1024 D = 1024 , 4L:1A, k = 4 k = 4 k = 4 , 15.10B and 30.20B tok), (2) 0.79B active / 12.5B total (L = 16 L = 16 L = 16 , D = 1024 D = 1024 D = 1024 , 7L:1A, 15L:1A, & Pure-Linear ∞ \infty ∞ :1, k = 8 k = 8 k = 8 , 15.10B and 60.40B tok), (3) 1.80B active / 31.5B total (L = 15 L = 15 L = 15 , D = 1792 D = 1792 D = 1792 , 4L:1A, k = 8 k = 8 k = 8 , 5.03B and 36.20B tok ≈ 1.0 × \approx1.0\times ≈ 1.0 × Chinchilla), and (4) 3.74B active / 63.1B total (L = 30 L = 30 L = 30 , D = 1792 D = 1792 D = 1792 , 4L:1A & 15L:1A, k = 8 k = 8 k = 8 , 5.03B/30.2B RoPE on 64 chips and 120.80B NoPE on 512 chips).
Hybrid LatentMoE backbone and multi-scale parameterization. All four pre-training scales are instantiated from a single TPU-aligned hybrid linear-softmax LatentMoE decoder family implemented in open-source MaxText [1 ] (Table 10 ). Every weight matrix (embeddings, projections, routers, and experts) uses fixed truncated-normal initialization (standard deviation σ i n i t = 0.02 \sigma_{\mathrm{init}} = 0.02 σ init = 0.02 , clipped at ± 3 σ i n i t \pm3\sigma_{\mathrm{init}} ± 3 σ init ), and each decoder block applies four RMSNorm layers (ϵ = 10 − 6 \epsilon= 10^{-6} ϵ = 1 0 − 6 ): pre- and post-sublayer norms around both the sequence mixer and the FFN, with post-norms applied to the sublayer output prior to residual addition, followed by a final stack RMSNorm.
Table 10. Exact model architecture and multi-scale parameterization across our four hybrid LatentMoE pre-training rungs (V = 100,352 V = 100{,}352 V = 100 , 352 untied vocabulary, E = 256 E = 256 E = 256 routed experts on layers 1 , … , L − 1 1,\ldots,L-1 1 , … , L − 1 , dense SwiGLU on layer 0).
Layer 0 uses a dense SwiGLU FFN [51 ] of intermediate width F d e n s e F_{\mathrm{dense}} F dense with no routed experts to prevent early-layer routing collapse. Layers 1 , … , L − 1 1,\ldots,L-1 1 , … , L − 1 pair one full-width shared SwiGLU expert of intermediate width F s h F_{\mathrm{sh}} F sh with LatentMoE: a shared linear down-projection compresses the normalized residual stream from R D \mathbb{R}^{D} R D to an MXU-aligned latent width ℓ < D \ell< D ℓ < D , E = 256 E = 256 E = 256 routed SwiGLU experts (intermediate width F m o e = 2 D F_{\mathrm{moe}} = 2D F moe = 2 D ) execute inside R ℓ \mathbb{R}^{\ell} R ℓ , and a shared up-projection maps the weighted expert sum from R ℓ \mathbb{R}^{\ell} R ℓ back to R D \mathbb{R}^{D} R D . Routing scores are still computed on the uncompressed residual in R D \mathbb{R}^{D} R D in FP32, using top-k k k softmax weights rescaled to sum to k k k together with document-pool routing (restricting each document to a pool size in [ k , E ] [k,E] [ k , E ] ). Along depth, each ( x + 1 ) (x+1) ( x + 1 ) -layer cycle places linear delta mixers (4-tap causal depthwise Conv1D with SiLU, L 2 L_2 L 2 -normalized queries/keys, low-rank decay and sigmoid output-gate projections of rank r p r o j r_{\mathrm{proj}} r proj , and per-group output RMSNorm at ϵ = 10 − 5 \epsilon= 10^{-5} ϵ = 1 0 − 5 ) on the first x x x layers and gated Grouped-Query Attention (GQA) [2 , 47 ] (H q = 8 H_q = 8 H q = 8 , H k v = 4 H_{kv} = 4 H k v = 4 ) on every ( x + 1 ) (x+1) ( x + 1 ) -th layer, with QK-RMSNorm, a 2 d h e a d 2d_{\mathrm{head}} 2 d head -widened query projection (for per-head dimension d h e a d d_{\mathrm{head}} d head ) split into attention queries and an elementwise sigmoid output gate, and either RoPE (θ b a s e = 5 × 10 5 \theta_{\mathrm{base}} = 5 \times10^{5} θ base = 5 × 1 0 5 ) [55 ] or NoPE with Scalable Softmax [40 ] .
Rather than choosing intermediate widths that pad 256 × \times × 256 systolic tiles, we step across two 256-divisible width tiers (D = 1024 = 4 × 256 D = 1024 = 4 \times256 D = 1024 = 4 × 256 for Rungs 1–2; D = 1792 = 7 × 256 D = 1792 = 7 \times256 D = 1792 = 7 × 256 for Rungs 3–4, a 1.75× \times × width step) and scale depth L L L , active expert count k k k , and shared-expert width F s h F_{\mathrm{sh}} F sh within each tier:
Embeddings and untied LM head (2 V D 2VD 2 V D linear scaling): All four scales share a fixed vocabulary V = 100,352 V = 100{,}352 V = 100 , 352 with untied V × D V \times D V × D input embedding and D × V D \times V D × V output LM-head matrices.
Input token embeddings are multiplied by D \sqrt{D} D and passed through RMSNorm ( D ) \operatorname{RMSNorm}(D) RMSNorm ( D ) before Layer 0 so initial residual-stream RMS remains invariant across D D D under fixed σ i n i t = 0.02 \sigma_{\mathrm{init}} = 0.02 σ init = 0.02 initialization. Because V V V is fixed, combined embedding and LM-head parameters 2 V D 2VD 2 V D grow strictly linearly with D D D (205.5M → \to → 359.7M, 1.75× \times × ), whereas non-embedding capacity grows with both width and depth; accordingly, 2 V D 2VD 2 V D drops monotonically from 46.6% of active (2.71% of total) parameters at Rung 1 to 9.6% (0.57%) at Rung 4.
Sublayer widths across tiers (D D D : 1024 → \to → 1792): Stepping from Tier A to Tier B scales the LatentMoE bottleneck 1.5× \times × (ℓ \ell ℓ : 512 → \to → 768, keeping ℓ / D ∈ [ 0.43 , 0.50 ] \ell/D \in[0.43, 0.50] ℓ / D ∈ [ 0.43 , 0.50 ] so expert All-to-All dispatch and grouped-matmul K K K -tiles stay compressed and 256-aligned) and routed expert width 1.75× \times × (F m o e = 2 D F_{\mathrm{moe}} = 2D F moe = 2 D : 2048 → \to → 3584). In parallel, global GQA head dimension (d h e a d d_{\mathrm{head}} d head : 128 → \to → 256), linear value width (D v D_v D v : 2048 → \to → 4096), low-rank gate bottleneck (r p r o j r_{\mathrm{proj}} r proj : 256 → \to → 512), and output RMSNorm groups (8 → \to → 16, fixing 256 channels/group) all double (2× \times × ). Baseline KDA doubles d k d_k d k : 128 → \to → 256 (H k = 8 H_k = 8 H k = 8 ), whereas SLA-R2 (H k = 4 H_k = 4 H k = 4 ) and SLA-R4 (H k = 2 H_k = 2 H k = 2 ) hold d k = 256 d_k = 256 d k = 256 fixed at every scale to preserve unpadded 256 × \times × 256 MXU state updates while doubling packed value width R d v Rd_v R d v .
Within-tier depth, sparsity, and iso-parameter compensation: Within D = 1024 D = 1024 D = 1024 (Rung 1 → \to → Rung 2), depth increases from L = 10 L = 10 L = 10 to 16, active routed experts double (k k k : 4 → \to → 8), and Layer 0 widens (F d e n s e F_{\mathrm{dense}} F dense : $4D \to 8D$). Within D = 1792 D = 1792 D = 1792 (Rung 3 → \to → Rung 4), depth doubles (L L L : 15 → \to → 30) and shared-expert width doubles (F s h F_{\mathrm{sh}} F sh : D → 2 D D \to2D D → 2 D ). Finally, because 4:1 key-head sharing in SLA-R4 reduces per-layer linear mixer parameters (7.63M → \to → 6.31M at D = 1024 D = 1024 D = 1024 ; 27.05M → \to → 20.48M at $D = 1792), Iso-Param SLA-R4 widens F s h F_{\mathrm{sh}} F sh by +256 (1024 → \to → 1280 on Rungs 1–2; 1792 → \to → 2048 on Rung 3) to match baseline parameter count.
Cohort labels 0.35B/6.5B and 1.80B/31.5B reflect the 8-layer proxy (6.08B total, 0.38B active) and half-63.1B shorthand; extending Rung 1 to L = 10 L = 10 L = 10 (2 × \times × 4L: 1A cycles) and Rung 3 to L = 15 L = 15 L = 15 (3 × \times × 4L: 1A cycles, F s h = 1792 F_{\mathrm{sh}} = 1792 F sh = 1792 ) ensures exact cycle divisibility and yields the parameter counts above. Active parameters are shown both with 2 V D 2VD 2 V D matching the 0.79B and 3.74B convention) and as FLOP-bearing active parameters (1 V D 1VD 1 V D , excluding input token lookup; pure non-embedding/non-head active parameters are 0.24B, 0.59B, 1.55B, 3.38B).
End-to-end MoE training throughput. We pre-train four token-choice sparse MoE families (6.5B–63.1B total and 0.35B–3.74B active parameters; 256 routed experts, top-8 routing, one shared expert [16 , 52 ] ) with AdamW [38] on packed Dolma 3.5 [54 , 60 , 61 ] at T t r a i n = 8 , 192 T_{\mathrm{train}} = 8,192 T train = 8 , 192 , using 64–512 TPU v7x chips per run across a 2,048-chip cluster (Table 15 ). Relative to MaxText Pallas KDA (used because open-source Tokamax Mosaic KDA restricts C ≤ 64 C \le64 C ≤ 64 and omits packed multi-document segment resets; even against an idealized zero-overhead Tokamax D = 1792 D = 1792 D = 1792 kernel from Table 6 c, 63.1B SLA-R4 remains 1.13× \times × faster end-to-end), SLA trains 8%–24% faster across matched scales (1.10× \times × –1.18× \times × at 6.5B/31.5B, 1.16× \times × –1.24× \times × at 12.5B in Table 11 a, and 1.11× \times × –1.15× \times × at 63.1B on 512 chips in Table 12 ). At native T t r a i n = 32 T_{\mathrm{train}} = 32 T train = 32 K with 4.19M tokens/step (EP=128), SLA fits in per-core
Table 12. Measured 512-chip TPU v7x 63.1B MoE (3.74B active, 8 × 8 × 8, 16.78M tok/step) at 120.80B tokens (≈ 1.6× Chinchilla, NoPE T t r a i n T_{\mathrm{train}} T train = 8,192; SSMax [40 ] query scale α ∈ { 0.25 , 0.5 } \alpha\in\{0.25, 0.5\} α ∈ { 0.25 , 0.5 } , § 3.5 ; 64K RULER QA cand4_acc at N N N = 512, {sup}*N N N = 128; full 64-chip XProf and RoPE ablations in Table 15 , Table 13 , and Table 17 ).
Table 15. Matched RoPE (θ max = 5 × 10 5 \theta_{\max} = 5 \times10^{5} θ m a x = 5 × 1 0 5 ) 4-rung scaling ladder (0.35B to 63.1B MoE; see Table 11 for 12.5B Rung 2), 0.35B 3-seed & 30.2B (4.4× Chinchilla), 31.5B at 36.2B tok (1.0× Chinchilla), 12.5B 8K → 32K continued pre-training, and 12.5B 15L:1A dilution control on 64-chip TPU v7x (under remat_32k with 4-step gradient accumulation [GA=4], KDA/0L:1A hit NaN at steps 1–43 and required noremаt reruns, whereas 2-seed SLA-R4 (iso-param) completed all 4,300 steps at 3,040 ms with 12.54 → 12.08 Tail PPL, 59.2%/48.2% 32K 4/8-N, and 58.5/62.6% N = 512 N=512 N = 512 64K QA, though router-remat spikes raised 8K PPL to 17.99).
Table 11: Measured 12.5B MoE (0.79B active, 16L, D = 1024 D = 1024 D = 1024 , D v = 2048 D_v = 2048 D v = 2048 , T t r a i n = 8,192 T_{\mathrm{train}} = 8{,}192 T train = 8 , 192 ) pre-training across (a) 60.40B-token (3.8× Chinchilla [27 ] , 8.39M tok/step) v7x arms and (b) 15.10B-token (1.0× Chinchilla, 2.10M/step) and ∼90-min Iso-Wall ablations (n ∈ { 2 , 3 } n \in\{2, 3\} n ∈ { 2 , 3 } random seeds where noted; 4-choice 64K RULER QA, N = 128 N = 128 N = 128 , SE = 4 pts; full controls in Table 15 –Table 16 ; bold: best within each cohort at equal tokens, with perplexity compared on matched seeds s0,s1 in (b); GDN-2 uses our unfused TPU port; seeds s0/s1/s2 use different data offsets, so perplexities are comparable only on matched seeds).
Table 16. Zero-shot 9-task lm-eval accuracy (11,793 real validation items; norm9 macro-average using byte-length normalized acc_norm on multiple-choice tasks and exact-word acc on WinoGrande/LAMBADA, with zero-shot row-scaled FP8 state norm9 in parentheses for SLA rows; 1 SE ≈ 0.65 pts) and real-text 4-candidate RULER QA accuracy (SQuAD 1-hop & HotpotQA 2-hop, N = 128 N=128 N = 128 /length, 25% chance, 1 SE ≈ 4.0 pts) across 8K → 32K → 64K against 0L:1A GQA+RoPE and MaxText KDA baselines.
HBM (≤ 94.74 GiB) while KDA (96.74 GiB) and full attention (95.48 GiB) exceed the 95 GiB cap; in 8K → 32K continued pre-training at 12.5B, where all three fit, SLA is 1.08×–1.15× faster than KDA (Table 15 ).
Hardware XProf operator attribution on 64-chip TPU v7x. XProf attributes most of SLA’s step-time saving to linear attention ops (Figure 7 a). Table 13 provides the complete hardware XProf HLO category breakdown (xplane traces captured over steps 15–17 across all 8 TensorCores of host 0, normalized by 24 core-steps to ms/step) for 63.1B MoE (FSDP=2, EP=64) on 64 TPU v7x chips. Mapping directly to Figure 7 a (where the top bar groups ICI async-done wait with MoE/dense matmuls: 366.6+245.8 = 612.4 ms, 354.1+266.8 = 620.9 ms, and 343.1+237.2 = 580.3 ms), the three direct linear attention op categories save −218.7 ms (87.2%) and reduced mixer memory/sync traffic saves −23.5 ms (9.4%) of exposed ICI async-done wait, totaling −242.2 ms (96.6%) of the −250.8 ms step-time reduction between SLA-R4 and KDA Base (with −8.6 ms [3.4%] in MoE/FFN matmul overlap).
Table 13. Hardware XProf per-step HLO category breakdown (ms/step) on 64-chip TPU v7x 63.1B MoE (FSDP=2, EP=64, 2.10M tok/step).
Single seed (s0). Full 64K QA breakdown (SQuAD, HotpotQA) and 64-chip RoPE controls are in Table 17 . {sup}†Deferring the fused conv AllReduce across gradient accumulation yields 1,843 ms.
Table 17.
Table 14 complements this with a symmetric 4-arm GA=4 (8.39M tok/step) XProf audit on 12.5B MoE (EP=128): deferring causal-conv weight AllReduce outside the GA=4 microbatch loop yields 0.0% speedup on MaxText KDA (2,118.9 → 2,119.7 ms) while cutting SLA-R4 (iso-param) to 1,686.0 ms in 3-step XProf (1,661.9 ms unprofiled min; 1,812.0 ± 14.1 ms over full 7,200-step n n n = 2 training runs due to host checkpoint/validation wrapper hooks) via a 1.96× faster Pallas linear attention kernel (528.4 → 269.1 ms).
At 6.3B tokens (RoPE validation checkpoint on the same 64-chip slice), 4L:1A KDA Base and 4L:1A SLA-R4 reach 9.90 vs. 10.15 8K Dolma PPL, 10.27 vs. 9.75 32K Tail PPL, and 14.5% vs. 37.5% 32K 8-Needle accuracy.
Table 14. Symmetric 4-arm GA=4 (8.39M tok/step) hardware XProf leaf-op breakdown (ms/step) on 64-chip TPU v7x 12.5B MoE (EP=128).
Pre-Training Quality and Long-Context Extrapolation
In-domain quality. At 12.5B parameters and 60.4B tokens with three paired seeds (Table 11 a), SLA-R2 and SLA-R4 are within seed noise of KDA on 11-set Dolma perplexity and on zero-shot 1m-eval [17 ] (Table 16 ); single-seed 63.1B runs are also close (Table 12 ). The 15.1B-token cohort (Table 11 b) shows where this tiny cost comes from. Seeds s0, s1, and s2 use different data offsets that shift perplexity by up to 0.9, so we compare only on matched seeds. First, over three seeds, SLA-R2 (Exact Inv., BF16 state, μ \mu μ = 0.25) improves on KDA Base on every seed (17.533 vs. 17.638; paired difference −0.10 ± 0.08) and matches the same-kernel G G G = 4 FP32 control without momentum (17.538, Table 18 ), so BF16 state and momentum cost no measurable quality. On seeds s0, s1, widening KDA’s keys to d k d_k d k = 256 with R R R = 2 head sharing (A1, [4 ] [256, 512], G G G = 256: 524.8 ms, 17.694 PPL) isolates head sharing from decay grouping; moving from G G G = 256 to G G G = 4 with MXU block-doubling at identical R R R = 2 sharing cuts step time by a further −27.5 ms (497.3 ms unfused, −15.5 ms further to 481.8 ms fused) while staying within 0.06–0.11 PPL of A1 and lifting 32K 8-Needle from 23.8% to 52.3%–58.0% and 64K HotpotQA from 54.3% to 73.8%–74.6%. The
Table 18.
optional 1-step Neumann inverse is the one approximation with a clear cost: A3 reaches 17.970 (+0.22 over A2), and on s0 the fused Neumann arm A5 reaches 18.016 versus 17.913 for fused Exact Inv., which is why all flagship pre-training runs use Exact Inv.. Second, GDN-2 [25 ] reaches the best per-token perplexity in this cohort (17.592 on s0, s1, versus 17.806 for fused SLA-R2). SLA’s advantage is throughput: in the same 90-minute budget, fused SLA-R2 trains on 53% more tokens and reaches 16.660 perplexity, versus 17.459 for GDN-2 on the same seed. Because this comparison uses our unfused TPU port of GDN-2, it reflects kernel maturity as much as architecture; combining GDN-2’s decoupled erase and write gates with SLA’s TPU mapping is a natural next step.
GDN-2 TPU comparison details. GDN-2 [25 ] decouples the delta rule’s scalar gate into channel-wise erase and write gates and inherits KDA’s per-channel decay. On TPU v7x, in our unified single-harness full-layer benchmark (B = 1 B = 1 B = 1 , 8K/32K F+B), our unfused XLA port of per-channel GDN-2 takes 63.86/251.39 ms at D = 1792 D = 1792 D = 1792 (15,217 MiB temporary HBM) and 34.15/132.83 ms at D = 2048 D = 2048 D = 2048 (Mamba-3 MIMO R = 4 R = 4 R = 4 isolated kernel for reference: 9.27 ms). End to end, unfused per-channel GDN-2 raises MoE step time by 1.23×–1.59× (1.349 vs. 1.093 s on 128-chip TPU v4; 778.1 ± 3.9 vs. 488.3 ms for fused SLA-R2 on 64-chip TPU v7x).
On quality, GDN-2 is the strongest arm per token in our 15.1B-token cohort (17.592 ± 0.188 vs. 17.806 for fused SLA-R2 and 17.893 for KDA on the same seeds s0, s1; Table 11 b), but extrapolates worse (19.3% vs. 52.3% 32K 8-needle). Under equal pure-step compute on seed s0 (5,582 s for 7,200 GDN-2 steps vs. 5,371 s for 11,000 fused SLA-R2 steps), SLA reaches lower perplexity (16.660 vs. 17.459), whereas at equal 15.1B tokens on s0 GDN-2 leads fused SLA-R2 by 0.45 (17.459 vs. 17.913). This comparison reflects the lack of a fused Pallas/Mosaic TPU kernel for GDN-2 as much as the architectures; combining GDN-2’s decoupled update with SLA’s head sharing, tile-grouped gating, and fused chunk inversion is a promising direction for future work.
Length extrapolation. We test zero-shot extrapolation from 8K training to 32K–64K on RULER [28 ] needle retrieval and 4-choice SQuAD [49 ] and HotpotQA [69 ] (Figure 7 b–Figure 7 c). Unclamped KDA degrades beyond the training length, severely at 63.1B, because about 1% of its channels learn horizons far beyond 8K (Table 17 ). SLA extrapolates without clamping: at 12.5B its 32K tail perplexity stays flat and 64K HotpotQA reaches 74%–78%. The advantage is limited, however. Clamping KDA’s horizons to 8K, with or without retraining, gives similar results at 12.5B. At 63.1B, clamped KDA with SSMax [40 ] on the softmax layers beats SLA with SSMax on 64K QA, although SLA is better on 32K needle retrieval. With 128–512 questions per QA set (standard error 2–4 points), many differences in Table 11 –Table 12 are within noise.
Causal τ e f f \tau_{\mathrm{eff}} τ eff dispersion and NoPE SSMax. Table 17 audits per-channel retention horizons
τ e f f ( i ) = E t [ − 1 ln f t , i ] \tau_{\mathrm{eff}}(i) = \mathbb{E}_{t}\left[-\frac{1}{\ln f_{t,i}}\right] τ eff ( i ) = E t [ − ln f t , i 1 ]
(in tokens, for channel i i i with learned decay f t , i f_{t,i} f t , i ) and Within-Tile Std (the RMS standard deviation of token-mean log-decay E t [ ln f t , i ] \mathbb{E}_{t}[\ln f_{t,i}] E t [ ln f t , i ] across channels i i i within each d k / 4 d_k/4 d k /4 tile block, equal to 0 when G ∈ { 1 , 4 } G \in\{1,4\} G ∈ { 1 , 4 } ) alongside zero-retraining inference interventions across 6.5B, 12.5B, and 63.1B MoEs.
Across all three scales, G = d k G = d_k G = d k KDA develops 0.8%–1.3% ultra-slow τ e f f ( i ) ≈ 131 \tau_{\mathrm{eff}}(i) \approx131 τ eff ( i ) ≈ 131 K channels (within-tile log-decay RMS std = 3.84–4.97) that inject OOD state spikes into hybrid layers: clamping τ e f f ( i ) ≤ τ max = 8 , 192 \tau_{\mathrm{eff}}(i) \leq\tau_{\max} = 8,192 τ eff ( i ) ≤ τ m a x = 8 , 192 at inference with zero retraining cuts 63.1B KDA 32K Tail PPL by 45.9× (353.42 → 7.70), lifts 32K 8-Needle across every scale (14.1% → 69.9% at 6.5B, 15.2% → 57.8% at 12.5B 7L, 15.2% → 42.2% at 12.5B 15L, 15.6% → 48.4% at 63.1B), and rescues +14.0 to +56.2 pts on 64K QA, whereas post-hoc G = 1 G = 1 G = 1 tile-mean projection collapses (57.8–78.1 Tail PPL). Live training-time softplus saturation (TrainMax=8K) likewise stabilizes OOD retrieval at both 6.5B (69.5/73.4% QA) and 12.5B (60.4B tok, n = 2 n = 2 n = 2 : 10.75 Tail, 49.6% 32K 8-N, 74.5/72.8% N = 512 N = 512 N = 512 QA) while retaining G = 128 G = 128 G = 128 ’s VPU gate tax and 4× FP32 state.
Conversely, on 63.1B SLA-R4 (120.8B tok, NoPE), 24-layer state-norm telemetry confirms linear states contract cleanly from 8K to 32K (||S t S_t S t ||F _F F = 2.04 → 1.62, max row norm 1.59 → 1.15, 0.00% rows near E4M3 448), and rescaling each global softmax query q q q on the 6 NoPE full-attention layers at evaluation sequence length T T T via zero-retraining SSMax [40 ] ,
q ← q ( 1 + α ln ( T T t r a i n ) ) q \leftarrow q\left(1+\alpha\ln\left(\frac{T}{T_{\mathrm{train}}}\right)\right) q ← q ( 1 + α ln ( T train T ) )
Table 17: Multi-scale causal τ e f f \tau_{\mathrm{eff}} τ eff dispersion, zero-retraining τ max = 8,192 \tau_{\max}=8{,}192 τ m a x = 8 , 192 inference clamping, live TrainMaxTau=8K saturation, post-hoc G = 4 G=4 G = 4 projection, and NoPE SSMax (α = 0.5 \alpha=0.5 α = 0.5 ) on 32K Tail PPL, 32K 8-N, and 64K RULER QA (N = 128 N=128 N = 128 unless noted N = 512 N=512 N = 512 ). The clamp evaluator (a3_causal_clamp_eval) scores its own held-out Dolma slice and paired N = 128 N=128 N = 128 QA subset, so absolute Tail PPL and QA values differ from the standalone scorers of Table 11 , Table 12 , Table 15 , and Table 16 (by 1–2 QA items, 0.8–1.6 pts) while within-row intervention deltas remain exact.
(with T t r a i n = 8,192 T_{\mathrm{train}=8{,}192} T train = 8 , 192 and SSMax scale α = 0.5 \alpha=0.5 α = 0.5 , unrelated to α s a f e \alpha_{\mathrm{safe}} α safe ; zero linear clamp), cuts 32K Tail PPL to 8.45 (SLA-R2) and 8.41 (SLA-R4) and restores 32K 8-Needle to 84.4% (SLA-R2) and 83.2% (SLA-R4) with zero linear gate clamping (whereas linear τ max = 8,192 \tau_{\max}=8{,}192 τ m a x = 8 , 192 clamping alone without SSMax moves 63.1B SLA-R2 32K 8-N from 18.6% to 21.9%, and SSMax alone on unclamped KDA still diverges at 380.66 Tail PPL and 18.8% 8-Needle, cleanly separating global NoPE softmax dilution from KDA’s linear gate drift).
Decay granularity. Sweeping G ∈ { 1 , 4 , 16 , 64 , 256 } G\in\{1,4,16,64,256\} G ∈ { 1 , 4 , 16 , 64 , 256 } with everything else fixed (Table 18 ), G = 4 G=4 G = 4 improves 8K perplexity over G = 1 G=1 G = 1 on all three paired seeds, but only by 0.15 on average, while G = 1 G=1 G = 1 extrapolates better (59.0% vs. 49.2% 32K 8-needle). Finer groups (G ≥ 16 G\geq16 G ≥ 16 ) extrapolate much worse. A scalar decay per head, as in GDN, is therefore a strong alternative; we choose G = 4 G=4 G = 4 for its multiple memory horizons and slightly better in-domain fit, not for long-context robustness.
Specifically, Table 18 dissects the 5-point G ∈ { 1 , 4 , 16 , 64 , 256 } G\in\{1,4,16,64,256\} G ∈ { 1 , 4 , 16 , 64 , 256 } sweep (with decay sub-tile width b = d k / G = 256 / G b=d_k/G=256/G b = d k / G = 256/ G ) on identical [4, 256] Exact Inv. FP32 kernel using both matched-seed (s0, s1) means and per-seed paired differences against G = 1 G=1 G = 1 . On matched s0, s1 seeds, 8K 11-set PPL favors finer G G G without a monotone trend (17.946, 17.749, 17.873, 17.748, 17.694 for G = 1 , 4 , 16 , 64 , 256 G=1,4,16,64,256 G = 1 , 4 , 16 , 64 , 256 ; all G ≥ 4 G\geq4 G ≥ 4 lie within 0.18 PPL of each other), and G = 4 G=4 G = 4 improves 11-set PPL over G = 1 G=1 G = 1 on 3/3 paired seeds (Δ \Delta Δ PPL = −0.34, −0.06, −0.06; mean −0.151). Conversely, zero-shot OOD retrieval favors larger tile width b = 256 / G b=256/G b = 256/ G : G = 1 G=1 G = 1 (b = 256 b=256 b = 256 ) achieves the highest 32K 8-N (59.0%) and 64K QA (67.8/71.2%, τ max ≤ 56 \tau_{\max}\leq56 τ m a x ≤ 56 K across seeds), G = 4 G=4 G = 4 (b = 64 b=64 b = 64 ) trades a small OOD step (49.2% 8-N; 64.5/68.9% QA, where seed s2 developed one slow tile with τ max = 124 \tau_{\max}=124 τ m a x = 124 K, scoring 58.6/64.5% vs. 67.5/71.2% on s0, s1) for 4× intra-head timescales, and G ≥ 16 G\geq16 G ≥ 16 (b ≤ 16 b\leq16 b ≤ 16 , τ max = 131 \tau_{\max}=131 τ m a x = 131 K) degrades sharply to 27.1%/19.0% 8-N and 60.3/61.9% → 53.5/50.8% QA.
Table 18: Unconfounded 5-point [4, 256, 512] decay-granularity sweep on the identical chunk_kda Exact Inv. FP32 kernel (μ = 0 \mu=0 μ = 0 , 12.5B MoE, 7L : 1A, 15.10B tok), reporting matched s0, s1 11-set PPL (valid across all 5 granularities), paired per-seed Δ \Delta Δ PPL vs. G = 1 G=1 G = 1 , 8K → 32K Tail PPL, 32K 4/8-N, lm-eval norm9, and N = 512 N=512 N = 512 (1 SE = 2.0 pts) 64K RULER QA.
Per-seed 11-set PPL (s0, s1, s2): G = 1 G=1 G = 1 (18.210, 17.681, 17.178), G = 4 G=4 G = 4 (17.873, 17.625, 17.117), G = 16 G=16 G = 16 (18.067, 17.679, 17.132), G = 64 G=64 G = 64 (17.851, 17.074, 17.073), G = 256 G=256 G = 256 (17.850, 17.538, s2 11-set not run). Seed s2 used a deterministic OLMO_DATA_STEP_OFFSET stream slice (17.07–17.18 PPL across all G G G ). On s0, s1, G = 4 G=4 G = 4 64K QA is 67.5 ± 1.8%/71.2 ± 0.1% (τ max ≤ 85 \tau_{\max}\leq85 τ m a x ≤ 85 K); on s2, one G = 4 G=4 G = 4 tile hit τ max = 124 \tau_{\max}=124 τ m a x = 124 K (58.6/64.5%). For G = 256 G=256 G = 256 , N = 128 N=128 N = 128 64K QA is 57.3%/59.1%.
FP8 state. SLA states quantize well: zero-retrain row-scaled E4M3 state quantization changes 8K/32K loss by less than 10 − 4 10^{-4} 1 0 − 4 nats (§ 3.4 ), and FP8 output QAT with Exact Inv. (A6) is within 0.09 perplexity of BF16 on s0 (18.006 vs. 17.913).
Conditioning Recurrent State in Low Precision: Oblique Whitening and Polar-2+
Diluting softmax attention from 7L:1A (12.5%) to 15L:1A (6.25%) cuts 131K KV+state HBM to 16.47 GiB (Table 8 ) while forcing middle linear layers (L5, L8, L11) to expand active retention 10×–35× (ρ e f f = N e f f / d k ∈ [ 0.101 , 0.152 ] \rho_{\mathrm{eff}} = N_{\mathrm{eff}}/d_k \in[0.101, 0.152] ρ eff = N eff / d k ∈ [ 0.101 , 0.152 ] vs. ≤ 0.015 \le0.015 ≤ 0.015 at 4L:1A, where N e f f N_{\mathrm{eff}} N eff is the number of key-value associations a layer effectively retains, estimated from its decay horizons) under coherent keys (κ 2 ( I C + K c K c T ) ∈ [ 20.6 , 62.9 ] \kappa_2(I_C + K_cK_c^{\mathsf{T}}) \in[20.6, 62.9] κ 2 ( I C + K c K c T ) ∈ [ 20.6 , 62.9 ] ; Figure 8 c). Whereas GPU second-order memories require 30-step Conjugate Gradient (CG) [64 ] or diagonal approximations [62 ] , a 256 × 256 MXU runs five [128, 128] matmuls across 64 chunks in 0.62 ms forward (0.90 ms F+B). Here we formalize when key whitening helps in low precision and detail Polar-2+ , a five-matmul BF16-safe inverse-square-root primitive.
Figure 8. Polar-2+ minimax contraction, overwrite recall, and checkpoint retention audit. (a) Scalar inverse-square-root contraction λ z 2 ( λ ) 2 \lambda z_2(\lambda)^2 λ z 2 ( λ ) 2 on normalized eigenvalue λ ∈ [ 0.02 , 1.0 ] \lambda\in[0.02, 1.0] λ ∈ [ 0.02 , 1.0 ] (Proposition 4.2 ): unscaled polynomial factor α = 1.000 \alpha= 1.000 α = 1.000 hits 1.0492 > 1, whereas Polar-2+ (safety factor α s a f e = 0.968 \alpha_{\mathrm{safe}} = 0.968 α safe = 0.968 ) bounds the peak to 0.9831 < 1, keeping minimum eigenvalue λ min ( Σ c ) ≥ 0.0388 \lambda_{\min}(\Sigma_c) \ge0.0388 λ m i n ( Σ c ) ≥ 0.0388 of key preconditioner Σ c \Sigma_c Σ c across 4,096 BF16 chunks (524K tok). (b) 8-seed Hadamard overwrite accuracy (overwrite multiplicity r = 3 r = 3 r = 3 ) at memory load ρ e f f = N / d k = 0.25 \rho_{\mathrm{eff}} = N/d_k = 0.25 ρ eff = N / d k = 0.25 (N = 16 N = 16 N = 16 stored keys, d k = 64 d_k = 64 d k = 64 ). (c) Trained 63.1B (4L : 1A) and 12.5B (15L : 1A) checkpoint load ρ e f f \rho_{\mathrm{eff}} ρ eff .
Proposition 4.1 (Oblique Whitening vs. Euclidean & RLS). Let K ∈ R N × d k K \in\mathbb{R}^{N \times d_k} K ∈ R N × d k stack N ≤ d k N \le d_k N ≤ d k unit keys k 1 , … , k N ∈ R d k k_1, \ldots, k_N \in\mathbb{R}^{d_k} k 1 , … , k N ∈ R d k with mutual coherence ν c o h = max i ≠ j ∣ ⟨ k i , k j ⟩ ∣ \nu_{\mathrm{coh}} = \max_{i \ne j} |\langle k_i, k_j \rangle| ν coh = max i = j ∣ ⟨ k i , k j ⟩ ∣ and Gram matrix G = K K T G = KK^{\mathsf{T}} G = K K T satisfying κ 2 ( G ) > 1 \kappa_2(G) > 1 κ 2 ( G ) > 1 (proofs in Appendix A).
[Prior Delta [62 , 67 ] ] For equiangular keys (∣ ⟨ k i , k j ⟩ ∣ = ν c o h |\langle k_i, k_j \rangle| = \nu_{\mathrm{coh}} ∣ ⟨ k i , k j ⟩ ∣ = ν coh for all i ≠ j i \ne j i = j ), zero-mean values v j ∈ R d v v_j \in\mathbb{R}^{d_v} v j ∈ R d v of expected squared norm s v 2 = E ∥ v j ∥ 2 s_v^2 = \mathbb{E}\lVert v_j\rVert^2 s v 2 = E ∥ v j ∥ 2 , and load ρ e f f = N / d k \rho_{\mathrm{eff}} = N/d_k ρ eff = N / d k , single-pass Euclidean delta incurs readout error Ω ( ρ e f f ( d k ν c o h 2 ) s v 2 ) \Omega(\rho_{\mathrm{eff}}(d_k\nu_{\mathrm{coh}}^2)s_v^2) Ω ( ρ eff ( d k ν coh 2 ) s v 2 ) , and diagonal column-energy preconditioning diag ( p ( K ) ) − 1 / 2 k t \operatorname{diag}(p(K))^{-1/2}k_t diag ( p ( K ) ) − 1/2 k t [62 ] (where p ( K ) ∈ R d k p(K) \in\mathbb{R}^{d_k} p ( K ) ∈ R d k has m m m -th entry p m ( K ) = ∑ i = 1 N K i , m 2 p_m(K) = \sum_{i=1}^{N}K_{i,m}^2 p m ( K ) = ∑ i = 1 N K i , m 2 ) fails under orthogonal key rotation K ↦ K Q K \mapsto KQ K ↦ K Q (Q ∈ O ( d k ) Q \in O(d_k) Q ∈ O ( d k ) ).
[Prior RLS [64 ] ] MesaNet (with recursive least-squares key covariance H t = γ H t − 1 + k t k t T H_t = \gamma H_{t-1} + k_tk_t^{\mathsf{T}} H t = γ H t − 1 + k t k t T and scalar forget factor γ ∈ ( 0 , 1 ] \gamma\in(0,1] γ ∈ ( 0 , 1 ] ) averages r ≥ 2 r \ge2 r ≥ 2 overwrites v ( 1 ) , … , v ( r ) ∈ R d v v^{(1)}, \ldots, v^{(r)} \in\mathbb{R}^{d_v} v ( 1 ) , … , v ( r ) ∈ R d v of the same key as 1 r ∑ m = 1 r v ( m ) \frac{1}{r}\sum_{m=1}^{r}v^{(m)} r 1 ∑ m = 1 r v ( m ) at γ = 1 \gamma= 1 γ = 1 , and suppresses stale values only at γ < 1 \gamma< 1 γ < 1 only by decaying write-once keys stored t − j t-j t − j steps earlier (at step j < t j < t j < t ) at rate γ t − j \gamma^{t-j} γ t − j .
[Ours ] Oblique whitened keys K ~ c ∈ R C × d k \widetilde{K}_c \in\mathbb{R}^{C \times d_k} K c ∈ R C × d k and causal oblique keys k ~ t ∈ R d k \widetilde{k}_t \in\mathbb{R}^{d_k} k t ∈ R d k (Eq. (13) ) inside Eq. (4) decorrelate intra-chunk writes while erasing selectively along k ~ t \widetilde{k}_t k t .
Non-circular normalization and Polar-2+ formulation. Matrix-sign polynomials (Polar Express [3 ] , CANS [21 ] ) collapse singular values to 1, while coupled NS–5 [4 , 24 , 26 , 37 ] uses 10 [ C , C ] [C, C] [ C , C ] matmuls [Prior Polynomials]. Here K c ∈ R C × d k K_c \in\mathbb{R}^{C \times d_k} K c ∈ R C × d k stacks the chunk’s keys, Σ c − 1 → 0 \Sigma_{c-1} \to0 Σ c − 1 → 0 is the d k × d k d_k \times d_k d k × d k key preconditioner carried across chunks (Eq. (14) ); initialized at Σ 0 = I d k \Sigma_0 = I_{d_k} Σ 0 = I d k and relaxed toward I d k I_{d_k} I d k at rate ρ p ∈ [ 0 , 1 ] \rho_p \in[0,1] ρ p ∈ [ 0 , 1 ] , η c ∈ ( 0 , 1 ] \eta_c \in(0,1] η c ∈ ( 0 , 1 ] is the whitening step size, and ε β > 0 \varepsilon_\beta> 0 ε β > 0 is the erasure safety margin (§ 2.3 ). To avoid circular regularization, we form the raw chunk Gram matrix G 0 ∈ R C × C \mathcal{G}_0 \in\mathbb{R}^{C \times C} G 0 ∈ R C × C , its Frobenius normalization G ˉ 0 \bar{\mathcal{G}}_0 G ˉ 0 , spectral upper bound u ^ 0 \hat{u}_0 u ^ 0 , and adaptive ridge ε r e g \varepsilon_{\mathrm{reg}} ε reg :
G 0 = I C + η c K c Σ c − 1 K c T , G ˉ 0 = G 0 ∥ G 0 ∥ F , u ^ 0 = max ( ∥ G 0 ∥ F ∥ G ˉ 0 2 ∥ F 1 / 2 , 1 ) ≥ λ max ( G 0 ) , \mathcal{G}_0=I_C+\eta_cK_c\Sigma_c^{-1}K_c^{\mathsf T},\qquad\bar{\mathcal{G}}_0=\frac{\mathcal{G}_0}{\|\mathcal{G}_0\|_F},\qquad\hat{u}_0=\max\left(\|\mathcal{G}_0\|_F\|\bar{\mathcal{G}}_0^2\|_F^{1/2},1\right)\ge\lambda_{\max}(\mathcal{G}_0), G 0 = I C + η c K c Σ c − 1 K c T , G ˉ 0 = ∥ G 0 ∥ F G 0 , u ^ 0 = max ( ∥ G 0 ∥ F ∥ G ˉ 0 2 ∥ F 1/2 , 1 ) ≥ λ m a x ( G 0 ) ,
and
ε r e g = max ( 0.02 , 0.0205 0.9795 u ^ 0 − 1 ) . \varepsilon_{\mathrm{reg}}=\max\left(0.02,\frac{0.0205}{0.9795}\hat{u}_0-1\right). ε reg = max ( 0.02 , 0.9795 0.0205 u ^ 0 − 1 ) .
Setting regularized Gram G r e g = G 0 + ε r e g I C \mathcal{G}_{\mathrm{reg}}=\mathcal{G}_0+\varepsilon_{\mathrm{reg}}I_C G reg = G 0 + ε reg I C and scale u 0 = u ^ 0 + ε r e g u_0=\hat{u}_0+\varepsilon_{\mathrm{reg}} u 0 = u ^ 0 + ε reg maps the eigenvalues of T 0 = G r e g / u 0 T_0=\mathcal{G}_{\mathrm{reg}}/u_0 T 0 = G reg / u 0 into [ 0.02 , 1.0 ] [0.02,1.0] [ 0.02 , 1.0 ] while reusing G ˉ 0 2 \bar{\mathcal{G}}_0^2 G ˉ 0 2 (T 0 2 T_0^2 T 0 2 adds zero matmuls; 5 matmuls total), with step-size clamp
β t ≤ 2 − ε β max ( k t T k ~ t , ε β ) . \beta_t\le\frac{2-\varepsilon_\beta}{\max(k_t^{\mathsf T}\tilde{k}_t,\varepsilon_\beta)}. β t ≤ max ( k t T k ~ t , ε β ) 2 − ε β .
Letting K < t K_{<t} K < t stack the chunk keys strictly preceding token t t t , M ℓ ∈ R C × C \mathcal{M}_\ell\in\mathbb{R}^{C\times C} M ℓ ∈ R C × C denote the degree-2 polynomial factors for stages ℓ ∈ { 0 , 1 } \ell\in\{0,1\} ℓ ∈ { 0 , 1 } with T 1 = M 0 T 0 M 0 T_1=\mathcal{M}_0T_0\mathcal{M}_0 T 1 = M 0 T 0 M 0 and minimax coefficients ( a 0 , b 0 , e 0 ) = ( 4.92864 , − 11.29513 , 7.70099 ) (a_0,b_0,e_0)=(4.92864,-11.29513,7.70099) ( a 0 , b 0 , e 0 ) = ( 4.92864 , − 11.29513 , 7.70099 ) and ( a 1 , b 1 , e 1 ) = ( 2.00410 , − 1.38700 , 0.38894 ) (a_1,b_1,e_1)=(2.00410,-1.38700,0.38894) ( a 1 , b 1 , e 1 ) = ( 2.00410 , − 1.38700 , 0.38894 ) , and W i s q ∈ R C × C \mathcal{W}_{\mathrm{isq}}\in\mathbb{R}^{C\times C} W isq ∈ R C × C denote the safety-scaled (α s a f e = 0.968 \alpha_{\mathrm{safe}}=0.968 α safe = 0.968 ) inverse-square-root factor, Polar-2+ computes:
Low-Precision Conditioning: Oblique Whitening & BF16-Safe Polar-2+
K ~ c = η c W i s q K c Σ c − 1 , k ~ t ∝ ( I d k − η c Σ c − 1 K < t T K < t ) Σ c − 1 k t (13) \tilde{K}_c=\sqrt{\eta_c}\mathcal{W}_{\mathrm{isq}}K_c\Sigma_{c-1},\qquad\tilde{k}_t\propto\left(I_{d_k}-\eta_c\Sigma_{c-1}K_{<t}^{\mathsf T}K_{<t}\right)\Sigma_{c-1}k_t \tag*{(13)} K ~ c = η c W isq K c Σ c − 1 , k ~ t ∝ ( I d k − η c Σ c − 1 K < t T K < t ) Σ c − 1 k t (13)
Σ c = ( 1 − ρ p ) ( Σ c − 1 − K ~ c T K ~ c ) + ρ p I d k (14) \Sigma_c=(1-\rho_p)\left(\Sigma_{c-1}-\tilde{K}_c^{\mathsf T}\tilde{K}_c\right)+\rho_p I_{d_k} \tag*{(14)} Σ c = ( 1 − ρ p ) ( Σ c − 1 − K ~ c T K ~ c ) + ρ p I d k (14)
M ℓ = a ℓ I C + b ℓ T ℓ + e ℓ T ℓ 2 , W i s q = α s a f e u 0 − 1 / 2 M 1 M 0 . (15) \mathcal{M}_\ell=a_\ell I_C+b_\ell T_\ell+e_\ell T_\ell^2,\qquad\mathcal{W}_{\mathrm{isq}}=\alpha_{\mathrm{safe}}u_0^{-1/2}\mathcal{M}_1\mathcal{M}_0. \tag*{(15)} M ℓ = a ℓ I C + b ℓ T ℓ + e ℓ T ℓ 2 , W isq = α safe u 0 − 1/2 M 1 M 0 . (15)
Symmetric K ~ c \tilde{K}_c K ~ c runs the chunk-whitening kernels (Table 19 ), the 4,096-chunk (ρ p = 0 \rho_p=0 ρ p = 0 ) BF16 Woodbury test, and lagged block-32 causal whitening (Appendix § 9 ). Since dense W i s q \mathcal{W}_{\mathrm{isq}} W isq mixes chunk tokens, packed LM training uses causal Schur-oblique k ~ t \tilde{k}_t k ~ t in Eq. (13) (k t T k ~ t = 1 k_t^{\mathsf T}\tilde{k}_t=1 k t T k ~ t = 1 ) with Polar-2+ in the chunk-boundary Woodbury tail (ρ p = 1 \rho_p=1 ρ p = 1 resets Σ c = I d k \Sigma_c=I_{d_k} Σ c = I d k for SLA Tree-CP, while unscaled α = 1.000 \alpha=1.000 α = 1.000 still diverges at step 299 via tail gradients; for ρ p = 0.02 \rho_p=0.02 ρ p = 0.02 cross-chunk Σ c \Sigma_c Σ c , multiplicative Sylvester congruence B c Σ c − 1 B c T \mathcal{B}_c\Sigma_c^{-1}\mathcal{B}_c^{\mathsf T} B c Σ c − 1 B c T with contraction matrix B c ∈ R d k × d k \mathcal{B}_c\in\mathbb{R}^{d_k\times d_k} B c ∈ R d k × d k [Appendix A] replaces additive Eq. (14) , which otherwise hits NaN at step 282).
Table 19. Measured TPU v7x chunk-whitening latency (64 chunks = 8,192 tokens, C = 128 C=128 C = 128 , H k = 4 H_k=4 H k = 4 , d k = 256 d_k=256 d k = 256 ; MesaNet CG-10: 1.74/11.16 ms), exact conditioned Gram κ p o s t \kappa_{\mathrm{post}} κ post (≤ 1.123 \le1.123 ≤ 1.123 in BF16), and 4,096-chunk (524K-token, ρ p = 0 \rho_p=0 ρ p = 0 ) BF16 Woodbury stress-test λ min \lambda_{\min} λ m i n .
Proposition 4.2 (Exact Equioscillation and Safety Margin of Polar-2+). Writing m ℓ ( x ) = a ℓ + b ℓ x + e ℓ x 2 m_\ell(x)=a_\ell+b_\ell x+e_\ell x^2 m ℓ ( x ) = a ℓ + b ℓ x + e ℓ x 2 , let
z 2 ( λ ) = m 1 ( λ m 0 ( λ ) 2 ) m 0 ( λ ) z_2(\lambda)=m_1\left(\lambda m_0(\lambda)^2\right)m_0(\lambda) z 2 ( λ ) = m 1 ( λ m 0 ( λ ) 2 ) m 0 ( λ ) be the scalar map that Eq. (15) applies to an eigenvalue λ \lambda λ of T 0 T_0 T 0 (since T 1 = M 0 T 0 M 0 T_1=\mathcal{M}_0T_0\mathcal{M}_0 T 1 = M 0 T 0 M 0 has eigenvalue λ m 0 ( λ ) 2 \lambda m_0(\lambda)^2 λ m 0 ( λ ) 2 ), and let κ p o s t = κ 2 ( W i s q G r e g W i s q ) \kappa_{\mathrm{post}}=\kappa_2(\mathcal{W}_{\mathrm{isq}}\mathcal{G}_{\mathrm{reg}}\mathcal{W}_{\mathrm{isq}}) κ post = κ 2 ( W isq G reg W isq ) be the post-whitening condition number. On λ ∈ [ 0.02 , 1.0 ] \lambda\in[0.02,1.0] λ ∈ [ 0.02 , 1.0 ] , z 2 z_2 z 2 equioscillates with λ z 2 ( λ ) 2 ∈ [ 0.9520 , 1.0492 ] \lambda z_2(\lambda)^2\in[0.9520,1.0492] λ z 2 ( λ ) 2 ∈ [ 0.9520 , 1.0492 ] , exceeding 1 for unscaled α = 1 \alpha=1 α = 1 and driving recursive Woodbury indefinite. Scaling by α s a f e = 0.968 \alpha_{\mathrm{safe}}=0.968 α safe = 0.968 bounds exact λ ( α s a f e z 2 ( λ ) ) 2 ∈ [ 0.8920 , 0.9831 ] \lambda(\alpha_{\mathrm{safe}}z_2(\lambda))^2\in[0.8920,0.9831] λ ( α safe z 2 ( λ ) ) 2 ∈ [ 0.8920 , 0.9831 ] (1.69% BF16 contractive margin below 1; exact κ p o s t ≤ 1.102 \kappa_{\mathrm{post}}\le1.102 κ post ≤ 1.102 , BF16 5-matmul κ p o s t ≤ 1.123 \kappa_{\mathrm{post}}\le1.123 κ post ≤ 1.123 ).
When whitening helps vs. unwhitened SLA. Polar-2+ runs 2.66× \times × faster than Panel-16 Cholesky and 12.4× \times × –30.8× \times × faster than MesaNet CG-10/30 (0.897 ms F+B; Table 19 ), keeping 4,096-chunk (524K-token) BF16 Woodbury SPD (λ min ≥ 0.0388 \lambda_{\min} \ge0.0388 λ m i n ≥ 0.0388 ). It trains 2-seed 12.5B 15L:1A with zero NaNs and matches 10-matmul NS-5 (s1: 17.936 vs. 17.970 PPL), whereas α = 1.000 \alpha= 1.000 α = 1.000 diverges at step 299 and BF16 Cholesky hits NaN at step 54 (Appendix § 9 ). Because its language-modeling benefit shrinks to 0.004 nats by 15B tokens and unwhitened G = 4 G = 4 G = 4 SLA already saturates PPL at 60.4B 15L:1A+MS (15.443 vs. 15.624 at +3.7% step time), unwhitened SLA is our default mixer.
SPD-SLA-W serves three high-collision regimes:
correlated r ≥ 2 r \ge2 r ≥ 2 overwrite (κ 2 ( G ) ≈ 48 \kappa_2(\mathcal{G}) \approx48 κ 2 ( G ) ≈ 48 ; Figure 8 b, Appendix Table 20), reaching 60.3 dB BF16 SNR and 80.1% noisy r = 3 r = 3 r = 3 recall vs. 2.67 dB and 50.0% for MesaNet;
uniform-decay 12.5B 15L:1A (15.1B tok, Table 15 ), edging 15L SLA-R2 on 32K 8-N (39.8% vs. 34.8%) and N = 512 N = 512 N = 512 64K HotpotQA (69.5% vs. 68.2%); and
63.1B 15L:1A, reaching 12.42 vs. 13.01 32K Tail PPL at 5.0B tok (Table 15 ) and 11.82 Tail PPL with 57.8%/64.8% 64K SQuAD/HotpotQA at 30.2B (+12.5/ + 25.0 pts over 0L:1A; Table 16 ).
Related Work
Decay Granularity in Linear Recurrences. Linear attentions replace the growing softmax KV cache with a fixed-size state updated once per token [12 , 33 ] . Using the decay-group view of § 2.1 , most follow-up models differ in how finely their forget gate is resolved across key channels: RetNet [58 ] , Mamba-2 [14 ] , and GDN [68 ] use one decay rate per head, whereas GLA [66 ] , RWKV-6/7 [42 , 43 ] , KDA [71 ] , and GDN-2 [25 ] give every key channel its own rate; Mamba [22 ] , HGRN2 [46 ] , xLSTM [7 ] , and Mamba-3 [34 ] explore further gate and state designs. SLA takes a middle point with four decay rates per 256-channel head. This is coarse enough that the decay factors out of 64-channel MXU matmuls. In our sweep it slightly lowers perplexity relative to one rate per head across all three paired seeds but extrapolates somewhat worse, so the choice between the two is close (§ 3.5 ).
Delta Rules, Higher-Order Memories, and Momentum. DeltaNet [50 , 67 ] and test-time-trained memories [8 , 9 , 59 ] view the state update as one step of online regression, which improves associative recall over decay-only models [5 , 30 ] . Later work adds multiple Householder reflections [53 ] , negative eigenvalues [20 ] , decoupled per-channel erase and write gates [25 ] , second-order preconditioning [62 , 64 ] , log-linear state hierarchies [23 ] , and heavy-ball momentum [9 , 41 , 44 ] . These works improve what the update can express; SLA instead keeps GDN’s single-reflection update and changes its shape for systolic hardware. The two directions are complementary: GDN-2 reaches lower per-token perplexity than SLA in our runs (§ 3.5 ), making a fused TPU kernel that combines GDN-2’s decoupled gates with SLA’s systolic mapping a natural next step.
Sharing Key Heads Across Value Heads. GQA [2 ] shares key and value heads across query heads in softmax attention. For linear attention, Kimi Linear [71 ] , Qwen3-Next [48 ] , and fla [65 ] already share query-key heads across value heads at d k = 128 d_k = 128 d k = 128 , while Gated Attention [47 ] establishes sigmoid output gating. SLA uses the same idea at d k = 256 d_k = 256 d k = 256 with per-value-head normalization, choosing the sizes so that one key head fills a full MXU tile.
Hardware-Aware Kernels. FlashAttention [13 , 15 , 70 ] re-architects softmax attention around the GPU memory hierarchy. For linear recurrences, fla [65 ] provides chunkwise WY kernels [10 ] for GPU tensor cores, and MaxText and Tokamax [1 , 19 ] port KDA to TPUs. These kernels optimize a fixed model shape; SLA also changes the model so that every intra-chunk stage maps onto the MXU and fits in one fused kernel (distinct from works that abbreviate sparse-linear or softmax-linear hybrids as SLA).
Context Parallelism for Linear Recurrences. Head-sharded Ulysses [29 ] requires at least as many key heads as devices, ring-based methods [36 , 45 , 56 ] pass the state through O ( P ) O(P) O ( P ) sequential hops, and LASP-2 [57 ] gathers chunk states with one AllGather. Because every gated delta chunk transition with scalar β t \beta_t β t is affine, chunks can also be combined with a parallel prefix scan [11 , 39 ] ; we use this standard construction for both SLA and KDA (§ 3.3 ).
Conclusion
Summary. Recent gated-delta models were shaped around GPUs, where per-channel gate rescaling and sequential in-block triangular solves are cheap relative to matmuls. On systolic accelerators, where matrix units outpace vector lanes by two orders of magnitude, kernel fusion alone cannot hide a vector-bound recurrence. Systolic Linear Attention (SLA) resolves this mismatch by co-designing the recurrence around the systolic array: sharing 256-wide keys across value heads fills full tiles, grouping forget gates into tiles factors decay out of matmuls, and block-doubling keeps chunk inversion and its backward pass on the MXU in one VMEM kernel. On TPU v7x, a fused SLA-R4 layer runs 1.6 × 1.6\times 1.6 × –2.5 × 2.5\times 2.5 × faster than Tokamax KDA (1.4 × 1.4\times 1.4 × –2.4 × 2.4\times 2.4 × with exact inversion) with 52%–61% less temporary HBM, and across 6.5B–63.1B MoE models on up to 512 chips per run, SLA trains 8%–24% faster than MaxText KDA at comparable perplexity without requiring linear-attention gate clamping at any scale (8 K → 64 K 8\mathrm{K} \to64\mathrm{K} 8 K → 64 K zero-shot on RoPE backbones, and 83.2%–84.4% 32K 8-Needle on 63.1B NoPE with global SSMax alone).
Future Work. SLA targets accelerators with wide matrix tiles and a large matrix-to-vector gap; on GPUs, small Tensor Core tiles and fast vector lanes already saturate narrow states. Because our speedups combine tile-grouped decay with head sharing, BF16 state storage, and VMEM fusion while GDN-2 uses an unfused port despite lower per-token perplexity, next steps include building fused TPU kernels for wide-key, shared-head KDA and GDN-2 to isolate each factor and combine GDN-2’s decoupled gates with SLA’s systolic mapping, closing SLA’s ≈ 0.1 \approx0.1 ≈ 0.1 perplexity gap to wide-key KDA (0.3 with the 1-step Neumann inverse) on matched seeds, and decoupling chunk-boundary momentum from C C C . Finally, extending multi-seed runs beyond 8K Dolma, where BF16 MoE routing variance (≈ 0.18 \approx0.18 ≈ 0.18 perplexity) masks small gaps, 63.1B runs are single-seed, and 64K QA sets are small, will sharpen long-context comparisons.
Proofs of Whitening and Polar-2+
Part 1 (Euclidean Interference and Diagonal Non-Invariance). Let unit keys k 1 , … , k N ∈ R d k k_1,\ldots,k_N \in\mathbb{R}^{d_k} k 1 , … , k N ∈ R d k satisfy ∣ ⟨ k i , k j ⟩ ∣ = ν c o h |\langle k_i,k_j\rangle|=\nu_{\mathrm{coh}} ∣ ⟨ k i , k j ⟩ ∣ = ν coh for i ≠ j i\ne j i = j (coherence ν c o h \nu_{\mathrm{coh}} ν coh ; e.g., an equiangular tight frame or shared-subspace correlated dictionary) and draw independent zero-mean values v j ∈ R d v v_j \in\mathbb{R}^{d_v} v j ∈ R d v with covariance
E [ v j v j T ] = s v 2 d v I d v \mathbb{E}[v_jv_j^{\mathsf T}]=\frac{s_v^2}{d_v}I_{d_v} E [ v j v j T ] = d v s v 2 I d v
(for d v × d v d_v \times d_v d v × d v identity I d v I_{d_v} I d v ), so s v 2 = tr ( s v 2 d v I d v ) = E ∥ v j ∥ 2 2 s_v^2=\operatorname{tr}\left(\frac{s_v^2}{d_v}I_{d_v}\right)=\mathbb{E}\lVert v_j\rVert_2^2 s v 2 = tr ( d v s v 2 I d v ) = E ∥ v j ∥ 2 2 . Under single-pass Euclidean delta updates with β t = 1 \beta_t=1 β t = 1 from S 0 = 0 S_0=0 S 0 = 0 , updating S t = ( I d k − k t k t T ) S t − 1 + k t v t T S_t=(I_{d_k}-k_tk_t^{\mathsf T})S_{t-1}+k_tv_t^{\mathsf T} S t = ( I d k − k t k t T ) S t − 1 + k t v t T writes later values j > i j>i j > i with projection ⟨ k j , k i ⟩ v j \langle k_j,k_i\rangle v_j ⟨ k j , k i ⟩ v j along k i k_i k i : for i = N − 1 i=N-1 i = N − 1 ,
S N T k N − 1 − v N − 1 = ⟨ k N , k N − 1 ⟩ ( v N − S N − 1 T k N ) . S_N^{\mathsf T}k_{N-1}-v_{N-1}=\langle k_N,k_{N-1}\rangle(v_N-S_{N-1}^{\mathsf T}k_N). S N T k N − 1 − v N − 1 = ⟨ k N , k N − 1 ⟩ ( v N − S N − 1 T k N ) .
and summing independent cross-key leakage over Θ ( N ) \Theta(N) Θ ( N ) interfering keys yields
E ∥ S N T k i − v i ∥ 2 2 = Ω ( N ν c o h 2 s v 2 ) = Ω ( ρ e f f ( d k ν c o h 2 ) s v 2 ) \mathbb{E}\left\lVert S_N^{\mathsf T}k_i-v_i\right\rVert_2^2=\Omega(N\nu_{\mathrm{coh}}^2s_v^2)=\Omega\left(\rho_{\mathrm{eff}}(d_k\nu_{\mathrm{coh}}^2)s_v^2\right) E S N T k i − v i 2 2 = Ω ( N ν coh 2 s v 2 ) = Ω ( ρ eff ( d k ν coh 2 ) s v 2 )
when N ν c o h 2 ≤ 1 / 2 N\nu_{\mathrm{coh}}^2\le1/2 N ν coh 2 ≤ 1/2 (ρ e f f = N / d k \rho_{\mathrm{eff}}=N/d_k ρ eff = N / d k , reducing to Ω ( ρ e f f s v 2 ) \Omega(\rho_{\mathrm{eff}}s_v^2) Ω ( ρ eff s v 2 ) at isotropic ν c o h 2 = 1 / d k \nu_{\mathrm{coh}}^2=1/d_k ν coh 2 = 1/ d k ). For the diagonal column-energy preconditioner K diag ( p ( K ) ) − 1 / 2 K\operatorname{diag}(p(K))^{-1/2} K diag ( p ( K ) ) − 1/2 where p m ( K ) = ∑ i = 1 N K i , m 2 p_m(K)=\sum_{i=1}^{N}K_{i,m}^2 p m ( K ) = ∑ i = 1 N K i , m 2 , rotating K ↦ K Q K\mapsto KQ K ↦ K Q with Q = diag ( ξ ) Q H ∈ O ( d k ) Q=\operatorname{diag}(\xi)Q_H\in O(d_k) Q = diag ( ξ ) Q H ∈ O ( d k ) for random Rademacher signs ξ ∈ { ± 1 } d k \xi\in\{\pm1\}^{d_k} ξ ∈ { ± 1 } d k followed by a normalized Hadamard matrix Q H ∈ { ± d k − 1 / 2 } d k × d k Q_H\in\{\pm d_k^{-1/2}\}^{d_k\times d_k} Q H ∈ { ± d k − 1/2 } d k × d k equalizes all column energies in expectation,
E ξ [ p m ( K diag ( ξ ) Q H ) ] = 1 d k ∥ K ∥ F 2 \mathbb{E}_{\xi}\left[p_m(K\operatorname{diag}(\xi)Q_H)\right]=\frac{1}{d_k}\lVert K\rVert_F^2 E ξ [ p m ( K diag ( ξ ) Q H ) ] = d k 1 ∥ K ∥ F 2
(and by Schur-Horn some rotation Q ∈ O ( d k ) Q\in O(d_k) Q ∈ O ( d k ) makes Q T K T K Q Q^{\mathsf T}K^{\mathsf T}KQ Q T K T K Q exactly constant-diagonal), reducing diag ( p ( K Q ) ) − 1 / 2 \operatorname{diag}(p(KQ))^{-1/2} diag ( p ( K Q ) ) − 1/2 to a scalar multiple of I d k I_{d_k} I d k that leaves κ 2 ( ( K Q ) ( K Q ) T ) = κ 2 ( K K T ) \kappa_2((KQ)(KQ)^{\mathsf T})=\kappa_2(KK^{\mathsf T}) κ 2 (( K Q ) ( K Q ) T ) = κ 2 ( K K T ) unchanged; by contrast, ( I C + η c K c K c T ) − 1 / 2 K c (I_C+\eta_cK_cK_c^{\mathsf T})^{-1/2}K_c ( I C + η c K c K c T ) − 1/2 K c transforms equivariantly (K c ↦ K c Q ⟹ K ~ c ↦ K ~ c Q K_c\mapsto K_cQ\Longrightarrow\widetilde K_c\mapsto\widetilde K_cQ K c ↦ K c Q ⟹ K c ↦ K c Q ) for all Q ∈ O ( d k ) Q\in O(d_k) Q ∈ O ( d k ) .
Part 2 (Gated RLS Overwrite-Retention Trade-off). Under gated RLS [64 ] , with key covariance H t ∈ R d k × d k H_t\in\mathbb{R}^{d_k\times d_k} H t ∈ R d k × d k and key-value moment B t ∈ R d k × d v B_t\in\mathbb{R}^{d_k\times d_v} B t ∈ R d k × d v (local to this paragraph), H t = γ H t − 1 + k t k t T H_t=\gamma H_{t-1}+k_tk_t^{\mathsf T} H t = γ H t − 1 + k t k t T and B t = γ B t − 1 + k t v t T B_t=\gamma B_{t-1}+k_tv_t^{\mathsf T} B t = γ B t − 1 + k t v t T with scalar forget factor γ ∈ ( 0 , 1 ] \gamma\in(0,1] γ ∈ ( 0 , 1 ] and ridge ϵ I d k \epsilon I_{d_k} ϵ I d k (ϵ > 0 \epsilon>0 ϵ > 0 ); readout on query k ∈ R d k k\in\mathbb{R}^{d_k} k ∈ R d k is B t T H t − 1 k B_t^{\mathsf T}H_t^{-1}k B t T H t − 1 k . If a unit key k k k is written r ≥ 2 r\ge2 r ≥ 2 times, with value v ( m ) v^{(m)} v ( m ) at step t m t_m t m (t 1 < ⋯ < t r = t t_1<\cdots<t_r=t t 1 < ⋯ < t r = t ) and weights w m = γ t − t m w_m=\gamma^{t-t_m} w m = γ t − t m , Sherman-Morrison on H t = ϵ I d k + ( ∑ m = 1 r w m ) k k T H_t=\epsilon I_{d_k}+(\sum_{m=1}^{r}w_m)kk^{\mathsf T} H t = ϵ I d k + ( ∑ m = 1 r w m ) k k T gives
lim ϵ → 0 + B t T H t − 1 k = ∑ m = 1 r w m v ( m ) ∑ m = 1 r w m . \lim_{\epsilon\to0^+}B_t^{\mathsf T}H_t^{-1}k=\frac{\sum_{m=1}^{r}w_mv^{(m)}}{\sum_{m=1}^{r}w_m}. ϵ → 0 + lim B t T H t − 1 k = ∑ m = 1 r w m ∑ m = 1 r w m v ( m ) .
Letting v ˉ 1 : r − 1 = 1 r − 1 ∑ m = 1 r − 1 v ( m ) \bar v_{1:r-1}=\frac{1}{r-1}\sum_{m=1}^{r-1}v^{(m)} v ˉ 1 : r − 1 = r − 1 1 ∑ m = 1 r − 1 v ( m ) denote the mean of the first r − 1 r-1 r − 1 values, readout at γ = 1 \gamma=1 γ = 1 (w m ≡ 1 w_m\equiv1 w m ≡ 1 ) equals 1 r ∑ m = 1 r v ( m ) \frac{1}{r}\sum_{m=1}^{r}v^{(m)} r 1 ∑ m = 1 r v ( m ) with error r − 1 r ∥ v ( r ) − v ˉ 1 : r − 1 ∥ 2 \frac{r-1}{r}\lVert v^{(r)}-\bar v_{1:r-1}\rVert_2 r r − 1 ∥ v ( r ) − v ˉ 1 : r − 1 ∥ 2 relative to v ( r ) v^{(r)} v ( r ) . Already for r = 2 r=2 r = 2 consecutive writes and any target relative tolerance ϵ ∈ ( 0 , 1 / 2 ) \epsilon\in(0,1/2) ϵ ∈ ( 0 , 1/2 ) , the error is γ 1 + γ ∥ v ( 2 ) − v ( 1 ) ∥ 2 \frac{\gamma}{1+\gamma}\lVert v^{(2)}-v^{(1)}\rVert_2 1 + γ γ ∥ v ( 2 ) − v ( 1 ) ∥ 2 , so making it at most ϵ ∥ v ( 2 ) − v ( 1 ) ∥ 2 \epsilon\lVert v^{(2)}-v^{(1)}\rVert_2 ϵ ∥ v ( 2 ) − v ( 1 ) ∥ 2 requires γ ≤ ϵ / ( 1 − ϵ ) \gamma\le\epsilon/(1-\epsilon) γ ≤ ϵ / ( 1 − ϵ ) , which simultaneously decays any non-overwritten key stored t − j t-j t − j steps earlier (at step j < t j<t j < t ) by γ t − j \gamma^{t-j} γ t − j relative to the regularizer ϵ I d k \epsilon I_{d_k} ϵ I d k , causing rapid forgetting in finite precision (verified in Table 20 ). Under oblique whitened delta, intra-chunk Gram whitening orthogonalizes concurrent keys while the oblique erasure ( I d k − β t k ~ t k ~ t T ) (I_{d_k}-\beta_t\tilde k_t\tilde k_t^{\mathsf T}) ( I d k − β t k ~ t k ~ t T ) subtracts the current state projection v ^ t = ( F t S ~ t − 1 ) T k t \hat v_t=(F_t\tilde S_{t-1})^{\mathsf T}k_t v ^ t = ( F t S ~ t − 1 ) T k t selectively along k ~ t \tilde k_t k ~ t without decaying orthogonal associations.
Table 20. Measured associative recall SNR (dB, reported as BF16 [FP32] ) and Mod-2 parity accuracy across linear attention update rules (d k = 256 d_k = 256 d k = 256 , C = 128 C = 128 C = 128 , 8 seeds, κ 2 ( G ) ≈ 48 \kappa_2(G) \approx48 κ 2 ( G ) ≈ 48 ); κ f o r g e t = κ 2 ( F c ) \kappa_{\mathrm{forget}} = \kappa_2(F_c) κ forget = κ 2 ( F c ) is the condition number of the chunk forget gate.
Part 3 (Polar-2+ Non-Circular Normalization, Equioscillation, and BF16 Safety). Let Σ c − 1 ≻ 0 \Sigma_{c-1}\succ0 Σ c − 1 ≻ 0 (initialized from Σ 0 = I d k \Sigma_0=I_{d_k} Σ 0 = I d k ) and let Y c = K c Σ c − 1 1 / 2 ∈ R C × d k Y_c=K_c\Sigma_{c-1}^{1/2}\in\mathbb{R}^{C\times d_k} Y c = K c Σ c − 1 1/2 ∈ R C × d k have singular values σ i ≥ 0 \sigma_i\ge0 σ i ≥ 0 (unrelated to the sigmoid σ \sigma σ ), so the unregularized Gram G 0 = I C + η c Y c Y c T \mathcal{G}_0=I_C+\eta_cY_cY_c^{\mathsf T} G 0 = I C + η c Y c Y c T (η c ∈ ( 0 , 1 ] \eta_c\in(0,1] η c ∈ ( 0 , 1 ] ) has eigenvalues ω 0 , i = 1 + η c σ i 2 \omega_{0,i}=1+\eta_c\sigma_i^2 ω 0 , i = 1 + η c σ i 2 . Computing G 0 = G 0 / ∥ G 0 ∥ F \mathcal{G}_0=\mathcal{G}_0/\lVert\mathcal{G}_0\rVert_F G 0 = G 0 / ∥ G 0 ∥ F and
u ^ 0 = max ( ∥ G 0 ∥ F ∥ G 0 2 ∥ F 1 / 2 , 1 ) = max ( ( ∑ j ω 0 , j 4 ) 1 / 4 , 1 ) ≥ max i ω 0 , i \hat u_0=\max\left(\lVert\mathcal{G}_0\rVert_F\lVert\mathcal{G}_0^2\rVert_F^{1/2},1\right)=\max\left(\left(\sum_j\omega_{0,j}^4\right)^{1/4},1\right)\ge\max_i\omega_{0,i} u ^ 0 = max ( ∥ G 0 ∥ F ∥ G 0 2 ∥ F 1/2 , 1 ) = max ( j ∑ ω 0 , j 4 ) 1/4 , 1 ≥ i max ω 0 , i
yields a non-circular upper bound on λ max ( G 0 ) \lambda_{\max}(\mathcal{G}_0) λ m a x ( G 0 ) from one [ C , C ] [C,C] [ C , C ] matmul G ˉ 0 2 \bar{G}_0^2 G ˉ 0 2 . Setting adaptive ridge ε r e g = max ( 0.02 , 0.0205 0.9795 u ^ 0 − 1 ) \varepsilon_{\mathrm{reg}}=\max\left(0.02,\frac{0.0205}{0.9795}\hat{u}_0-1\right) ε reg = max ( 0.02 , 0.9795 0.0205 u ^ 0 − 1 ) , G r e g = G 0 + ε r e g I C \mathcal{G}_{\mathrm{reg}}=\mathcal{G}_0+\varepsilon_{\mathrm{reg}}I_C G reg = G 0 + ε reg I C with shifted eigenvalues ω i = ω 0 , i + ε r e g \omega_i=\omega_{0,i}+\varepsilon_{\mathrm{reg}} ω i = ω 0 , i + ε reg , and u 0 = u ^ 0 + ε r e g ≥ max i ω i u_0=\hat{u}_0+\varepsilon_{\mathrm{reg}}\ge\max_i\omega_i u 0 = u ^ 0 + ε reg ≥ max i ω i guarantees normalized eigenvalues λ i = ω i / u 0 ∈ [ 0.02 , 1.0 ] \lambda_i=\omega_i/u_0\in[0.02,1.0] λ i = ω i / u 0 ∈ [ 0.02 , 1.0 ] for all u ^ 0 ≥ 1 \hat{u}_0\ge1 u ^ 0 ≥ 1 while expanding
Γ 0 2 = ( G 0 + ε r e g I C ) 2 u 0 2 = ∥ G 0 ∥ F 2 G ˉ 0 2 + 2 ε r e g G 0 + ε r e g 2 I C u 0 2 \Gamma_0^2=\frac{(\mathcal{G}_0+\varepsilon_{\mathrm{reg}}I_C)^2}{u_0^2}=\frac{\|\mathcal{G}_0\|_F^2\bar{G}_0^2+2\varepsilon_{\mathrm{reg}}\mathcal{G}_0+\varepsilon_{\mathrm{reg}}^2I_C}{u_0^2} Γ 0 2 = u 0 2 ( G 0 + ε reg I C ) 2 = u 0 2 ∥ G 0 ∥ F 2 G ˉ 0 2 + 2 ε reg G 0 + ε reg 2 I C
with zero additional matmuls (5 matmuls total across stages ℓ ∈ { 0 , 1 } \ell\in\{0,1\} ℓ ∈ { 0 , 1 } , where M ℓ = m ℓ ( T ℓ ) \mathcal{M}_\ell=m_\ell(T_\ell) M ℓ = m ℓ ( T ℓ ) evaluates m 0 ( λ ) = a 0 + b 0 λ + e 0 λ 2 m_0(\lambda)=a_0+b_0\lambda+e_0\lambda^2 m 0 ( λ ) = a 0 + b 0 λ + e 0 λ 2 and m 1 ( x ) = a 1 + b 1 x + e 1 x 2 m_1(x)=a_1+b_1x+e_1x^2 m 1 ( x ) = a 1 + b 1 x + e 1 x 2 ).
Clamping β t ≤ ( 2 − ε β ) / max ( k ~ t ⊤ k ~ t , ε β ) \beta_t\le(2-\varepsilon_\beta)/\max(\tilde{k}_t^\top\tilde{k}_t,\varepsilon_\beta) β t ≤ ( 2 − ε β ) / max ( k ~ t ⊤ k ~ t , ε β ) keeps the nontrivial eigenvalue 1 − β t k ~ t ⊤ k ~ t 1-\beta_t\tilde{k}_t^\top\tilde{k}_t 1 − β t k ~ t ⊤ k ~ t of I d k − β t k ~ t k ~ t ⊤ I_{d_k}-\beta_t\tilde{k}_t\tilde{k}_t^\top I d k − β t k ~ t k ~ t ⊤ in [ − 1 + ε β , 1 ] [-1+\varepsilon_\beta,1] [ − 1 + ε β , 1 ] whenever k ~ t ⊤ k ~ t > 0 \tilde{k}_t^\top\tilde{k}_t>0 k ~ t ⊤ k ~ t > 0 (in packed LM training, intra-chunk causal Schur-oblique projection sets
k ~ t = ( I d k − η c Σ c − 1 K < t ⊤ K < t ) Σ c − 1 k t k t ⊤ ( I d k − η c Σ c − 1 K < t ⊤ K < t ) Σ c − 1 k t \tilde{k}_t=\frac{(I_{d_k}-\eta_c\Sigma_{c-1}K_{<t}^{\top}K_{<t})\Sigma_{c-1}k_t}{k_t^\top(I_{d_k}-\eta_c\Sigma_{c-1}K_{<t}^{\top}K_{<t})\Sigma_{c-1}k_t} k ~ t = k t ⊤ ( I d k − η c Σ c − 1 K < t ⊤ K < t ) Σ c − 1 k t ( I d k − η c Σ c − 1 K < t ⊤ K < t ) Σ c − 1 k t
so k t ⊤ k ~ t = 1 k_t^\top\tilde{k}_t=1 k t ⊤ k ~ t = 1 depends strictly on rows < t <t < t , while symmetric Polar-2+ governs the chunk-boundary Woodbury tail, two-sided chunk-whitening kernels, and lagged block-32 causal whitening). Substituting W i s q = α s a f e u 0 − 1 / 2 M 1 M 0 = α s a f e u 0 − 1 / 2 z 2 ( T 0 ) W_{\mathrm{isq}}=\alpha_{\mathrm{safe}}u_0^{-1/2}\mathcal{M}_1\mathcal{M}_0=\alpha_{\mathrm{safe}}u_0^{-1/2}z_2(T_0) W isq = α safe u 0 − 1/2 M 1 M 0 = α safe u 0 − 1/2 z 2 ( T 0 ) and K ~ c = η c W i s q K c Σ c − 1 \tilde{K}_c=\sqrt{\eta_c}W_{\mathrm{isq}}K_c\Sigma_{c-1} K ~ c = η c W isq K c Σ c − 1 (Eq. (13) ) into the unrelaxed Woodbury update Σ ~ c = Σ c − 1 − K ~ c ⊤ K ~ c \tilde{\Sigma}_c=\Sigma_{c-1}-\tilde{K}_c^\top\tilde{K}_c Σ ~ c = Σ c − 1 − K ~ c ⊤ K ~ c (Eq. (14) at ρ p = 0 \rho_p=0 ρ p = 0 ) gives
Σ ~ c = Σ c − 1 1 / 2 ( I d k − η c Y c ⊤ W i s q 2 Y c ) Σ c − 1 1 / 2 , \tilde{\Sigma}_c=\Sigma_{c-1}^{1/2}(I_{d_k}-\eta_cY_c^\top W_{\mathrm{isq}}^2Y_c)\Sigma_{c-1}^{1/2}, Σ ~ c = Σ c − 1 1/2 ( I d k − η c Y c ⊤ W isq 2 Y c ) Σ c − 1 1/2 ,
whose i i i -th singular-direction multiplier is
1 − η c σ i 2 u 0 − 1 α s a f e 2 z 2 ( λ i ) 2 = 1 − η c σ i 2 1 + ε r e g + η c σ i 2 λ i ( α s a f e z 2 ( λ i ) ) 2 . (16) 1-\eta_c\sigma_i^2u_0^{-1}\alpha_{\mathrm{safe}}^2z_2(\lambda_i)^2
=1-\frac{\eta_c\sigma_i^2}{1+\varepsilon_{\mathrm{reg}}+\eta_c\sigma_i^2}\lambda_i\left(\alpha_{\mathrm{safe}}z_2(\lambda_i)\right)^2. \tag*{(16)} 1 − η c σ i 2 u 0 − 1 α safe 2 z 2 ( λ i ) 2 = 1 − 1 + ε reg + η c σ i 2 η c σ i 2 λ i ( α safe z 2 ( λ i ) ) 2 . (16)
Direct evaluation of the scalar map z 2 z_2 z 2 (Proposition 4.2 ) on λ ∈ [ 0.02 , 1.0 ] \lambda\in[0.02,1.0] λ ∈ [ 0.02 , 1.0 ] gives minimax equioscillation λ z 2 ( λ ) ∈ [ 0.9757 , 1.0243 ] \sqrt{\lambda}z_2(\lambda)\in[0.9757,1.0243] λ z 2 ( λ ) ∈ [ 0.9757 , 1.0243 ] , hence λ z 2 ( λ ) 2 ∈ [ 0.9520 , 1.0492 ] \lambda z_2(\lambda)^2\in[0.9520,1.0492] λ z 2 ( λ ) 2 ∈ [ 0.9520 , 1.0492 ] (and < 0.952 <0.952 < 0.952 on ( 0 , 0.02 ) (0,0.02) ( 0 , 0.02 ) since z 2 ( 0 ) = a 1 a 0 ≈ 9.877 z_2(0)=a_1a_0\approx9.877 z 2 ( 0 ) = a 1 a 0 ≈ 9.877 ). Thus unscaled α = 1 \alpha=1 α = 1 exceeds 1 by + 4.92 % +4.92\% + 4.92% near equioscillation peaks, driving Eq. (16) negative when η c σ i 2 ≫ 1 \eta_c\sigma_i^2\gg1 η c σ i 2 ≫ 1 . Setting α s a f e = 0.968 \alpha_{\mathrm{safe}}=0.968 α safe = 0.968 (α s a f e 2 = 0.937024 \alpha_{\mathrm{safe}}^2=0.937024 α safe 2 = 0.937024 ) maps exact λ ( α s a f e z 2 ( λ ) ) 2 \lambda(\alpha_{\mathrm{safe}}z_2(\lambda))^2 λ ( α safe z 2 ( λ ) ) 2 onto [ 0.8920 , 0.9831 ] [0.8920,0.9831] [ 0.8920 , 0.9831 ] across [ 0.02 , 1.0 ] [0.02,1.0] [ 0.02 , 1.0 ] , giving exact κ 2 ( W i s q G r e g W i s q ) = 0.9831 / 0.8920 = 1.102 \kappa_2(W_{\mathrm{isq}}\mathcal{G}_{\mathrm{reg}}W_{\mathrm{isq}})=0.9831/0.8920=1.102 κ 2 ( W isq G reg W isq ) = 0.9831/0.8920 = 1.102 (≤ 1.123 \le1.123 ≤ 1.123 in 5-matmul BF16 hardware execution).
At ρ p = 0 \rho_p=0 ρ p = 0 , 4,096 correlated BF16 chunks (524,288 tokens) maintain λ min ( Σ c ) ≥ 0.0388 > 0 \lambda_{\min}(\Sigma_c)\ge0.0388>0 λ m i n ( Σ c ) ≥ 0.0388 > 0 ; when evolving Σ c \Sigma_c Σ c across thousands of packed BF16 LM gradient steps with ρ p < 1 \rho_p<1 ρ p < 1 , replacing additive subtraction with multiplicative Sylvester congruence B c Σ c − 1 B c ⊤ \mathcal{B}_c\Sigma_{c-1}\mathcal{B}_c^\top B c Σ c − 1 B c ⊤ (with contraction matrix B c = I d k − η c 2 Σ c − 1 K c ⊤ W i s q 2 K c \mathcal{B}_c=I_{d_k}-\frac{\eta_c}{2}\Sigma_{c-1}K_c^\top W_{\mathrm{isq}}^2K_c B c = I d k − 2 η c Σ c − 1 K c ⊤ W isq 2 K c , or B c = I d k − C ^ c \mathcal{B}_c=I_{d_k}-\hat{C}_c B c = I d k − C ^ c using row-sum-normalized key covariance C ^ c ∝ η c K c ⊤ K c \hat{C}_c\propto\eta_cK_c^\top K_c C ^ c ∝ η c K c ⊤ K c in our kernel) enforces PSD preservation unconditionally (Appendix § 9 ).
Associative Recall, Gated RLS Trade-Off, and Gate Conditioning
Table 20 reports synthetic associative-recall SNR (dB) in both BF16 and FP32 alongside Mod-2 parity accuracy and the chunk forget-gate condition number κ f o r g e t = κ 2 ( F c ) \kappa_{\mathrm{forget}}=\kappa_2(F_c) κ forget = κ 2 ( F c ) (d k = 256 d_k=256 d k = 256 , C = 128 C=128 C = 128 , 8 seeds, correlated keys with κ 2 ( G ) ≈ 48 \kappa_2(\mathcal{G})\approx48 κ 2 ( G ) ≈ 48 ). Separating BF16 from FP32 clarifies precision ceilings: exact inverse-square-root whitening reaches > 140 >140 > 140 dB (r = 2 , 3 r=2,3 r = 2 , 3 overwrite) and 51.9 dB (write-once) in FP32, whereas BF16 readout caps SNR near 59–60 dB due to 7-bit mantissa rounding (20 log 10 ( 2 7 12 ) ≈ 53 20\log_{10}(2^7\sqrt{12})\approx53 20 log 10 ( 2 7 12 ) ≈ 53 –60 dB). Notably, within-layer Euclidean delta rules reach 58.40–60.97 dB on the final r = 3 r=3 r = 3 overwrite via recency-weighted last-write erasure, whereas at r = 2 r=2 r = 2 KDA’s channel decay (κ f o r g e t = 1.1 × 10 7 \kappa_{\mathrm{forget}}=1.1\times10^7 κ forget = 1.1 × 1 0 7 ) collapses to 0.07 dB (and write-once − 0.84 -0.84 − 0.84 dB) and unwhitened SLA (κ f o r g e t = 1.00 \kappa_{\mathrm{forget}}=1.00 κ forget = 1.00 ) drops to 24.10 dB vs. 60.33 dB for SPD-SLA-W. Sweeping gated MesaNet’s forget factor γ ∈ { 1.00 , 0.98 , 0.80 } \gamma\in\{1.00,0.98,0.80\} γ ∈ { 1.00 , 0.98 , 0.80 } directly confirms Proposition 4.1(2): γ = 1.00 \gamma=1.00 γ = 1.00 and γ = 0.98 \gamma=0.98 γ = 0.98 suffer severe stale-value averaging (1.90 and 4.13 dB at r = 3 r=3 r = 3 ), whereas γ = 0.80 \gamma=0.80 γ = 0.80 improves overwrite SNR (29.53 dB) only by degrading BF16 write-once retention from 10.63 dB down to 5.78 dB.
Negative Results and Reproducibility Details
We record three negative results observed during hardware-algorithm co-design alongside baseline software versions:
Tier-1 VMEM Fusion Without Shape Co-Design (1.03 × 1.03\times 1.03 × ). Fusing Ops 2–7a on GPU-shaped 8 × 128 × 256 8 \times128 \times256 8 × 128 × 256 KDA yields only 3% forward-backward speedup on TPU v7x (7.108 → 6.912 ms; Table 5 ) because the sequential in-block substitution of the triangular solve and the 50%-padded FP32 state scans remain VPU- and compute-bound inside VMEM.
Unscaled Minimax Polynomials, BF16 Cholesky, and Additive Woodbury Drift in Live LM Training. Using exact-arithmetic minimax coefficients (α = 1.000 \alpha= 1.000 α = 1.000 ) causes λ z 2 ( λ ) \lambda_{z2}(\lambda) λ z 2 ( λ ) to reach 1.0492 > 1, driving BF16 Woodbury covariance Σ c \Sigma_c Σ c indefinite (λ min = − 0.0412 \lambda_{\min} = -0.0412 λ m i n = − 0.0412 ; Table 19 ) and diverging 12.5B MoE pre-training to 258.5 PPL by step 299, while direct BF16 Cholesky/Woodbury hits NaN at step 54. Scaling by α s a f e = 0.968 \alpha_{\mathrm{safe}} = 0.968 α safe = 0.968 bounds the exact peak to 0.9831, preserves λ min ≥ 0.0388 \lambda_{\min} \ge0.0388 λ m i n ≥ 0.0388 across 4,096 chunks (524K tokens at ρ p = 0 \rho_p = 0 ρ p = 0 ), and trains cleanly for 7,200 steps across 2 seeds at ρ p = 1.0 \rho_p = 1.0 ρ p = 1.0 (0 NaNs; seed-matched s1: 17.936 vs. 17.970 PPL for NS-5, 2-seed Polar-2+ mean 17.970 on the +MS arms [s0 18.004]; the uniform-decay arm reaches 17.575 in Table 15 ). When evolving Σ c \Sigma_c Σ c across thousands of packed BF16 LM gradient steps with ρ p = 0.02 \rho_p = 0.02 ρ p = 0.02 , additive subtraction Σ c − 1 − K ~ c T K ~ c \Sigma_{c-1} - \widetilde{K}_c^{\mathsf{T}}\widetilde{K}_c Σ c − 1 − K c T K c still accumulates BF16 rounding drift (NaN at step 282), whereas multiplicative Sylvester congruence B c Σ c − 1 B c T B_c\Sigma_{c-1}B_c^{\mathsf{T}} B c Σ c − 1 B c T preserves PSD unconditionally (and strictly causal block-32 lagged two-sided Polar-2+, polar2_causal_blk32, completes all 7,200 steps [1510B tokens] on 12.5B 15L:1A with 0 NaNs, compressing κ 2 ( I C + M c ) \kappa_2(I_C + M_c) κ 2 ( I C + M c ) from 11.9–79.3 to 1.12–1.40 across all 15 linear layers and reaching 17.684 11-set PPL, 11.49 32K Tail PPL, 49.0% norm9 [49.1% FP8], and 68.8% 64K HotpotQA).
Global First-Order Neumann Inversion at C ≥ 128 C \ge128 C ≥ 128 . The unscaled global truncation ( I C + M C ) − 1 ≈ I C − M C (I_C + M_C)^{-1} \approx I_C - M_C ( I C + M C ) − 1 ≈ I C − M C degrades local delta order and incurs +0.014 nats validation loss on Dolma 3. The block variant A ~ N 1 \widetilde{A}_{\mathrm{N1}} A N1 (§ 2.4.3 ) is exact for all lags t − j ≤ 16 t-j \le16 t − j ≤ 16 ; the row-scaled variant A N 1 r o w A_{\mathrm{N1}}^{\mathrm{row}} A N1 row used in A3/A5 is a contractive version of this global truncation, and its cost is the +0.22 perplexity A2→A3 step in Table 11 b.