betaFeedback

WorksComputer Science

LinQuant: Functional Invariances for Low-Precision Linear Recurrences

Abstract

Linear recurrences are emerging as practical alternatives to softmax attention in large language models, appearing in both standalone and hybrid architectures. However, despite their rapid emergence, low-precision inference remains much less developed for these models than for softmax attention. During prefill, chunkwise-parallel algorithms cast much of the recurrence as dense matrix multiplications, suggesting a natural path to low-precision execution. Simply quantizing these operations in their native form, however, can be fragile. In this work, we identify functional invariances that preserve the full-precision computation while making the operands more amenable to quantization. Building on this observation, we introduce LinQuant, a framework for low-precision recurrent prefill based on seven such invariances, including rotations, diagonal rescalings, and alternative placements of recurrent decay and write factors. We derive when these transformations remain valid across common recurrence families spanning linear, gated, and delta-rule updates, and show that they provide complementary degrees of freedom for improving low-precision accuracy. We implement these transformations in INT8 and INT4 kernels and evaluate them on the recurrences of Nemotron-H, Qwen3.5, and Kimi Linear, recovering most of the accuracy lost to quantization while accelerating recurrent prefill. We also augment SSDi8, an existing low-precision prefill pipeline for Mamba-2, improving perplexity at matched prefill latency on pretrained Mamba-2 and Nemotron-H models. Altogether, these results demonstrate the practical utility of invariance-based quantization for recurrent models.

github.com/ntumm120/LinQuant

Introduction

Linear recurrences offer an efficient, sub-quadratic alternative to softmax attention for sequence modeling ([12, 8] (Yang et al. 2025; Team et al. 2025)). They summarize past tokens in a fixed-size state, avoiding the growing key-value cache of softmax attention, and are increasingly used in both standalone and hybrid language models ([18, 1, 20, 3]). For prefill, these models are typically evaluated using chunkwise-parallel algorithms, which process blocks of tokens in parallel through dense matrix multiplications while propagating a recurrent state between them ([8] (Yang & Zhang 2024)). This mirrors the blockwise organization of efficient attention kernels such as FlashAttention and makes recurrent prefill naturally amenable to low-precision execution.

Low-precision attention has been studied extensively, with optimized integer and floating-point kernels now forming a mature part of the Transformer inference stack ([23] (Zhang et al. 2025)). In contrast, low-precision kernels for linear recurrences remain largely unexplored. Prior work has studied quantization of Mamba-family models ([7] (Xu et al. 2025)), including the surrounding projections, activations and state readout within the recurrence. SSDi8 executes the chunkwise recurrent computation in low precision, demonstrating an efficient INT8 implementation specific to Mamba-2 ([13]). We instead study the space more generally, investigating low-precision computation across the design space of modern linear recurrent architectures.

Quantizing a sequence mixer can reduce its accuracy. One way to mitigate this loss is through functional invariances. Such transformations preserve the full-precision operator while changing the numerics presented to a quantizer (Figure 11, appendix). We distinguish two broad classes. Mixer-agnostic invariances act on the residual stream or individual linear layers and do not depend on the algebra of the sequence operator ([2, 16] (Xiao et al. 2023)). Because these transformations are largely independent of the mixer, they transfer naturally to recurrent architectures and have already been adopted by methods such as MambaQuant and Quamba2 ([7] (Xu et al. 2025)). Mixer-specific invariances instead rely on the sequence operator itself, and can be classified as either mixer-external or mixer-internal. The former materialize as changes of basis and can be applied outside of the sequence-mixer, while the latter are functions of a recurrence’s algorithmic realization (Figure 1). Transformer PTQ has exploited such invariances within softmax attention, for example through transformations of its query, key, and value representations ([2, 16]). In contrast, the mixer-specific invariances of linear recurrences have yet to be characterized.

Schematic of a typical recurrent model at different resolutions

Figure 1. Schematic of a typical recurrent model at different resolutions. The full-precision invariances can be classified as mixer-external changes of basis and mixer-internal allocations of decay and write factors. These invariances change the distribution of operands exposed to quantization; their applicability depends on the recurrence (§ 4). Style inspired by [16 Figure 1].

Figure 11. DeltaNet write error as the gate moves from values (a=0a = 0) to keys (a=1a = 1). Solid: both operands quantized; dashed: one operand. Lines and bands show head medians and interquartile ranges.

Linear recurrences present a qualitatively different invariance structure from softmax attention. Their transformations must remain consistent with a state that is updated and carried across chunks, while decay, gating, and delta-rule updates impose additional algebraic constraints. At the same time, these same mechanisms create new degrees of freedom that have no direct attention analogue. We therefore ask which attention-inspired invariances survive in recurrences, what new restrictions arise from the recurrent computation, and what additional invariances are unique to linear recurrences. Our main contributions are summarized as follows:

  • We introduce LinQuant, a framework for computing low-precision linear recurrences based on both mixer-agnostic and mixer-specific functional invariances, and characterize their applicability across common recurrent architectures.

  • Across a controlled set of recurrences, we show how recurrence structure, query-key normalization, scale granularity, and state-storage basis determine which transformations help, and examine how their benefits change when combined.

  • We implement INT8 chunkwise kernels for Mamba-2, Gated DeltaNet, and Kimi Delta Attention that exploit these invariances to recover most of the quantization loss while accelerating recurrent prefill in pretrained Nemotron-H, Qwen3.5, and Kimi Linear models.

  • We also augment SSDi8 with query-key transformations, improving perplexity at matched prefill latency on Mamba-2 and Nemotron-H models.

Nr.TransformationLAMamba-2GLADNGDNKDA
(1/7)RQK\mathrm{R}_{QK}✓✓(✓)✓✓(✓)
(2/7)DQK\mathrm{D}_{QK}✓✓✓×××
(3/7)RV\mathrm{R}_{V}✓✓✓✓✓✓
(4/7)DV\mathrm{D}_{V}✓✓✓✓✓✓
(5/7)(Γ⃗[i])η,(Γ⃗[i])1−η(\vec{\Gamma}_{[i]})^{\eta},(\vec{\Gamma}_{[i]})^{1-\eta}×✓(✓)×✓(✓)
(6/7)(E[i]η,E[i]1−η)(\mathbf{E}_{[i]}^{\eta},\mathbf{E}_{[i]}^{1-\eta})×✓✓×✓✓
(7/7)(B[i]ξ,B[i]1−ξ)(\mathbf{B}_{[i]}^{\xi},\mathbf{B}_{[i]}^{1-\xi})(✓)(✓)(✓)✓✓✓

Table 1. Applicability of the seven invariances introduced in § 4 to common linear recurrences. ×: not applicable. ✓: applicable. (✓): applicable under constraints.

Background and preliminaries

Linear Recurrences

Causal softmax attention (Vaswani et al., 2017) computes ot=V1:tTsoftmax⁡(K1:tqt/Dk)\mathbf{o}_t = \mathbf{V}_{1:t}^{\mathsf{T}}\operatorname{softmax}(\mathbf{K}_{1:t}\mathbf{q}_t/\sqrt{D_k}), from queries and keys qt,kt∈RDk\mathbf{q}_t,\mathbf{k}_t \in\mathbb{R}^{D_k} and values vt∈RDv\mathbf{v}_t \in\mathbb{R}^{D_v}, where DkD_k and DvD_v denote the state (key/query) and value dimensions, respectively. Linear attention replaces the nonlinear softmax operator with an unnormalized linear kernel [12]. In its simplest form, associativity allows past keys and values to be summarized in a fixed-size matrix-valued state St∈RDv×Dk\mathbf{S}_t \in\mathbb{R}^{D_v \times D_k}, St=St−1+vtktT,ot=Stqt\mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}, \quad\mathbf{o}_t = \mathbf{S}_t\mathbf{q}_t.

Modern linear recurrences extend this basic update with decay, write gating, and delta-rule corrections (Dao & Gu, 2024; Yang et al., 2024b;a; Team et al., 2025). We capture these axes through the common recurrence

St=St−1Λt(I−δβtktktT)+βtvtktT,(1)\mathbf{S}_t = \mathbf{S}_{t-1}\boldsymbol{\Lambda}_t\left(\mathbf{I}-\delta\boldsymbol{\beta}_t\mathbf{k}_t\mathbf{k}_t^{\mathsf{T}}\right)+\boldsymbol{\beta}_t\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}, \tag*{(1)}

where βt>0\beta_t > 0 (βt∈(0,2)\beta_t \in(0,2) for the delta rule), ∥Λt∥≤1\lVert\boldsymbol{\Lambda}_t\rVert\le1, and δ∈{0,1}\delta\in\{0,1\} is fixed for a given recurrence. (2) separates three structural choices.

Decay controls how the previous state is forgotten before new information is written. A scalar gate Λt=αtI\boldsymbol{\Lambda}_t = \alpha_t\mathbf{I} uses one decay rate per recurrent head [8] (Yang et al. 2025), whereas a diagonal gate Λt=diag⁡(αt)\boldsymbol{\Lambda}_t = \operatorname{diag}(\boldsymbol{\alpha}_t) allows the decay to vary across state channels (Yang et al., 2024b; Team et al., 2025). Write gating scales the new association vtktT\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}. Delta correction [22] (Yang et al. 2024c) removes the component of the existing state aligned with the current key before writing the new association. Together, these choices organize the space of modern recurrences as shown in Table 3 in Appendix § 10.§ 1. We later instantiate this taxonomy as a controlled mixer zoo, varying the recurrent update while holding the surrounding model architecture and training setup fixed.

modelupdate
Linear attention ([12])St=St−1+vtktTS_t = S_{t-1} + v_t k_t^{\mathsf{T}}
Mamba-2 ([8])St=αtSt−1+vtktTS_t = \alpha_t S_{t-1} + v_t k_t^{\mathsf{T}}
Gated Linear Attention (GLA) ((Yang et al., 2024b))St=St−1diag⁡(αt)+vtktTS_t = S_{t-1}\operatorname{diag}(\alpha_t) + v_t k_t^{\mathsf{T}}
DeltaNet ([22] (Yang et al. 2024c))St=St−1(I−βtktktT)+βtvtktTS_t = S_{t-1}\left(I - \beta_t k_t k_t^{\mathsf{T}}\right) + \beta_t v_t k_t^{\mathsf{T}}
Gated DeltaNet ((Yang et al., 2025))St=αtSt−1(I−βtktktT)+βtvtktTS_t = \alpha_t S_{t-1}\left(I - \beta_t k_t k_t^{\mathsf{T}}\right) + \beta_t v_t k_t^{\mathsf{T}}
Kimi Delta Attention (KDA) ((Team et al., 2025))St=St−1diag⁡(αt)(I−βtktktT)+βtvtktTS_t = S_{t-1}\operatorname{diag}(\alpha_t)\left(I - \beta_t k_t k_t^{\mathsf{T}}\right) + \beta_t v_t k_t^{\mathsf{T}}

Table 3. State-update rules for recurrent models in the mixer-zoo design space. Colors identify decay and write gating.

Chunkwise-parallel computation

Chunkwise implementations parallelize (2) within chunks of CC tokens [8] (Yang & Zhang 2024; Tumma et al. 2026), while processing chunks sequentially. Chunk [i][i] contains positions iC+1,…,(i+1)CiC+1,\ldots,(i+1)C, with incoming state S[i]:=SiC\mathbf{S}_{[i]} := \mathbf{S}_{iC}, queries/keys Q[i],K[i]∈RC×Dk\mathbf{Q}_{[i]},\mathbf{K}_{[i]} \in\mathbb{R}^{C \times D_k}, and values/outputs V[i],O[i]∈RC×Dv\mathbf{V}_{[i]},\mathbf{O}_{[i]} \in\mathbb{R}^{C \times D_v}. Indices r,sr,s denote positions within chunk [i][i], nn denotes a channel. For example, α[i]r:=αiC+r\alpha_{[i]r} := \alpha_{iC+r} and β[i]r:=βiC+r\beta_{[i]r} := \beta_{iC+r}. We collect the write gates in B[i]:=diag⁡(β[i]1,…,β[i]C)∈RC×C\mathbf{B}_{[i]} := \operatorname{diag}(\beta_{[i]}^1,\ldots,\beta_{[i]}^C) \in\mathbb{R}^{C \times C}. Throughout this work, we also refer to matrix multiplications as “contractions”. In the following, we lay out chunkwise parallel formulations for the recurrences covered by (2).

Scalar decay. For no delta correction (δ=0\delta=0) and scalar decay Λt=αtI\boldsymbol{\Lambda}_t=\alpha_t\mathbf{I}, define the cumulative decay γ[i]rs→r:=∏u=s+1rα[i]u\gamma_{[i]r}^{s\to r} := \prod_{u=s+1}^{r}\alpha_{[i]u} and γ[i]rr→r:=1\gamma_{[i]r}^{r\to r} := 1. The prefix and tail vectors Γ[i]→r∈RC\boldsymbol{\Gamma}_{[i]}^{\to r} \in\mathbb{R}^{C} have entries (Γ[i]→r)s:=γ[i]rs→r(\boldsymbol{\Gamma}_{[i]}^{\to r})_s := \gamma_{[i]r}^{s\to r} and (Γ[i]→r)r:=γ[i]rr→C(\boldsymbol{\Gamma}_{[i]}^{\to r})_r := \gamma_{[i]r}^{r\to C}. With the causal decay matrix (Γ[i])rs:=γ[i]rs→r(\boldsymbol{\Gamma}_{[i]})_{rs} := \gamma_{[i]r}^{s\to r} for $s \le r @page 3/46 starts_mid=0 ends_mid=1

Nr.TransformationLAMamba-2GLADNGDNKDA
(1/7)RQK\mathrm{R}_{QK}✓✓(✓)✓✓(✓)
(2/7)DQK\mathrm{D}_{QK}✓✓✓×××
(3/7)RV\mathrm{R}_{V}✓✓✓✓✓✓
(4/7)DV\mathrm{D}_{V}✓✓✓✓✓✓
(5/7)(Γ⃗[i])η,(Γ⃗[i])1−η(\vec{\Gamma}_{[i]})^{\eta},(\vec{\Gamma}_{[i]})^{1-\eta}×✓(✓)×✓(✓)
(6/7)(E[i]η,E[i]1−η)(\mathbf{E}_{[i]}^{\eta},\mathbf{E}_{[i]}^{1-\eta})×✓✓×✓✓
(7/7)(B[i]ξ,B[i]1−ξ)(\mathbf{B}_{[i]}^{\xi},\mathbf{B}_{[i]}^{1-\xi})(✓)(✓)(✓)✓✓✓

Table 1 (PDF p. 3). Applicability of the seven invariances introduced in § 4 to common linear recurrences. ×: not applicable. ✓: applicable. (✓): applicable under constraints.

Background and preliminaries

Linear Recurrences

Causal softmax attention (Vaswani et al., 2017) computes ot=V1:tTsoftmax⁡(K1:tqt/Dk)\mathbf{o}_t = \mathbf{V}_{1:t}^{\mathsf{T}}\operatorname{softmax}(\mathbf{K}_{1:t}\mathbf{q}_t/\sqrt{D_k}), from queries and keys qt,kt∈RDk\mathbf{q}_t,\mathbf{k}_t \in\mathbb{R}^{D_k} and values vt∈RDv\mathbf{v}_t \in\mathbb{R}^{D_v}, where DkD_k and DvD_v denote the state (key/query) and value dimensions, respectively. Linear attention replaces the nonlinear softmax operator with an unnormalized linear kernel [12]. In its simplest form, associativity allows past keys and values to be summarized in a fixed-size matrix-valued state St∈RDv×Dk\mathbf{S}_t \in\mathbb{R}^{D_v \times D_k}, St=St−1+vtktT,ot=Stqt\mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}, \quad\mathbf{o}_t = \mathbf{S}_t\mathbf{q}_t.

Modern linear recurrences extend this basic update with decay, write gating, and delta-rule corrections (Dao & Gu, 2024; Yang et al., 2024b;a; Team et al., 2025). We capture these axes through the common recurrence

St=St−1Λt(I−δβtktktT)+βtvtktT,(2)\mathbf{S}_t = \mathbf{S}_{t-1}\boldsymbol{\Lambda}_t\left(\mathbf{I}-\delta\boldsymbol{\beta}_t\mathbf{k}_t\mathbf{k}_t^{\mathsf{T}}\right)+\boldsymbol{\beta}_t\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}, \tag*{(2)}

where βt>0\beta_t > 0 (βt∈(0,2)\beta_t \in(0,2) for the delta rule), ∥Λt∥≤1\lVert\boldsymbol{\Lambda}_t\rVert\le1, and δ∈{0,1}\delta\in\{0,1\} is fixed for a given recurrence. (2) separates three structural choices.

Decay controls how the previous state is forgotten before new information is written. A scalar gate Λt=αtI\boldsymbol{\Lambda}_t = \alpha_t\mathbf{I} uses one decay rate per recurrent head [8] (Yang et al. 2025), whereas a diagonal gate Λt=diag⁡(αt)\boldsymbol{\Lambda}_t = \operatorname{diag}(\boldsymbol{\alpha}_t) allows the decay to vary across state channels (Yang et al., 2024b; Team et al., 2025). Write gating scales the new association vtktT\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}. Delta correction [22] (Yang et al. 2024c) removes the component of the existing state aligned with the current key before writing the new association. Together, these choices organize the space of modern recurrences as shown in Table 3 in Appendix § 10.§ 1. We later instantiate this taxonomy as a controlled mixer zoo, varying the recurrent update while holding the surrounding model architecture and training setup fixed.

Chunkwise-parallel computation

Chunkwise implementations parallelize (2) within chunks of CC tokens [8] (Yang & Zhang 2024; Tumma et al. 2026), while processing chunks sequentially. Chunk [i][i] contains positions iC+1,…,(i+1)CiC+1,\ldots,(i+1)C, with incoming state S[i]:=SiC\mathbf{S}_{[i]} := \mathbf{S}_{iC}, queries/keys Q[i],K[i]∈RC×Dk\mathbf{Q}_{[i]},\mathbf{K}_{[i]} \in\mathbb{R}^{C \times D_k}, and values/outputs V[i],O[i]∈RC×Dv\mathbf{V}_{[i]},\mathbf{O}_{[i]} \in\mathbb{R}^{C \times D_v}. Indices r,sr,s denote positions within chunk [i][i], nn denotes a channel. For example, α[i]r:=αiC+r\alpha_{[i]r} := \alpha_{iC+r} and β[i]r:=βiC+r\beta_{[i]r} := \beta_{iC+r}. We collect the write gates in B[i]:=diag⁡(β[i]1,…,β[i]C)∈RC×C\mathbf{B}_{[i]} := \operatorname{diag}(\beta_{[i]}^1,\ldots,\beta_{[i]}^C) \in\mathbb{R}^{C \times C}. Throughout this work, we also refer to matrix multiplications as “contractions”. In the following, we lay out chunkwise parallel formulations for the recurrences covered by (2).

Scalar decay. For no delta correction (δ=0\delta=0) and scalar decay Λt=αtI\boldsymbol{\Lambda}_t=\alpha_t\mathbf{I}, define the cumulative decay γ[i]rs→r:=∏u=s+1rα[i]u\gamma_{[i]r}^{s\to r} := \prod_{u=s+1}^{r}\alpha_{[i]u} and γ[i]rr→r:=1\gamma_{[i]r}^{r\to r} := 1. The prefix and tail vectors Γ[i]→r∈RC\boldsymbol{\Gamma}_{[i]}^{\to r} \in\mathbb{R}^{C} have entries (Γ[i]→r)s:=γ[i]rs→r(\boldsymbol{\Gamma}_{[i]}^{\to r})_s := \gamma_{[i]r}^{s\to r} and (Γ[i]→r)r:=γ[i]rr→C(\boldsymbol{\Gamma}_{[i]}^{\to r})_r := \gamma_{[i]r}^{r\to C}. With the causal decay matrix (Γ[i])rs:=γ[i]rs→r(\boldsymbol{\Gamma}_{[i]})_{rs} := \gamma_{[i]r}^{s\to r} for $s \le r and zero otherwise, the chunkwise parallel realization reads Dao & Gu [8]

S[i+1]=γ[i]CS[i]+V[i]TB[i](Γ[i]→⊙K[i]),S_{[i+1]} = \gamma^{C}_{[i]}S_{[i]} + V_{[i]}^{\mathsf{T}}B_{[i]}(\Gamma^{\rightarrow}_{[i]} \odot K_{[i]}),
O[i]=(Γ[i]←⊙Q[i])S[i]T+((Q[i]K[i]T)⊙Γ[i]→)B[i]V[i].O_{[i]} = (\Gamma^{\leftarrow}_{[i]} \odot Q_{[i]})S^{\mathsf{T}}_{[i]} + ((Q_{[i]}K^{\mathsf{T}}_{[i]}) \odot\Gamma^{\rightarrow}_{[i]})B_{[i]}V_{[i]}.

Diagonal decay. In the absence of delta correction (δ=0\delta= 0) but with diagonal decay Λt=diag⁡(αt)\Lambda_t = \operatorname{diag}(\alpha_t), define a channel-wise diagonal decay γ[i],n←:=∏u=s+1rαu,n[i]\gamma^{\leftarrow}_{[i],n} := \prod_{u=s+1}^{r}\alpha^{[i]}_{u,n} and γ[i],n→:=∏u=r+1eαu,n[i]\gamma^{\rightarrow}_{[i],n} := \prod_{u=r+1}^{e}\alpha^{[i]}_{u,n}. The stacked prefix-and-tail-decay now are Γ[i]←,Γ[i]→∈RC×Dk\Gamma^{\leftarrow}_{[i]}, \Gamma^{\rightarrow}_{[i]} \in\mathbb{R}^{C \times D_k}, with entries (Γ[i]←)rn=γ[i],n←(\Gamma^{\leftarrow}_{[i]})_{rn} = \gamma^{\leftarrow}_{[i],n} and (Γ[i]→)rn=γ[i],n→(\Gamma^{\rightarrow}_{[i]})_{rn} = \gamma^{\rightarrow}_{[i],n}. For αt>0\alpha_t > 0, the chunkwise parallel form reads Yang et al. (2024b); Team et al. (2025)

S[i+1]=S[i]diag⁡(γ[i]C)+(B[i]V[i])T(Γ[i]→⊙K[i]),S_{[i+1]} = S_{[i]}\operatorname{diag}(\gamma^{C}_{[i]}) + (B_{[i]}V_{[i]})^{\mathsf{T}}(\Gamma^{\rightarrow}_{[i]} \odot K_{[i]}),
O[i]=(Γ[i]←⊙Q[i])S[i]T+tril⁡((Γ[i]←⊙Q[i])(K[i]⊘Γ[i]←T))B[i]V[i].O_{[i]} = (\Gamma^{\leftarrow}_{[i]} \odot Q_{[i]})S^{\mathsf{T}}_{[i]} + \operatorname{tril}((\Gamma^{\leftarrow}_{[i]} \odot Q_{[i]})(K_{[i]} \oslash\Gamma^{\leftarrow\mathsf{T}}_{[i]}))B_{[i]}V_{[i]}.

Delta correction. For delta correction (δ=1\delta= 1) without decay, the chunkwise parallel form reads Yang et al. (2024c)

S[i+1]=S[i]+F[i]TK[i],S_{[i+1]} = S_{[i]} + F_{[i]}^{\mathsf{T}}K_{[i]},
O[i]=Q[i]S[i]T+tril⁡(Q[i]K[i]T)F[i],O_{[i]} = Q_{[i]}S^{\mathsf{T}}_{[i]} + \operatorname{tril}(Q_{[i]}K^{\mathsf{T}}_{[i]})F_{[i]},

where F[i]F_{[i]} are delta-corrected pseudo-values generated by

F[i]=(I+tril⁡(B[i]K[i]K[i]T,−1))−1B[i](V[i]−K[i]S[i]T).F_{[i]} = (I + \operatorname{tril}(B_{[i]}K_{[i]}K^{\mathsf{T}}_{[i]}, -1))^{-1}B_{[i]}(V_{[i]} - K_{[i]}S^{\mathsf{T}}_{[i]}).

Appendix D.1 gives the decaying delta forms Yang et al. (2025); Team et al. (2025).

LinQuant: Functional invariances for low-precision recurrences

The chunkwise parallel forms in Section 2.2 expose a set of matrix multiplications and a state carried between chunks. Importantly, these chunkwise forms are not unique: the same full-precision recurrence can be expressed through functionally equivalent realizations that expose different quantities to low precision. LinQuant exploits this freedom. We first define the low-precision realization used throughout the paper, then derive seven functional invariances: four axis transformations and three absorptions. Table 1 summarizes their applicability; proofs are in Appendix § 11.

Low-precision recurrence realization

Across the recurrences in Table 3, we find that the chunkwise computation can be organized into four matrix contractions: a state write, state read, local score, and intra-chunk output. Together with a recurrent state pass between chunks, this decomposition defines the low-precision boundaries we study. Consider for example the recurrence underlying Mamba-2. Using the notation of Section 2.2 and writing QQ for quantization, with scale factors implicit, we consider the following low-precision chunkwise realization:

H^[i]=Q(B[i]V[i])TQ(Γ[i]→⊙K[i]),(state write),\widehat{H}_{[i]} = Q(B_{[i]}V_{[i]})^{\mathsf{T}}Q(\Gamma^{\rightarrow}_{[i]} \odot K_{[i]}), \qquad\text{(state write)},
S^[i+1]=Q(γ[i]CS[i]+H^[i]),(state pass),\widehat{S}_{[i+1]} = Q(\gamma^{C}_{[i]}S_{[i]} + \widehat{H}_{[i]}), \qquad\text{(state pass)},
O^[i]inter=Γ[i]←⊙(Q(Q[i])S^[i]T),(state read),\widehat{O}^{\mathrm{inter}}_{[i]} = \Gamma^{\leftarrow}_{[i]} \odot(Q(Q_{[i]})\widehat{S}^{\mathsf{T}}_{[i]}), \qquad\text{(state read)},
A^[i]=(Q(Q[i])Q(K[i])T)⊙Γ[i]→,(local score),\widehat{A}_{[i]} = (Q(Q_{[i]})Q(K_{[i]})^{\mathsf{T}}) \odot\Gamma^{\rightarrow}_{[i]}, \qquad\text{(local score)},
O^[i]intra=A^[i]Q(B[i]V[i]),(intra-chunk output).\widehat{O}^{\mathrm{intra}}_{[i]} = \widehat{A}_{[i]}Q(B_{[i]}V_{[i]}), \qquad\text{(intra-chunk output)}.

For models that employ the delta-rule, the Gram and WY solve are executed in full precision. Delta-rule writes and intra-chunk outputs replace the gated values B[i]V[i]B_{[i]}V_{[i]} by the pseudo-values F[i]F_{[i]}, whose computation also includes a low-precision matrix multiplication between the keys and the incoming state (Appendix § 12.§ 1).

LinQuant exploits the fact that functional invariances of these operators generally do not commute with quantization QQ. These functional invariances fall into two classes: changes of basis, which change the representation along the key or value axes, and absorptions, which redistribute decay or write factors within an equivalent chunkwise computation. Appendix § 13.2 studies sensitivity to the stored-state and write-side quantization boundaries.

Mixer-external invariances: changes of basis

Both the key- and the value axis of the recurrent state St∈RDv×DkS_t \in\mathbb{R}^{D_v \times D_k} expose a choice of basis which leaves the full precision recurrence unchanged. We consider two such transformations along each axis: orthogonal rotations and positive diagonal rescalings. Notably, changes of basis along the key axis must remain compatible with the recurrent dynamics, whereas the value axis admits the same equivariance across all recurrences in Equation (1).

(1/7) Key-side rotation. A joint orthogonal rotation of queries and keys, (qt,kt)→(RQKqt,RQKkt)(q_t, k_t) \to(R_{QK}q_t, R_{QK}k_t), RQK∈O(Dk)R_{QK} \in O(D_k), preserves query/key inner products and is a standard invariance of full precision softmax attention exploited by rotation-based PTQ methods such as QuaRot and SpinQuant ([2, 16]). In low precision, random rotations spread energy across channels, reducing outlier concentration and improving quantization robustness.

For a linear recurrence, the invariance holds if RQKΛt=ΛtRQKR_{QK}\Lambda_t = \Lambda_tR_{QK} ([17]), and implicitly applies a change of basis to the state’s key axis, which transforms like St→StRQKTS_t \to S_tR_{QK}^{\mathsf{T}}. Invariance thus holds with no decay or scalar decay. With diagonal decay, a general rotation does not commute with Λt\Lambda_t, so the recurrent state cannot be carried in a globally rotated basis. However, we can still rotate queries and keys together after applying their decay factors when computing within-chunk scores. The recurrent state remains unrotated (Appendix § 12.§ 1).

In this work, we use random signed Hadamard rotations. Recurrent models may apply per-channel causal convolutions and nonlinearities between their input projections and the recurrence, which generally do not commute with a dense rotation. We thus apply RQKR_{QK} online after these operations.

(2/7) Key-side rescaling. The key axis also admits a reciprocal diagonal transformation, (qt,kt)→(DQKqt,DQK−1kt)(q_t, k_t) \to(D_{QK}q_t, D_{QK}^{-1}k_t), DQK≻0D_{QK} \succ0 diagonal, which likewise preserves query-key inner products. Related rescalings underlie SmoothAttention in QServe and, more broadly, methods such as SmoothQuant ([15] (Xiao et al. 2023)).

For a linear recurrence, the state implicitly transforms like St→StDQK−1S_t \to S_tD_{QK}^{-1}. Unlike a dense rotation, DQKD_{QK} commutes with diagonal decay, making the transformation valid for linear writes under no, scalar, or diagonal decay. It does not, however, preserve delta corrected writes.

DQKD_{QK} transports magnitude between queries and keys. In this work, we parameterize DQK=diag⁡(d)D_{QK} = \operatorname{diag}(d) by dn=(max⁡t∣kt,n∣/max⁡t∣qt,n∣)κd_n = (\max_t |k_{t,n}|/\max_t |q_{t,n}|)^\kappa, where the maxima are computed over calibration tokens within each head. The exponent κ∈[0,1]\kappa\in[0,1] controls the rescaling strength. Specifically, κ=12\kappa= \frac{1}{2} balances the magnitudes of queries and keys. We leave other parametrizations to future work.

(3/7) Value-side rotation. The value axis admits an analogous orthogonal transformation, vt→RVvtv_t \to R_Vv_t, RV∈O(Dv)R_V \in O(D_v). In softmax attention, value rotations are used for KV-cache quantization ([16]). The operator, however, is not invariant to value rotations, which have to be undone after the softmax. Linear recurrences exhibit the same equivariance. Intuitively, this is because the recurrence acts independently along the value axis, which is implicitly rotated like St→RVStS_t \to R_VS_t.

We note that, compared to softmax attention, the recurrent setting reveals another degree of freedom. The state can, in between chunks, either be stored in its canonical basis, Q(St)Q(S_t), or in a rotated basis, Q(RVSt)Q(R_VS_t). These realizations are equivalent in full precision but expose different representations to QQ. Rotated storage is what the rotation produces by itself, whereas canonical storage returns each chunk’s write to the original basis before it joins the carried state. This comes at the cost of an inverse transform per chunk, and keeps the state in the basis whose per-channel scales fit it.

(4/7) Value-side rescaling. Finally, the value axis admits the positive diagonal analogue vt→DVvtv_t \to D_Vv_t, DV≻0D_V \succ0 diagonal. The recurrent state transforms accordingly as St→DVStS_t \to D_VS_t, with the inverse rescaling applied before any normalization. If an output projection directly follows the recurrence, the inverse can instead be absorbed into it. As with RVR_V, this transformation is valid across all recurrence families considered here.

In this work, we parametrize DV=diag⁡(d)D_V = \operatorname{diag}(d) by dp=((∏j=1Dvmax⁡t∣vt,j∣)1/Dv/max⁡t∣vt,p∣)κd_p = \left((\prod_{j=1}^{D_v} \max_t |v_{t,j}|)^{1/D_v}/\max_t |v_{t,p}|\right)^\kappa, where the maxima are computed over calibration tokens and the product runs over value channels within each head. This parametrization moves range between value channels. At κ=1\kappa= 1, the ranges of the value channels are equalized.

Mixer-internal invariances: absorptions

The first four invariances discussed in the previous section change the basis along the two axes of the recurrent state. The remaining three redistribute multiplicative factors within a chunkwise parallel implementation of the recurrence. In particular, decay and write factors can often be divided between the two sides of a matrix product without changing the full-precision operator.

(5/7) Scalar-decay absorption. For scalar decay, the tail-decay Γ[i]\mathbf{\Gamma}_{[i]} can be split between the value and key sides of the state write:

H[i]=((Γ[i])η⊙(B[i]V[i]))T((Γ[i])1−η⊙K[i]),η∈[0,1].\mathbf{H}_{[i]} = \left((\mathbf{\Gamma}_{[i]})^\eta\odot(\mathbf{B}_{[i]}\mathbf{V}_{[i]})\right)^{\mathsf{T}}\left((\mathbf{\Gamma}_{[i]})^{1-\eta} \odot\mathbf{K}_{[i]}\right), \qquad\eta\in[0,1].

Thus, η\eta continuously interpolates between equivalent realizations of the same write. At η=0\eta= 0, all tail decay remains on the keys, which we call key-side decay absorption. At η=1\eta= 1, it moves entirely to the values, giving value-side decay absorption, the endpoint used by SSDi8 [13]. Intermediate values split the decay between the two sides. The same split applies to delta-rule writes, replacing B[i]V[i]\mathbf{B}_{[i]}\mathbf{V}_{[i]} by the corresponding pseudo-values F[i]\mathbf{F}_{[i]}.

(6/7) Diagonal-decay absorption. Diagonal decay varies across state channels and therefore cannot, in general, be transferred entirely to the value side. Nevertheless, we can factor each token’s decay into a scalar magnitude and a relative channelwise pattern. This decomposition is not unique: any positive scalar factor yields an exact redistribution. Here, we use the channelwise maximum. Appendix § 13.§ 1 examines alternative choices.

For diagonal tail decay Γ[i]\mathbf{\Gamma}_{[i]}, define the per-token scalar magnitude and normalized decay

e[i]r:=max⁡n(Γ[i])rn,E[i]:=diag⁡(e[i]1,…,e[i]C),Γˉ[i]tail:=E[i]−1Γ[i].e^{r}_{[i]} := \max_n \left(\mathbf{\Gamma}_{[i]}\right)_{rn}, \qquad\mathbf{E}_{[i]} := \operatorname{diag}(e^{1}_{[i]},\ldots,e^{C}_{[i]}), \qquad\mathbf{\bar{\Gamma}}^{\mathrm{tail}}_{[i]} := \mathbf{E}_{[i]}^{-1}\mathbf{\Gamma}_{[i]}.

Then the state write can be expressed as

H[i]=(E[i]ηB[i]V[i])T(E[i]1−ηΓˉ[i]tail⊙K[i]),η∈[0,1].\mathbf{H}_{[i]} = \left(\mathbf{E}^{\eta}_{[i]}\mathbf{B}_{[i]}\mathbf{V}_{[i]}\right)^{\mathsf{T}}\left(\mathbf{E}^{1-\eta}_{[i]}\mathbf{\bar{\Gamma}}^{\mathrm{tail}}_{[i]} \odot\mathbf{K}_{[i]}\right), \qquad\eta\in[0,1].

Here, η\eta moves the magnitude of the diagonal decay. At η=0\eta= 0, the standard key-side form is recovered. At η=1\eta= 1, each token’s scalar decay magnitude moves to the value side, while the relative channel-dependent decay necessarily remains on the key.

(7/7) Write-factor absorption. For a linear write, the write factor admits an analogous split:

H[i]=(B[i]ξV[i])T(Γ[i]⊙B[i]1−ξK[i]),ξ∈[0,1].\mathbf{H}_{[i]} = \left(\mathbf{B}^{\xi}_{[i]}\mathbf{V}_{[i]}\right)^{\mathsf{T}}\left(\mathbf{\Gamma}_{[i]} \odot\mathbf{B}^{1-\xi}_{[i]}\mathbf{K}_{[i]}\right), \qquad\xi\in[0,1].

The endpoints ξ=1\xi= 1 and ξ=0\xi= 0 place the write factor entirely on the value and key sides, respectively.

For delta-corrected writes, the write gate also enters the triangular system used to construct the pseudo-values. Without decay, split it between the two key factors of the Gram matrix:

L[i](σ):=tril⁡((B[i]σK[i])(B[i]1−σK[i])T,−1),σ∈[0,1].\mathbf{L}^{(\sigma)}_{[i]} := \operatorname{tril}\left((\mathbf{B}^{\sigma}_{[i]}\mathbf{K}_{[i]})(\mathbf{B}^{1-\sigma}_{[i]}\mathbf{K}_{[i]})^{\mathsf{T}}, -1\right), \qquad\sigma\in[0,1].

Using B[i]σ\mathbf{B}^{\sigma}_{[i]} on the corresponding right-hand side yields transformed pseudo-values F[i](σ)\mathbf{F}^{(\sigma)}_{[i]} satisfying

F[i](σ)=B[i]σ−1F[i],\mathbf{F}^{(\sigma)}_{[i]} = \mathbf{B}^{\sigma-1}_{[i]}\mathbf{F}_{[i]},

where F[i]\mathbf{F}_{[i]} denotes the pseudo-values produced by the standard formulation in Section 2.2. A second parameter τ\tau splits the complementary factor B[i]1−σ\mathbf{B}^{1-\sigma}_{[i]} between the transformed write operands:

V[i](σ,τ)=B[i](1−σ)(1−τ)F[i](σ),K[i](σ,τ)=B[i](1−σ)τK[i],\mathbf{V}^{(\sigma,\tau)}_{[i]} = \mathbf{B}^{(1-\sigma)(1-\tau)}_{[i]}\mathbf{F}^{(\sigma)}_{[i]}, \qquad\mathbf{K}^{(\sigma,\tau)}_{[i]} = \mathbf{B}^{(1-\sigma)\tau}_{[i]}\mathbf{K}_{[i]},
H[i]=(V[i](σ,τ))TK[i](σ,τ)=F[i]TK[i],τ∈[0,1].\mathbf{H}_{[i]} = \left(\mathbf{V}^{(\sigma,\tau)}_{[i]}\right)^{\mathsf{T}}\mathbf{K}^{(\sigma,\tau)}_{[i]} = \mathbf{F}^{\mathsf{T}}_{[i]}\mathbf{K}_{[i]}, \qquad\tau\in[0,1].

Thus, σ\sigma determines how the write factor is distributed inside the Gram construction, while τ\tau determines how the remaining compensation is distributed in the subsequent write. At τ=0\tau= 0, the compensation remains on the pseudo-values. At τ=1\tau= 1, it moves to the keys. Setting σ=1\sigma= 1 recovers the standard triangular formulation, while σ=1/2\sigma= 1/2 places square-root write factors on the two Gram operands. Appendix § 11 proves these equivalences and extends the construction to decayed delta-rule recurrences.

Together, the four changes of basis and three absorptions define seven functional degrees of freedom for realizing the same recurrent computation. Their applicability depends on the recurrence structure, while their numerical behavior depends on the resulting low-precision representation, which we explore in the following section.

Experiments

We first use a controlled model suite to examine how recurrence structure and quantization choices affect the utility of each invariance (§ 5.1). We then study the practical utility of these invariances in production-scale models (Nemotron-H, Qwen3.5, and Kimi Linear) by implementing integer prefill kernels that employ these invariances for Mamba-2, Gated DeltaNet, and Kimi Delta Attention (§ 5.2).

Analyzing LinQuant

Model suite. To isolate the impact of each component that comprises the linear recurrence, we set up a controlled mixer zoo. All models share the same backbone and training recipe. It consists of Linear Attention, Mamba-2, GLA, DeltaNet, Gated DeltaNet, and KDA at 340M and 1.3B parameters, trained with 15B and 100B tokens of SlimPajama, respectively. The 340M suite furthermore contains models that vary write gating βt\beta_t and QK normalization. Appendix § 10.§ 1 gives the full setup.

Quantization settings. We evaluate how the identified invariances affect quantization error using WikiText-2 test NLL, comparing against full precision and a naive quantizer without invariance transforms. We use Hadamard rotations and diagonal rescalings discussed in § 4.2, calibrated using WikiText-2. Several normalized decaying and delta-rule models already incur little quantization error at INT8 with per-channel value scales (Table 10). We therefore use INT6 for the main controlled comparisons: it interpolates between INT8 and INT4, the two precisions for which we implement low precision kernels, and makes the impact of the invariances easier to distinguish. The production experiments evaluate INT8 and INT4 kernels, which use native integer Tensor Core instructions to accelerate recurrent prefill. Appendix § 10.§ 1 varies the calibration corpus and seeds. The end-of-chunk states are passed in FP32, while quantizing a copy to an INT8 readout. Appendix § 10.§ 1 gives the quantization and calibration setup, and Appendix § 13 reports INT8, INT4, 1.3B, and zero-shot results.

RecurrenceFPINT8, ⟨T,C⟩\langle T, C\rangle NaiveINT8, ⟨T,C⟩\langle T, C\rangle StackINT4, ⟨T,C⟩\langle T, C\rangle NaiveINT4, ⟨T,C⟩\langle T, C\rangle StackINT4, ⟨T,T⟩\langle T, T\rangle NaiveINT4, ⟨T,T⟩\langle T, T\rangle Stack
340M
LA17.11719.48217.26573.22624.736498.76832.750
LA+β\beta16.68817.54116.79883.12022.495726.57073.118
Mamba-215.55315.61215.56125.81216.42785.44117.141
Mamba-2+β\beta15.42015.443–21.05916.07055.01016.748
GLA15.10415.124–32.43120.58788.88421.214
GLA+β\beta15.30815.322–24.89618.76670.12219.410
DeltaNet15.67815.698–24.52317.685168.07720.072
GDN15.00415.019–20.54016.09882.81417.421
KDA14.48914.511–28.732–111.44930.205
LA{sup}†`17.23017.79717.388314.03937.950436.48866.355
LA+β\beta{sup}†`16.98018.74417.902515.21769.456663.57694.058
Mamba-2{sup}†`16.72116.91116.735316.76223.555362.19925.470
Mamba-2+β\beta{sup}†`16.18916.26616.194202.73218.344285.46922.110
GLA{sup}†`16.07016.28216.099173.81046.248263.19654.814
GLA+β\beta{sup}†`15.89116.00915.935172.78142.062244.26445.162
1.3B
LA+β\beta10.27610.76310.34960.51813.9874185.73144.904
Mamba-2+β\beta9.7559.774–11.76710.185295.66110.971
GLA+β\beta9.1899.205–12.50910.786160.47011.423
DeltaNet9.9179.934–13.75311.322957.27515.235
GDN9.7279.731–11.91610.329424.98616.016
KDA9.4609.467–20.15019.877352.92925.376

Table 10. Precision sensitivity on WikiText-2. INT8 uses per-channel value scales. INT4 includes both value-scale policies. Keys use per-tensor scales throughout. The stack is held-out-selected. Dashes denote no reported promoted stack. Quantized settings retain the FP32 carry and INT8 read copy. †: no QK normalization.

Scale geometries. The queries/read-keys and values can be quantized with different granularities, denoted by ⟨bQK,bV⟩\langle b_{\mathrm{QK}}, b_{\mathrm{V}}\rangle. Here, bQKb_{\mathrm{QK}} gives the scale-granularity for queries and read-keys, while bVb_{\mathrm{V}} gives that of the values. Both can be C (per-channel), H (per head) or T (per-tensor). The scale groupings considered are directly influenced by the chunkwise kernels. The scale for a value channel is shared by all terms contributing to that output channel, so it can be applied once after the sum, in the kernel’s existing output-scaling step. In query–key products and state reads, per-channel key scales instead vary across the terms being summed and therefore cannot be factored out of the contraction. We compare the three scale regimes ⟨H,C⟩\langle\mathrm{H}, \mathrm{C}\rangle, ⟨T,C⟩\langle\mathrm{T}, \mathrm{C}\rangle, and ⟨T,T⟩\langle\mathrm{T}, \mathrm{T}\rangle. The first two use the per-channel value scales supported by our kernels, while ⟨T,T⟩\langle\mathrm{T}, \mathrm{T}\rangle isolates the effect of sharing value scales. We therefore discuss the per-channel results first, then examine how sharing value scales changes the benefit of each invariance. In the state write contraction, the key dimension stays an outer dimension. We exploit this by quantizing write keys per channel in ⟨H,C⟩\langle\mathrm{H}, \mathrm{C}\rangle, where each contraction quantizes its own copy of the keys, so the write can take the finest scale it can absorb. In the other two regimes, write keys are quantized per tensor and reused by each contraction.

We examine key-axis transformations (1-2), value-axis transformations (3-4), and absorptions (5-7) individually and in selected combinations, studying how their benefits depend on scale granularity and storage basis. Appendices E.2 and G.2 report additional combination results.

(1-2) Key-axis transformations. With per-channel value scales, the benefit of RQKR_{QK} depends on the recurrence structure. For recurrences with no decay or scalar decay, it rotates the carried state along the key axis and reduces quantization error. With diagonal decay, it is restricted to local contractions and provides little benefit (Figure 2a). In contrast, DQKD_{QK} commutes with diagonal decay and reduces quantization error where applicable. QK normalization also reduces quantization error (Figure 2b). Coarsening key scales increases the benefit of RQKR_{QK}, while DQKD_{QK} remains useful with per-head scales (Figure 2c). The diagonal rescaling is generally best shared between queries and keys (Figure 2d).

Figure 2. Key-axis transformations at INT6. (a) Recovered excess NLL by decay and write rule. (b) Excess NLL with and without QK normalization (†). Panels (a,b) use ⟨T,C⟩\langle\mathrm{T}, \mathrm{C}\rangle and κ=1/2\kappa= 1/2 for DQK\mathrm{D}_{\mathrm{QK}}. (c) Recovery under ⟨H,C⟩\langle\mathrm{H}, \mathrm{C}\rangle and its change under ⟨T,C⟩\langle\mathrm{T}, \mathrm{C}\rangle, with selected κ\kappa. (d) κ\kappa sweep, one curve per model.

(3-4) Value-axis transformations. Both RVR_V and DVD_V apply to every recurrence. With per-channel value scales, each alone provides little benefit in the INT6 controlled study (Figure 3a). Combining them can nevertheless recover much of the quantization loss. When a few value channels dominate, rotation alone can increase the relative quantization error of smaller channels. Rescaling first balances their ranges before rotation mixes them, reducing this error (Figure 3b). The combination consistently improves on naive quantization and rotation alone in the 1.3B controlled models at INT4 (Appendix G.6), and yields substantial gains at INT8 in production-scale pretrained models (§ 5.2). These gains are achieved with rotated state storage, avoiding inverse transforms inside the kernel. When values instead share a scale (⟨T,T⟩\langle T,T\rangle), the individual transformations also become effective, each recovering a substantial portion of the excess NLL (Figure 3a). Their gains are similar, but only RVR_V introduces a choice of storage basis. For rotation alone, the benefit of canonical storage is largest in Linear Attention (Figure 3c), but it requires inverse transforms inside the kernel. When the two transformations give similar quality, this favors DVD_V, which folds into the quantization scales.

Value-axis transformations

Figure 3. Value-axis transformations. (a) Recovered excess NLL at INT6 under ⟨T,C⟩\langle T,C\rangle and its change under ⟨T,T⟩\langle T,T\rangle (RVR_V with canonical storage). (b) Relative quantization MSE of each value channel cc of the state write, E[(Q(vc)−vc)2]/E[vc2]\mathbb{E}[(Q(v_c)-v_c)^2]/\mathbb{E}[v_c^2], at INT8 with per-channel scales in GDN. Channels ranked by maximum absolute calibration activation, largest first; lines and bands show the median and interquartile range over heads and layers. (c) Recovery lost at INT6 by storing the state rotated instead of canonically under RVR_V, relative to naive.

(5-7) Absorptions. With per-channel value scales, write-gate placement depends on the recurrence and the key scales. For gated linear writes, keeping the gate on values (ξ=1\xi= 1) is preferred under ⟨T,C⟩\langle\mathrm{T}, \mathrm{C} \rangle (Figure 4a), while ⟨H,C⟩\langle\mathrm{H}, \mathrm{C} \rangle can favor splitting it between keys and values (Figure 9). For DeltaNet and Gated DeltaNet, write-gate absorption recovers a substantial portion of the excess NLL even under ⟨T,C⟩\langle\mathrm{T}, \mathrm{C} \rangle (Figure 4a). Scalar-decay placement has relatively little effect with per-channel value scales (Figure 4b and Figure 8). With shared value scales (⟨T,T⟩\langle\mathrm{T}, \mathrm{T} \rangle), write-gate absorption helps both linear and delta writes (Figure 4a). For delta writes, the selected settings use σ∈{0,12}\sigma\in\left\{0, \frac{1}{2}\right\} and τ=1\tau= 1 (Figure 10). Compared with the standard WY placement (σ,τ)=(1,1)(\sigma, \tau) = (1, 1) used in Flash Linear Attention (Yang & Zhang, 2024), this moves more of the gate factor onto the transformed keys. The standard floating-point placement is therefore not always best after quantization. Scalar-decay placement also matters more with shared value scales: placing decay on keys increases excess NLL relative to placing it on values (Figure 4b). For diagonal decay, however, absorption provides little gain across the scale regimes (Figure 4b and Figure 8).

Absorptions at INT6

Figure 4. Absorptions at INT6 under ⟨T,C⟩\langle T,C\rangle and their change under ⟨T,T⟩\langle T,T\rangle. (a) Recovered excess NLL from the selected non-default write-factor allocation (ξ\xi or (σ,τ)(\sigma,\tau)). Negative: the default is better. (b) NLL increase from placing decay on keys (η=0\eta=0) rather than values (η=1\eta=1), relative to η=1\eta=1.

Figure 8. Response to the decay allocation η\eta (decay on the key at 0, on the value at 1) for the decaying cells, INT6 operands and INT8 stored state: percent of the naive excess NLL recovered, one curve per scale regime. Scalar-decay placement matters most under ⟨T,T⟩\langle\mathrm{T},\mathrm{T}\rangle. Diagonal decay and per-channel value scales are less sensitive.

Six gated linear models showing recovered percentage versus write-factor allocation

Figure 9. Write-factor allocation on the six gated linear models, with INT6 operands and INT8 stored state: B[i]ξB^{\xi}_{[i]} multiplies values and B[i]1−ξB^{1-\xi}_{[i]} multiplies keys. Percent of the naive excess NLL recovered. Each curve shows one scale regime. † marks models without QK normalization.

Delta write-factor allocation for DN, GDN, and KDA

Figure 10. Delta write-factor allocation with INT6 operands and INT8 stored state. Curves vary the Gram exponent σ\sigma at fixed τ\tau, which allocates the remaining factor between the transformed write operands: solid τ=1\tau= 1, dashed τ=1/2\tau= 1/2, dotted τ=0\tau= 0. Percent of the naive excess NLL recovered. Each colour denotes one scale regime.

Combined transformations. Finally, combining transformations generally reduces quantization error beyond the best individual transformation (Table 7 and Table 8). The gains are not additive, since some transformations address overlapping sources of error (Appendix § 13.2).

RecurrenceFP⟨T,C⟩\langle\mathrm{T}, \mathrm{C}\rangle Naive⟨T,C⟩\langle\mathrm{T}, \mathrm{C}\rangle Best single⟨T,C⟩\langle\mathrm{T}, \mathrm{C}\rangle Stack⟨T,T⟩\langle\mathrm{T}, \mathrm{T}\rangle Naive⟨T,T⟩\langle\mathrm{T}, \mathrm{T}\rangle Best single⟨T,T⟩\langle\mathrm{T}, \mathrm{T}\rangle Stack
LA17.11725.70619.42818.09730.69423.15118.138
LA+β\beta16.68819.63117.49917.08928.38119.77317.388
Mamba-215.55316.02915.72215.60516.44616.03615.633
Mamba-2+β\beta15.42015.59515.48515.44916.23015.60915.473
GLA15.10415.65115.30215.30216.04815.64615.309
GLA+β\beta15.30815.54015.41815.41516.04615.54715.440
DeltaNet15.67815.92515.77815.76920.32316.13815.871
GDN15.00415.16815.05515.04217.06415.23615.103
KDA14.48914.84614.843–15.83714.88514.906
LA{sup}†`17.23024.91618.07918.07131.58222.48618.336
LA+β\beta{sup}†`16.98030.71921.04220.738100.24735.44120.429
Mamba-2{sup}†`16.72120.84217.08616.84821.47917.50616.924
Mamba-2+β\beta{sup}†`16.18917.66516.30616.26418.84417.21316.326
GLA{sup}†`16.07020.28216.74016.73920.97717.16816.907
GLA+β\beta{sup}†`15.89119.22216.83416.83820.52317.87316.647

Table 7. INT6 WikiText-2 perplexity for the 340M zoo. Keys use per-tensor scales. Value scales are per channel ⟨T,C⟩\langle\mathrm{T}, \mathrm{C}\rangle or per tensor ⟨T,T⟩\langle\mathrm{T}, \mathrm{T}\rangle. Best single is the descriptive comparison defined in the text. The stack is selected on held-out data and uses canonical storage. A dash denotes no reported promoted stack. †: no QK normalization.

RecurrenceFP⟨\langleT, C⟩\rangle Naive⟨\langleT, C⟩\rangle Best single⟨\langleT, C⟩\rangle Stack⟨\langleT, T⟩\rangle Naive⟨\langleT, T⟩\rangle Best single⟨\langleT, T⟩\rangle Stack
LA+β\beta10.27612.10110.76810.625148.17312.58810.842
Mamba-2+β\beta9.7559.8399.8059.78912.6879.8959.802
GLA+β\beta9.1899.3079.2679.26010.1159.3169.263
DeltaNet9.91710.0509.9799.97346.63910.40610.111
GDN9.7279.7849.7519.75211.4019.8679.788
KDA9.4609.6039.593–10.7419.6429.653

Table 8. INT6 WikiText-2 perplexity for the 1.3B zoo. Keys use per-tensor scales. Value scales are per channel ⟨\langleT, C⟩\rangle or per tensor ⟨\langleT, T⟩\rangle. Best single is the descriptive comparison defined in the text. The stack is selected on held-out data and uses canonical storage. A dash denotes no reported promoted stack.

LinQuant in a production setting

In this section, we implement INT8 and INT4 chunkwise kernels that exploit these invariances and evaluate them in the production-scale hybrids Nemotron-H (Blakeman et al., [4]), Qwen3.5 (Qwen Team, [21]), and Kimi Linear (Team et al., 2025). These models use Mamba-2, Gated DeltaNet, and Kimi Delta Attention recurrences, respectively. Our INT8 Triton kernels support all six recurrence families in our mixer zoo, with INT8 state reads and an FP32 carry (Appendix § 12.2). For these hybrids, we also provide packed INT4 kernels (Appendix § 12.§ 4). Here, we quantize only the recurrence, using per-channel value scales, and keep the rest of each model unchanged. For each model, we compare individual transformations and selected combinations (Table 24) and choose the configuration with the lowest held-out cross-entropy among those with no additional kernel overhead (Table 2a, Appendix § 15). We also incorporate the key-axis invariances into SSDi8 (Kim et al., [13]), an existing low-precision prefill pipeline for Mamba-2 (Table 2b).

ModelRecurrenceFPW4A16BaselineLinQuantRec.Inv.
aa Recurrence only, LinQuant kernels. Baseline: naive INT8
Nemotron-H BaseMamba-26.035–6.1626.054851–4
Nemotron-H ReasoningMamba-28.429–8.7238.471861, 5
Qwen3.5-4BGated DeltaNet8.056–8.1148.066823, 4
Qwen3.5-9BGated DeltaNet6.983–7.0296.990843, 4
Kimi Linear 48BKDA5.519–5.5515.530653, 4
bb W4A8 pipeline. Baseline: SSDi8
Mamba-2 1.3BMamba-210.42010.71710.92810.849371, 2
Mamba-2 2.7BMamba-29.0609.2869.4419.383371, 2
Mamba-2 8BMamba-27.2507.3807.6207.435771, 2
Nemotron-H BaseMamba-26.0306.0896.3386.178641, 2
Nemotron-H ReasoningMamba-28.4238.5118.9218.650661, 2

Table 2.

TransformationNH BaseNH Reas.Qwen 4BQwen 9BKimi 48B
FP perplexity6.0358.4298.0566.9835.519
Naive INT8 perplexity6.1628.7238.1147.0295.551
(1) RQK\mathrm{R}_{QK}3679-18-26–
(2) DQK\mathrm{D}_{QK}-22-15–––
(3) RV\mathrm{R}_{V}, rotated storagediv.div.57-164
(3) RV\mathrm{R}_{V}, canonical storage-34-54634346
(4) DV\mathrm{D}_{V}00-11-4
(5) decay on keys (η=0\eta=0)-67–––
(7) write gate on keys (σ=0\sigma=0)––2324–
RVDV\mathrm{R}_{V}\mathrm{D}_{V}3352828358
RVDV\mathrm{R}_{V}\mathrm{D}_{V}, κ=12\kappa=\frac{1}{2}2452848465
RVDV\mathrm{R}_{V}\mathrm{D}_{V}, canonical storage178––40
RVDV+σ=0\mathrm{R}_{V}\mathrm{D}_{V}+\sigma=0––7980–
RQK+DQK\mathrm{R}_{QK}+\mathrm{D}_{QK}3442–––
RQK+η=0\mathrm{R}_{QK}+\eta=03186–––
RQK+RVDV\mathrm{R}_{QK}+\mathrm{R}_{V}\mathrm{D}_{V}7999–––
RQK+RVDV+η=0\mathrm{R}_{QK}+\mathrm{R}_{V}\mathrm{D}_{V}+\eta=077102–––
RQK+DQK+RVDV\mathrm{R}_{QK}+\mathrm{D}_{QK}+\mathrm{R}_{V}\mathrm{D}_{V}8595–––
RQK+DQK+RVDV+η=0\mathrm{R}_{QK}+\mathrm{D}_{QK}+\mathrm{R}_{V}\mathrm{D}_{V}+\eta=08398–––

Table 24. Share of the naive INT8 gap to full precision (in %) that each transformation recovers on WikiText-2, with only the recurrence quantized (⟨H,C⟩\langle H, C\rangle static scales, LinQuant kernels). Means over the five calibration draws of each entry (four for RQK\mathrm{R}_{QK} on Qwen3.5-4B, where the job of the fifth draw has no paired naive run); the first two rows are means over the draws of a model. Bold: the selected stack (lowest held-out cross-entropy without inverse state rotations inside the recurrence). NH: Nemotron-H; div.: diverged; –: not applicable or not run. RVDV\mathrm{R}_{V}\mathrm{D}_{V} rescales the values before rotating them and uses rotated storage and κ=1\kappa=1 unless stated; DQK\mathrm{D}_{QK} uses κ=12\kappa=\frac{1}{2}.

Table 2: WikiText-2 perplexity. (a) Recurrence-only INT8. (b) SSDi8’s W4A8 pipeline, with value-side decay in both variants. Rec.: percentage of the baseline gap to FP (a) or W4A16 (b) recovered by LinQuant. W4A16: 4-bit weights, FP16 activations. Inv.: invariance numbers from § 4.

Quantization quality. The selected combinations recover most of the naive INT8 perplexity gap on every evaluated hybrid, consistently across calibration draws. As in the controlled study, the useful transformations depend on the recurrence: key-axis transformations combine with value-axis transformations on Nemotron-H Base and with key-side scalar decay on Nemotron-H Reasoning, whereas Qwen3.5 and Kimi Linear benefit most from the value-axis combinations discussed in § 5.1. Moving the write gate onto the keys also helps Qwen3.5 on its own. Overall, six of the seven invariances improve at least one pretrained model, either individually or when combined with others. Diagonal-decay absorption has little effect (Appendix § 13.§ 1).

Speedup. In the kernels, rescalings and absorptions fold into existing scales and operands, while the selected value rotations fuse into neighboring convolution and normalization passes. We fuse input quantization into the preceding convolution, activation, and normalization kernels, avoiding a separate quantization pass. We time each mixer from its projection outputs through the recurrence against FLA’s BF16 kernels on the same segment. At the mixer shapes of the three hybrids, our INT8 paths are up to 2.0× faster for Mamba-2, 1.59× for Gated DeltaNet, and 1.34× for Kimi Delta Attention (Figure 5a and Figure 5b). The GDN and KDA speedups are largest at small batches, where fused input preparation saves more launch overhead; this advantage shrinks as computation dominates (Appendix § 15.§ 6).

Kernel and prefill performance plots

Figure 5. Kernel and prefill performance on an RTX A5000 at T=2048T = 2048. (a, b) Speedup of our INT8 and packed INT4 mixer paths over the BF16 kernels of FLA at pretrained model shapes, timed from the projection outputs through the recurrence with all quantization included (Appendix § 15.§ 6). (c) Canonical RVR_{V} storage latency relative to naive INT8 (B=H=8B = H = 8, Dk=Dv=128D_k = D_v = 128). (d) Mamba-2 prefill latency with RQKR_{QK} and value-side decay, relative to matched SSDi8.

Integer operands accelerate the matrix products of a recurrence. The Mamba-2 recurrence consists of such products alone, whereas the delta rule additionally prepares every chunk with a Gram matrix and a triangular solve, which we keep in floating point. This preparation takes over a third of FLA’s recurrence time for Gated DeltaNet and about half for Kimi Delta Attention, which bounds the gain. Canonical storage and local key rotations under diagonal decay require additional work inside the kernel (Figure 5c, Appendix § 12.§ 4). Packed INT4 operands further reduce runtime but incur a larger accuracy loss.

Integration into SSDi8. We also apply the key-axis transformations within SSDi8 ([13]), which combines W4 projections with an INT8 Mamba-2 recurrence and an INT8 fixed-point state pass. Unlike our kernels, SSDi8 quantizes shared queries and keys once and reuses them across contractions. We therefore fuse RQKR_{QK} and DQKD_{QK} into its operand-preparation pass. Query–key rotation improves perplexity on every evaluated Mamba-2 and Nemotron-H model, and rescaling improves most configurations further (Table 20). These gains come without a measured latency increase: rotation leaves end-to-end prefill latency unchanged on the evaluated Mamba-2 models (Figure 5d), and fused rescaling leaves Mamba-2 1.3B layer latency unchanged (Table 16). We compare against a W4A16 reference in Appendix § 14.2 to distinguish the remaining weight-quantization loss from the recurrence-quantization loss. The SSDi8 integration also improves long-context perplexity and retrieval on Nemotron-H Base (Appendix § 14.§ 4).

Rescaling exponent κ\kappa1.3B8B
14\frac{1}{4}10.8587.435
12\frac{1}{2}10.8497.435
34\frac{3}{4}10.8667.441
ImplementationLatency (ms)Relative latency
Without rescaling1.4981.00
Fused rescaling1.4981.00
Separate elementwise rescaling–1.04

Table 16. Production query–key rescaling controls. Top: WikiText-2 perplexity with static value-side decay absorption. Bottom: Mamba-2 1.3B layer latency at B=1B = 1, relative to the baseline without rescaling. A dash denotes an unreported absolute time.

ConfigurationScaleMamba-2 1.3BMamba-2 2.7BMamba-2 8BNemo-H BaseNemo-H Reason.
PPL ↓\downarrowAvg. ↑\uparrowPPL ↓\downarrowAvg. ↑\uparrowPPL ↓\downarrowAvg. ↑\uparrowPPL ↓\downarrowAvg. ↑\uparrowPPL ↓\downarrowAvg. ↑\uparrow
FP16–10.4200.5959.0600.6387.2500.7076.0300.7588.4230.731
W4A16–10.7170.5889.2860.6327.3800.7006.0890.7508.5110.730
SSDi8 (official)static-(H,Dv)(H,D_v)10.923 ±\pm0.0070.5889.4300.6267.6290.696––––
SSDi8 (matched)static-(H,Dv)(H,D_v)10.928 ±\pm0.004–9.441 ±\pm0.017–7.620 ±\pm0.009–6.3380.7458.9210.726
Value (RQKR_{QK})static-(H,Dv)(H,D_v)10.860 ±\pm0.0070.5909.379 ±\pm0.0110.6297.456 ±\pm0.0090.7036.1860.7518.6660.727
+ DQKD_{QK}static-(H,Dv)(H,D_v)10.849 ±\pm0.0110.5889.383 ±\pm0.0070.6277.435 ±\pm0.0080.6996.1780.7538.6500.728
Value (RQKR_{QK})dynamic-(H,1)(H,1)10.829 ±\pm0.0120.5889.371 ±\pm0.0120.6317.447 ±\pm0.0080.7046.1840.7518.6570.727
+ DQKD_{QK}dynamic-(H,1)(H,1)10.812 ±\pm0.0070.5879.363 ±\pm0.0040.6347.423 ±\pm0.0070.697––––
Key (RQKR_{QK})static-(H,Dk)(H,D_k)10.849 ±\pm0.0080.5899.367 ±\pm0.0110.6307.455 ±\pm0.0060.7036.1700.7508.7330.727
+ DQKD_{QK}static-(H,Dk)(H,D_k)10.830 ±\pm0.0070.5899.372 ±\pm0.0030.6307.437 ±\pm0.0040.6976.1750.7538.6700.728
Key (RQKR_{QK})dynamic-(H,1)(H,1)10.828 ±\pm0.010–9.360 ±\pm0.004–7.444 ±\pm0.009–––––
+ DQKD_{QK}dynamic-(H,1)(H,1)10.807 ±\pm0.0050.5859.368 ±\pm0.0050.6307.422 ±\pm0.0070.702––––

Table 20. SSDi8 quality comparisons, including W4A16 and matched-baseline references. Perplexity is mean ±\pm std over four calibration sets for Mamba-2; rows without ±\pm and Nemotron-H are single runs. Avg.: six-task accuracy on the default calibration draw. Value/key denotes decay placement; scales apply to that operand. Dashes indicate unreported results.

We also compare the two placements of scalar decay. Placing decay on keys achieves comparable perplexity without materializing expanded keys, but the cost of requantizing operands makes value-side absorption faster at the tested Mamba-2 shapes. We therefore keep decay on values for these models, although wider value dimensions can reverse the runtime preference (Figure 14b, Appendix § 14.§ 1).

Cost of decay placement at $B=8$

Figure 14. Cost of decay placement at B=8B = 8. (a) Mamba-2 1.3B stages at T=2048T = 2048 with static or dynamic (′') scales. (b) Key-side latency relative to value-side latency over DvD_v, with static scales and 64 heads.

Conclusion

In this work, we introduce LinQuant, a framework that exploits seven functional invariances for low-precision linear recurrences, and characterize their applicability across a wide range of recurrent mixers. These consist of extra-kernel changes of basis and intra-kernel absorptions, namely alternative allocations of decay and write factors between operands. Our controlled study shows their benefits depend on recurrence structure, QK normalization, scale granularity, and the basis used to quantize the state. Changes of basis and write-gate allocation are particularly effective under coarse scales. Combining invariances also improves quantization with per-channel value scales, where individual transformations can offer little benefit. These findings transfer to production-scale models: our INT8 kernels recover most of the quantization loss on Nemotron-H, Qwen3.5, and Kimi Linear while accelerating recurrent prefill. We also augment SSDi8, improving perplexity at matched prefill latency. Although our implementation focuses on integer prefill, the introduced invariances provide a broader design space (e.g. floating-point) for future low-precision implementations.

Acknowledgements

This work was supported by the Hector Foundation, Max Planck ETH Center for Learning Systems, and the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

AI Use Statement

In this work, we use generative AI tools to formulate mathematical claims, provide critical ingredients for proving them, to design or give feedback on our methodology and experiments, to create or edit code, implement methods, to propose or refine hypotheses, to support qualitative and thematic data analysis, interpret results, and to do a literature review. Additionally, we used generative AI tools for LaTeX formatting, to draft parts of the appendix, and for brainstorming. We have not used generative AI tools for generating synthetic datasets, to help develop theoretical models or conceptual frameworks, to assist with translation, or to clean or reformat datasets. We have reviewed all AI-assisted work, checked the code as well as the reformatted LaTeX tables. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Extended related work

Linear recurrent and hybrid language models. A growing line of work replaces softmax attention with recurrent sequence mixers whose state size [19] and per-token decode cost do not grow with context length. This is achieved by implicitly summarizing past tokens in a fixed-size hidden state, instead of explicitly in a KV-cache [12, 22] (Yang et al. 2024b). Selective state-space models such as Mamba and Mamba-2 introduce input-dependent recurrent dynamics and hardware-efficient implementations [10, 8].

More expressive recurrences modify how information is retained and written into the hidden state. DeltaNet corrects stored associations by removing existing associations from its state (Yang et al., 2024c), while Gated DeltaNet combines this correction mechanism with scalar forgetting (Yang et al., 2025), and KDA uses channelwise decay (Team et al., 2025). Preconditioned DeltaNet further incorporates curvature-aware updates (Tumma et al., 2026), while DeltaProduct enhances the state transition operator (Siems et al., 2025). Large-scale hybrid models combine recurrent and softmax attention layers [4] (Team et al. 2025).

Our work studies how the structure of these different recurrences enables and constrains functional invariances. We use these invariances for post-training quantization (PTQ).

PTQ of softmax Transformers. Weight-only PTQ methods such as GPTQ and AWQ reduce quantization error through reconstruction-based compensation or channel protection [9, 14]. For joint weight and activation quantization, SmoothQuant transfers activation range into weights (Xiao et al., 2023), while QServe combines quantization with efficient serving [15]. Rotation-based methods redistribute concentrated energy across channels: QuIP# uses Hadamard incoherence processing for weight quantization (Tseng et al., 2024), QuaRot extends rotations to weights, activations, and the KV cache [2], and SpinQuant learns rotations from calibration data [16]. LinQuant extends the study of functional invariances to linear recurrences.

Quantization of recurrent language models. Quamba develops an 8-bit PTQ recipe for selective state-space models [6]; MambaQuant introduces variance-aligned rotations (Xu et al., 2025); and Quamba2 extends PTQ across Mamba and Mamba-2 [7]. These methods motivate distinguishing projection transformations from transformations of the recurrence itself. Quamba2 uses projection rotations, which SSDi8 retains and our transformations complement. MambaQuant also smooths and rotates the contraction reading a materialized scan output. Its Mamba-1 recurrence has diagonal decay, so a general state-axis rotation cannot be carried through the update. Our framework characterizes these restrictions across recurrence families and identifies additional choices of chunkwise placement and storage basis (Section 3). Table 2b demonstrates compatibility with Hadamard-transformed W4 projections through SSDi8; it does not compare the complete MambaQuant or Quamba2 recipes.

SSDi8 targets low-precision execution inside the chunkwise Mamba-2 computation [13]. It reuses quantized keys and queries, absorbs decay into values, and carries an INT8 state, all to maintain a persistent INT8 computational path. This decay placement is the value-side endpoint of our scalar-decay absorption family. Our production pipeline builds on SSDi8 by adding state-axis rotations and rescalings (Section 4.2).

Recurrent-state compression beyond quantization. Chahine et al. [5] prune the hidden state during training using the free choice of key-axis basis, which we identify as invariance (1). Similarly, Nazari & Rusch [17] use this freedom to prune the state of pretrained linear recurrences. Low-rank and sparse recurrent connectivity has also been studied outside language modeling (Tumma et al., 2024). In LinQuant, we instead retain state dimensions and use equivalent representations to improve quantization, addressing a complementary source of recurrent efficiency.

Experimental setup

This appendix gives the setup and timing methodology for the controlled study and the SSDi8 experiments. Their results appear in Appendices § 13 and § 14. Appendix § 15 gives the setup and results for recurrence-only quantization of pretrained hybrids.

Controlled study

Table 3: State-update rules for recurrent models in the mixer-zoo design space. Colors identify decay and write gating.

Models and training. The 340M zoo contains fifteen recurrence variants, or cells: LA, Mamba-2, and GLA with and without write gating and QK normalization, plus DN, GDN, and KDA with QK normalization. All models use the FLA implementation ((Yang & Zhang, 2024)) and share a 24-layer architecture with a SwiGLU MLP, an output gate on the mixer, and tied embeddings. At 340M parameters, the model width is 1024 with 8 heads and Dk=Dv=128D_k = D_v = 128. At 1.3B, the width is 2048 with 16 heads and the same head dimensions.

Each cell is an instance of Equation (1) with the decay and write rule of Table 3. Appendix § 12.§ 1 gives the quantized chunkwise forms. QK normalization applies ℓ2\ell_2 normalization to qtq_t and ktk_t independently within each head.

Every model is trained from scratch on SlimPajama using FLA and a Llama tokenizer, with 15B tokens for the 340M models and 100B for the 1.3B models. All runs use sequence length 2048 and AdamW with peak learning rate 4×10−44 \times10^{-4}, 1024 warm-up steps, and cosine decay to one tenth of the peak. We clip gradients at 1.

Quantization. Quantization is symmetric, with static scales calibrated by the maximum absolute activation over eight WikiText-2 training windows of 2048 tokens. The state write, state read, and local-score contractions use INT8 operands with INT32 accumulation (except the 16-token diagonal blocks of the local score under diagonal decay, which compute the decay-weighted dot products in FP32 using dequantized queries and keys). The intra-chunk output uses FP16 arithmetic on INT8-quantized values. The controlled study applies fake quantization at these boundaries: operands are rounded to the integer grid of their scale, and the contractions are evaluated in floating point with the rounding of the kernels’ intermediate precisions. The kernels of Appendix § 12.2 execute the same boundaries on integers (§ 5.2). An FP32 state is carried between chunks, and reads consume an INT8 copy with one scale per head and value channel, calibrated on the carry. We call this copy the stored state, and vary operand precision and stored-state precision separately. INT6 and INT4 configurations restrict the integer range of the operands within the same boundaries and kernels. The stored state remains INT8 unless stated otherwise. Chunk size is 64, with 16-token sub-blocks for diagonal decay, as in FLA.

Evaluation. We evaluate perplexity on WikiText-2 test at sequence length 2048. Transformation parameters are selected using cross-entropy on eight further 2048-token WikiText-2 training samples. They are drawn with a different seed than the eight calibration windows. The test split is used only for the reported perplexity. Excess NLL is log⁡(PPLquant/PPLFP)\log(\mathrm{PPL}_{\mathrm{quant}}/\mathrm{PPL}_{\mathrm{FP}}), and NLL recovered is the difference between the naïve quantizer’s NLL and that of the transformed configuration. We evaluate zero-shot accuracy with the Language Model Evaluation Harness ((Sutawika et al., 2024)) on LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, and WinoGrande.

Transformation selection. We use the diagonal rescaling parameterizations of § 4.2 and sweep κ∈{14,12,34,1}\kappa\in\left\{\frac{1}{4}, \frac{1}{2}, \frac{3}{4}, 1\right\}, including both DQKD_{QK} alone and its composition with RQKR_{QK} on linear-write cells. The full DVD_V sweep is evaluated under shared value scales.

For decaying cells, we sweep η∈{0,14,12,34,1}\eta\in\{0,\frac{1}{4},\frac{1}{2},\frac{3}{4},1\}. Linear write-factor absorption uses ξ∈{1,12,0}\xi\in\{1,\frac{1}{2},0\}, while delta cells use (σ,τ)∈{1,12,0}2(\sigma,\tau) \in\{1,\frac{1}{2},0\}^{2}.

We form the stack by combining transformations whose selected settings improve held-out cross-entropy by at least 0.003 nats over the naive quantizer. We use canonical state storage unless stated otherwise, and retain the naive quantizer if no transformation qualifies. The zoo results correspond to plain Hadamard rotations: the sign vector of the zoo implementation is applied after the transform, where it cancels under symmetric quantizers. The sensitivity to random input-side signs, to the calibration corpus, and to the calibration draw is reported below. This screening uses individual gains rather than an exhaustive search over joint settings, so it can miss transformations that help only in combination. For the pretrained hybrids, whose naive INT8 gaps are of the order of the 0.003-nat threshold, we instead select among single transformations and their tested compositions by minimum held-out cross-entropy (Appendix § 15.§ 1).

Calibration corpus and seeds. We recalibrated the 340M zoo on Pile validation data and evaluated on WikiText-2 test, covering all 15 models, both shared-key scale regimes, and FP and INT8 states. At each operand precision, this gives 60 model-and-quantization settings. Changing the calibration corpus rarely changes the preferred transformation.

The combinations selected on WikiText-2 still improve on naive quantization under Pile calibration in 36 of the 37 INT8 settings where a transformation was selected, and all 58 such INT6 settings. The exception is a small query–key rescaling gain on Mamba-2 that becomes negligible. Repeating 18 configurations with a second calibration draw also gives similar perplexities (median absolute difference 0.004, maximum 0.042).

Three input-side Hadamard sign patterns gave similar results across the 15 models, except that all improved the two LA models without QK normalization relative to the plain Hadamard. The production experiments use random input-side signs.

Timing. We compare the transformation stack with canonical RV\mathrm{RV} against naive INT8 on the A5000, A100, and H100 across batch sizes, with T=2048T = 2048, H=8H = 8, and Dk=Dv=128D_k = D_v = 128. We report ratios of median call times after autotuning and warm-up. Both calls include gate processing, operand quantization, and recurrent computation, including preparation for delta cells. Forward rotations performed before the call are excluded. Appendix § 12.§ 4 reports these ratios, also for the key-side stack and rotated storage, together with a comparison against the BF16 kernels of FLA.

Mamba-2 in SSDi8

We evaluate pretrained Mamba-2 and Nemotron-H models using SSDi8’s W4A8 projection and INT8-recurrence pipeline. Unless otherwise stated, the matched baseline uses value-side decay absorption, static per-(H,Dv)(H,D_v) value scales, per-group INT8 keys and queries, and an INT8 fixed-point state pass.

Calibration. The controlled study calibrates activations on WikiText-2 (Appendix § 10.§ 1). In production, static activation scales and DQKD_{QK} are calibrated on 512 Pile sequences of 512 tokens for Mamba-2 and 2048 tokens for Nemotron-H. The activation scales are measured on the FP16 model with the projection-level Hadamard transformations applied. Production weight quantization then uses 128 WikiText-2 training samples with GPTQ after fusing these transformations. Each layer uses maximum absolute activation measurements within the same scale axes as the quantized kernel (Algorithm 3). In the recurrent core, we observe the raw operands V, K, Q, decay-weighted operands Γ⃗[i]⊙V[i]\vec{\Gamma}_{[i]} \odot\mathrm{V}_{[i]}, Γ⃗[i]⊙K[i]\vec{\Gamma}_{[i]} \odot\mathrm{K}_{[i]}, local state contribution H, and chunk product Z=QKTZ = \mathrm{QK}^{\mathsf{T}}.

For DQKD_{QK}, we measure channel maxima of the Mamba-2 key and query streams, conventionally denoted B and C, after convolution and SiLU. Rescaling is applied at this same point during inference. Moving it before the nonlinear preprocessing would not preserve the function.

Evaluation. We evaluate WikiText-2 test perplexity at sequence length 2048, using the standard tokenizer for each model: EleutherAI/gpt-neox-20b for Mamba-2 1.3B and 2.7B, and NVIDIA SentencePiece for Mamba-2 8B. Downstream evaluation uses the Language Model Evaluation Harness (Sutawika et al., 2024) on SSDi8’s six zero-shot tasks: LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, and WinoGrande. For Mamba-2, rows with static-(H,Dv)(H,D_v) value-side scales run without SSDi8’s output-projection mean correction, as do the tables marked “no compensation”. The remaining Mamba-2 rows apply it with step factor 0.15, and the Nemotron-H rows with 0.1.

Timing. Unless otherwise indicated, latency measurements use an NVIDIA RTX A5000 with batch size B=8B = 8 and sequence length T=2048T = 2048. For component measurements, we instantiate a single Mamba-2 mixer layer with random weights and time each kernel or sub-operation using CUDA events. Each run uses 30 warm-up forward passes for Triton autotuning and allocation, followed by 200 measured forward passes. We synchronize the stream after each measured forward pass, sum calls belonging to the same stage, and report the median stage latency in microseconds.

The seven invariances: statements and proofs

We prove the invariances of § 4 for Equation (1) in exact arithmetic, with positive decay and write gates and consistently transformed initial states. Table 1 summarizes their applicability.

(1/7) Key-axis rotation RQKR_{QK}. Let R=RQK\mathbf{R} = R_{QK} be orthogonal and commute with every Λt\Lambda_t. With qt′=Rqt\mathbf{q}'_t = \mathbf{R}\mathbf{q}_t, kt′=Rkt\mathbf{k}'_t = \mathbf{R}\mathbf{k}_t, and St−1′=St−1RT\mathbf{S}'_{t-1} = \mathbf{S}_{t-1}\mathbf{R}^{\mathsf{T}}, the update and readout satisfy

St′=(St−1Λt(I−δβtktktT)+βtvtktT)RT=StRT,St′qt′=Stqt.\mathbf{S}'_t = \left(\mathbf{S}_{t-1}\Lambda_t(I-\delta\beta_t\mathbf{k}_t\mathbf{k}_t^{\mathsf{T}})+\beta_t\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}\right)\mathbf{R}^{\mathsf{T}} = \mathbf{S}_t\mathbf{R}^{\mathsf{T}}, \qquad \mathbf{S}'_t\mathbf{q}'_t = \mathbf{S}_t\mathbf{q}_t.

Commutation is automatic for absent or scalar decay. For fixed diagonal decay, R\mathbf{R} must preserve the common decay eigenspaces. Transforming the decay itself would generally make it dense. Local rotations of both decay-weighted score operands remain valid and leave the carried state unchanged. A global key rotation preserves the state quantizer’s scale axes but can change its calibrated maxima.

(2/7) Key-axis rescaling DQKD_{QK}. Let D=DQK≻0\mathbf{D} = D_{QK} \succ0 be diagonal, with qt′=Dqt\mathbf{q}'_t = \mathbf{D}\mathbf{q}_t, kt′=D−1kt\mathbf{k}'_t = \mathbf{D}^{-1}\mathbf{k}_t, and St−1′=St−1D−1\mathbf{S}'_{t-1} = \mathbf{S}_{t-1}\mathbf{D}^{-1}. Since D\mathbf{D} commutes with diagonal decay, a linear write gives

St′=(St−1Λt+βtvtktT)D−1=StD−1,St′qt′=Stqt.\mathbf{S}'_t = \left(\mathbf{S}_{t-1}\Lambda_t+\beta_t\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}\right)\mathbf{D}^{-1} = \mathbf{S}_t\mathbf{D}^{-1}, \qquad \mathbf{S}'_t\mathbf{q}'_t = \mathbf{S}_t\mathbf{q}_t.

For a delta write, matching the erasure terms requires D−2ktktTD−1=ktktTD−1\mathbf{D}^{-2}\mathbf{k}_t\mathbf{k}_t^{\mathsf{T}}\mathbf{D}^{-1} = \mathbf{k}_t\mathbf{k}_t^{\mathsf{T}}\mathbf{D}^{-1}, or D2kt=kt\mathbf{D}^2\mathbf{k}_t = \mathbf{k}_t, excluding a general rescaling. The transformation acts after any QK normalization.

(3/7–4/7) Value-axis rotation and rescaling. For any invertible T\mathbf{T}, set vt′=Tvt\mathbf{v}'_t = \mathbf{T}\mathbf{v}_t and St−1′=TSt−1\mathbf{S}'_{t-1} = \mathbf{T}\mathbf{S}_{t-1}. For either write rule and any diagonal decay,

St′=T(St−1Λt(I−δβtktktT)+βtvtktT)=TSt,T−1ot′=ot.\mathbf{S}'_t = \mathbf{T}\left(\mathbf{S}_{t-1}\Lambda_t(I-\delta\beta_t\mathbf{k}_t\mathbf{k}_t^{\mathsf{T}})+\beta_t\mathbf{v}_t\mathbf{k}_t^{\mathsf{T}}\right) = \mathbf{T}\mathbf{S}_t, \qquad \mathbf{T}^{-1}\mathbf{o}'_t = \mathbf{o}_t.

Choose T=RV\mathbf{T} = R_V or T=DV\mathbf{T} = D_V. Canonical storage undoes T\mathbf{T} on each chunk’s write and local output. Delta pseudo-values additionally require the incoming-state correction in the transformed value basis. With per-channel scales, diagonal rescaling cancels because (dpvp)/(dpsp)=vp/sp(d_pv_p)/(d_ps_p) = v_p/s_p, where sps_p is the channel scale. Consistently transformed scales therefore leave quantization unchanged, apart from finite-precision rounding.

(5/7–7/7) Decay and linear-write absorptions. All three follow from the token-wise identity

c[i]rv[i]r(k[i]r)T=(c[i]rv[i]r)((c[i]r)1−ηk[i]r)T,c[i]r>0.c^{r}_{[i]} \mathbf{v}^{r}_{[i]}\left(\mathbf{k}^{r}_{[i]}\right)^{\mathsf{T}} = \left(c^{r}_{[i]}\mathbf{v}^{r}_{[i]}\right)\left((c^{r}_{[i]})^{1-\eta}\mathbf{k}^{r}_{[i]}\right)^{\mathsf{T}}, \qquad c^{r}_{[i]} > 0.

Apply it to each chunk-write term with c[i]rc^{r}_{[i]} equal to the scalar tail decay, the diagonal-decay magnitude e[i]re^{r}_{[i]}, or the write gate β[i]r\beta^{r}_{[i]}. Other factors retain their placement. In particular, the relative channel-dependent decay remains on the key. For delta-rule decay absorption, use the corresponding pseudo-values in place of gated values.

(7/7) Delta write-factor absorption. Using the interval weights of Section 2.2, define the strictly lower-triangular Gram matrix

C[i],rs=1s<r∑nK[i],rnK[i],snγ[i],ns→r.C_{[i],rs} = 1_{s<r}\sum_n K_{[i],rn}K_{[i],sn}\gamma^{s\to r}_{[i],n}.

Scalar decay is broadcast across channels. Without decay, γ≡1\gamma\equiv1. The gate allocation is a diagonal similarity transformation:

L[i](σ)=B[i]σC[i]B[i]1−σ=B[i]σ−1L[i](1)B[i]1−σ,\mathbf{L}_{[i]}^{(\sigma)}=\mathbf{B}_{[i]}^{\sigma}\mathbf{C}_{[i]}\mathbf{B}_{[i]}^{1-\sigma}=\mathbf{B}_{[i]}^{\sigma-1}\mathbf{L}_{[i]}^{(1)}\mathbf{B}_{[i]}^{1-\sigma},
(I+L[i](σ))−1B[i]σ=B[i]σ−1(I+L[i](1))−1B[i].\left(\mathbf{I}+\mathbf{L}_{[i]}^{(\sigma)}\right)^{-1}\mathbf{B}_{[i]}^{\sigma}=\mathbf{B}_{[i]}^{\sigma-1}\left(\mathbf{I}+\mathbf{L}_{[i]}^{(1)}\right)^{-1}\mathbf{B}_{[i]}.

Writing F[i]:=F[i](1)\mathbf{F}_{[i]}:=\mathbf{F}_{[i]}^{(1)}, the complete pseudo-values satisfy

F[i](σ)=(I+L[i](σ))−1B[i]σ(V[i]−(Γ[i]←⊙K[i])S[i]T)=B[i]σ−1F[i].\mathbf{F}_{[i]}^{(\sigma)}=\left(\mathbf{I}+\mathbf{L}_{[i]}^{(\sigma)}\right)^{-1}\mathbf{B}_{[i]}^{\sigma}\left(\mathbf{V}_{[i]}-\left(\mathbf{\Gamma}_{[i]}^{\leftarrow}\odot\mathbf{K}_{[i]}\right)\mathbf{S}_{[i]}^{\mathsf{T}}\right)=\mathbf{B}_{[i]}^{\sigma-1}\mathbf{F}_{[i]}.

For the transformed operands of (7/7), the remaining gate powers sum to 1−σ1-\sigma, giving

(V[i](σ,τ))T(Γ[i](σ,τ)⊙K[i](σ,τ))=(F[i](σ))TB[i]1−σ(Γ[i]←⊙K[i])=F[i]T(Γ[i]←⊙K[i]).\begin{aligned} \left(\mathbf{V}_{[i]}^{(\sigma,\tau)}\right)^{\mathsf{T}}\left(\mathbf{\Gamma}_{[i]}^{(\sigma,\tau)}\odot\mathbf{K}_{[i]}^{(\sigma,\tau)}\right)=\left(\mathbf{F}_{[i]}^{(\sigma)}\right)^{\mathsf{T}}\mathbf{B}_{[i]}^{1-\sigma}\left(\mathbf{\Gamma}_{[i]}^{\leftarrow}\odot\mathbf{K}_{[i]}\right) \\ &=\mathbf{F}_{[i]}^{\mathsf{T}}\left(\mathbf{\Gamma}_{[i]}^{\leftarrow}\odot\mathbf{K}_{[i]}\right). \end{aligned}

The same token-wise cancellation preserves the local output. The incoming-state read is unchanged. At (σ,τ)=(1,1)(\sigma,\tau)=(1,1), this recovers the standard WY form (Yang et al., 2024c; 2025; Yang & Zhang, 2024).

Channel permutations. Channel permutations within a head also preserve the recurrence. On the key axis, the rotation argument applies with a permutation matrix P\mathbf{P} and consistently permuted decay Λt′=PΛtPT\Lambda'_t=\mathbf{P}\Lambda_t\mathbf{P}^{\mathsf{T}}, which remains diagonal. This holds for both linear and delta writes. Value-axis permutations follow from the equivariance above, with the inverse applied to the output. We do not explore permutations in our study: under our per-tensor, per-head, and per-channel quantizers, they only reorder quantized entries and their associated scales, leaving quantization error unchanged. They can matter under groupwise quantization, where channel order determines which entries share a scale (Zhang et al., 2026).

Low-precision realizations and kernel implementation

This appendix describes the quantized chunkwise forms, the controlled and production kernels, and the placement and fusion of transformations. Figure 1 provides an architectural overview of these transformation locations.

Quantized chunkwise forms

Section 3.1 gives the scalar-decay linear recurrence underlying Mamba-2. The other five families retain its state write, state pass, state read, local score, and intra-chunk output, but form their operands differently. Throughout this subsection, Q\mathrm{Q} denotes quantization at the indicated boundary, with scale factors implicit. The write, read, and local-score contractions use INT8 operands with INT32 accumulation, with the corresponding scales applied to the accumulated products. The intra-chunk output uses FP16 arithmetic on INT8-quantized values.

Decay factors. We use the interval weights γ[i]←\gamma_{[i]}^{\leftarrow} and stacked prefix and tail weights Γ[i]←,Γ[i]→\mathbf{\Gamma}_{[i]}^{\leftarrow},\mathbf{\Gamma}_{[i]}^{\rightarrow} from Section 2.2.

No decay, linear write (LA). Setting γ≡1\gamma\equiv1 removes all decay factors. With an optional write gate, the chunk computation becomes

H^[i]=Q ⁣(B[i]V[i])TQ ⁣(K[i]),\widehat{\mathbf{H}}_{[i]}=\mathrm{Q}\!\left(\mathbf{B}_{[i]}\mathbf{V}_{[i]}\right)^{\mathsf{T}}\mathrm{Q}\!\left(\mathbf{K}_{[i]}\right),
S[i+1]=S[i]+H^[i],S^[i+1]=Q ⁣(S[i+1]),\mathbf{S}_{[i+1]}=\mathbf{S}_{[i]}+\widehat{\mathbf{H}}_{[i]},\qquad\widehat{\mathbf{S}}_{[i+1]}=\mathrm{Q}\!\left(\mathbf{S}_{[i+1]}\right),
O^[i]inter=Q ⁣(Q[i])S^[i]T,\widehat{\mathbf{O}}_{[i]}^{\mathrm{inter}}=\mathrm{Q}\!\left(\mathbf{Q}_{[i]}\right)\widehat{\mathbf{S}}_{[i]}^{\mathsf{T}},
A^[i]=(Q ⁣(Q[i])Q ⁣(K[i])T)⊙M,\widehat{\mathbf{A}}_{[i]}=\left(\mathrm{Q}\!\left(\mathbf{Q}_{[i]}\right)\mathrm{Q}\!\left(\mathbf{K}_{[i]}\right)^{\mathsf{T}}\right)\odot\mathbf{M},
O^[i]intra=A^[i]Q ⁣(B[i]V[i]).\widehat{\mathbf{O}}_{[i]}^{\mathrm{intra}}=\widehat{\mathbf{A}}_{[i]}\mathrm{Q}\!\left(\mathbf{B}_{[i]}\mathbf{V}_{[i]}\right).

Diagonal decay, linear write (GLA). For positive decay gates, the masked product in Section 2.2 expands as

[tril⁡((Γ[i]r←⋅⊙Q[i])(K[i]Γ[i]r←⋅)T)]rs=Mrs∑nQ[i],rnK[i],snγ[i],ns→rγ[i],n.(3)\left[\operatorname{tril}\left(\left(\Gamma_{[i]}^{r\leftarrow\cdot}\odot Q_{[i]}\right)\left(\frac{K_{[i]}}{\Gamma_{[i]}^{r\leftarrow\cdot}}\right)^{\mathsf T}\right)\right]_{rs}=M_{rs}\sum_n Q_{[i],rn}K_{[i],sn}\gamma_{[i],n}^{s\to r}\gamma_{[i],n}. \tag*{(3)}

The right-hand side also applies when decay gates are zero. Quantizing the contraction operands for positive gates gives

H^[i]=Q(B[i]V[i])TQ(Γ[i]r←⋅⊙K[i]),\widehat{H}_{[i]}=\mathcal{Q}\left(B_{[i]}V_{[i]}\right)^{\mathsf T}\mathcal{Q}\left(\Gamma_{[i]}^{r\leftarrow\cdot}\odot K_{[i]}\right),
S[i+1]=S[i]diag⁡(γ[i],1C,…,γ[i],DkC)+H^[i],S^[i+1]=Q(S[i+1]),S_{[i+1]}=S_{[i]}\operatorname{diag}\left(\gamma_{[i],1}^{C},\ldots,\gamma_{[i],D_k}^{C}\right)+\widehat{H}_{[i]},\qquad\widehat{S}_{[i+1]}=\mathcal{Q}\left(S_{[i+1]}\right),
O^[i]inter=Q(Γ[i]r←⋅⊙Q[i])S^[i]T,\widehat{O}_{[i]}^{\mathrm{inter}}=\mathcal{Q}\left(\Gamma_{[i]}^{r\leftarrow\cdot}\odot Q_{[i]}\right)\widehat{S}_{[i]}^{\mathsf T},
A^[i],rs=∑nQ(Q[i],rnγ[i],nr←a/γ[i],na)Q(K[i],snγ[i],na/γ[i],ns),\widehat{A}_{[i],rs}=\sum_n\mathcal{Q}\left(Q_{[i],rn}\gamma_{[i],n}^{r\leftarrow a}/\gamma_{[i],n}^{a}\right)\mathcal{Q}\left(K_{[i],sn}\gamma_{[i],n}^{a}/\gamma_{[i],n}^{s}\right),
O^[i]intra=A^[i]Q(B[i]V[i]).(4)\widehat{O}_{[i]}^{\mathrm{intra}}=\widehat{A}_{[i]}\mathcal{Q}\left(B_{[i]}V_{[i]}\right). \tag*{(4)}

where the displayed local score applies to causal pairs s≤rs\le r. Entries above the causal diagonal are zero. The decay lies inside the reduction over channels, as in Equation (3). For cross-block scores, the kernel anchors the two decay factors at the first position aa of the block containing rr, so that both γ[i],nr←a/γ[i],na\gamma_{[i],n}^{r\leftarrow a}/\gamma_{[i],n}^{a} and γ[i],na/γ[i],ns\gamma_{[i],n}^{a}/\gamma_{[i],n}^{s} are at most one, then quantizes and multiplies the anchored operands. Within each 16-token block, it dequantizes the queries and keys and applies γ[i],ns→r\gamma_{[i],n}^{s\to r} inside the FP32 sum. A query–key rotation can act on the anchored operands in registers, but cannot generally be applied once to the unweighted QQ and KK (§ 4).

Delta write (DeltaNet, GDN, KDA). Delta cells first construct pseudo-values that account for interactions among writes and with the incoming state. The Gram matrix, the triangular solve, and the WY intermediates are computed in FP32. Only the incoming-state correction uses an INT8-quantized copy of W[i]W_{[i]} against the INT8 copy of the incoming state. For absent or scalar decay, let Γ[i]\Gamma_{[i]} denote the causal decay mask. The compensated write-factor placement (σ,τ)(\sigma,\tau) gives

G[i]=tril⁡((B[i]σK[i])(B[i]1−σK[i])T)⊙Γ[i],−1,T[i]=(I+G[i])−1,G_{[i]}=\operatorname{tril}\left(\left(B_{[i]}^{\sigma}K_{[i]}\right)\left(B_{[i]}^{1-\sigma}K_{[i]}\right)^{\mathsf T}\right)\odot\Gamma_{[i],-1},\qquad T_{[i]}=\left(I+G_{[i]}\right)^{-1},
W[i]=T[i][B[i]σ(Γ[i]τ⊙K[i])],U[i]=T[i]B[i]σV[i],W_{[i]}=T_{[i]}\left[B_{[i]}^{\sigma}\left(\Gamma_{[i]}^{\tau}\odot K_{[i]}\right)\right],\qquad U_{[i]}=T_{[i]}B_{[i]}^{\sigma}V_{[i]},
F^[i](σ)=U[i]−Q(W[i])S^[i]T,\widehat{F}_{[i]}^{(\sigma)}=U_{[i]}-\mathcal{Q}\left(W_{[i]}\right)\widehat{S}_{[i]}^{\mathsf T},
V[i](σ,τ)=B[i](1−σ)(1−τ)F^[i](σ),K[i](σ,τ)=B[i](1−σ)τK[i].V_{[i]}^{(\sigma,\tau)}=B_{[i]}^{(1-\sigma)(1-\tau)}\widehat{F}_{[i]}^{(\sigma)},\qquad K_{[i]}^{(\sigma,\tau)}=B_{[i]}^{(1-\sigma)\tau}K_{[i]}.

Here U[i],W[i]U_{[i]},W_{[i]} are WY intermediates, and F^[i](σ)\widehat{F}_{[i]}^{(\sigma)} approximates the complete pseudo-values F[i](σ)F_{[i]}^{(\sigma)} using the quantized incoming-state correction. The transformed write operands V[i](σ,τ),K[i](σ,τ)V_{[i]}^{(\sigma,\tau)},K_{[i]}^{(\sigma,\tau)} enter the chunk step:

H^[i]=Q(V[i](σ,τ))TQ(Γ[i]r←⋅⊙K[i](σ,τ)),\widehat{H}_{[i]}=\mathcal{Q}\left(V_{[i]}^{(\sigma,\tau)}\right)^{\mathsf T}\mathcal{Q}\left(\Gamma_{[i]}^{r\leftarrow\cdot}\odot K_{[i]}^{(\sigma,\tau)}\right),
S[i+1]=S[i]P[i]+H^[i],S^[i+1]=Q(S[i+1]),S_{[i+1]}=S_{[i]}P_{[i]}+\widehat{H}_{[i]},\qquad\widehat{S}_{[i+1]}=\mathcal{Q}\left(S_{[i+1]}\right),
O^[i]inter=Γ[i]r←⋅⊙(Q(Q[i])S^[i]T),\widehat{O}_{[i]}^{\mathrm{inter}}=\Gamma_{[i]}^{r\leftarrow\cdot}\odot\left(\mathcal{Q}\left(Q_{[i]}\right)\widehat{S}_{[i]}^{\mathsf T}\right),
A^[i]=(Q(Q[i])Q(K[i](σ,τ))T)⊙Γ[i],\widehat{A}_{[i]}=\left(\mathcal{Q}\left(Q_{[i]}\right)\mathcal{Q}\left(K_{[i]}^{(\sigma,\tau)}\right)^{\mathsf T}\right)\odot\Gamma_{[i]},
O^[i]intra=A^[i]Q(V[i](σ,τ)).\widehat{O}_{[i]}^{\mathrm{intra}}=\widehat{A}_{[i]}\mathcal{Q}\left(V_{[i]}^{(\sigma,\tau)}\right).

where P[i]P_{[i]} is II for DeltaNet, γ[i]CI\gamma_{[i]}^{C}I for GDN, and diag⁡(γ[i],1C,…,γ[i],DkC)\operatorname{diag}\left(\gamma_{[i],1}^{C},\ldots,\gamma_{[i],D_k}^{C}\right) for KDA. For KDA, both the Gram and local-score contractions place the channel-dependent decay inside the reduction, the local score using the anchored construction of Equation (4) in place of a scalar mask applied after the product, and the state read quantizes the decay-weighted query Q(Γ[i]r←⋅⊙Q[i])\mathcal{Q}\left(\Gamma_{[i]}^{r\leftarrow\cdot}\odot Q_{[i]}\right). At (σ,τ)=(1,1)(\sigma,\tau)=(1,1), with no decay and quantization removed, V[i](σ,τ)=F[i]V_{[i]}^{(\sigma,\tau)} = F_{[i]} recovers the pseudo-values in Section 2.2. Setting σ=12\sigma= \frac{1}{2} instead assigns B[i]1/2B_{[i]}^{1/2} to each Gram operand.

Absorptions. For a linear write with scalar decay, the tail weights Γ[i]\Gamma_{[i]} and write gate B[i]B_{[i]} can each be split between the two operands before quantization. With elementwise powers of the tail vector and broadcasting across channels, the write is

Q((Γ[i]η)⊙(B[i]ξV[i]))TQ((Γ[i]1−η)⊙(B[i]1−ξK[i])).Q\left((\Gamma_{[i]}^{\eta}) \odot(B_{[i]}^{\xi}V_{[i]})\right)^{\mathsf{T}} Q\left((\Gamma_{[i]}^{1-\eta}) \odot(B_{[i]}^{1-\xi}K_{[i]})\right).

Here η=1\eta= 1 and ξ=1\xi= 1 put the respective factors on the value, while zero puts them on the key. Under diagonal decay, only the scalar maximum e[i]r=max⁡n(Γ[i]r)ne_{[i]}^{r} = \max_n (\Gamma_{[i]}^{r})_n moves. The relative channel-dependent decay remains on the key. Delta cells use pseudo-values for decay absorption and the compensated (σ,τ)(\sigma,\tau) construction for write strength, as proved in Appendix § 11.

Kernel implementation

Controlled study. Diagonal rescalings are applied to the operands before quantization, and absorptions redistribute token-wise factors already used by the write. Algorithm 1 shows one chunk step for scalar or no decay with canonical state storage. Unused axis transformations are set to the identity. Matrices stack tokens as rows, so the rotations of § 4.2 act by right multiplication with RTR^{\mathsf{T}}.

Here, QbQ_b quantizes operands to bb-bit integers and deq⁡\operatorname{deq} restores the associated scales. Scale factors not shown explicitly are folded into the surrounding operations. The full-precision carry is SS, and S^[i]\hat{S}_{[i]} is its INT8 read copy. The tail weights are gg, the prefix weights pp, the causal decay mask Γ\Gamma, and the chunk decay γC\gamma^{C}.

Algorithm 1 One INT8 chunk step with LinQuant transformations (per head).

  1. once: Q←QDQKQ \leftarrow QD_{QK}, K←KDQK−1K \leftarrow KD_{QK}^{-1}, V←VDVV \leftarrow VD_V ▷ rescale operands

  2. once: Q←QRQKTQ \leftarrow QR_{QK}^{\mathsf{T}}, K←KRQKK \leftarrow KR_{QK}, V←VRVTV \leftarrow VR_V^{\mathsf{T}} ▷ rotate operands

  3. if δ=0\delta= 0 then

  4. (K,V)←(B[i]1−ξK,B[i]ξV)(K,V) \leftarrow(B_{[i]}^{1-\xi}K,B_{[i]}^{\xi}V) ▷ write-gate absorption

  5. else

  6. (K,V)←(K[i](σ,τ),V[i](σ,τ))(K,V) \leftarrow(K_{[i]}^{(\sigma,\tau)},V_{[i]}^{(\sigma,\tau)}) from Algorithm 2

  7. end if

  8. wk←g1−ηw_k \leftarrow g^{1-\eta}, wv←gηw_v \leftarrow g^{\eta} ▷ decay absorption

  9. O←p⊙deq⁡(Qb(Q)S^[i])O \leftarrow p \odot\operatorname{deq}\left(Q_b(Q)\hat{S}_{[i]} \right) ▷ state read

  10. A←deq⁡(Qb(Q)Qb(K)T)⊙ΓA \leftarrow\operatorname{deq}\left(Q_b(Q)Q_b(K)^{\mathsf{T}}\right) \odot\Gamma ▷ local score

  11. Oloc←AQb(V)O_{\mathrm{loc}} \leftarrow A Q_b(V) ▷ intra-chunk output

  12. Oloc←OlocRVO_{\mathrm{loc}} \leftarrow O_{\mathrm{loc}}R_V ▷ restore value basis

  13. O←O+OlocO \leftarrow O + O_{\mathrm{loc}} ▷ combine outputs

  14. K^,V^←Qb(wk⊙K),Qb(wv⊙V)\hat{K},\hat{V} \leftarrow Q_b(w_k \odot K), Q_b(w_v \odot V) ▷ quantize write operands

  15. H←deq⁡(V^TK^)H \leftarrow\operatorname{deq}(\hat{V}^{\mathsf{T}}\hat{K}) ▷ state write

  16. H←RVTHH \leftarrow R_V^{\mathsf{T}}H ▷ restore value basis

  17. S←SγC+HS \leftarrow S\gamma^{C} + H. S^[i+1]←Q8(S)\hat{S}_{[i+1]} \leftarrow Q_8(S) ▷ carry and read copy

Rescalings and rotations are fused with quantization; inverse value rescaling is applied to the output. The arithmetic and quantization boundaries follow Appendix § 12.§ 1.

For diagonal decay, Algorithm 1 uses the channel-dependent forms of Appendix § 12.§ 1; query–key rotations act on the decay-weighted operands before quantization.

Canonical storage under RVR_V adds matrix products inside the call. Each chunk’s rotated write is multiplied by RVTR_V^{\mathsf{T}} before it joins the FP32 carry, and the local output is returned to the original basis. The state read stays canonical. For canonical storage, delta cells right-multiply the incoming-state correction Q(W[i])S^[i]T\mathcal{Q}(W_{[i]})\hat{S}_{[i]}^{\mathsf{T}} by RVTR_V^{\mathsf{T}} before subtracting it from U[i]U_{[i]}, so both terms use the same rotated value basis. With rotated storage, the inverse rotations of the local output and state write are replaced by one O←ORVO \leftarrow OR_V at the end of the chunk. This incurs the accuracy penalty discussed in

Appendix E.1. The canonical delta kernels hold the full state and the Dv×DvD_v \times D_v rotation operand per program.

Algorithm 2 WY preparation with compensated write-factor placement (σ,τ)(\sigma,\tau), per head and chunk. M<M_{<} is the strictly lower causal mask, including scalar decay when present. Diagonal decay is applied inside the Gram reduction as described above. The Gram matrix, solve, and WY intermediates are FP32. S^in\hat{S}_{\mathrm{in}} is the INT8-quantized incoming state and W[i]\mathbf{W}_{[i]} is the only quantized operand of the preparation. The resulting transformed write operands are used in the chunk step.

  1. K[i]σ←B[i]σK[i], K[i]1−σ←B[i]1−σK[i]\mathbf{K}^{\sigma}_{[i]} \leftarrow\mathbf{B}^{\sigma}_{[i]}\mathbf{K}_{[i]},\ \mathbf{K}^{1-\sigma}_{[i]} \leftarrow\mathbf{B}^{1-\sigma}_{[i]}\mathbf{K}_{[i]} ▷ Gram factors

  2. G[i]←(K[i]σ(K[i]1−σ)T)⊙M<\mathbf{G}_{[i]} \leftarrow\left(\mathbf{K}^{\sigma}_{[i]}(\mathbf{K}^{1-\sigma}_{[i]})^{\mathsf{T}}\right)\odot M_{<} ▷ Gram matrix, fp32

  3. T[i]←(I+G[i])−1\mathbf{T}_{[i]} \leftarrow\left(\mathbf{I}+\mathbf{G}_{[i]}\right)^{-1} ▷ unit lower-triangular solve, fp32

  4. W[i]←T[i](K[i]σ⊙Γ[i]), U[i]←T[i]B[i]σV[i]\mathbf{W}_{[i]} \leftarrow\mathbf{T}_{[i]}(\mathbf{K}^{\sigma}_{[i]}\odot\mathbf{\Gamma}_{[i]}),\ \mathbf{U}_{[i]} \leftarrow\mathbf{T}_{[i]}\mathbf{B}^{\sigma}_{[i]}\mathbf{V}_{[i]} ▷ WY intermediates, fp32

  5. C[i]←Qb(W[i])S^inT\mathbf{C}_{[i]} \leftarrow Q_b(\mathbf{W}_{[i]})\hat{\mathbf{S}}_{\mathrm{in}}^{\mathsf{T}} ▷ incoming-state correction

  6. Canonical RV\mathbf{R}_V: C[i]←C[i]RVT\mathbf{C}_{[i]} \leftarrow\mathbf{C}_{[i]}\mathbf{R}_V^{\mathsf{T}} ▷ match the rotated value basis of U[i]\mathbf{U}_{[i]}

  7. F^[i](σ)←U[i]−C[i]\hat{\mathbf{F}}^{(\sigma)}_{[i]} \leftarrow\mathbf{U}_{[i]}-\mathbf{C}_{[i]} ▷ complete pseudo-values

  8. K[i](σ,τ)←B[i](1−σ)τK[i], V[i](σ,τ)←B[i](1−σ)(1−τ)F^[i](σ)\mathbf{K}^{(\sigma,\tau)}_{[i]} \leftarrow\mathbf{B}^{(1-\sigma)\tau}_{[i]}\mathbf{K}_{[i]},\ \mathbf{V}^{(\sigma,\tau)}_{[i]} \leftarrow\mathbf{B}^{(1-\sigma)(1-\tau)}_{[i]}\hat{\mathbf{F}}^{(\sigma)}_{[i]} ▷ transformed write operands

  9. return K[i](σ,τ),V[i](σ,τ)\mathbf{K}^{(\sigma,\tau)}_{[i]},\mathbf{V}^{(\sigma,\tau)}_{[i]} for Algorithm 1

Production Mamba-2. The production implementation adds state-axis transformations to SSDi8’s dataflow, as summarized in Algorithm 3. The added DQK\mathbf{D}_{QK} rescaling is applied as a per-channel multiply inside the fused split-and-quantize pass over the convolution output, followed by RQK\mathbf{R}_{QK} in the same pass. Neither transformation requires a separate kernel launch or memory pass in this implementation.

Algorithm 3 The Mamba-2 kernel of LinQuant: SSDi8’s dataflow with RQK\mathbf{R}_{QK} and DQK\mathbf{D}_{QK}. Keys and queries are shared by the GG groups and read by the HH heads. g(h)g(h) denotes the group of head hh. w\mathbf{w} denotes the within-chunk decay weights (tail factors for the write, prefix factors for the read), and γ[i]\boldsymbol{\gamma}_{[i]} denotes the chunk decay.

Once per layer (the fused pass over the convolution output)

  1. K←KDQK−1RQKT, Q←QDQKRQKT. K^←Q8(K,sK), Q^←Q8(Q,sQ)\mathbf{K} \leftarrow\mathbf{K}\mathbf{D}_{QK}^{-1}\mathbf{R}_{QK}^{\mathsf{T}},\ \mathbf{Q} \leftarrow\mathbf{Q}\mathbf{D}_{QK}\mathbf{R}_{QK}^{\mathsf{T}}.\ \hat{\mathbf{K}} \leftarrow Q_8(\mathbf{K},s_K),\ \hat{\mathbf{Q}} \leftarrow Q_8(\mathbf{Q},s_Q) ▷ per group. The quantized queries and keys are reused by three stages

Per chunk [i][i] and head hh

  1. V^←Q8(wtail⊙V,sV)\hat{\mathbf{V}} \leftarrow Q_8(\mathbf{w}_{\mathrm{tail}}\odot\mathbf{V},s_V), tile by tile in registers ▷ decay on the value. Static per-(H,Dv)(H,D_v) scale or dynamic per tile. Key side: K^←Q8(wtail⊙K)\hat{\mathbf{K}} \leftarrow Q_8(\mathbf{w}_{\mathrm{tail}}\odot\mathbf{K}) from an fp re-read instead

  2. H←V^K^g(h)T\mathbf{H} \leftarrow\hat{\mathbf{V}}\hat{\mathbf{K}}_{g(h)}^{\mathsf{T}} ▷ write: INT8×INT8, INT32 accumulation, one rescale

  3. S^[i+1]←fx(γ[i]S^[i])+Q8(H,sS)\hat{\mathbf{S}}_{[i+1]} \leftarrow f_x(\boldsymbol{\gamma}_{[i]}\hat{\mathbf{S}}_{[i]})+Q_8(\mathbf{H},s_S) ▷ state pass in INT8 fixed point, per-(H,Dv)(H,D_v) scale

  4. Z^←Q8(Q^g(h)K^g(h)T)\hat{\mathbf{Z}} \leftarrow Q_8(\hat{\mathbf{Q}}_{g(h)}\hat{\mathbf{K}}_{g(h)}^{\mathsf{T}}) ▷ local score: INT32 accumulation, requantized per group

  5. O←wpref⊙deq⁡(Q^g(h)S^[i]T)+(deq⁡(Z^)⊙Γ)V\mathbf{O} \leftarrow\mathbf{w}_{\mathrm{pref}}\odot\operatorname{deq}(\hat{\mathbf{Q}}_{g(h)}\hat{\mathbf{S}}_{[i]}^{\mathsf{T}})+(\operatorname{deq}(\hat{\mathbf{Z}})\odot\mathbf{\Gamma})\mathbf{V} ▷ read on INT8. Intra-chunk output fp16, the scale of Z^\hat{\mathbf{Z}} folded into Γ\mathbf{\Gamma}

Transformation placement and rotation fusion. We implement online rotations with structured signed Hadamards fused into existing operand passes. Query–key rotation follows convolution in the split-and-quantize pass. Value rotations must be undone before incompatible downstream operations: Mamba-2 restores each head’s basis in the recurrence-output kernel before its full-width gated RMSNorm. For GDN and KDA, we fuse the forward value rotation into convolution and the inverse into output normalization, with rescaling in the same passes. Intervening nonlinearities prevent folding these rotations into projection weights.

Applying decay in registers. Keys are shared by groups, while decay weights can differ across heads. A direct implementation of key-side absorption could therefore materialize a larger, head-indexed tensor of weighted keys. Our write kernel avoids this intermediate tensor. Each program loads its group’s key tile and its head’s decay weights, then forms the weighted entries in registers:

kr,n′=wh,r kg(h),r,nk'_{r,n} = w_{h,r}\,k_{g(h),r,n}

The weighted tile is quantized and clipped before the INT8 tensor-core contraction. Programs for heads in the same group read the same underlying key tile and apply their own weights, so head-specific values never need to be stored as an expanded floating-point tensor.

Kernel latency

Setup. All timings use the protocol of Appendix § 10.§ 1 (T=2048T = 2048, H=8H = 8, Dk=Dv=128D_k = D_v = 128, median call time after autotuning and warm-up) and are medians over three runs. The comparison with the BF16 kernels of FLA is measured on an RTX A5000.

INT8 operands. The harness of the controlled study changes scale granularity and transformation per run, so its kernels take BF16 operands and quantize them between the load and the tensor-core product. A deployed layer instead emits INT8 operands from the kernel that produces queries, keys, and values, as the production Mamba-2 path does (§ 5.2). The kernels of LA, Mamba-2, DeltaNet, and Gated DeltaNet therefore also accept INT8 operands, and the launch configurations of all six families are autotuned. For KDA, the chunk preparation forms the intra-chunk query–key product on INT8 queries and keys and passes INT8 operands to the state and output kernels.

Table 4 compares these kernels with FLA’s BF16 implementation. All six families have lower wall-clock call times, with the advantage of Mamba-2 and Gated DeltaNet increasing with batch size. At batch size one, launch overhead is substantial, and GPU timings show much smaller differences. KDA’s larger wall-clock speedup there mainly reflects overhead in the FLA operator. The LA, Mamba-2, DeltaNet, and Gated DeltaNet rows start from prequantized inputs, whereas GLA and KDA include quantization. When input quantization is included for every family, LA, Mamba-2, DeltaNet, and Gated DeltaNet all retain a speedup. The † rows include quantization after applying channel-dependent decay. For DeltaNet (‡), our path uses FLA’s fused GDN preparation; matching preparation reduces the speedup at B=8B = 8 to 1.11×. GDN provides the matched-preparation comparison. Appendix § 15.§ 6 reports pretrained-model shapes.

RecurrenceB=1B = 148163264128GPU time
LA1.571.791.901.841.781.801.811.71–1.79
Mamba-21.561.511.851.932.072.162.191.73–1.99
GLA†1.221.191.211.201.191.181.201.13–1.18
DN‡1.301.381.331.401.441.461.441.39–1.44
GDN1.071.291.311.361.381.401.421.37–1.42
KDA†1.491.211.191.181.171.181.171.13–1.18

Table 4. INT8 speedup over FLA BF16 on the RTX A5000, by batch size. GPU time sums kernel durations over B=4B = 4–3232. Input and preparation differences are described above.

INT4 operands. The controlled study evaluates INT6 and INT4 operands in INT8 containers, which gives no latency benefit below INT8. The tensor cores of the RTX A5000 and the A100 provide a 4-bit integer multiply-accumulate with twice as many products per instruction as for INT8. On the H100, this instruction does not return the 4-bit product, and INT4 operands stay in INT8 containers. The kernels of LA, Mamba-2, DeltaNet, and Gated DeltaNet therefore also accept two INT4 operands packed per byte and run the integer products of the recurrence on this instruction, while the stored state remains INT8. The KDA kernel forms packed operands in chunk preparation. GLA has no INT4 path. The resulting speedup (Table 5) is largest for the linear-write kernels, whose time is dominated by integer products, whereas the delta-rule kernels also spend substantial time in floating-point chunk preparation. At batch size one, launch overhead dominates and packed operands make little difference. Packed operands also halve operand memory. What remains open at INT4 is accuracy: the transformations recover a large part of the loss, but no pretrained model reaches its INT8 accuracy (Appendix § 15.§ 4).

Table 5: Packed INT4 speedup over INT8 and FLA BF16 on the RTX A5000. Columns give B=1B = 1 and the range over B=4B = 4–128; KDA (†) includes operand quantization.

RecurrenceAgainst INT8 operands, B=1B = 1Against INT8 operands, B=4B = 4–128Against FLA BF16, B=1B = 1Against FLA BF16, B=4B = 4–128
LA0.951.11–1.241.491.98–2.24
Mamba-20.971.10–1.141.511.65–2.48
DN1.041.07–1.091.431.46–1.59
GDN0.971.04–1.071.041.33–1.52
KDA†0.981.02–1.051.461.21–1.24

Table 5.

Overhead of the transformations. Table 6 compares the key-side stack (RQKR_{QK}) and stacks that additionally use RVR_V with rotated or canonical storage. Rescalings and absorptions fold into existing operands. Timings exclude forward operand rotations and the final inverse rotation outside the call; production timings include their fused passes (Appendix § 15.§ 6). Key rotations under diagonal decay are inside the call and included. Both KDA paths use unfused chunk preparation. For LA, Mamba-2, DeltaNet, and Gated DeltaNet, key-axis transformations and value rotation with rotated storage generally add little kernel overhead. The main exception is rotated storage at batch size one on the A100. GLA and KDA incur additional cost because diagonal decay requires key rotations inside the kernel. Canonical value storage is slower across all families and GPUs because it adds inverse state transforms.

Table 6: Transformation overhead relative to naive INT8 (1×: equal time). Both calls use the same preparation; rotations outside the recurrence call are excluded, as detailed below.

StackRecurrenceRTX A5000, B=1B = 1RTX A5000, 4RTX A5000, 8A100-SXM4-40GB, B=1B = 1A100-SXM4-40GB, 4A100-SXM4-40GB, 8H100, B=1B = 1H100, 4H100, 8
Key-side stackLA1.001.011.010.990.991.000.990.990.99
Mamba-21.011.001.021.001.001.000.980.980.99
GLA1.291.441.431.131.271.281.081.251.28
DN0.991.031.010.991.001.000.980.991.00
GDN1.001.021.011.001.001.000.990.991.00
KDA1.141.221.201.051.121.141.011.131.18
Rotated storageLA1.001.021.011.130.991.000.990.990.99
Mamba-21.001.021.021.101.001.000.980.980.99
GLA1.291.431.421.311.271.281.081.251.28
DN0.991.011.001.080.990.990.980.981.02
GDN0.991.011.001.001.001.000.991.001.02
KDA1.101.211.231.061.121.141.001.151.18
Canonical storageLA1.931.641.862.451.451.372.531.761.46
Mamba-21.671.501.772.031.361.282.081.641.37
GLA1.831.741.682.041.491.541.801.631.47
DN2.872.381.832.972.141.842.402.271.94
GDN2.572.331.942.521.971.742.062.091.77
KDA2.111.681.452.041.631.461.801.721.56

Table 6.

Controlled-study results

This appendix reports the controlled-study results for § 5.1, using the setup in Appendix § 10.§ 1. We organize them by key-axis transformations (1/7–2/7), value-axis transformations (3/7–4/7), and absorptions (5/7–7/7), followed by combined transformations, precision controls, and downstream accuracy. The complete per-configuration grids can be regenerated from the records in the supplementary material.

Individual transformations

Key-axis transformations (1/7)–(2/7). Figure 6 compares query–key rescaling alone and in combination with rotation across linear-write cells and scale regimes.

Figure 6. Response to the exponent κ\kappa of DQKD_{QK}, alone (solid) and composed with RQKR_{QK} (dashed, hollow), per linear cell and regime, INT6-quantized operands, INT8 stored state: percent of the naive excess NLL recovered. At κ=0\kappa= 0 the dashed curve is RQKR_{QK} alone.

Value-axis transformations (3/7)–(4/7). The value-rescaling sweep in Figure 7 isolates the role of scale granularity: consistent per-channel scales absorb the rescaling, while shared scales allow it to change quantization error.

Figure 7. Response to the exponent κ\kappa of DV\mathrm{DV}, percent of the naive excess NLL recovered: per-channel scales absorb a consistent diagonal rescaling, whereas shared scales allow the exponent to change the error. The full grid κ∈{14,12,34,1}\kappa\in\left\{\frac{1}{4},\frac{1}{2},\frac{3}{4},1\right\} was run only under coarse scales ⟨T,T⟩\langle\mathrm{T},\mathrm{T}\rangle.

State-storage basis. Figure 3c summarizes the storage-basis effect. Canonical storage helps LA most, with smaller effects in decaying and delta-rule cells, but adds inverse transforms inside the kernel (Appendix § 12.§ 4). Table 9 gives the per-model results under per-channel value scales.

precisioncellfpnaiveRQKR_{QK}RVR_V rot.RVR_V canon.η∗\eta^{*}β∗\beta^{*}DQK∗D^{*}_{QK}DQK+RQK∗D_{QK}+R^{*}_{QK}DV∗D^{*}_{V}stackstack, rot.
INT8LA17.11719.48217.73622.91519.502––17.42417.26519.48517.265–
LA+β\beta16.68817.54116.92821.51017.519–17.69316.88816.79717.54016.798–
Mamba-215.55315.61215.59215.63115.60815.613–15.56115.55615.61015.561–
Mamba-2+β\beta15.42015.44315.43315.44215.43915.44515.44615.43115.42715.444––
GLA15.10415.12415.12615.12315.12115.125–15.11615.12115.125––
GLA+β\beta15.30815.32215.32315.32215.32115.32115.32515.31115.31315.320––
DeltaNet15.67815.69815.69315.68815.690–15.695––15.696––
GDN15.00415.01915.01215.01515.01515.01915.017––15.018––
KDA14.48914.51114.51414.50814.50714.50714.508––14.509––
LA, no norm17.23017.79718.18317.93917.791––17.37217.38517.78917.388–
LA+β\beta, no norm16.98018.74419.44019.92618.988–20.26318.13517.90318.75117.902–
Mamba-2, no norm16.72116.91116.74416.90616.90716.901–16.80016.73516.91416.735–
Mamba-2+β\beta, no norm16.18916.26616.19816.25916.26316.26316.28116.24416.19416.26516.194–
GLA, no norm16.07016.28216.28916.27816.27816.280–16.10016.10416.27916.099–
GLA+β\beta, no norm15.89116.00916.01516.00916.00716.00916.02415.93515.93616.01215.935–
INT6LA17.11725.70620.81828.44225.691––19.42818.09725.68118.097–
LA+β\beta16.68819.63117.66922.61919.668–21.80817.49917.07919.62417.08919.236
Mamba-215.55316.02915.87616.04416.02316.026–15.72215.60316.02215.605–
Mamba-2+β\beta15.42015.59515.49915.58415.58615.59615.64115.48515.44615.59215.449–
GLA15.10415.65115.70015.65015.64215.659–15.30215.32115.65315.302–
GLA+β\beta15.30815.54015.56015.54315.54215.54215.62415.41815.42415.54115.415–
DeltaNet15.67815.92515.77815.92115.918–15.843––15.92715.769–
GDN15.00415.16815.05515.17315.17215.16015.128––15.16515.042–
KDA14.48914.84614.85014.84314.84514.85014.846––14.846––
LA, no norm17.23024.91624.08925.03224.897––18.07918.59624.85418.071–
LA+β\beta, no norm16.98030.71929.67132.04930.834–40.03221.04220.77530.72120.73821.141
Mamba-2, no norm16.72120.84217.08620.82420.81320.757–18.17816.85820.84416.848–
Mamba-2+β\beta, no norm16.18917.66516.30617.66117.66017.57317.94016.94316.26817.66516.264–
GLA, no norm16.07020.28220.49220.26020.26420.28919.55716.74016.79120.26316.739–
GLA+β\beta, no norm15.89119.22219.38319.22619.22319.23319.55716.83416.90219.21316.838–
INT4LA17.11723.72644.33376.04173.632––35.89624.91072.89324.73624.803
LA+β\beta16.68883.12032.21886.78783.585–89.67229.42622.50383.45022.495–
Mamba-215.55325.81220.87725.80325.72825.995–18.16816.55625.81016.42716.425
Mamba-2+β\beta15.42021.05917.58220.98021.01821.13122.19817.17216.06821.06116.070–
GLA15.10432.43135.26432.40632.35032.538–20.60021.49032.43220.587–
GLA+β\beta15.30824.89626.20325.00424.96225.04026.19218.76019.16124.89018.766–
DeltaNet15.67824.52318.75724.09824.010–21.416––24.47617.68517.680
GDN15.00420.54016.28220.51820.57020.48519.677––20.52516.09816.084
KDA14.48928.73229.71728.75028.59928.68928.257––28.717––
LA, no norm17.230314.039513.517317.889311.775––37.95047.213312.87837.950–
LA+β\beta, no norm16.980515.217151.013595.755467.922–735.86368.78479.739514.37969.45671.150
Mamba-2, no norm16.721316.76235.612315.117315.802315.758–107.19523.947316.22123.55523.540
Mamba-2+β\beta, no norm16.189202.73219.859202.726200.767188.882208.87864.85018.533202.45918.34418.351
GLA, no norm16.070173.810181.020173.014173.332173.702–46.24750.952173.54046.248–
GLA+β\beta, no norm15.891172.781178.699171.474171.329172.893164.10457.65063.135173.14342.06242.049

Table 9. WikiText-2 perplexity for the 340M zoo under per-tensor key scales and per-channel value scales. Operands are quantized to 8, 6, or 4 bits. The stored state remains INT8. An asterisk marks parameters selected by held-out cross-entropy. The stack combines promoted transformations and uses canonical storage. “stack, rot.” uses the same transformations with rotated storage. A dash denotes an inapplicable or untested setting, or a stack with no promoted transformation.

Absorptions (5/7)–(7/7). The absorption sweeps compare where token-wise factors enter the quantized computation. Scalar-decay placement has relatively small effects in the deployment-oriented regimes (Figure 8). For gated linear writes, the value-side endpoint ξ=1\xi= 1 performs best under ⟨T,C⟩\langle T,C\rangle, while the other regimes can favor an interior split. For the delta cells under coarse scales, the selection procedure moves the compensation to the keys (τ=1\tau= 1) and chooses σ∈{0,12}\sigma\in\{0,\frac{1}{2}\}, with small differences between placements. Figure 11 shows the corresponding error of the state write.

Write error under write-gate allocation. Figure 11 isolates how an exact reallocation changes quantization error in the state write of DeltaNet 340M, using INT6 per-tensor write scales. Moving the write gate from the pseudo-values toward the keys reduces error from the pseudo-values while increasing error from the keys. The net write error falls substantially, with the key-side endpoint

improving over the value-side endpoint in all 32 measured heads. Here σ=1−a\sigma= 1 - a and τ=1\tau= 1, with scales recalibrated for each allocation. We measure relative Frobenius error over 256 held-out chunks at four layers. The result explains the allocations selected under ⟨T,T⟩\langle T, T\rangle (Figure 10).

Choice of scalar decay factor. Diagonal-decay absorption (6/7) factors a scalar magnitude out of the per-channel decay and allocates it between keys and values. Our experiments use the channelwise maximum, which keeps the remaining channel-dependent factors at most one. However, the slowest channel in a head barely decays, so the scalar factor is close to one and captures little of the decay: only 0.6% of the mean log decay in Kimi Linear and roughly 1–2% in the controlled models. Moving a factor this close to one has little effect on quantization, explaining the negligible gains in both groups. Sweeping the allocation on Kimi Linear changes perplexity by less than 0.01 at each tested precision.

Choosing a smaller scalar factor captures more decay but leaves the remaining channel-dependent factors above one, so they can grow within a chunk. We tested factors chosen from channel quantiles on the two GLA models and KDA in the 340M zoo, across scale regimes, precisions, and allocation exponents. All approximately 100 settings performed worse than the channelwise maximum. Thus, choosing the maximum leaves little decay to redistribute, while choosing a smaller factor worsens the numerical range of the remaining channel-dependent factors.

Combined transformations and precision controls

Figure 12 compares the two shared-key regimes at INT6. Key scales are held fixed and only value scales change, from per channel on the left to per tensor on the right. State-axis transformations can help in either panel, whereas the larger value-axis gains emerge when values share a scale.

Figure 12. Percentage of the naive quantizer’s excess NLL recovered per cell and invariance at its selected setting (negative: the invariance increases the loss), shared keys with per-channel values (left) and with per-tensor values (right).

Scale granularity and combined transformations. Figure 2c and Figure 3a isolate the effects of coarsening key and value scales. Table 7 and Table 8 summarize INT6 quality at both model sizes, retaining every recurrence reported in the corresponding full grids. Coarse value scales particularly affect the delta-rule models. Value-axis transformations recover much of this loss. Combining transformations can improve further, although the selected stack does not outperform every individual setting in every configuration.

Each individual invariance uses its held-out-selected parameters. “Best single” is the lowest reported test perplexity among these individual settings, including either storage basis for value rotation but excluding compositions. It is a descriptive oracle comparison, not a selection rule. The stack retains the original held-out selection and canonical storage.

Table 9 retains the full grid of the 340M zoo under ⟨T,C⟩\langle T, C\rangle, with every individual transformation at INT8, INT6, and INT4.

Precision sensitivity. Table 10 contrasts INT8 and INT4. With per-channel value scales, several normalized decaying and delta-rule models are already close to full precision at INT8, leaving little room for a promoted transformation. At INT4, the stack can reduce perplexity substantially but does not consistently recover full-precision quality. The gap is particularly large with coarse value scales in the 1.3B zoo.

The write-scale control in Table 11 shows no improvement from dynamic scaling in the tested configurations. These accuracy experiments use 4-bit integer ranges within INT8 kernels. Our packed INT4 kernels also provide a runtime benefit while producing bit-identical states (Appendix § 12.§ 4), but recovering INT8-level accuracy remains an open problem.

ModelStatic write scalesDynamic write scales
LA22.726.2
Mamba-216.116.4
GDN18.418.8

Table 11. WikiText-2 perplexity for the static and dynamic per-chunk write-scale controls at INT4, with per-head read-key scales and INT8 stored states (340M models).

State precision and write-scale controls. State quantization affects accumulating recurrences most strongly (Table 12). Keeping the read state in floating point reduces error in LA and gated LA, while changing little in the decaying cells. Carrying the requantized state into subsequent updates adds error beyond that of an INT8 read copy, supporting the distinction between read precision and carry precision.

cellper-tensor keysper-head keys
INT8INT6INT8INT6
fp modelfp stateINT8 storedfp stateINT8 storedfp stateINT8 storedINT8 carryfp stateINT8 storedINT8 carry
LA17.11717.97619.48224.40325.70617.29819.14019.85019.95422.34923.963
LA+β\beta16.68816.82517.54119.18519.63116.71917.46618.19617.37318.20919.118
Mamba-215.55315.57915.61215.98816.02915.56815.60115.70015.76315.79515.914
Mamba-2+β\beta15.42015.43515.44315.58515.59515.43215.43915.44415.51815.52415.545
GLA15.10415.12415.12415.65115.65115.12015.11915.16915.30715.31215.360
GLA+β\beta15.30815.32215.32215.54415.54015.32015.31815.42215.45715.45615.566
DeltaNet15.67815.69615.69815.92115.92515.69515.69715.83915.87015.86716.012
GDN15.00415.01815.01915.17115.16815.01815.01715.04815.12015.12215.153
KDA14.48914.50814.51114.84714.84614.50414.50414.60214.76014.76414.861
LA, no norm17.23017.63717.79724.75624.91617.31917.52518.21018.29418.50219.443
LA+β\beta, no norm16.98017.89518.74429.79730.71917.55518.39019.03120.62121.44622.277
Mamba-2, no norm16.72116.90316.91120.82720.84216.75216.76116.93017.10617.11517.328
Mamba-2+β\beta, no norm16.18916.26116.26617.64117.66516.20416.20916.37716.38416.39116.577
GLA, no norm16.07016.27816.28220.28320.28216.19516.19716.36818.08718.09118.326
GLA+β\beta, no norm15.89116.00916.00919.24619.22215.93715.94016.03116.70916.70316.818

Table 12. State-quantization controls for the 340M zoo, with all invariances disabled. Entries report WikiText-2 perplexity. For per-tensor keys, we compare an unquantized state with the kernels' FP32 carry and INT8 read copy. For per-head keys, we additionally test an INT8 carry that feeds requantization error into later updates. Quantizing the read copy has the largest effect in non-decaying cells. Quantizing the carry adds further loss in the tested settings.

Refining write-key scales alone does not resolve the error from coarse read keys (Table 13). At INT6 it recovers part of the loss in linear-write models, with much smaller effects in delta-rule models. The FP32 carry, INT8 read copy, INT32 local-score accumulation, and FP16 intra-chunk output product remain fixed.

RecurrenceFPPer-channel values: SharedPer-channel values: Fine writePer-tensor values: SharedPer-tensor values: Fine write
LA17.11725.70622.42230.69426.380
LA+β\beta16.68819.63118.31228.38126.888
Mamba-215.55316.02915.79616.44616.242
Mamba-2+β\beta15.42015.59515.53616.23016.161
GLA15.10415.65115.32516.04815.678
GLA+β\beta15.30815.54015.46316.04615.952
DeltaNet15.67815.92515.91620.32320.261
GDN15.00415.16815.14517.06417.059
KDA14.48914.84614.78715.83715.766
LA†17.23024.91620.59731.58225.976
LA+β\beta†16.98030.71929.583100.24792.325
Mamba-2†16.72120.84220.21521.47920.817
Mamba-2+β\beta†16.18917.66517.38018.84418.460
GLA†16.07020.28219.78820.97720.411
GLA+β\beta†15.89119.22218.18220.52319.410

Table 13. INT6 WikiText-2 perplexity with shared per-tensor keys or separate per-channel write-key scales, without invariance transforms. Read keys remain per-tensor; †: no QK normalization.

Interactions between invariances. The controlled study first selects transformations that improve held-out cross-entropy individually, then evaluates their combination (Appendix § 10.§ 1). This procedure can miss transformations that help only together. Qwen3.5 provides a clear example: value rescaling and rotation can each have little effect alone, yet their combination removes most of the quantization loss. Conversely, moving the write gate onto the keys helps by itself but adds no benefit to the value-axis combination (Table 24). The pretrained-model selection therefore evaluates individual transformations and their tested combinations directly on held-out data.

The controlled zoo likewise shows that gains are not always additive. Across INT8, INT6, and INT4, every evaluated combination of two or more transformations improves on naive quantization, and most improve beyond their strongest member. However, the gain is typically smaller than the sum of the individual gains, consistent with transformations addressing overlapping sources of error. These comparisons include both canonical and rotated state storage.

Downstream accuracy

Tables 14 and 15 retain LAMBADA and the six-task average for every reported recurrence. The stack largely closes the gap to full precision for 340M LA and gated LA. At 1.3B, gated LA recovers most of the LAMBADA accuracy lost to naive quantization, although a gap remains. The other normalized recurrent models change little under this INT8 setting.

RecurrenceLAMBADA FPLAMBADA NaiveLAMBADA StackSix-task average FPSix-task average NaiveSix-task average Stack
LA0.2670.0420.2580.3980.3560.395
LA+β\beta0.2830.0950.2680.4000.3670.398
Mamba-20.3240.3100.3160.4100.4090.409
Mamba-2+β\beta0.3160.307–0.4130.412–
GLA0.3350.331–0.4190.417–
GLA+β\beta0.3320.328–0.4160.415–
DeltaNet0.2730.283–0.4020.406–
GDN0.3610.349–0.4180.413–
KDA0.3410.339–0.4130.413–
LA†0.2660.2420.2500.3970.3910.394
LA+β\beta†0.2730.1500.2220.4000.3720.380
Mamba-2†0.2900.2830.2860.4020.4010.401
Mamba-2+β\beta†0.3040.2990.2980.4050.4050.404
GLA†0.3020.3010.3020.4060.4060.405
GLA+β\beta†0.3140.3110.3110.4060.4060.405

Table 14. Zero-shot accuracy for the 340M zoo with INT8 operands and ⟨T,C⟩\langle T, C\rangle scales. The six-task average includes LAMBADA, HellaSwag, ARC-e, ARC-c, PIQA, and WinoGrande, with normalized accuracy for HellaSwag and ARC-c. A dash denotes an unreported stack evaluation. †: no QK normalization.

RecurrenceLAMBADA FPLAMBADA NaiveLAMBADA StackSix-task average FPSix-task average NaiveSix-task average Stack
LA+β\beta0.4650.1690.4080.5070.4470.493
Mamba-2+β\beta0.4800.469–0.5220.521–
GLA+β\beta0.5310.518–0.5440.545–
DeltaNet0.4540.458–0.5050.508–
GDN0.5110.505–0.5350.532–
KDA0.5220.514–0.5400.538–

Table 15. Zero-shot accuracy for the 1.3B zoo with INT8 operands and ⟨T,C⟩\langle T, C\rangle scales. The six-task average includes LAMBADA, HellaSwag, ARC-e, ARC-c, PIQA, and WinoGrande, with normalized accuracy for HellaSwag and ARC-c. A dash denotes an unreported stack evaluation.

Mamba-2 in SSDi8

This appendix reports the SSDi8 experiments in § 5.2. We first verify the baseline and examine transformations within its dataflow, then give the quality comparisons, weight-only reference, and longer-context evaluations. Setup and timing methodology are in Appendix § 10.2; recurrence-only quantization of pretrained hybrids is covered separately in Appendix § 15.

Ablations

Comparison with the official SSDi8 release. Our implementation agrees with the official SSDi8 release within 0.02 WikiText-2 perplexity and 2% full-model prefill latency across Mamba-2 1.3B, 2.7B, and 8B (RTX A5000, B=8B = 8, T=2048T = 2048). Both use value-side absorption, static per-(H,Dv)(H, D_v) scales, no recurrent rotation, and no output mean correction.

For Nemotron-H, where no official SSDi8 implementation is available, we use our matched value-side baseline.

Query–key rotation and diagonal rescaling. Query–key rotation lowers perplexity at both decay endpoints and under both scale policies, with the largest gains at 8B (Table 19). The endpoints remain close in quality after rotation.

Decay onGranularity1.3B None1.3B RQK\mathbf{R}_{QK}2.7B None2.7B RQK\mathbf{R}_{QK}8B None8B RQK\mathbf{R}_{QK}
Valuestatic-(H,Dv)(H, D_v)10.928±0.00410.928 \pm0.00410.860±0.00710.860 \pm0.0079.441±0.0179.441 \pm0.0179.379±0.0119.379 \pm0.0117.620±0.0097.620 \pm0.0097.456±0.0097.456 \pm0.009
Valuedynamic-(H,1)(H, 1)10.894±0.00510.894 \pm0.00510.829±0.01210.829 \pm0.0129.433±0.0069.433 \pm0.0069.371±0.0129.371 \pm0.0127.642±0.0137.642 \pm0.0137.447±0.0087.447 \pm0.008
Keystatic-(H,Dk)(H, D_k)10.925±0.00310.925 \pm0.00310.849±0.00810.849 \pm0.0089.424±0.0169.424 \pm0.0169.367±0.0119.367 \pm0.0117.601±0.0097.601 \pm0.0097.455±0.0067.455 \pm0.006
Keydynamic-(H,1)(H, 1)10.884±0.00810.884 \pm0.00810.828±0.01010.828 \pm0.0109.424±0.0119.424 \pm0.0119.360±0.0049.360 \pm0.0047.634±0.0097.634 \pm0.0097.444±0.0097.444 \pm0.009

Table 19. Mamba-2 W4A8 absorption endpoint and rotation ablation, WikiText-2 perplexity, mean ±\pm std over four calibration sets, random Hadamard RQK\mathbf{R}_{QK}, without DQK\mathbf{D}_{QK}. Static value-side runs omit output-projection mean correction; the other configurations include it.

The exponent sweep favors κ=12\kappa= \frac{1}{2}, with 14\frac{1}{4} close behind (Table 16). Incorporating rescaling into the existing kernel leaves measured layer latency unchanged, whereas a separate elementwise operation adds overhead.

Value rotation. Value rotation alone does not improve this pipeline. With full-precision projections and per-channel value scales, rotated state storage increases perplexity, while canonical storage largely removes that penalty without improving on the unrotated baseline (Table 17). We have not evaluated value rotation combined with rescaling in the full W4A8 pipeline.

Transformation and state basisFP projections
No rotation10.496
RV\mathbf{R}_{V}, rotated storage10.533
RV\mathbf{R}_{V}, canonical storage10.497

Table 17. Value-rotation diagnostics on Mamba-2 1.3B: WikiText-2 perplexity with full-precision projections and an INT8 fixed-point state. Canonical storage is evaluated by fake quantization.

To test the effect of RMSNorm directly, we measure each layer in isolation, excluding error propagated from earlier layers.

Error after normalization. We measure error before and after RMSNorm with full-precision inputs to each layer, excluding propagated error (Table 18). Value rotation slightly lowers error before normalization but increases it afterward. Canonical storage reduces this penalty without eliminating it, even with a full-precision state. The full-precision transformation remains exact.

ModelStateSettingBefore RMSNormAfter RMSNorm
Mamba-2 1.3BINT8No rotation0.01300.0128
RV\mathbf{R}_{V}, rotated storage0.01190.0169
RV\mathbf{R}_{V}, canonical storage0.01190.0158
FPNo rotation0.01260.0123
RV\mathbf{R}_{V}0.01140.0154
Mamba-2 2.7BINT8No rotation0.01360.0136
RV\mathbf{R}_{V}, rotated storage0.01230.0168
RV\mathbf{R}_{V}, canonical storage0.01240.0159
FPNo rotation0.01320.0130
RV\mathbf{R}_{V}0.01190.0153

Table 18. Mean isolated-layer INT8 error before and after RMSNorm, with per-channel value scales. With a full-precision state, the two storage bases coincide.

Figure 13 shows amplification across most layers, not just the final layers. Smaller outputs contribute disproportionately but do not fully explain the pattern. Since value rotation alone gives no quality gain and canonical storage adds inverse transforms, the SSDi8 configuration leaves the value axis unchanged.

Figure 13. Isolated-layer INT8 error before and after RMSNorm, and mean output magnitude. Each layer receives full-precision inputs and uses an INT8 state.

Decay allocation. We compare decay placement along two axes: quantization quality and execution cost. Table 19 compares key- and value-side configurations under static and dynamic scales, with and without query–key rotation. Static value-side runs omit the output-projection mean correction used by the other configurations (Appendix § 10.2), so the static comparison does not isolate decay placement. Figure 14a isolates the affected kernel stage, and Figure 14b varies the operand dimensions.

The endpoints have similar perplexity under the tested scale policies (Table 19). Without rotation, key-side absorption is slightly better in each comparison. Static scales are per channel on the absorbed operand, while dynamic scales are per (chunk, HH, 1) and computed inside the kernel. These results support either placement on quality grounds, consistent with the limited sensitivity to scalar-decay placement at INT8 in the zoo.

Key-side absorption applies head-specific decay to shared keys in registers, avoiding a head-expanded tensor (Appendix § 12.2). Its remaining cost is in the state write, which reloads the absorbed operand before weighting and quantization; other stages are unchanged (Figure 14a).

Value-side absorption is faster at the tested Mamba-2 geometry (Dk=128D_k = 128, Dv=64D_v = 64). Dynamic key scales add another pass over the weighted operand to find its maximum. Wider values can reverse the preferred endpoint at Dk=64D_k = 64, while the key-side penalty persists at Dk=128D_k = 128 (Figure 14b). Thus, both operand dimensions and the accumulation and rescaling layout affect the cost of decay placement.

Quality comparisons and references

Table 20 combines the W4A8 grid with the W4A16 reference and downstream accuracy. The two decay endpoints perform similarly after rescaling.

Weight-only and downstream references. W4A16 keeps activations in FP16 while retaining W4 projections. A substantial perplexity gap remains on the smaller Mamba-2 models even without activation quantization; LinQuant reduces the additional W4A8 gap (Table 20). Each run uses independently quantized weights under the same GPTQ procedure, so this comparison is a reference, not an additive error decomposition.

We report the lower-perplexity of two near-identical W4A16 Mamba-2 runs and one run per Nemotron-H model. On Mamba-2, only the key-side configurations include output mean correction, so the value-side comparison isolates the added transformations. All Nemotron-H configurations include this correction.

The INT8 pipelines remain close to W4A16 on the six short-context tasks, without consistent accuracy gains from the transformations (Table 20). The longer-context retrieval tests below show a clearer benefit.

Hadamard signs. Random Hadamard signs improve most of the query–key rotation comparisons over deterministic signs (Table 21). The main experiments use random signs.

ModelDatasetNo rotationDet.Rand.
Mamba-2 1.3BWikiText-210.93010.88210.857
Mamba-2 1.3BPile7.4157.3977.382
Mamba-2 2.7BWikiText-29.4209.4009.374
Mamba-2 2.7BPile6.5726.5576.555
Mamba-2 8BWikiText-27.6197.4637.467
Mamba-2 8BPile6.2936.2136.206

Table 21. Perplexity with deterministic (Det.) or random-signed (Rand.) query–key Hadamards: W4A8 projections, static value-side decay, no rescaling, one calibration draw.

Longer contexts and retrieval

Context length. Table 22 evaluates Nemotron-H Base up to its full 8192-token context. SSDi8 uses value-side decay without transformations; LinQuant uses RQK+DQKR_{QK} + D_{QK} with key-side decay. The gap to FP16 grows with context length, but LinQuant consistently removes about half of SSDi8’s gap.

WindowFP16SSDi8LinQuantSSDi8LinQuant
20486.0306.3366.1750.3060.145
40965.6786.0285.8430.3500.165
81925.4715.9205.6860.4490.215
Position0–6464–256256–10241024–20482048–40964096–8192
SSDi80.0570.0370.0470.0570.0680.098
LinQuant0.0350.0200.0230.0260.0340.048

Table 22. Nemotron-H Base: WikiText-2 perplexity by context length (top) and NLL gap to FP16 by position within 8192-token windows (bottom). These runs are separate from Table 2.

Error by position. The gap also grows with token position within a sequence, while remaining smaller with LinQuant (Table 22, bottom).

Retrieval. We evaluate seven RULER [11] retrieval tasks at 4k and 8k context, using 100 matched prompts per task and length, greedy generation, and string-match scoring. The tasks cover single-needle, multi-key, and multi-query retrieval. Generation repeatedly runs chunkwise prefill on the growing sequence; cached decoding is not evaluated.

The retrieval gap widens at full context, especially when distinguishing multiple keys. LinQuant recovers much of SSDi8’s lost accuracy (Table 23).

LengthModelS1S2S3MK1MK2MK3MQMean
4kFP16100.0100.0100.0100.0100.0100.096.599.5
W4A16100.0100.0100.0100.0100.0100.096.599.5
SSDi897.0100.099.086.085.095.088.893.0
LinQuant100.0100.0100.093.0100.0100.093.398.0
8kFP16100.0100.0100.098.0100.0100.089.098.1
W4A16100.0100.0100.098.0100.0100.097.599.4
SSDi886.0100.095.040.058.033.085.071.0
LinQuant100.0100.0100.061.094.095.088.391.2

Table 23. RULER accuracy (%) on Nemotron-H Base. S: single-needle; MK: multi-key; MQ: multi-query. W4A16 uses W4 projections and FP16 activations. Prompt lengths are 3.7k–4.0k at 4k (2.1k–2.2k for MK3) and 7.5k–8.0k at 8k.

Pretrained hybrids

This appendix expands the pretrained-model results in § 5.2: individual and combined transformations, lower precisions, kernel performance, and the role of value-channel imbalance.

Setup

Models. We use the public checkpoints of five hybrid language models that interleave linear recurrences with softmax attention. Nemotron-H 8B Base and Reasoning ([4]) contain 24 Mamba-2 mixers among 52 layers, each with 128 heads, Dv=64D_v = 64, Dk=128D_k = 128, and keys shared by groups of 16 heads. Qwen3.5-4B and 9B ([21]) contain 24 Gated DeltaNet layers among 32, and Kimi Linear 48B ((Team et al., 2025)) contains 20 KDA layers among 27. The Qwen3.5 and Kimi Linear recurrences use 32 heads with Dk=Dv=128D_k = D_v = 128 and normalized queries and keys.

What is quantized. Only the recurrence is quantized. Its operands are INT8 with static scales in the ⟨H,C⟩\langle H, C\rangle geometry of § 5.1: queries and read keys per head, and values, write keys, and state per channel. As in the controlled study, each chunk reads an INT8 copy of the state, while the carry between chunks stays in FP32. All other layers keep the BF16 weights and activations of the checkpoint, so the full-precision reference is the unmodified model and every difference to it is caused by the recurrence, unlike in Table 2b, where the whole mixer runs in W4A8. The recurrences run in the INT8 kernels of § 5.2 with chunk length 64 and the Hadamard rotations and diagonal rescalings of the controlled study.

Calibration, selection, and evaluation. Scales and rescalings are calibrated on 8 windows of 2048 tokens from the validation split of the Pile, and every configuration is evaluated on five independent calibration draws. Each job evaluates the naive quantizer and a set of transformations on the same windows and the same GPU, so every transformation is paired with a naive run. Statistics are taken over calibration draws, with repeated runs of the same draw averaged first. The candidates are the single invariances that apply to a recurrence and their compositions (Table 24). The selected stack is the candidate with the lowest mean cross-entropy on 8 further Pile windows, drawn with a sixth seed and shared by all draws, among configurations without inverse state rotations inside the recurrence. This excludes canonical storage when a value rotation is used (Appendix § 15.§ 6), and changes the selection only on Kimi Linear (Appendix § 15.2). The promotion threshold of the controlled study is large relative to these models’ INT8 quantization gaps. We therefore select by minimum held-out cross-entropy rather than requiring a fixed improvement per transformation (Appendix § 10.§ 1). We report perplexity on the WikiText-2 test set with 2048-token windows. The recovered share is the paired perplexity gain of a configuration divided by the gap between naive INT8 and full precision over the same draws.

Invariances one by one

Table 24 reports every candidate at INT8. Six of the seven invariances improve at least one pretrained model with per-channel value scales, alone or within a stack; the exception is diagonal-decay absorption (Appendix § 13.§ 1).

Mamba-2 (Nemotron-H). Query–key rotation provides the largest individual gain on both Nemotron-H models. Query–key rescaling helps the Base model in combination, and key-side scalar-decay absorption further improves the Reasoning model; each addition improves its paired baseline across all five calibration draws. The value axis illustrates a stronger interaction: rotation alone is harmful, but rescaling the values first makes rotated storage accurate and yields substantial recovery. Some combinations nearly match full precision on the WikiText-2 test set. We nevertheless report the combinations selected on held-out Pile data, which prefer key rotation with scalar-decay absorption on the Reasoning model. Recoveries slightly above 100% reflect small test-set differences, not evidence of a systematic improvement over the unquantized model.

Gated DeltaNet (Qwen3.5). The largest gains are on the value axis. A few channels account for most of the squared value magnitude, making rotation alone unreliable, particularly with rotated storage. Applying DV\mathrm{D}_{V} first balances the channel ranges and makes RV\mathrm{R}_{V} effective on both models (Appendix § 15.6). The composition outperforms canonical rotation alone without the additional inverse state transforms. Rescaling alone is absorbed by the per-channel scales. Moving the write gate onto the keys also helps consistently across calibration draws, but adds no further benefit to the selected value-axis combination. Scalar-decay allocation has little effect at INT8 or INT6. Query–key rotation, harmful at INT8 here, becomes the strongest individual transformation at INT4 (Appendix § 15.§ 4).

KDA (Kimi Linear). All tested value-rotation configurations improve on every calibration draw. Their held-out losses are nearly tied, and excluding inverse state rotations inside the recurrence selects RVDV\mathrm{R}_{V}\mathrm{D}_{V} with κ=12\kappa=\frac{1}{2} and rotated storage. Allowing canonical storage gives only a negligible reduction in held-out loss at substantially higher execution cost (Table 28). Rescaling contributes little on this model, consistent with its more balanced value channels. Write-gate and diagonal-decay allocations likewise have little effect at INT8 or INT6. At INT4, a balanced write-gate allocation modestly improves perplexity, while moving the gate fully onto the keys worsens it.

PathB=1B=1481632
Gated DeltaNet kernel (Qwen3.5)
FLA BF161.0052.7375.1659.97019.694
INT8 recurrence0.862 (1.17)2.558 (1.07)4.807 (1.07)9.262 (1.08)18.225 (1.08)
+ RV\mathbf{R}_{V} as two separate passes0.980 (1.03)2.915 (0.94)5.563 (0.93)10.791 (0.92)21.323 (0.92)
+ RV\mathbf{R}_{V} fused into convolution and normalization0.822 (1.22)2.533 (1.08)4.824 (1.07)9.362 (1.06)18.543 (1.06)
KDA kernel (Kimi Linear)
FLA BF161.5604.8599.25618.09835.923
INT8 recurrence1.279 (1.22)4.269 (1.14)8.275 (1.12)16.217 (1.12)32.033 (1.12)
+ RVDV\mathbf{R}_{V}\mathbf{D}_{V} fused, rotated storage1.207 (1.29)4.244 (1.15)8.269 (1.12)16.266 (1.11)32.307 (1.11)
+ RVDV\mathbf{R}_{V}\mathbf{D}_{V}, canonical storage2.654 (0.59)8.475 (0.57)16.589 (0.56)33.009 (0.55)66.168 (0.54)

Table 28. Value convolution through output normalization at Qwen3.5/Kimi Linear shapes (32 heads, Dk=Dv=128D_k=D_v=128): milliseconds, with BF16 speedup in parentheses. This segment differs from Table 27; compare rows here to isolate rotation overhead.

Lower bit widths

Table 25 extends the comparison to INT6 and INT4 and reports paired gains over calibration draws. Every selected INT8 combination improves on every draw. At lower precision, the transformations continue to reduce error but recover a smaller share on the delta-rule hybrids. Query–key transformations become particularly important: they prevent the collapse of naive INT4 quantization on Nemotron-H and provide the largest gains on Qwen3.5. Packed INT4 kernels also reduce runtime (Appendix § 12.§ 4), but none of these models reaches its INT8 accuracy at INT4. INT4 therefore trades some accuracy for lower runtime; INT8 remains the more accurate operating point.

Table caption: Selected stack per model and operand precision (recurrence only, ⟨H,C⟩\langle H, C\rangle static scales, INT8 state readout). Gain: paired perplexity gain over the naive run, mean ±\pm standard deviation over the five calibration draws. Rec.: share of the naive gap recovered. Both are omitted where naive quantization collapses. Draws: calibration draws on which the stack improves on its paired naive run.

ModelOperandsFPNaiveLinQuantGainRec. (%)DrawsSelected stack
Nemotron-H BaseINT86.0356.1626.0540.108 ±\pm 0.010855/5RQK+DQK+RVDV\mathrm{R}_{QK} + \mathrm{D}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}
INT66.0357.0366.1150.921 ±\pm 0.094925/5RQK+DQK+RVDV\mathrm{R}_{QK} + \mathrm{D}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}
INT46.03561847.367––5/5RQK+DQK\mathrm{R}_{QK} + \mathrm{D}_{QK}
Nemotron-H ReasoningINT88.4298.7238.4710.253 ±\pm 0.072865/5RQK\mathrm{R}_{QK}, η=0\eta=0
INT68.42910.1418.5161.626 ±\pm 0.311955/5RQK+DQK\mathrm{R}_{QK} + \mathrm{D}_{QK}
INT48.428562979.785––5/5RQK+DQK\mathrm{R}_{QK} + \mathrm{D}_{QK}
Qwen3.5-4BINT88.0568.1148.0660.048 ±\pm 0.006825/5RVDV\mathrm{R}_{V}\mathrm{D}_{V}
INT68.0568.2178.1270.090 ±\pm 0.004565/5RQK+RVDV\mathrm{R}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}, η=0\eta=0, σ=0\sigma=0
INT48.05613.4039.9073.496 ±\pm 0.082655/5RQK+RVDV\mathrm{R}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}
Qwen3.5-9BINT86.9837.0296.9900.039 ±\pm 0.001845/5RVDV\mathrm{R}_{V}\mathrm{D}_{V}, κ=12\kappa=\frac{1}{2}
INT66.9837.1217.0500.071 ±\pm 0.006525/5RQK+RVDV\mathrm{R}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}
INT46.98311.5399.5751.964 ±\pm 0.102435/5RQK+RVDV\mathrm{R}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}
Kimi Linear 48BINT85.5195.5515.5300.021 ±\pm 0.004655/5RVDV\mathrm{R}_{V}\mathrm{D}_{V}, κ=12\kappa=\frac{1}{2}
INT65.5195.6165.5940.022 ±\pm 0.004235/5RVDV\mathrm{R}_{V}\mathrm{D}_{V}
INT45.5198.2888.0330.255 ±\pm 0.03395/5RVDV\mathrm{R}_{V}\mathrm{D}_{V}, κ=12\kappa=\frac{1}{2}

Table 25. Selected stack per model and operand precision (recurrence only, ⟨H,C⟩\langle H, C\rangle static scales, INT8 state readout). Gain: paired perplexity gain over the naive run, mean ±\pm standard deviation over the five calibration draws. Rec.: share of the naive gap recovered. Both are omitted where naive quantization collapses. Draws: calibration draws on which the stack improves on its paired naive run.

The operand layout of SSDi8

The two parts of Table 2 differ in projection precision and in the recurrence’s quantized dataflow. Table 26 tests an SSDi8-like layout on Nemotron-H while retaining BF16 weights: shared query and

Table caption: Nemotron-H recurrences in the operand layout of SSDi8 with BF16 weights (fake quantization): WikiText-2 perplexity and share of the naive gap recovered (in %), with decay absorbed on the value or key side. Means over three calibration draws.

TransformationBase, value side PPLBase, value side %Base, key side PPLBase, key side %Reasoning, value side PPLReasoning, value side %Reasoning, key side PPLReasoning, key side %
FP6.0356.0358.4288.428
Naive INT86.3576.3328.9878.907
RQK\mathrm{R}_{QK}6.115756.127698.499878.47590
RQK+DQK\mathrm{R}_{QK} + \mathrm{D}_{QK}, κ=0.1\kappa=0.16.106786.120718.492898.46991
RQK+DQK\mathrm{R}_{QK} + \mathrm{D}_{QK}, κ=0.5\kappa=0.56.103796.114738.522838.49586
RV\mathrm{R}_{V}div.div.div.div.div.div.div.div.
RVDV\mathrm{R}_{V}\mathrm{D}_{V}6.33186.30798.97728.9031
RQK+RVDV\mathrm{R}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}6.092826.099788.505868.48488
RQK+RVDV\mathrm{R}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}, canonical storage6.098806.106768.508868.49087
RQK+DQK+RVDV\mathrm{R}_{QK} + \mathrm{D}_{QK} + \mathrm{R}_{V}\mathrm{D}_{V}, κ=0.1\kappa=0.16.081866.090828.479918.46592

Table 26. Nemotron-H recurrences in the operand layout of SSDi8 with BF16 weights (fake quantization): WikiText-2 perplexity and share of the naive gap recovered (in %), with decay absorbed on the value or key side. Means over three calibration draws.

key scales, per-channel value scales, an INT8 fixed-point state pass, and chunk length 128. It uses fake quantization, calibrated on 128 Pile sequences of 512 tokens. This layout has a larger naive quantization gap, most of which query–key rotation recovers. Rescaling improves on rotation in 11 of 12 paired runs, and the value-axis composition adds a further gain on the Base model. Value rotation alone remains harmful. These experiments show that the useful combinations extend beyond our kernel layout, although they do not isolate the effects of scale granularity, state precision, chunk length, and calibration. The value-axis composition is not included in the full W4A8 pipeline.

Latency at the geometry of the hybrids

We time synthetic inputs at pretrained-mixer shapes on an RTX A5000 (T=2048T = 2048, median of three runs). The segments include the recurrence and specified surrounding operations, excluding projections and full-model execution.

Timed segments. Table 27 compares two segments against BF16 FLA. The layer segment includes input preparation through the recurrence output: convolution and SiLU for Mamba-2; additionally QK normalization for GDN, and decay-gate activation for KDA. Quantization is fused into these input passes. Figure 5a and Figure 5b plot this segment. The recurrence call starts from BF16 operands and includes quantization.

PathB=1B = 14816
Mamba-2 (Nemotron-H), layer segment
FLA BF161.1923.3646.67213.099
LinQuant INT80.588 (2.03)1.777 (1.89)3.436 (1.94)6.748 (1.94)
LinQuant INT40.573 (2.08)1.720 (1.96)3.312 (2.01)6.526 (2.01)
Mamba-2 (Nemotron-H), recurrence call
FLA BF160.8462.8445.67011.085
LinQuant INT80.403 (2.10)1.403 (2.03)2.703 (2.10)5.336 (2.08)
Gated DeltaNet (Qwen3.5), layer segment
FLA BF161.1413.1456.07711.933
LinQuant INT8 recurrence, FLA input preparation1.141 (1.00)2.926 (1.07)5.689 (1.07)11.175 (1.07)
LinQuant INT8 recurrence, fused input preparation, BF16 outputs0.775 (1.47)2.503 (1.26)4.868 (1.25)9.550 (1.25)
LinQuant INT80.718 (1.59)2.328 (1.35)4.575 (1.33)8.894 (1.34)
LinQuant INT40.707 (1.61)2.225 (1.41)4.380 (1.39)8.525 (1.40)
Gated DeltaNet (Qwen3.5), recurrence call
FLA BF160.6462.1064.0597.889
LinQuant INT80.607 (1.06)1.903 (1.11)3.639 (1.12)7.038 (1.12)
LinQuant INT40.574 (1.13)1.745 (1.21)3.298 (1.23)6.432 (1.23)
KDA (Kimi Linear), layer segment
FLA BF161.6345.0169.89519.569
LinQuant INT81.219 (1.34)4.372 (1.15)8.594 (1.15)16.961 (1.15)
LinQuant INT41.196 (1.37)4.214 (1.19)8.244 (1.20)16.319 (1.20)
KDA (Kimi Linear), recurrence call
FLA BF161.1194.1108.03115.931
LinQuant INT81.005 (1.11)3.568 (1.15)6.982 (1.15)13.647 (1.17)
LinQuant INT40.976 (1.15)3.449 (1.19)6.666 (1.20)13.073 (1.22)

Table 27. Latency at pretrained-model mixer shapes (ms). Parentheses give speedup over BF16 FLA on the same segment. All rows include input quantization; reported INT4 paths use packed operands.

Mamba-2 at the Nemotron-H shape. At the Nemotron-H shape, our INT8 path roughly halves the time of FLA’s BF16 layer segment, and packed INT4 provides a further modest reduction (Table 27). The recurrence alone is also faster with input quantization included. Our kernel reads shared keys directly, without expanding them for each head.

Gated DeltaNet and KDA at the Qwen3.5 and Kimi Linear geometry. GDN benefits from both integer recurrence products and fused input preparation, with the latter contributing more to the layer-segment speedup (Table 27). Fusing convolution, activation, normalization, and quantization removes separate passes. Packed INT4 further reduces runtime.

KDA gains less because chunk preparation must apply channel-dependent decay and quantize the resulting operands, offsetting part of the faster state and output computation.

The falling GDN and KDA speedups with batch size in Figure 5b reflect the changing contribution of launch overhead. Fusing FLA’s separate input-preparation kernels saves a larger fraction of runtime at small batches. As the batch grows, computation dominates and this relative saving shrinks; both paths remain faster than BF16. Table 27 separates these preparation gains from the recurrence-call speedup.

Value rotation without an extra pass. We use the fusion described in Appendix § 12.2. Table 28 shows why it matters: separate rotation passes erase GDN’s gain at larger batches, whereas fusion preserves a speedup.

With fusion and rotated storage, both delta-rule paths remain faster than BF16. Canonical storage adds inverse transforms and, for KDA, requires unfused chunk preparation and a separate output-rescaling pass. We therefore exclude it from production selection (Appendix § 15.§ 1).

Chunk preparation in floating point. The delta rule prepares each chunk by forming the key Gram matrix, solving a unit lower-triangular system, and multiplying the result onto keys and values to obtain W[i]\mathbf{W}_{[i]} and U[i]\mathbf{U}_{[i]} (Appendix § 12). This preparation accounts for over a third of FLA’s Gated DeltaNet recurrence time at the Qwen3.5 shape, limiting the gain from the remaining integer products.

Quantizing these preparation steps changes perplexity little but usually slows the recurrence (Table 29). Requantization adds work, and forward substitution remains sequential. We therefore retain floating-point preparation. The solve variants either merge four 16×1616 \times16 blocks, also invert these blocks on integers, or use forward substitution over all 64 chunk positions.

Integer stepGDN 1.3BQwen3.5-9BTime
Floating-point preparation9.7266.9911.00
Gram matrix9.7276.9911.01–1.02
Solve: block merges9.7276.9901.04–1.05
Solve: block merges and diagonal blocks9.7276.9901.05
Solve: forward substitution over the chunk9.7286.9921.58–1.63
Products forming W and U9.7286.9921.16–1.19
Pseudo-values9.7276.9920.97–0.99
Carry9.7276.9911.11–1.14
Decay and gate factors9.7276.9900.99
Local scores9.7276.9911.04–1.05
All steps, solve by block merges9.7276.9911.25–1.28
All steps, block merges and diagonal blocks9.7286.9931.29–1.33
All steps, forward substitution9.7276.9911.79–1.86

Table 29. Quantizing one additional operation at a time in GDN: WikiText-2 perplexity and runtime relative to floating-point preparation (RTX A5000, B=4,8,16B = 4, 8, 16; above 1: slower). Operands and state are INT8. FP perplexities: 9.727 and 6.983.

Value channels and the value-axis transformations

Table 30 compares value rescaling and rotation in the controlled models and pretrained hybrids at INT8. It adds their composition to the 1.3B zoo and reports statistics of the state-write value operands.

Table 30. Value-axis transformations at INT8 with per-channel scales: 1.3B controlled models (⟨T,C⟩\langle T, C\rangle, one calibration draw) and pretrained hybrids (⟨H,C⟩\langle H, C\rangle, means over five draws). Top: operand statistics defined in the text, measured on the first draw; rotation columns give the change in SQNR (dB). Bottom: WikiText-2 perplexity with rotated state storage and naive excess NLL (Gap, 10−310^{-3} nats). Parentheses give the rescaling exponent κ\kappa; div.: diverged. NH: Nemotron-H.

Value-operand statistics
ModelTop-4 (%)IISQNR+RVR_V+RVDVR_V D_V (1)
LA504.034.8+2.8+3.6
LA+β\beta504.531.5+3.9+4.1
Mamba-2292.031.2+1.7+1.5
Mamba-2+β\beta352.229.4+1.6+1.3
GLA252.136.6+2.9+3.1
GLA+β\beta232.034.8+2.5+2.7
DeltaNet212.730.0+1.4+2.5
GDN232.128.0+1.5+1.6
KDA182.332.7+3.7+4.1
NH Base752,76622.2+1.8+2.9
NH Reas.731,94322.4+1.7+2.6
Qwen 4B741833.5+2.6+2.6
Qwen 9B683433.0+2.8+3.0
Kimi 48B334.429.6+5.8+7.0
WikiText-2 perplexity and naive excess NLL
ModelGapFPNaiveRVR_VRVDVR_V D_V (12\frac{1}{2})RVDVR_V D_V (1)
LA11810.15311.42915.74512.31612.107
LA+β\beta4610.27610.75917.73011.47411.046
Mamba-23.59.7759.8099.8079.8059.822
Mamba-2+β\beta1.99.7559.7749.7649.7659.771
GLA0.99.0619.0699.0669.0659.065
GLA+β\beta1.79.1899.2049.1939.1939.195
DeltaNet1.99.9179.9369.9289.9249.926
GDN0.49.7279.7319.7289.7269.729
KDA0.99.4609.4689.4679.4679.470
NH Base216.0356.162div.6.1316.120
NH Reas.348.4298.723div.8.5708.569
Qwen 4B7.28.0568.1148.0818.0658.066
Qwen 9B6.66.9837.0297.0306.9906.991
Kimi 48B5.85.5195.5515.5315.5305.532

Room to recover. Most controlled models are nearly lossless at INT8 with per-channel values, leaving little room for end-to-end improvement. Nevertheless, rotation and its composition with rescaling improve value-operand SQNR in both the zoo and pretrained hybrids (Table 30).

Why the rotation needs the rescaling. A simple rounding-noise model helps interpret these observations. Approximating errors as independent and uniform within each quantization bin, per-channel scales give each value channel noise variance proportional to its squared range pc2p_c^2. If the rotated channels have similar quantization-noise variance, un-rotation spreads this noise approximately evenly across the original channels. If it lowers the total noise of the head by a factor GG, the noise of channel cc changes by pˉ2/(Gpc2)\bar{p}^{2}/(G p_{c}^{2}), with the mean taken over the channels of the head. Averaged over channels, this factor is I/GI/G, where I=pˉ2⋅pˉ−2≥1I = \bar{p}^{2} \cdot\bar{p}^{-2} \ge1 is the ratio of the arithmetic to the harmonic mean of the squared ranges: the less balanced the ranges, the more noise the rotation moves into the small channels while it lowers the total. DVD_V divides each channel by its range raised to κ\kappa before the rotation. At κ=1\kappa= 1 the rotation mixes channels of equal range, and undoing the rescaling approximately restores noise proportional to pc2p_c^2. This model describes operand rounding; it does not account for propagated state error or guarantee an end-to-end gain.

The measurements follow this account: rescaling helps when a few channels dominate, as in Nemotron-H and Qwen3.5, and matters less when channels are balanced, as in Kimi Linear. State storage also matters: in LA, canonical storage largely removes the remaining rotation penalty.

Figure 15 shows how this happens across the three pretrained recurrence families. In Gated DeltaNet, rotation reduces error in the largest channels but redistributes it into smaller channels, where the relative error can increase sharply. Rescaling first balances the channel ranges; the median relative-error curve then stays below naive quantization across all channel ranks. Kimi Linear shows the same benefit from rescaling, despite its more balanced channels. On Nemotron-H, rescaling removes the large penalty in small channels, but the resulting relative error remains close to the naive baseline. Here, relative quantization MSE is E[(Q(vc)−vc)2]/E[vc2]\mathbb{E}[(Q(v_c)-v_c)^2]/\mathbb{E}[v_c^2] for channel cc.

Figure 15. Value channels of the state write in the recurrent layers of Qwen3.5-9B (Gated DeltaNet), Nemotron-H Base (Mamba-2), and Kimi Linear (KDA) at INT8 with static per-channel scales, on eight calibration windows. Channels are ranked by maximum absolute calibration activation, largest first. Left: channel range over the geometric mean of the head, before and after DVD_{\mathrm{V}}. Right: relative quantization MSE of each original channel for naive quantization, RVR_{\mathrm{V}} alone, and DVD_{\mathrm{V}} before RVR_{\mathrm{V}}. Lines: median over the heads of all recurrent layers (768, 3072, and 640 heads); bands: interquartile range.

Agreement between the groups. DeltaNet and Kimi Linear have similarly balanced value channels, and the composition narrows their INT8 perplexity gaps. At INT4, rescaling before rotation (κ=12\kappa= \frac{1}{2}, rotated storage) improves on naive quantization in rotation alone in all nine tested control models. On DeltaNet, neither transformation helps alone. These results connect the controlled and pretrained studies despite their different quantization gaps.

Operand statistics. For each head, Top-4 is the percentage of the sum of squared values contributed by its four largest channels. The imbalance II is defined above and averaged over heads and layers. We estimate the value operand’s signal-to-quantization-noise ratio (SQNR) under uniform rounding noise as

SQNR=12qmax⁡2∑cmc∑cpc2,\mathrm{SQNR} = 12q_{\max}^{2}\frac{\sum_c m_c}{\sum_c p_c^{2}},

where pcp_c and mcm_c are the range and mean square of channel cc, and qmax⁡=127q_{\max} = 127. We average the inverse ratio over layers before converting to decibels. Table 30 reports this estimate and its change under rotation and rescaling followed by rotation.

References

  1. [1]Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, Tarek Dakhran, Song Duong, Alfred Eng, Fernando Fernandes, Marc Härkönen, Anne Harrington, Ramin Hasani, Saniya Karwa, Yuri Khrustalev, Maxime Labonne, Mathias Lechner, Valentine Lechner, Simon Lee, Zetian Li, Noel Loo, Jacob Marks, Edoardo Mosca, Samuel J. Paech, Paul Pak, Rom N. Parnichkun, Alex Quach, Ryan Rogers, Daniela Rus, Nayan Saxena, Bettina Schlager, Tim Seyde, Jimmy T. H. Smith, Aditya Tadimeti, and Neehal Tumma. Lfm2 technical report, 2025. URL https://arxiv.org/abs/2511.23404.
  2. [2]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated llms. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/b5b939436789f76108b9d0da5e81a f7c-Abstract-Conference.html.arxiv.org/abs/2404.00456
  3. [3]Maximilian Beck, Korbinian Pöppel, Markus Spannring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, 2024. URL https://arxiv.org/abs/2405.04517.
  4. [4]Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficĕk, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, et al. Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models. arXiv preprint arXiv:2504.03624, 2025.arxiv.org/abs/2504.03624
  5. [5]Makram Chahine, Philipp Nazari, Daniela Rus, and T Konstantin Rusch. The curious case of in-training compression of state space models. arXiv preprint arXiv:2510.02823, 2025.arxiv.org/abs/2510.02823
  6. [6]Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, and Diana Marc̆ulescu. Quamba: A post-training quantization recipe for selective state space models. arXiv preprint arXiv:2410.13229, 2024.arxiv.org/abs/2410.13229
  7. [7]Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, Mohamed S. Abdelfattah, and Diana Marc̆ulescu. Quamba2: A robust and scalable post-training quantization framework for selective state space models, 2025. URL https://arxiv.org/abs/2503.22879.
  8. [8]Tri Dao and Albert Gu. Transformers are smms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024.arxiv.org/abs/2405.21060
  9. [9]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTO: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcBBPnfWxS.arxiv.org/abs/2210.17323
  10. [10]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=tEYskw1VY2.arxiv.org/abs/2312.00752
  11. [11]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?, 2024. URL https://arxiv.org/abs/2404.06654.
  12. [12]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html.arxiv.org/abs/2006.16236
  13. [13]Hyunwoo Kim, Byoungchan Ko, Minseok Kang, Minwoo Kim, Dongjin Lee, Jaehoon Lee, Sungho Yoon, and Dahuin Jung. Ssd8: Accurate and efficient 8-bit quantization for state space duality. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2608.21952
  14. [14]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device llm compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, 2024. URL https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452c8fa96dd64e9ba4aa95cc1ef21-Abstract-Conference.html.DOI
  15. [15]Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. QServe: W4A8KV4 quantization and system co-design for efficient llm serving. In Proceedings of Machine Learning and Systems, volume 7, 2025. URL https://proceedings.mlsys.org/paper_files/paper/2025/hash/fbe2bf74a42ece8070d8bf07317d7a6-A bstract-Conference.html.
  16. [16]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: Llm quantization with learned rotations. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=qoQ6DGE6FZ.arxiv.org/abs/2405.16406
  17. [17]Philipp Nazari and T Konstantin Rusch. On state reduction in linear attention. arXiv preprint arXiv:2602.04852, 2026.arxiv.org/abs/2602.04852
  18. [18]NVIDIA. Nvidia nemotron 3: Efficient and open intelligence, 2025. URL https://arxiv.org/abs/2512.20856.
  19. [19]Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, and Stefano Massaroli. Quantifying memory utilization with effective state-size, 2025. URL https://arxiv.org/abs/2504.19561.
  20. [20]Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaj u Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. Rwkv-7 "goose" with expressive dynamic state evolution, 2025. URL https://arxiv.org/abs/2503.14456.
  21. [21]Qwen Team. Qwen3.5. https://huggingface.co/Qwen/Qwen3.5-4B-Base, 2026.
  22. [22]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers, 2021. URL https://arxiv.org/abs/2102.11174.
  23. [23]Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision, 2024. URL https://arxiv.org/abs/2407.08608.

Paper details

vlnc:W_ct3hxfxw
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents