WorksComputer Science
Quantizing Recurrent Language Models Is Easier Than You Think
Abstract
Linear recurrences are becoming an increasingly common building block in large language models, replacing the growing KV cache of softmax attention with a fixed-size recurrent state that enables efficient decoding. However, the quantization stack has largely been developed around softmax Transformers, whose numerical properties and inference state differ from those of recurrent models. Our goal is to understand these differences and study the design space of quantization methods for recurrent models. In our analysis, we differentiate between two targets. First, extra-state quantization concerns weights and activations outside the recurrent state, whose distributions are shaped indirectly by the sequence mixer during training. In controlled 1.3B parameter models, we find that decayed recurrences develop weaker residual outlier channels than softmax attention and are therefore easier to quantize without incoherence processing. Second, intra-state quantization concerns the recurrent state itself during decode, where repeated updates create a different numerical problem from KV cache quantization. We instantiate our findings as LoReS, a state quantization pipeline combining value-axis scale grouping to match output channel magnitude, decay-based mixed-precision to protect slowly decaying heads or channels, and stochastic rounding to limit error accumulation under repeated requantization. Across Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B, LoReS supports robust 8-bit state quantization and lower effective precision on shorter decode horizons. We implement LoReS in vLLM and demonstrate end-to-end decode speedups.
github.com/camail-official/LoReS
Introduction
Softmax attention has shaped the architecture of large language models, as well as the deployment stack built around them. This is particularly true for post-training quantization (PTQ). A large body of prior work investigates quantization of the weights and activations of softmax Transformers [16, 47, 3, 25], while a complementary line studies quantization of the key-value cache that grows during autoregressive generation [26, 20, 15].
Concurrently, the space of sequence mixers used in foundation models is expanding. Linear attention and state-space models replace softmax attention with linear recurrences [22, 14, 49, 51, 43], and recent large-scale hybrid models interleave recurrent and softmax layers within the same architecture [4, 39, 32]. As these hybrid models become increasingly common, the inference stack developed for softmax Transformers must be adapted accordingly.
Compared with softmax attention, however, the behavior of recurrent language models under quantization remains poorly understood. Existing work has begun to adapt quantization to recurrent architectures (Chiang et al., 2025b;a; Xu et al., 2025; Kim et al., 2026; Chandiramani et al., 2026), but has focused on particular architectures and quantization targets. In this work, we instead ask the more general question: how does replacing softmax attention with a linear recurrent sequence mixer change the quantization problem? We argue that this replacement matters along two distinct axes.
Table 1. State-update rules for recurrent models in the mixer-zoo design space. Colors identify decay, write gating, and delta correction.
First, the choice of sequence mixer changes the training dynamics of the model and therefore the distributions of the weights and activations learned throughout the network. We refer to quantization of these tensors as extra-state weight/activation (W/A) quantization. Second, softmax attention and recurrent mixers maintain fundamentally different objects during inference. Softmax attention stores a growing cache of keys and values, whereas a recurrent mixer repeatedly updates a fixed-size state in place. We refer to quantization of this object as (intra-)state quantization. Together, these two settings separate the implicit effect of the sequence mixer on the rest of the model from the explicit effect of quantizing the mixer state itself. Our main contributions can be summarized as follows:
For extra-state W/A quantization, we show that decayed linear recurrences develop substantially weaker residual outlier channels than softmax attention, making them more robust to W/A quantization before incoherence processing. Incoherence processing largely closes this gap.
For intra-state quantization, we analyze how scale grouping, recurrent decay, rounding rule, and storage format impact quantization error, and reveal failure modes of random rotations.
We validate these findings in controlled 1.3B-parameter experiments, and demonstrate their transfer to the hybrids Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B. We instantiate our findings as LoReS (Low-bit Recurrent State), combining value-axis grouping, stochastic rounding and decay-based mixed precision for integer state quantization.
We implement LoReS in vLLM, demonstrating end-to-end decode speedups at scale.
Background and preliminaries
Softmax attention. Given queries , keys , and values , softmax self attention (Vaswani et al., [45]) computes
During autoregressive generation, the model stores previously computed keys and values in a key-value (KV) cache and reuses them at subsequent decode steps. As a result, the sequence mixer state grows linearly with sequence length (Figure 1).

Figure 1. We consider two quantization targets. Extra-state quantization acts on weights and activations outside the mixer state, while intra-state quantization acts on the state used during inference. A recurrent state is repeatedly updated and requantized in place, whereas entries in a KV cache are quantized once and subsequently reused.
Linear attention. Linear attention replaces the nonlinear softmax operator with an unnormalized linear kernel (Katharopoulos et al., [22]; Schlag et al., [34]). In its simplest form, associativity allows keys and values to be summarized into a fixed-size matrix-valued state , satisfying
Unlike the KV cache, this recurrent state has constant size and is updated in place after every token. Modern recurrent sequence mixers refine this basic recurrence through mechanisms such as forgetting, gated writes, and delta-rule corrections (Dao & Gu, [14]; Yang et al. [49, 51]; Team et al. [39]). To isolate these architectural choices, we construct a mixer zoo in which the sequence mixer varies while the surrounding layer design and training recipe are fixed. In particular, all models share the backbone around the mixer, including the query, key, and value projections, causal convolutions, normalization, RoPE (Su et al., [36]), and output projection, apart from the explicitly varied query/key normalization (which we denote by ). The recurrent variants in the zoo follow
where , , and the vector has entries . In our notation, the indicator controls delta correction. The recurrence composes three mechanisms:
Decay is a multiplicative forget gate applied to the state before each write. A scalar gate defines one decay rate per attention head, as in Mamba-2 [14] and Gated DeltaNet [51], while a diagonal gate defines a separate rate per key channel, as in Gated Linear Attention [49] and Kimi Delta Attention [39].
Write gating controls the strength of the outer-product update .
Delta correction, introduced in DeltaNet [34, 50], applies a rank-one correction along the current key through before writing the new association.
Table 1 classifies linear attention models using this taxonomy. Our mixer zoo additionally contains softmax attention with and without query/key RMSNorm [53].
Quantizing Linear Attention
Figure 1 distinguishes two quantization targets. Extra-state quantization concerns the weights and activations (W/A) outside the mixer state, while intra-state quantization concerns the state maintained during decode. The distinction between quantizing softmax attention and linear recurrences is implicit in the former, through the representations learned during training, and explicit in the latter, through the object that is stored and updated during inference.
Extra-state quantization. Softmax and recurrent models expose the same class of weights and activations to a W/A quantizer, so the same general PTQ machinery can be applied to both [16, 47, 3, 25]. However, changing the sequence mixer can change the distributions of these tensors learned during training. Prior work on Mamba models has consequently developed architecture-specific treatments for their activation statistics (Chiang et al., 2025b;a; Xu et al., 2025).
We instead study this dependence directly: using a common quantization framework and controlled architecture, § 3 asks how the sequence mixer itself changes W/A quantization.
Intra-state quantization. Here the distinction between sequence mixers is explicit. KV cache quantization methods are designed around keys and values that are appended once and then reused. They exploit this structure through choices such as separate scaling and grouping for keys and values, outlier treatment, channel reordering, and a recent token window kept at higher precision [26, 20, 15]. A linear recurrence instead stores a fixed-size matrix that is updated after every decoded token. Quantizing this state therefore requires repeatedly dequantizing, updating, and requantizing the same object, allowing quantization error to propagate temporally through subsequent updates. Previous work on recurrent state quantization during decode has focused primarily on Mamba models. Q-Mamba [40] develops scaling methods that account for channel distributions in quantized Mamba states, while Quamba2 [9] stores states in INT8 during generation. Nemotron 3 Super [6] studies low precision state storage using stochastic rounding. SSDi8 [23] considers a related but different target, quantizing the chunked Mamba-2 SSD computation used during prefill to maintain a low-precision execution path.
Concurrent work proposes state-quantization methods using decay-based mixed precision [54, 52], optimized scaling [52], and reduced requantization frequency [30]. Our work instead focuses on a broader analysis of quantization in linear recurrent language models, covering both extra-state W/A and intra-state quantization. § 4 studies how scale grouping, recurrent decay, and repeated requantization influence quantization error. It furthermore reveals failure modes of random rotations and explores integer versus floating-point formats.
Extra-State Weight and Activation Quantization
Experimental setup. In this section, we investigate how the choice of sequence mixer affects W/A quantization using a 1.3B parameter mixer zoo (training recipe in Appendix § 9.§ 1). We apply the same quantization stack to every model: GPTQ [16] with per-output-channel weight scales and dynamic activation scales [25] (details in Appendix § 9.§ 2). Following rotation-based PTQ methods [3, 25], we evaluate three orthogonal rotations to reduce activation outliers. R₁ rotates the residual stream, R₄ rotates the input to the MLP down projection, and Rₒ the input to the mixer’s output projection. We report relative WikiText-2 [28] perplexity changes, with absolute perplexities in Appendix § 11.§ 1.
Decayed recurrences are robust. Without rotations, we find that decayed linear recurrences are more robust to 4-bit activation quantization than their softmax counterpart (Table 2). Holding the other recurrent degrees of freedom fixed, decay consistently improves robustness, with diagonal decay performing best. Applying the full rotation stack reduces W4A4 degradation to a similar level across the zoo, suggesting that the advantage of decayed recurrences arises primarily from the representation they learn. The individual rotations further localize this difference: R₁, which acts on the residual stream, is more effective for softmax models. In decayed recurrences, R₄ becomes relatively more important. This motivates the following residual stream analysis:
| Architectural choices | No rotation | With rotation (W4A4) | |||||||
| Decay | ∥QK∥ | β | Δ | W4A8 | W4A4 | R₁ | R₄ | R₁R₄Rₒ | |
| − | − | − | − | 5.8 | 98.4 | 36.7 | 47.9 | 10.6 | |
| − | ✓ | − | − | 5.9 | 92.3 | 37.8 | 46.5 | 11.1 | |
| − | ✓ | − | − | 6.8 | 111.4 | 48.1 | 54.0 | 12.5 | |
| − | ✓ | ✓ | − | 6.9 | 82.4 | 47.2 | 35.9 | 12.4 | |
| − | ✓ | ✓ | ✓ | 6.1 | 68.2 | 46.5 | 28.5 | 12.7 | |
| Scal | ✓ | − | − | 6.0 | 56.6 | 40.3 | 24.7 | 11.9 | |
| Scal | ✓ | ✓ | − | 5.5 | 53.5 | 41.3 | 21.3 | 11.3 | |
| Scal | ✓ | ✓ | ✓ | 6.0 | 55.4 | 43.2 | 21.5 | 11.8 | |
| Diag | ✓ | − | − | 4.8 | 45.0 | 32.6 | 18.3 | 9.3 | |
| Diag | ✓ | ✓ | − | 5.4 | 49.2 | 35.7 | 20.7 | 10.6 | |
| Diag | ✓ | ✓ | ✓ | 5.8 | 47.8 | 37.0 | 19.5 | 11.2 |
Table 2. Relative perplexity change (%) on WikiText-2 in the 1.3B mixer zoo. The first two rows use softmax attention, ∥QK∥ denotes QK-norm. Subsequent rows are recurrent mixers. Rotation columns use W4A4.
Residual stream outliers. Figure 2 shows that softmax attention develops a dominant residual channel whose magnitude is approximately constant across tokens (and layers, Figure 9b). Decayed recurrences exhibit weaker outliers. Across the zoo, residual stream kurtosis closely tracks unrotated W4A4 degradation (Figure 9a), showing that models with stronger residual outliers are harder to quantize. We next test how strongly the models rely on this channel (Table 3). Removing it before the query and key projections sharply increases perplexity in the softmax models, while removing it for the value projection is less detrimental.
Figure 2. Residual stream of a representative layer.
| Architectural choices | removed | mean | |||||
| Decay | QK | V | QKV | QKV | |||
| − | − | − | − | 1691.3 | 7.6 | 2765.2 | 7.9 |
| − | ✓ | − | − | 419.3 | 9.8 | 389.3 | 2.7 |
| Diag | ✓ | ✓ | ✓ | 2.2 | 1.1 | 9.9 | 1.7 |
Table 3. Removing or replacing the dominant channel by its mean for QK, V, or the mixer input. Reported as relative WikiText-2 perplexity changes (%). The first two rows use softmax attention.

Figure 9. Subfigure (a) relates residual-stream kurtosis to unrotated W4A4 perplexity degradation. Legend entries denote decay///, as in Table 2. Subfigure (b) shows the mixer-input magnitude over 128 tokens.
Replacing the channel at the mixer input by its mean over tokens is far less harmful than removing it entirely. This approximately constant component thus provides a useful bias to the queries and keys. Decayed recurrent models are substantially less sensitive to the same intervention. Prior work connects large activation channels to attention sinks and the ability of softmax attention to concentrate probability mass on particular tokens [37, 38, 8]. We hypothesize that decayed recurrences have less need to develop the same residual structure because decay already provides a mechanism for focusing on a subset of tokens. We leave further investigation to future work.
State Quantization
We next study low precision storage of the recurrent state while keeping state updates in full precision. Our evaluation covers five 1.3B mixer-zoo models and the hybrids using teacher-forced decode over eight sequences of 2048 WikiText-2 tokens. Our core metric is the median token-level increase in cross-entropy relative to a full precision 32 bit trajectory. Appendix § 14.§ 1 shows that correlates with generation accuracy, supporting its use as a useful proxy loss. We find that allows for efficient iteration, making it a practical metric for the study of future quantization methods. Throughout this section, we refer to zoo variants by the architectures they model.
We organize our analysis around properties that distinguish recurrent state quantization from KV cache quantization. First, the state has two channel axes, whereas a KV cache has one channel axis and one time axis. § 4.1 addresses how this affects scale grouping. Second, linear recurrences typically employ a decay factor that suppresses error accumulation over time. § 4.2 uses this structure to allocate precision according to decay strength. Third, each KV cache entry is written and quantized once, whereas the recurrent state is dequantized, updated, and requantized after every decode step. § 4.3 studies how the rounding rule affects this accumulation. Finally, § 4.4 reveals failure modes of random rotations under repeated state quantization, and § 4.5 compares integer and floating point formats. The proofs of all statements are in Appendix § 10.
For the analysis below, we keep the keys, values, and gates fixed and let denote the difference between the quantized and full-precision states. By Equation (1), this error consists of a propagated component and fresh quantization error ,
Figure 3(a) shows how increases sharply as the state is quantized more often, demonstrating decode is the most sensitive regime.

Figure 3. (a) under INT6 end-of-chunk state quantization with chunk size . (b) Relative readout error under INT8 value- and key-axis grouping, against the row’s size relative to the largest row of its head. (c) Surviving fraction of a single INT4 error injected at . Slow, mid, and fast denote heads at the 98th, 85th, and 60th percentile of mean decay. (d) Distribution over heads of the remaining error for Mamba-2 under INT6 value-axis grouping with RTN and SR (the latter averaging eight draws).
Scale grouping
We compare two choices for scale grouping of the state : value-axis grouping, which assigns one scale to each value row, and key-axis grouping, which assigns one scale to each key column. Both reduce degradation relative to using a single scale for the entire state (see Table 12 in Appendix § 12). Towards comparing key- and value-axis grouping, consider the readout error introduced by a single quantization step. At bits, let and let group have quantization scale . We approximate errors as independent and uniform on .
| model | state | value | key |
| Linear | 2.358 | 1.230 | 0.967 |
| Mamba-2 | 0.516 | 0.027 | 0.052 |
| GLA | 0.377 | 0.005 | 0.034 |
| GDN | 0.158 | 0.004 | 0.025 |
| KDA | 0.361 | 0.004 | 0.033 |
Table 12. Median cross-entropy increase under INT6 with per-state, value-axis, and key-axis scaling.
Proposition 1 (Per-channel readout error). Define the readout error . Let and denote the row and column ranges and . The expected per-channel squared readout errors for value- and key-axis grouping are, respectively,
To interpret this result, recall that each value row of the state produces one output channel. Under value-axis grouping, the expected squared error in that output channel is proportional to the square of the row’s largest absolute entry. Under key-axis grouping, every channel has the same expected squared error, determined by the column ranges and the query. Large entries in one row can therefore set coarse quantization steps for other rows. Figure 3(b) supports this view empirically.
Value grouping is robust. Table 4 compares the two grouping axes. Value-axis grouping is preferred for every model at INT8 and for all models except Linear at INT6. At INT4, errors increase substantially, indicating that the bit budget is insufficient. This also exposes a limitation of the local approximation above, which does not consider error accumulation from repeated requantization (a phenomenon that is exacerbated in low-bit regimes). We discuss and mitigate this effect in § 4.3.
| model | INT8 value | INT8 key | INT6 value | INT6 key | INT4 value | INT4 key |
| Linear | 0.125 | 0.332 | 1.230 | 0.967 | 3.647 | 2.479 |
| Mamba-2 | 0.003 | 0.011 | 0.027 | 0.052 | 0.521 | 0.400 |
| GLA | 0.000 | 0.005 | 0.005 | 0.034 | 0.269 | 0.309 |
| GDN | 0.000 | 0.005 | 0.004 | 0.025 | 0.202 | 0.313 |
| KDA | 0.000 | 0.007 | 0.004 | 0.033 | 0.276 | 0.266 |
| Qwen-3.5 | 0.001 | 0.010 | 0.004 | 0.157 | 0.825 | 1.037 |
| Nemotron-H | 0.001 | 0.019 | 0.010 | 0.183 | 0.763 | 1.227 |
| Kimi-Linear | 0.001 | 0.001 | 0.005 | 0.009 | 0.262 | 0.056 |
Table 4. Median cross-entropy increase for value- and key-axis grouping under RTN.
Decay-Based Mixed Precision
Unlike static entries in a KV cache, a recurrent state can suppress previously introduced quantization error through decay. Unrolling Equation (2) yields the following result.
Proposition 2 (Decaying error). Assume the decay is bounded by for all . Then
This result shows that decay limits error accumulation by exponentially suppressing the contribution of earlier state quantization steps. Figure 3(c) tracks a single injected error and shows faster suppression in faster-decaying heads.
Decay based precision allocation. We use the above observation that decay attenuates injected error to assign more precision to more slowly decaying components. Using a calibration set, we rank heads (or channels for per-channel decayed models) by their average decay and promote the slowest-decaying half from the best INT4 configuration to the best INT6 configuration (Table 4). Table 5 shows that the resulting INT5 effective precision recovers most of the gap from INT4 to INT6.
| model | INT4 | INT5 | INT6 |
| Mamba-2 | 0.400 | 0.033 | 0.027 |
| GLA | 0.269 | 0.009 | 0.005 |
| GDN | 0.202 | 0.005 | 0.004 |
| KDA | 0.266 | 0.005 | 0.004 |
| Qwen-3.5 | 0.825 | 0.006 | 0.004 |
| Nemotron-H | 0.763 | 0.045 | 0.010 |
| Kimi-Linear | 0.056 | 0.006 | 0.005 |
Table 5. Decay-based mixed precision. INT5 promotes the slowest-decaying half of heads (or channels for GLA, KDA and Kimi). Entries are under RTN.
Rounding
Under repeated requantization, errors can reinforce or cancel one another. Unlike round-to-nearest (RTN), stochastic rounding (SR) is conditionally unbiased [19, 13] and can therefore inhibit error reinforcement. Indeed, Chandiramani et al. [6] find that SR helps state quantization in Mamba-2. The following result, obtained by unrolling Equation (2), formalizes this observation:
Proposition 3 (Repeated quantization). For scalar decay and , define . Under RTN and SR, respectively, the error at time satisfies
This result shows that rounding errors from different steps can reinforce one another under RTN. Under SR, these interactions vanish in expectation. Figure 3(d) evaluates the remainder empirically, isolating the interaction terms. It is predominantly positive under RTN, indicating reinforcement, but clusters around zero under SR, consistent with its vanishing expectation.
SR shows benefits in practice. Table 6 compares RTN and SR under value-axis grouping. It shows that SR consistently lowers for all three hybrids at both INT8 and INT6. On the mixer zoo, the results are mixed, likely because SR can introduce larger fresh rounding errors than RTN. We will later see that real decode tasks also favor SR for all three hybrids (Appendix § 14, Table 20).
| model | INT8 RTN | INT8 SR | INT6 RTN | INT6 SR |
| Mamba-2 | 0.0026 | 0.0015 | 0.0270 | 0.0352 |
| GLA | 0.0002 | 0.0001 | 0.0055 | 0.0020 |
| GDN | 0.0000 | 0.0001 | 0.0041 | 0.0048 |
| KDA | 0.0002 | 0.0002 | 0.0042 | 0.0043 |
| Qwen-3.5 | 0.0005 | 0.0001 | 0.0039 | 0.0016 |
| Nemotron-H | 0.0006 | 0.0001 | 0.0105 | 0.0026 |
| Kimi-Linear | 0.0005 | 0.0000 | 0.0046 | 0.0007 |
Table 6. Median cross-entropy increase for RTN and SR with value-axis grouping and symmetric grids. Bold marks best per precision. Table 14 contains more data.
| model | INT8 value RTN | INT8 value SR | INT8 key RTN | INT8 key SR | INT6 value RTN | INT6 value SR | INT6 key RTN | INT6 key SR | INT4 value RTN | INT4 value SR | INT4 key RTN | INT4 key SR |
| Linear | 0.125 | 0.223 | 0.332 | 0.135 | 1.230 | 1.654 | 0.967 | 1.019 | 3.647 | 4.338 | 2.479 | 3.884 |
| Mamba-2 | 0.003 | 0.001 | 0.011 | 0.001 | 0.027 | 0.035 | 0.052 | 0.011 | 0.521 | 0.813 | 0.400 | 0.342 |
| GLA | 0.000 | 0.000 | 0.005 | 0.001 | 0.005 | 0.002 | 0.034 | 0.009 | 0.269 | 0.103 | 0.309 | 0.266 |
| GDN | 0.000 | 0.000 | 0.005 | 0.001 | 0.004 | 0.005 | 0.025 | 0.019 | 0.202 | 0.391 | 0.313 | 0.650 |
| KDA | 0.000 | 0.000 | 0.007 | 0.000 | 0.004 | 0.004 | 0.033 | 0.010 | 0.276 | 0.218 | 0.266 | 0.328 |
| Qwen-3.5 | 0.001 | 0.000 | 0.010 | 0.877 | 0.004 | 0.002 | 0.157 | 7.493 | 0.825 | 2.632 | 1.037 | 8.818 |
| Nemotron-H | 0.001 | 0.000 | 0.019 | 6.513 | 0.010 | 0.003 | 0.183 | 7.574 | 0.763 | 0.520 | 1.227 | 9.453 |
| Kimi-Linear | 0.001 | 0.000 | 0.001 | 0.000 | 0.005 | 0.001 | 0.009 | 0.001 | 0.262 | 0.153 | 0.056 | 0.141 |
Table 14. Median cross-entropy increase for RTN and SR with value- and key-axis grouping and symmetric grids. Bold marks best per precision.
| rounding | Nemotron-H value axis | Qwen-3.5 value axis | Kimi-Linear value axis | Kimi-Linear key axis |
| RTN | 40.3 | 54.0 | 56.1 | 53.9 |
| SR | 48.4 | 79.5 | 62.6 | 60.5 |
| FP32 | 51.2 | 82.3 | 63.6 | 63.6 |
Table 20. AIME avg@32 accuracy (%) at INT8 state with symmetric absmax scales. Value-axis grouping unless stated otherwise.
Rotations
KV cache quantization methods such as QuaRot [3] and SpinQuant [25] use rotations to reduce outliers before quantization. Linear attention with scalar or no decay also admits the required symmetries [29, 44]. Diagonally decayed models, like KDA, require explicit rotations of the key axis before quantization. During decode, the setting of this paper, rotations furthermore influence temporal error propagation. We investigate this effect below.
For value-axis grouping, we denote by a within scale-group rotation of the key-axis. For key-axis grouping, rotates the value dimension. Figure 4(a) compares these within scale-group rotations under SR. While lowers with key-axis grouping, can increase it with value-axis grouping, particularly at low bit widths. LinQuant (Tumma et al., [44]) observes a similar rotation penalty during chunked prefill. During decode, this penalty can be exacerbated by the recurrent state being requantized after every token, not just at the chunk boundaries. We characterize the resulting error accumulation below.
Figure 4. (a) with rotation divided by without it ( for key-axis grouping, for value-axis grouping), under SR. Below one: rotation lowers the error. Data in Table 15. (b) INT4 SR quantization scale over the scale of an auxiliary full-precision (an “oracle”) copy of the recurrence against the decode step for GLA, mean within a head, median over 64 steps, 90th percentile over heads. (c) As (a) for value-axis grouping, with every scale taken from an oracle copy of the state (Appendix § 13).
| model | INT8 value | INT8 + | INT8 key | INT8 + | INT6 value | INT6 + | INT6 key | INT6 + | INT4 value | INT4 + | INT4 key | INT4 + |
| Linear | 0.2229 | 0.0380 | 0.1349 | 0.0299 | 1.6545 | 1.1161 | 1.0191 | 0.5310 | 4.3383 | 4.3309 | 3.8839 | 3.2264 |
| Mamba-2 | 0.0015 | 0.0005 | 0.0005 | 0.0001 | 0.0352 | 0.0412 | 0.0107 | 0.0034 | 0.8127 | 1.1788 | 0.3422 | 0.1511 |
| GLA | 0.0001 | 0.0000 | 0.0005 | 0.0001 | 0.0020 | 0.0019 | 0.0095 | 0.0029 | 0.1026 | 0.4735 | 0.2662 | 0.1129 |
| GDN | 0.0001 | 0.0001 | 0.0007 | 0.0000 | 0.0048 | 0.0108 | 0.0187 | 0.0034 | 0.3911 | 1.3706 | 0.6502 | 0.4049 |
| KDA | 0.0002 | 0.0000 | 0.0005 | 0.0001 | 0.0043 | 0.0026 | 0.0098 | 0.0035 | 0.2183 | 0.8500 | 0.3283 | 0.1758 |
| Qwen-3.5 | 0.0001 | 0.0006 | 0.8767 | 0.0002 | 0.0016 | 0.0236 | 7.4927 | 0.1205 | 2.6319 | 5.4741 | 8.8185 | 7.5207 |
| Nemotron-H | 0.0001 | 0.0000 | 6.5132 | 4.5026 | 0.0026 | 0.0011 | 7.5745 | 7.2219 | 0.5198 | 0.1926 | 9.4528 | 8.9028 |
| Kimi-Linear | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0007 | 0.0015 | 0.0007 | 0.0003 | 0.1533 | 1.1315 | 0.1414 | 0.0484 |
Table 15. Median cross-entropy increase under within-group rotations for SR with symmetric grids. Bold marks best per precision.
Scale feedback loops. The accumulated state error consists of the fresh rounding error and propagated earlier errors (Equation (2)). Following Chee et al. [7], we find random Hadamard rotations reduce one-step quantization error (Appendix § 12.§ 2, Table 16). This benefit need not persist under repeated requantization.
| model | INT8 key | INT8 + | INT8 value | INT8 + | INT6 key | INT6 + | INT6 value | INT6 + | INT4 key | INT4 + | INT4 value | INT4 + |
| Mamba-2 | 0.013 | 0.009 | 0.020 | 0.008 | 0.052 | 0.036 | 0.082 | 0.034 | 0.229 | 0.159 | 0.351 | 0.152 |
| GLA | 0.013 | 0.009 | 0.018 | 0.009 | 0.052 | 0.036 | 0.073 | 0.035 | 0.230 | 0.159 | 0.305 | 0.154 |
| GDN | 0.012 | 0.009 | 0.015 | 0.009 | 0.049 | 0.036 | 0.060 | 0.036 | 0.216 | 0.161 | 0.263 | 0.160 |
| KDA | 0.012 | 0.009 | 0.021 | 0.008 | 0.049 | 0.036 | 0.083 | 0.033 | 0.217 | 0.160 | 0.329 | 0.148 |
| Qwen-3.5 | 0.027 | 0.008 | 0.016 | 0.009 | 0.106 | 0.031 | 0.067 | 0.036 | 0.407 | 0.137 | 0.290 | 0.159 |
| Nemotron-H | 0.016 | 0.007 | 0.014 | 0.009 | 0.064 | 0.030 | 0.057 | 0.035 | 0.264 | 0.132 | 0.236 | 0.157 |
| Kimi-Linear | 0.015 | 0.009 | 0.021 | 0.008 | 0.060 | 0.037 | 0.084 | 0.034 | 0.259 | 0.162 | 0.330 | 0.152 |
Table 16. One-step quantization error of the state: median of over layers, heads, and decode steps under SR.
During decode, quantization scales are derived from an already noisy state. This can lead to growing scales and thus coarsen the quantization grid, which in turn can introduce larger rounding errors and increase the next scale. This feedback loop can outweigh the one-step benefit of rotations. Figure 4(b) illustrates this effect in GLA, where increases scale inflation under value-axis grouping. Figure 11 in Appendix § 13 extends this comparison to other recurrences. In order to isolate the influence of scale inflation on , we carry along a full precision copy of the state. We then derive oracle scales from this full precision state (Appendix § 13). This intervention substantially reduces the additional penalty caused by (Figure 4(c)). However, such oracle scales still leave a rotation penalty in some models:
Figure 11. Quantization scale over the scale of the oracle copy of the state against the decode step, INT4 SR, plain (dashed) and rotated (solid). Median of the per-step mean over the head’s scale groups within 64 steps, then the 90th percentile over heads (24 layers, 384 heads). Value-axis grouping with , key-axis grouping with .
Query alignment. The effect of fresh rounding error on the readout depends on its alignment with the query. For unit-norm queries, we define
with expectations over fresh SR draws. Rotations influence this alignment. Figure 5 showcases the effect at INT4 with oracle scales. For Qwen-3.5, which has a remaining rotation penalty even with oracle scales, unrotated error is often less aligned with the query. Rotation removes this advantage, leaking quantization error into usually suppressed query dimensions.

Figure 5. Query alignment under INT4 SR with oracle scales. Heads are split by whether increases or lowers readout error.
Format
The integer quantization grids considered in the above experiments are uniformly spaced. We compare them with FP8 in E4M3 format, which gives finer spacing near zero and coarser spacing at large magnitudes. Under SR with value-axis grouping, INT8 consistently yields lower than E4M3 (Table 7). With key-axis grouping, E4M3 improves over INT8 for the three hybrids but worsens all four decayed mixer-zero models. Across both axes and formats, value-grouped INT8 is best for almost all models, including all three hybrids.
| value | key | |||
| model | INT8 | E4M3 | INT8 | E4M3 |
| Mamba-2 | 0.0015 | 0.0019 | 0.0005 | 0.0010 |
| GLA | 0.0001 | 0.0007 | 0.0005 | 0.0007 |
| GDN | 0.0001 | 0.0016 | 0.0007 | 0.0013 |
| KDA | 0.0002 | 0.0006 | 0.0005 | 0.0009 |
| Qwen-3.5 | 0.0001 | 0.0002 | 0.8767 | 0.0004 |
| Nemotron-H | 0.0001 | 0.0015 | 6.5132 | 0.0429 |
| Kimi-Linear | 0.0000 | 0.0001 | 0.0002 | 0.0001 |
Table 7. Median cross-entropy increase for INT8 and E4M3 under SR.
| W/A/S | \multicolumn{5}{c}{Qwen-3.5} | \multicolumn{5}{c}{Nemotron-H} | \multicolumn{5}{c}{Kimi-Linear} | ||||||||||||
| GSM8K | MMLU-Pro | GPQA | AIME | RULER | GSM8K | MMLU-Pro | GPQA | AIME | RULER | GSM8K | MMLU-Pro | GPQA | AIME | RULER | |
| 16/16/32 | 95.9 | 82.6 | 80.1 | 82.3 | 94.6 | 92.3 | 70.6 | 54.1 | 51.2 | 88.0 | 89.1 | 70.3 | 63.5 | 63.6 | 93.7 |
| 16/16/8 | 96.4 | 82.2 | 79.6 | 79.5 | 94.7 | 92.0 | 70.1 | 53.5 | 48.4 | 88.3 | 89.2 | 70.2 | 63.8 | 62.6 | 93.8 |
| 8/16/32 | 96.5 | 82.5 | 80.1 | 83.1 | 94.6 | 92.9 | 70.4 | 54.0 | 51.7 | 88.4 | 89.8 | 70.4 | 63.9 | 64.3 | 93.5 |
| 8/16/8 | 96.0 | 82.3 | 79.1 | 80.1 | 94.5 | 92.5 | 70.0 | 53.1 | 49.4 | 88.2 | 89.6 | 70.2 | 61.9 | 62.4 | 94.0 |
| 8/8/32 | 96.8 | 82.4 | 80.3 | 82.1 | 94.6 | 93.3 | 70.3 | 54.8 | 50.9 | 88.4 | 88.9 | 70.1 | 63.3 | 62.3 | 93.7 |
| 8/8/8 | 96.5 | 82.2 | 79.8 | 78.6 | 94.7 | 92.6 | 70.1 | 53.5 | 49.5 | 88.5 | 88.4 | 69.7 | 62.6 | 63.6 | 93.9 |
| 4/8/32 | 97.1 | 80.7 | 75.0 | 73.5 | 94.9 | 90.9 | 67.8 | 49.0 | 43.3 | 85.4 | 86.4 | 65.9 | 53.6 | 56.9 | 93.4 |
| 4/8/8 | 96.1 | 80.1 | 73.6 | 70.9 | 94.8 | 91.8 | 67.0 | 49.7 | 41.0 | 85.1 | 88.9 | 65.5 | 52.6 | 55.2 | 93.3 |
Table 8. End-to-end accuracy (%). W/A/S gives weight, activation, and state precision in bits. GPQA and AIME are avg@32.
End-to-End Evaluation
To demonstrate the practical utility of our findings, we combine extra-state W/A quantization (§ 3) with LoReS, our intra-state quantization (§ 4) recipe. LoReS stores recurrent states on unrotated integer grids with value-axis grouping, symmetric absmax scales, and stochastic rounding. We use INT8 as the default state precision. For lower effective precisions, we use decay-based precision allocation to assign fewer bits to quickly decaying heads or channels. We evaluate the framework on the three large post-trained hybrid models. Appendix § 9.§ 3 gives the complete protocol.
Accuracy. We evaluate 8-shot GSM8K (Cobbe et al., [12]), MMLU-Pro (Wang et al., [46]), GPQA (Rein et al., [33]), and AIME (combining the 2024 and 2025 competitions for 60 questions), together with RULER (Hsieh et al., [21]) at a 32k context. Generation is capped at 8k tokens for GSM8K and 64k for MMLU-Pro, GPQA, and AIME.
Weight quantization uses GPTQ (Frantar et al., [16]) with groups of 128 input channels, except that W8A8 uses per-output-channel weight scales (Appendix § 9.§ 2). We apply W/A quantization with llm-compressor (AI & vLLM Project, [1]) and serve the resulting models with vLLM (Kwon et al., [24]).
Table 8 compares INT8 and FP32 states across W/A precisions. INT8 states preserve most of the accuracy at each W/A setting, while larger losses generally arise in configurations with 4-bit weights. The additional loss from state quantization does not increase consistently as W/A precision decreases, supporting the combination of extra-state and intra-state quantization.
Next, we examine decode length, which helps interpret the variation across benchmarks. Median response lengths follow AIME > GPQA > MMLU-Pro > GSM8K > RULER (Table 22). GSM8K and MMLU-Pro generally show small changes under state quantization. RULER likewise remains stable when state precision is reduced, despite its 32k-token input, because generation is short. GPQA and AIME tend to show somewhat larger losses, consistent with greater sensitivity over longer decode trajectories. We believe that incorporating quantization during training (e.g. via quantization-aware training or low-precision training) could reduce the remaining degradation, but leave further exploration of this to future work.
| benchmark | Qwen-3.5 FP32 | INT8 | (%) | Nemotron-H FP32 | INT8 | (%) | Kimi-Linear FP32 | INT8 | (%) |
| GSM8K | 2096 | 2212 | +5.5 | 592 | 588 | −0.7 | 89 | 88 | −1.1 |
| MMLU-Pro | 4050 | 4152 | +2.5 | 1112 | 1132 | +1.8 | 2080 | 2184 | +5.0 |
| GPQA | 9169 | 9134 | −0.4 | 5449 | 5312 | −2.5 | 7249 | 7356 | +1.5 |
| AIME | 25223 | 24950 | −1.1 | 10994 | 10964 | −0.3 | 11839 | 11516 | −2.7 |
| RULER | 16 | 16 | 0.0 | 21 | 20 | −4.8 | 12 | 12 | 0.0 |
Table 22. Median response length in generated tokens for the FP32-state baseline and state INT8 at W/A = 16/16. is relative to FP32 state.
As discussed in § 4, we use decay-based allocation to reduce state precision below INT8, promoting linear attention heads of Qwen-3.5 and Nemotron-H between adjacent precisions based on decay strength. Figure 6 shows that both models support lower effective precision on GSM8K and GPQA. On GSM8K, this permits 5 effective bits for Qwen-3.5 and 5.5 for Nemotron-H with less than a 5% relative drop in accuracy from the FP32-state baseline.
Figure 6. Decay-ranked promotion. Dotted lines mark the minimum bit budget within 5% of FP32 accuracy. GPQA and AIME are pass@1.
Speedup. We evaluate how state quantization affects decode speed in individual sequence mixer kernels and in end-to-end vLLM serving. Figure 8 shows that INT8 state storage accelerates all three sequence mixer kernels as the decode batch size increases. At a batch size of 8, Mamba-2 more than doubles throughput, while GDN and KDA also improve significantly. Appendix § 15.§ 1 describes the measurement and reports the same experiment on an RTX 6000.
Figure 8. H100 speedup of vLLM mixer kernels using INT8 versus FP32 states.
To evaluate the end-to-end efficiency of LoReS, we measure fixed-batch decoding of Nemotron-H and Qwen-3.5 with vLLM on one H100. We report time per output token (TPOT), excluding prefill. Appendix § 15.§ 2 gives the complete measurement protocol. Appendix § 15 reports additional results. Figure 7a shows that the speedup from INT8 state quantization grows with batch size, since each sequence carries its own recurrent state. Weight quantization contributes most at small batch sizes because weight traffic is shared across active sequences. Combining weight and state quantization therefore provides complementary speedups across serving loads (Figure 7b). Finally, since Qwen and Nemotron are hybrid models, longer prefill grows the KV cache and reduces the relative contribution of the recurrent state to total memory traffic.

Figure 7. Time-per-output-token (TPOT) speedup. Subfigure (a) shows the speedup of an INT8 state over an FP32 state with BF16 weights across batch sizes. Subfigure (b) shows the speedup relative to BF16 weights with an FP32 state across prefill lengths at several batch sizes. The lower segment shows the contribution of INT4 weights; the upper segment shows the additional contribution of INT8 states. Appendix § 15.§ 2 reports contributions separately as well as absolute TPOT.
Conclusion
In this work, we study PTQ of linear recurrent and hybrid language models, separating weight/activation quantization from state quantization during decode. First, we focus on extra-state quantization, and in controlled experiments find that decayed recurrences develop weaker residual outliers than softmax attention, making their activations easier to quantize. Next, we characterize the effects of scale grouping, decay, and repeated rounding on state error. We furthermore reveal failure modes of random rotations under repeated state-quantization and compare integer and floating point formats for state quantization. We showcase the practical utility of our analysis by combining extra-state W/A quantization with LoReS, an intra-state quantization scheme which combines value-axis grouping, stochastic rounding, and decay-based precision allocation. Empirically, LoReS supports low bit states on Qwen-3.5, Nemotron-H, and Kimi-Linear with modest accuracy loss, with lower precision at shorter decode horizons. We implement LoReS in vLLM and show that combining state and weight quantization improves decode throughput. Future work could explore quantization-aware or low-precision training as well as GPU-native microscaling formats to further improve the performance and efficiency of linear recurrences. Furthermore, the effect of rotations on state quantization warrants further investigation.
Acknowledgments
This work was supported by the Hector Foundation, Max Planck ETH Center for Learning Systems, and the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Extended Related Work
Linear recurrent and hybrid language models. A growing line of work replaces softmax attention with recurrent sequence mixers whose state size and per-token decode cost do not grow with context length. Linear attention uses associativity to express attention as a recurrent update over a fixed-size matrix state [22, 34]. Selective state-space models such as Mamba and Mamba-2 introduce input-dependent recurrent dynamics and hardware-efficient implementations [18, 14].
More recent linear recurrences increase the expressivity of the state update. Gated Linear Attention introduces data-dependent forgetting [49]. DeltaNet uses the delta rule to update stored associations [34, 50], while Gated DeltaNet combines delta updates with forgetting [51]. Preconditioned DeltaNet instead introduces curvature-aware preconditioning into DeltaNet-style updates [43]. Kimi Delta Attention extends this family with a diagonal decay term and is deployed in Kimi-Linear, which interleaves recurrent and attention layers [39]. ParaTTT [27] composes these linear recurrent primitives to enhance expressivity. Large-scale hybrid models such as Nemotron-H and Qwen-3.5 likewise combine recurrent and softmax attention layers [4, 32]. LFM2 is a hardware-aware hybrid architecture that combines gated short convolutions with grouped-query attention for efficient edge inference [2].
Post-training quantization of softmax Transformers. Post-training quantization (PTQ) of softmax Transformers has been studied extensively. Weight-only methods such as GPTQ use layerwise reconstruction to reduce low-bit weight quantization error [16]. Joint weight and activation quantization must additionally contend with heavy-tailed activation distributions. SmoothQuant transfers activation scale into the weights [47], while incoherence-based methods apply a change of basis to spread concentrated energy across channels. QuIP and QuIP# apply randomized incoherence processing for low-bit weight quantization [7, 41]. QuaRot extends rotational invariances to weights, activations, and the KV cache [3], while SpinQuant learns rotations from calibration data [25].
The activation outliers that motivate these methods have also been studied independently of quantization. Sun et al. [37] identify massive activations that can act as implicit bias terms. More recent work studies their relationship with attention sinks [38, 8]. Our extra-state analysis is complementary: instead of introducing another outlier treatment, we ask whether changing the sequence mixer alters the outlier structure learned during training in the first place.
A second line of Transformer PTQ targets the KV cache. KIVI observes different quantization geometry for keys and values and uses per-channel key quantization and per-token value quantization [26]. KVQuant similarly exploits asymmetric key/value structure together with outlier-aware quantization [20], while SKVQ combines channel reordering with groupwise quantization and a recent-token window retained at higher precision [15]. Rotation-based methods can also transform keys and values before cache quantization [3, 25]. These methods operate on cache entries that are written once and subsequently reused, in contrast to a recurrent state that is modified in place after every generated token.
Quantization of recurrent language models. Several recent methods adapt weight and activation PTQ to Mamba-family architectures. Quamba identifies sensitive feature maps inside the selective scan and large output activation outliers, and develops an 8-bit weight and activation quantization recipe around these statistics [10]. MambaQuant likewise targets Mamba-specific activation distributions using variance-aligned rotations [48]. Quamba2 extends PTQ to Mamba and Mamba-2 across several weight and activation precisions [9]. These works develop effective quantization recipes for particular recurrent architectures. Our extra-state study instead holds the surrounding architecture and quantization procedure fixed and asks how the sequence mixer itself changes the numerical properties of the learned weights and activations. Lin-Quant [44] studies functional invariances of linear recurrences for low-precision prefill, complementing our analysis of repeated state quantization during decode.
Q-Mamba directly quantizes Mamba state caches in addition to weights and activations. Its decoupled scale quantization accounts for outliers along multiple state dimensions and supports low-bit state caches [40]. Nemotron 3 Super studies reduced-precision Mamba state storage during autoregressive decode. It evaluates FP16 storage with stochastic rounding and block-scaled INT16, and shows that stochastic rounding helps control error accumulation across recurrent updates [6]. These works establish that low-precision state storage is feasible for Mamba-family models.
SSDi8 considers a related but distinct setting. It maintains an INT8 representation through the internal Mamba-2 SSD computation and quantizes intermediate activations inside the chunked SSD algorithm [23]. Its primary target is therefore low-precision execution of the SSD computation.
Our intra-state study takes a different perspective from these architecture-specific recipes. We treat recurrent state quantization as a numerical problem across several recurrence families and isolate three sources of error: the geometry of scale grouping, accumulation under repeated requantization, and attenuation through recurrent decay. We then use these properties to construct LoReS and evaluate its transfer to large hybrid language models.
Concurrent works DAMP [54] and STEPQuant [52] allocate precision based on decay, with STEPQuant optimizing scales to reduce output error. Concurrent work LeapQuant [30] reduces requantization frequency and preserves outliers in higher precision.
Recurrent-state compression beyond quantization. Quantization reduces the precision of each stored state, while an orthogonal line of work improves recurrent efficiency by changing the dimensionality or structure of the recurrence. Chahine et al. (2026) use ideas from balanced truncation and Hankel singular values to reduce state dimension during training, while Parnichkun et al. (2025) investigate effective state size in linear operators. For pretrained linear attention models, Nazari & Rusch (2026) identify low-rank recurrent states and structurally prune query and key dimensions after training. Related work has also studied low-rank and sparse recurrent structure outside language modeling [42].
Experimental Setup
Models and training
All mixer zoo checkpoints share the same 1.3B-parameter backbone: 24 layers, hidden size 2048, an MLP ratio of 4, and 16 mixer heads of head dimension 128 per layer, both along the key- and the value axis. All sequence mixer layers share the same scaffold, using short convolutions on queries, keys and values [11, 35, 17] followed by SiLU and RoPE. The output is generated using a gated RMSNorm before an output projection. All models are trained for 100B tokens with the same recipe and a learning rate of .
Quantization and calibration
For hybrid models, we use GPTQ for weight quantization [16] through llm-compressor [1]. We use groups of 128 input channels, except that W8A8 uses per-output-channel weight scales, and calibrate on 128 sequences of 2048 tokens from the WikiText-2 train split. Activations are quantized per token. The rotations described in § 3 are active during calibration, and foldable rotations are absorbed into the weights before quantization. The language-model head remains in BF16. For Kimi-Linear, every expert is calibrated on all calibration tokens, while the router remains in BF16.
Mixer zoo. We use GPTQ with symmetric per-output-channel weight scales and calibrate on 128 sequences of 2048 tokens from the WikiText-2 train split. Dynamic activation scales are computed per token, with groups of 128 channels at mixer output-projection inputs. Decay and write-gate projections use the same W/A precision as the other block linears, and the rotations use Hadamard matrices. The unquantized baseline uses BF16. Embeddings and the language-model head remain in BF16. We evaluate word perplexity on the full WikiText-2 test split with a maximum sequence length of 2048.
Evaluation protocols
Our generative evaluations follow a common protocol, with model-specific sampling parameters and reasoning modes. Qwen-3.5 uses temperature 0.6, top- 0.95, top- 20, following the “precise” thinking-mode preset in its model card ([32]). Nemotron-H uses temperature 0.6 and top- 0.95, as recommended in the model card ([4]). Kimi-Linear uses temperature 1.0 and top- 0.95. While Qwen-3.5 has reasoning enabled by default, we explicitly activate it for all evals but RULER for Nemotron-H.
Generation budgets. GSM8K decodes up to 8192 new tokens. AIME, GPQA (diamond), and MMLU-Pro decode up to 65536. GSM8K uses the 8-shot chain-of-thought prompt.
Answer-format instructions. Our GPQA evaluations use the following NeMo-Skills multiple-choice instruction for all models: “Answer the following multiple choice question. The last line of your response should be in the following format: ’Answer: A/B/C/D’ (e.g. ’Answer: A’).” The question and four lettered choices follow this instruction. For AIME, all models use the NeMo-Skills math instruction “Solve the following math problem. Make sure to put the answer (and only answer) inside .” MMLU-Pro uses the corresponding ten-choice instruction with the format line “Answer: A/B/C/D/E/F/G/H/I/J,” options listed in dataset order, and no few-shot examples. We omit the math harness’s base-model stop strings (Q:, Question:, and Problem:) because we find they can also occur within a reasoning trace.
Scoring. AIME results pool the 2024 and 2025 competitions, each containing 30 problems. AIME responses are graded with the NeMo-Skills math grader: the integer in the last valid expression, compared with the reference by Math-Verify. For GSM8K, we select the last boxed answer, then the last explicit answer cue, and otherwise fall back to Math-Verify’s mathematical extraction. Numeric answers are compared exactly, without integer rounding. Other expressions use the NeMo-Skills math grader ([4]). For GPQA and MMLU-Pro, a rule modeled on the NeMo-Skills multiple-choice grader extracts the choice from the last valid Answer: X line and otherwise uses the last expression, where X is A–D for GPQA and A–J for MMLU-Pro. Our implementation also accepts a parenthesized choice after the answer cue, as in Answer: (C). Responses without a valid choice are marked incorrect.
Proofs
Proof of Proposition 1. For fixed S and q, the readout error in component is . If an entry belongs to a group with scale , the high-resolution approximation gives , so
Independence and zero means imply for . Expanding the square therefore yields
Under value-axis grouping, every entry in row has scale . Substitution into equation (4) gives
Linear attention uses unit-norm queries, , which gives the final equality. Under key-axis grouping, every entry in column has scale , hence
These are the two expressions in equation (3).
Shared error propagation setup for Propositions 2 and 3. For the fixed update operands in Equation (2), define
The states start identically, so . Unrolling Equation (2) yields
Each product propagates an injection from step to step .
Proof of Proposition 2. Because , the matrix acts as the identity on the orthogonal complement of and has eigenvalue along . The ranges and therefore give . Since , we have and . Applying the triangle inequality to Equation (5) gives
This argument imposes no independence or alignment assumptions on the injections.
Proof of Proposition 3. Expanding the squared Frobenius norm in Equation (5) gives, for any rounding rule,
In particular, this is the RTN decomposition for the full recurrence, including diagonal decay and delta correction.
Let denote the history immediately before rounding at step , including the quantization scales. Under conditionally unbiased SR, . For , the earlier injection is -measurable, and the propagation matrices are fixed independently of rounding. Thus
Taking expectations in Equation (6) therefore yields
The quantization scales may depend on previous rounding errors; conditional unbiasedness suffices and does not require independent injections. For scalar decay without delta correction, and , where . Substituting into Equation (6) and Equation (7) proves the two identities in the proposition.
Under the contraction assumption of Proposition 2, the general SR identity also gives
To show that RTN reinforcement can persist under decay, let , where and has a single unit entry and is zero elsewhere. Start both states at zero and round elementwise to the fixed grid , with . RTN discards the write at every step, so and . Consequently,
Thus positive cross terms can occur over arbitrarily many successive steps, even with strict decay.
Extra-State Quantization
Quantization results and rotation ablations
Table 9 reports the full weight–activation precision ladder in absolute WikiText-2 perplexity, including the unquantized baseline. The main text (Table 2) shows relative changes for unrotated W4A8 and W4A4, together with W4A4 rotation ablations.
| Architectural choices | No rotation | Full stack R₁R₄Rₒ | ||||||||||||
| Decay | Baseline | W8A8 | W4A16 | W4A8 | W16A4 | W4A4 | W8A8 | W4A16 | W4A8 | W16A4 | W4A4 | |||
| − | − | − | − | 15.66 | 15.71 | 16.52 | 16.57 | 26.47 | 31.07 | 15.67 | 16.34 | 16.34 | 16.44 | 17.33 |
| − | ✓ | − | − | 15.66 | 15.70 | 16.53 | 16.58 | 25.87 | 30.11 | 15.67 | 16.40 | 16.40 | 16.46 | 17.40 |
| − | ✓ | − | − | 17.93 | 17.98 | 19.06 | 19.15 | 31.72 | 37.91 | 17.93 | 18.84 | 18.86 | 18.95 | 20.16 |
| − | ✓ | ✓ | − | 18.05 | 18.10 | 19.24 | 19.30 | 28.54 | 32.93 | 18.06 | 19.01 | 19.02 | 19.12 | 20.30 |
| − | ✓ | ✓ | ✓ | 17.72 | 17.77 | 18.72 | 18.80 | 26.32 | 29.80 | 17.72 | 18.61 | 18.61 | 18.73 | 19.97 |
| Scal | ✓ | − | − | 16.94 | 16.99 | 17.93 | 17.97 | 23.68 | 26.53 | 16.95 | 17.78 | 17.80 | 17.87 | 18.96 |
| Scal | ✓ | ✓ | − | 17.19 | 17.23 | 18.09 | 18.15 | 23.82 | 26.40 | 17.19 | 18.01 | 18.02 | 18.07 | 19.13 |
| Scal | ✓ | ✓ | ✓ | 17.10 | 17.15 | 18.09 | 18.12 | 23.90 | 26.58 | 17.12 | 17.94 | 17.94 | 18.03 | 19.12 |
| Diag | ✓ | − | − | 15.86 | 15.88 | 16.58 | 16.62 | 20.95 | 22.99 | 15.86 | 16.50 | 16.50 | 16.55 | 17.33 |
| Diag | ✓ | ✓ | − | 16.08 | 16.12 | 16.89 | 16.95 | 21.75 | 23.99 | 16.09 | 16.83 | 16.82 | 16.83 | 17.78 |
| Diag | ✓ | ✓ | ✓ | 16.55 | 16.58 | 17.46 | 17.51 | 22.13 | 24.46 | 16.55 | 17.30 | 17.29 | 17.42 | 18.40 |
Table 9. Complete absolute WikiText-2 perplexity ladder. Baseline is the unquantized BF16 model. 16-bit entries denote BF16.
Table 10 ablates the full rotation stack R₁R₄Rₒ on the 1.3B mixer zoo. Among the individual rotations, R₁ is more effective for the softmax models and for plain linear attention, whereas R₄ is more effective for every recurrent cell that carries a decay mechanism, a write gate, or a delta-rule correction. This shows that decayed linear recurrence PTQ error is not dominated by outliers in the residual stream.
| Architectural choices | (%) | ||||||||||
| Decay | none | R₁ | R₄ | Rₒ | R₁R₄ | R₁Rₒ | R₄Rₒ | R₁R₄Rₒ | |||
| − | − | − | − | 98.4 | 36.7 | 47.9 | 90.1 | 12.2 | 34.4 | 44.3 | 10.6 |
| − | ✓ | − | − | 92.3 | 37.8 | 46.5 | 86.0 | 12.9 | 35.7 | 42.8 | 11.1 |
| − | ✓ | − | − | 111.4 | 48.1 | 54.0 | 105.9 | 14.0 | 45.5 | 50.5 | 12.5 |
| − | ✓ | ✓ | − | 82.4 | 47.2 | 35.9 | 77.9 | 13.9 | 44.8 | 33.2 | 12.4 |
| − | ✓ | ✓ | ✓ | 68.2 | 46.5 | 28.5 | 65.0 | 13.9 | 43.7 | 26.3 | 12.7 |
| Scal | ✓ | − | − | 56.6 | 40.3 | 24.7 | 54.8 | 13.0 | 38.1 | 23.3 | 11.9 |
| Scal | ✓ | ✓ | − | 53.5 | 41.3 | 21.3 | 51.2 | 12.6 | 38.8 | 19.5 | 11.3 |
| Scal | ✓ | ✓ | ✓ | 55.4 | 43.2 | 21.5 | 52.8 | 13.1 | 40.6 | 19.8 | 11.8 |
| Diag | ✓ | − | − | 45.0 | 32.6 | 18.3 | 42.8 | 10.4 | 31.0 | 16.7 | 9.3 |
| Diag | ✓ | ✓ | − | 49.2 | 35.7 | 20.7 | 46.8 | 11.6 | 34.4 | 19.3 | 10.6 |
| Diag | ✓ | ✓ | ✓ | 47.8 | 37.0 | 19.5 | 45.6 | 12.5 | 34.8 | 18.1 | 11.2 |
Table 10. Relative WikiText-2 perplexity increase (%) under W4A4 with different rotation subsets on the 1.3B mixer zoo.
Residual outliers and interventions
Figure 9a relates residual-stream kurtosis to unrotated W4A4 degradation across the mixer zoo.
Figure 9b shows that the Transformer builds an outlier channel that persists across layers and complements Figure 2.
Table 11 extends Table 3 to every cell of the 1.3B mixer zoo. Removing the dominant channel, either at the mixer input or before individual projections, has a much larger effect in the softmax cells than in the recurrent cells.
| Architectural choices | none | removed | mean | |||||
| Decay | QK | V | QKV | QKV | ||||
| − | − | − | − | 15.66 | 280.51 | 16.86 | 448.70 | 16.89 |
| − | − | − | 15.66 | 81.30 | 17.18 | 76.61 | 16.08 | |
| − | − | − | 17.93 | 21.23 | 18.20 | 23.62 | 19.81 | |
| − | − | 18.05 | 18.89 | 18.21 | 20.64 | 18.62 | ||
| − | 17.72 | 18.42 | 18.10 | 21.91 | 17.96 | |||
| Scal | − | − | 16.94 | 17.41 | 17.12 | 18.21 | 17.16 | |
| Scal | − | 17.19 | 17.76 | 17.36 | 19.65 | 17.38 | ||
| Scal | 17.10 | 17.52 | 17.22 | 18.78 | 17.22 | |||
| Diag | − | − | 15.86 | 16.64 | 15.98 | 18.81 | 16.26 | |
| Diag | − | 16.08 | 16.85 | 16.35 | 19.93 | 16.27 | ||
| Diag | 16.55 | 16.91 | 16.74 | 18.19 | 16.83 |
Table 11. Absolute WikiText-2 perplexity under the dominant-channel interventions of Table 3. None is the unmodified BF16 model. Under removed, QK/V delete the channel only before the corresponding projections and QKV deletes it at the mixer input. mean replaces QKV at the mixer input by its token average.
State Quantization
Grouping, mixed precision, and rounding
This subsection reports the additional ablations referenced in § 4.
Table 12 compares per-axis with per-state scale grouping. Both value- and key-axis grouping outperform per-state grouping by a wide margin.
Table 13 compares the decay-based INT5 allocation from § 4.2 with a matched random allocation.
| model | decay | random |
| Mamba-2 | 0.033 | 0.203 |
| GLA | 0.009 | 0.073 |
| GDN | 0.005 | 0.064 |
| KDA | 0.005 | 0.108 |
| Qwen-3.5 | 0.006 | 0.267 |
| Nemotron-H | 0.045 | 0.158 |
| Kimi-Linear | 0.006 | 0.028 |
Table 13. Median cross-entropy increase for decay-based and random INT5 allocations under RTN.
Rotations and formats
| model | value INT8 RTN | value INT8 SR | value E4M3 RTN | value E4M3 SR | key INT8 RTN | key INT8 SR | key E4M3 RTN | key E4M3 SR |
| Linear | 0.1246 | 0.2229 | 0.0722 | 0.0273 | 0.3317 | 0.1349 | 0.0937 | 0.0102 |
| Mamba-2 | 0.0026 | 0.0015 | 0.0071 | 0.0019 | 0.0105 | 0.0005 | 0.0118 | 0.0010 |
| GLA | 0.0002 | 0.0001 | 0.0040 | 0.0007 | 0.0047 | 0.0005 | 0.0064 | 0.0007 |
| GDN | 0.0000 | 0.0001 | 0.0017 | 0.0016 | 0.0048 | 0.0007 | 0.0052 | 0.0013 |
| KDA | 0.0002 | 0.0002 | 0.0031 | 0.0006 | 0.0071 | 0.0005 | 0.0073 | 0.0009 |
| Qwen-3.5 | 0.0005 | 0.0001 | 0.0010 | 0.0002 | 0.0100 | 0.8767 | 0.0027 | 0.0004 |
| Nemotron-H | 0.0006 | 0.0001 | 0.0009 | 0.0015 | 0.0193 | 6.5132 | 0.0037 | 0.0429 |
| Kimi-Linear | 0.0005 | 0.0000 | 0.0015 | 0.0001 | 0.0013 | 0.0002 | 0.0037 | 0.0001 |
Table 17. Median cross-entropy increase for INT8 and E4M3 under RTN and SR with uniform grouping and symmetric grids. Bold marks best per row.
Scale feedback and query alignment under rotations
Scale feedback. § 4.4 partially attributes the penalty of under value-axis grouping to the quantization scale, which is computed from the stored state and therefore carries the accumulated rounding error. Table 18 reports an intervention. Every scale group takes its scale from an FP32 copy of the state that is advanced with the same operands but never rounded. Figure 10 shows the resulting ratios for both grouping axes.
| model | INT8 value | INT8 + | INT8 key | INT8 + | INT6 value | INT6 + | INT6 key | INT6 + | INT4 value | INT4 + | INT4 key | INT4 + |
| Mamba-2 | 0.0014 | 0.0007 | 0.0005 | 0.0001 | 0.0365 | 0.0284 | 0.0110 | 0.0035 | 0.4878 | 0.4407 | 0.2238 | 0.0541 |
| GLA | 0.0000 | 0.0000 | 0.0004 | 0.0001 | 0.0021 | 0.0011 | 0.0096 | 0.0031 | 0.0488 | 0.0538 | 0.1794 | 0.0432 |
| GDN | 0.0002 | 0.0002 | 0.0009 | 0.0001 | 0.0048 | 0.0065 | 0.0166 | 0.0022 | 0.1000 | 0.1644 | 0.3033 | 0.0640 |
| KDA | 0.0001 | 0.0000 | 0.0004 | 0.0001 | 0.0047 | 0.0023 | 0.0100 | 0.0025 | 0.0970 | 0.0799 | 0.2184 | 0.0517 |
Table 18. Median cross-entropy increase under within-group rotations for SR with symmetric grids, with every scale group taking its scale from the FP32 copy of the state; mixer-zoo models. Bold marks best per precision.

Figure 10. with rotation divided by without it, as in Figure 4(a), with every scale group taking its scale from the FP32 copy of the state.
Figure 11 extends the scale-inflation trajectories in Figure 4(b) to all four decayed recurrences. Rotations generally amplify scale inflation more under value-axis grouping than under key-axis grouping.
Query alignment. We replay the first eight recurrent layers on one 2048-token WikiText-2 sequence using INT4 SR with oracle scales. We sample all heads at four decode steps, spaced 512 steps apart starting at step 256. At each snapshot, we evaluate quantization error and alignment statistics from exact SR moments at each run’s own state before rounding. We average each head’s statistics over the snapshots and classify it as worse under if rotation increases its expected squared readout error.
Figure 12 shows that rotation hurts most often in heads whose plain error has low query alignment. This pattern is pronounced in Qwen-3.5, whereas most Mamba-2 heads have plain alignment near unity and benefit from rotation.
Figure 12. As Figure 5, more models.
End-to-End Evaluation
This section extends the decode results presented in § 5.
Configuration comparisons and promotion sweeps
Table 19 compares value- and key-axis grouping under stochastic rounding.
| model | FP32 | value INT8 | value INT6 | value INT4 | key INT8 | key INT6 | key INT4 |
| Qwen-3.5 | 95.9 | 96.4 | 95.3 | 8.0 | 47.8 | 1.5 | 0.7 |
| Nemotron-H | 92.3 | 92.0 | 91.2 | 3.5 | 0.9 | 1.4 | 1.1 |
Table 19. GSM8K accuracy (%) under SR. FP32 is the unquantized-state baseline.
Table 20 compares RTN and SR at INT8 state on AIME. Stochastic rounding outperforms round-to-nearest for all three models.
Proxy-to-generation transfer. For sequence and token , let and denote the token-level cross-entropies obtained with quantized and FP32 states, respectively. We define
where the median is taken over all sequence–token pairs. Figure 13 compares the teacher-forced proxy with full-test GSM8K accuracy for four INT6 configurations under round-to-nearest with symmetric grids. The two metrics follow the same overall trend.
Figure 13. Teacher-forced versus full-test GSM8K accuracy for four INT6 state configurations under round-to-nearest with symmetric grids. The horizontal axis is reversed so that better results lie to the right. (a) Nemotron-H. (b) Qwen-3.5.
Calibration-to-generation transfer under promotion. Figure 14 compares the teacher-forced proxy with generation accuracy as decay-ranked heads are promoted.
Figure 14. Promotion sweeps on GSM8K, GPQA, and AIME. Teal denotes , magenta accuracy, dotted lines the FP32-state baseline, and dashed lines the minimum precision under the 5% criterion.
Table 21 gives the complete accuracy sweeps. Its entries are exactly the accuracy points plotted in Figure 14 and Figure 6.
| effective bits | Qwen-3.5 GSM8K | Qwen-3.5 GPQA | Qwen-3.5 AIME | Nemotron-H GSM8K | Nemotron-H GPQA | Nemotron-H AIME |
| 4.0 | 8.0 | 5.1 | 0.0 | 3.5 | 7.6 | 0.0 |
| 4.5 | 27.8 | 9.6 | 0.0 | 6.2 | 16.2 | 0.0 |
| 5.0 | 93.1 | 55.6 | 25.0 | 77.0 | 35.9 | 6.7 |
| 5.5 | 94.2 | 62.6 | 43.3 | 91.4 | 41.4 | 26.7 |
| 6.0 | 95.3 | 68.2 | 50.0 | 91.2 | 48.5 | 26.7 |
| 6.5 | 95.8 | 77.8 | 76.7 | 91.6 | 47.5 | 38.3 |
| 7.0 | 96.1 | 78.3 | 78.3 | 92.9 | 51.5 | 38.3 |
| 7.5 | 96.1 | 76.8 | 83.3 | 92.3 | 51.5 | 46.7 |
| 8.0 | 96.4 | 80.8 | 80.0 | 92.0 | 51.0 | 51.7 |
| FP32 | 95.9 | 80.8 | 83.3 | 92.3 | 51.5 | 50.0 |
Table 21. Generation accuracy (%) as a function of effective bits. GPQA and AIME entries are pass@1.
Generation length
Table 22 shows that state INT8 can either lengthen or shorten the median response, depending on the model and benchmark. The accuracy results are therefore not an artifact of systematically shorter quantized generations. The AIME comparison below examines stochastic and nearest rounding.
Table 23 reports the generation lengths of the AIME runs behind Table 20. For Qwen-3.5 and Nemotron-H, stochastic rounding keeps mean response length close to the FP32-state baseline, whereas round-to-nearest lengthens responses and increases budget exhaustion. Kimi-Linear instead produces shorter responses under round-to-nearest with value-axis grouping, so budget exhaustion does not explain its accuracy loss. For all three models, stochastic rounding keeps mean response length closer to the FP32-state baseline than round-to-nearest.
| rounding | Qwen-3.5 tokens | capped | Nemotron-H tokens | capped | Kimi-Linear (value) tokens | capped | Kimi-Linear (key) tokens | capped |
| FP32 state | 29927 | 240 | 12106 | 26 | 22565 | 318 | 22565 | 318 |
| RTN | 46437 | 792 | 30136 | 459 | 21398 | 244 | 25728 | 322 |
| SR | 30362 | 275 | 12470 | 36 | 22121 | 316 | 22507 | 365 |
Table 23. Mean generated AIME tokens (and traces hitting the 65536-token budget) for the arms of Table 20. Same generations as the reported accuracies. per cell (60 problems 32 samples).
Implementation and Efficiency
Kernel Throughput
The state-update kernels tile each head’s state over value rows. Value-axis scales are local to each tile, so quantization and dequantization fuse into the decode kernel. Key-axis scales are not local to these tiles, complicating an efficient implementation. We benchmark vLLM’s FP32-state decode kernels against our INT8-state variants with equal state element counts: GDN and KDA use 32 heads with , while Mamba-2 uses 64 heads with and . The INT8 variants use stochastic rounding. A CUDA graph cycles through a working set larger than the L2 cache, and each kernel is launch-tuned at every batch size. The selected configurations are then timed in seven repetitions, and speedup is the ratio of median kernel times. Figure 15 compares the H100 with the lower-bandwidth RTX 6000. The larger gains on the RTX 6000 are consistent with the state traffic accounting for a larger share of decode cost.
Figure 15. Speedup of vLLM mixer kernels using INT8 versus FP32 states.
End-to-end throughput
We time Nemotron-H-8B-Reasoning-128K and Qwen-3.5 9B with vLLM on a dedicated H100 SXM5, using CUDA graphs and asynchronous decode scheduling. The weight settings are the vendor BF16 checkpoint and rotated, weight-only GPTQ at INT4 with group size 128. Activations remain in BF16. Online rotations remain active for INT4 weights during timing and use
Dao-AILab’s fast CUDA Hadamard kernels.1 We compare FP32 recurrent states with INT8 states using stochastic rounding and value-axis grouping.
Both sweeps use fixed batches of requests with synthetic token prompts and greedy decoding, forcing 32 output tokens per request. All prefills and the first output token finish before timing starts. We then measure engine wall time for the remaining 31 decode steps, synchronizing the GPU at both timing boundaries, and divide by 31 to obtain TPOT. Prefill, tokenization and HTTP overhead are excluded.
The batch-size sweep uses (up to 64 with INT4 weights) and two-token prompts. The context-length sweep uses and prompt lengths , plus 8160 tokens for Nemotron-H or 8192 for Qwen-3.5. Each cell has two warmup batches followed by 16 measured batches per repetition. FP32 and INT8 states are measured in seven paired repetitions on the same GPU, with fresh engine processes, matching prompts and seeds, and alternating measurement order. Reported TPOT is the median of the seven per-repetition means. Speedup is the ratio of the corresponding median TPOTs.
Launch configurations are tuned separately for the FP32 and INT8 decode kernels at each batch size. Attention block sizes are selected automatically, except in the INT4-weight pinned-block control, which fixes both state formats to the FP32-state block size: 1040 tokens for Nemotron-H and 528 for Qwen-3.5.
Figure 16 visualizes INT8-state speedup over an FP32 state at the same weight setting. Table 24 gives the corresponding TPOT for the automatic-block sweeps.
| W/A/S | batch size | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 | 256 |
| Nemotron-H | ||||||||||
| 16/16/32 | 6.31 | 6.41 | 6.62 | 6.96 | 7.63 | 8.78 | 11.40 | 16.83 | 29.35 | |
| 16/16/8 | 6.27 | 6.32 | 6.44 | 6.67 | 6.98 | 7.66 | 9.01 | 11.18 | 17.32 | |
| 4/16/32 | 4.46 | 4.52 | 4.73 | 4.99 | 5.62 | 6.77 | 9.86 | – | – | |
| 4/16/8 | 4.41 | 4.44 | 4.57 | 4.74 | 4.98 | 5.57 | 7.50 | – | – | |
| Qwen-3.5 | ||||||||||
| 16/16/32 | 6.99 | 7.05 | 7.15 | 7.39 | 7.77 | 8.44 | 9.60 | 12.19 | 18.23 | |
| 16/16/8 | 6.98 | 7.01 | 7.06 | 7.20 | 7.48 | 7.92 | 8.67 | 9.99 | 14.21 | |
| 4/16/32 | 5.70 | 5.79 | 5.89 | 6.11 | 6.46 | 7.22 | 9.57 | – | – | |
| 4/16/8 | 5.67 | 5.74 | 5.81 | 5.96 | 6.23 | 6.76 | 8.59 | – | – |
Table 24. Absolute TPOT (ms), reported as the median of seven repetitions.

Figure 16. INT8-state TPOT speedup over an FP32 state, by batch size. Top row Nemotron-H, bottom row Qwen-3.5. Columns BF16 weights, INT4 weights, and INT4 weights with the attention block size pinned to its FP32-state value.
Figure 17 and Table 25 report the context sweep.
Figure 17. INT8-state TPOT speedup over the same model and weight setting with an FP32 state, across prefill length. Bar shade gives the batch size.
Table 25: Absolute TPOT (ms, median of seven repetitions) as prefill length increases. The 8k column uses the longest benchmarked prompt: 8160 tokens for Nemotron-H and 8192 for Qwen-3.5.
| W/A/S | batch size | prefill length | |||||
| 16 | 32 | 512 | 1024 | 4096 | 8k | ||
| Nemotron-H | |||||||
| 16/16/32 | 2 | 6.4 | 6.4 | 6.5 | 6.4 | 6.3 | 6.4 |
| 16/16/32 | 4 | 6.6 | 6.6 | 6.7 | 6.6 | 6.7 | 6.8 |
| 16/16/32 | 8 | 7.0 | 7.0 | 7.1 | 6.9 | 7.2 | 7.4 |
| 16/16/32 | 16 | 7.7 | 7.7 | 7.8 | 7.8 | 8.1 | 8.5 |
| 16/16/32 | 32 | 8.9 | 9.0 | 9.0 | 9.1 | 9.7 | 10.5 |
| 16/16/8 | 2 | 6.3 | 6.3 | 6.4 | 6.3 | 6.2 | 6.3 |
| 16/16/8 | 4 | 6.5 | 6.5 | 6.5 | 6.6 | 6.7 | 6.7 |
| 16/16/8 | 8 | 6.7 | 6.7 | 6.8 | 6.8 | 7.0 | 7.1 |
| 16/16/8 | 16 | 7.2 | 7.2 | 7.3 | 7.4 | 7.7 | 8.1 |
| 16/16/8 | 32 | 7.9 | 7.9 | 8.1 | 8.2 | 8.8 | 9.5 |
| 4/16/32 | 2 | 4.5 | 4.5 | 4.6 | 4.6 | 4.6 | 4.6 |
| 4/16/32 | 4 | 4.7 | 4.7 | 4.8 | 4.9 | 4.8 | 4.9 |
| 4/16/32 | 8 | 4.9 | 5.0 | 5.1 | 5.2 | 5.2 | 5.4 |
| 4/16/32 | 16 | 5.6 | 5.6 | 5.8 | 5.8 | 6.0 | 6.4 |
| 4/16/32 | 32 | 6.7 | 6.8 | 6.9 | 6.9 | 7.5 | 8.3 |
| 4/16/8 | 2 | 4.4 | 4.4 | 4.5 | 4.4 | 4.5 | 4.5 |
| 4/16/8 | 4 | 4.5 | 4.5 | 4.7 | 4.6 | 4.7 | 4.8 |
| 4/16/8 | 8 | 4.7 | 4.7 | 4.9 | 4.8 | 5.0 | 5.2 |
| 4/16/8 | 16 | 5.1 | 5.1 | 5.3 | 5.2 | 5.5 | 6.0 |
| 4/16/8 | 32 | 5.7 | 5.8 | 5.9 | 5.9 | 6.6 | 7.4 |
| Qwen-3.5 | |||||||
| 16/16/32 | 2 | 6.8 | 6.8 | 6.8 | 6.9 | 7.0 | 7.1 |
| 16/16/32 | 4 | 6.9 | 6.9 | 7.0 | 7.1 | 7.2 | 7.5 |
| 16/16/32 | 8 | 7.2 | 7.2 | 7.3 | 7.4 | 7.7 | 8.1 |
| 16/16/32 | 16 | 7.6 | 7.6 | 7.8 | 7.9 | 8.5 | 9.2 |
| 16/16/32 | 32 | 8.2 | 8.3 | 8.5 | 8.7 | 9.9 | 11.4 |
| 16/16/8 | 2 | 6.8 | 6.8 | 6.8 | 6.9 | 7.0 | 7.1 |
| 16/16/8 | 4 | 6.8 | 6.8 | 7.0 | 7.0 | 7.2 | 7.4 |
| 16/16/8 | 8 | 7.0 | 7.0 | 7.2 | 7.2 | 7.6 | 7.9 |
| 16/16/8 | 16 | 7.3 | 7.3 | 7.5 | 7.6 | 8.2 | 9.0 |
| 16/16/8 | 32 | 7.7 | 7.8 | 8.0 | 8.3 | 9.4 | 11.0 |
| 4/16/32 | 2 | 5.8 | 5.8 | 5.9 | 6.0 | 6.1 | 6.2 |
| 4/16/32 | 4 | 5.9 | 5.9 | 6.1 | 6.2 | 6.3 | 6.5 |
| 4/16/32 | 8 | 6.1 | 6.2 | 6.4 | 6.4 | 6.8 | 7.2 |
| 4/16/32 | 16 | 6.5 | 6.5 | 6.7 | 6.9 | 7.5 | 8.3 |
| 4/16/32 | 32 | 7.2 | 7.3 | 7.6 | 7.8 | 8.9 | 10.6 |
| 4/16/8 | 2 | 5.7 | 5.7 | 5.9 | 6.0 | 6.1 | 6.2 |
| 4/16/8 | 4 | 5.8 | 5.8 | 6.0 | 6.1 | 6.2 | 6.5 |
| 4/16/8 | 8 | 6.0 | 6.0 | 6.3 | 6.3 | 6.6 | 7.0 |
| 4/16/8 | 16 | 6.3 | 6.3 | 6.5 | 6.7 | 7.3 | 8.0 |
| 4/16/8 | 32 | 6.8 | 6.8 | 7.1 | 7.4 | 8.5 | 10.0 |
Table 25.
Figure 18. INT8-weight timing without online rotations. TPOT speedup over BF16 weights with an FP32 state across prefill length. Teal denotes the INT8-weight contribution, salmon the additional INT8-state contribution, and shade batch size. Exported weights remain rotated.
AI Use Statement
In this work, we use generative AI tools to formulate mathematical claims, provide critical ingredients for proving them, to design or give feedback on our methodology and experiments, to create or edit code, implement methods, to propose or refine hypotheses, to support qualitative and thematic data analysis, interpret results, to help develop theoretical models or conceptual frameworks, and to do a literature review. Additionally, we used generative AI tools for LaTeX formatting, to draft parts of the appendix, and for brainstorming. We have not used generative AI tools for generating synthetic datasets, to assist with translation, or to clean or reformat datasets. We have reviewed all AI-assisted work, checked the code as well as the reformatted LaTeX tables. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.
References
- [1]Red Hat AI and vLLM Project. LLM Compressor, 8 2024. URL https://github.com/vllm-project/llm-compressor.
- [2]Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, Tarek Dakhran, Song Duong, Alfred Eng, Fernando Fernandes, Marc Härkönen, Anne Harrington, Ramin Hasani, Saniya Karwa, Yuri Khrustalev, Maxime Labonne, Mathias Lechner, Valentine Lechner, Simon Lee, Zetian Li, Noel Loo, Jacob Marks, Edoardo Mosca, Samuel J. Paech, Paul Pak, Rom N. Parnichkun, Alex Quach, Ryan Rogers, Daniela Rus, Nayan Saxena, Bettina Schlager, Tim Seyde, Jimmy T. H. Smith, Aditya Tadimeti, and Neehal Tumma. Lfm2 technical report, 2025. URL https://arxiv.org/abs/2511.23404.
- [3]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quotor: Outline-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240, 2024.
- [4]Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, et al. Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models. arXiv preprint arXiv:2504.03624, 2025.
- [6]Aakshita Chandiramani, Aaron Blakeman, Abdullah Olaoye, Abhibha Gupta, Abhilash Somasamudramath, Abhinav Khattar, Adeola Adesoba, Adi Renduchintala, Adil Asif, Aditya Agrawal, et al. Nemotron 3 super: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning. arXiv preprint arXiv:2604.12374, 2026.
- [7]Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in neural information processing systems, 36:4396–4429, 2023.
- [8]Yihong Chen, Zhouchen Lin, and Quanming Yao. Attention sinks induce gradient sinks: Massive activations as gradient regulators in transformers. arXiv preprint arXiv:2603.17771, 2026.
- [9]Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, Mohamed S Abdelfattah, and Diana Marculescu. Quamba2: A robust and scalable post-training quantization framework for selective state space models. arXiv preprint arXiv:2503.22879, 2025a.
- [10]Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, and Diana Marculescu. Quamba: A post-training quantization recipe for selective state space models. In International Conference on Learning Representations, volume 2025, pp. 101328–101354, 2025b.
- [11]François Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp. 1800–1807. Ieee, 2017.
- [12]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reïchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
- [13]Matteo Croci, Massimiliano Fasi, Nicholas J Higham, Theo Mary, and Mantas Mikaitis. Stochastic rounding: implementation, error analysis and applications. Royal Society Open Science, 9(3): 211631, 2022.
- [14]Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024.arxiv.org/abs/2405.21060
- [15]Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Skvq: Sliding-window key and value cache quantization for large language models. arXiv preprint arXiv:2405.06219, 2024.arxiv.org/abs/2405.06219
- [16]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.arxiv.org/abs/2210.17323
- [17]Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv:2212.14052, 2022.arxiv.org/abs/2212.14052
- [18]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752.
- [19]Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pp. 1737–1746. PMLR, 2015.
- [20]Coleman Hooper, Sehoon Kim, Hiva Mohammadzaheh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270–1303, 2024.arxiv.org/abs/2401.18079
- [21]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024.arxiv.org/abs/2404.06654
- [22]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html.arxiv.org/abs/2006.16236
- [23]Hyunwoo Kim, Byoungchan Ko, Minseok Kang, Minwoo Kim, Dongjin Lee, Jaehoon Lee, Sunghro Yoon, and Dahuin Jung. Ssdi8: Accurate and efficient 8-bit quantization for state space duality. In International Conference on Learning Representations, volume 2026, pp. 97014–97042, 2026.arxiv.org/abs/2608.21952
- [24]Woosuk Kwon, Z uohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.arxiv.org/abs/2309.06180
- [25]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Sp inquant: Llm quantization with learned rotations. In International Conference on Learning Representations, volume 2025, pp. 92009–92032, 2025.
- [26]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.DOI
- [27]Noel Loo, Neehal Tumma, T. Konstantin Rusch, Mathias Lechner, Alexander Amini, and Daniela Rus. Paratt: Parallelizing test-time training layers with deep memories, 2026. URL https://arxiv.org/abs/2610.38255.
- [28]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.arxiv.org/abs/1609.07843
- [29]Philipp Nazari and T. Konstantin Rusch. On state reduction in linear attention, 2026. URL https://arxiv.org/abs/2602.04852.
- [30]Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra, Hongtao Zhang, William X Zheng, Baris Kaciski, Song Han, et al. Leapquant: Efficient linear attention with accurate recurrent state quantization. arXiv preprint arXiv:2609.38166, 2026.arxiv.org/abs/2609.38166
- [32]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
- [33]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanze Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.arxiv.org/abs/2311.12022
- [34]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9355–9366. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/schlag21a.html.arxiv.org/abs/2102.11174
- [35]David So, Wojciech Mánke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Searching for efficient transformers for language modeling. Advances in neural information processing systems, 34:6010–6022, 2021.
- [36]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
- [37]Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762, 2024.arxiv.org/abs/2402.17762
- [38]Shangwen Sun, Alfredo Canziani, Yann LeCun, and Jiachen Zhu. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks. arXiv preprint arXiv:2603.05498, 2026.arxiv.org/abs/2603.05498
- [39]Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025.
- [40]Chen Tianqi, Yuanteng Chen, Peisong Wang, Weixiang Xu, Zeyu Zhu, and Jian Cheng. Q-mamba: Towards more efficient mamba models via post-training quantization. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 10594–10610, 2025.
- [41]Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. Proceedings of machine learning research, 235:48630, 2024.arxiv.org/abs/2402.04396
- [42]Neehal Tumma, Mathias Lechner, Noel Loo, Ramin Hasani, and Daniela Rus. Leveraging low-rank and sparse recurrent connectivity for robust closed-loop control, 2023. URL https://arxiv.org/abs/2310.03915.
- [43]Neehal Tumma, Noel Loo, and Daniela Rus. Preconditioned deltanet: Curvature-aware sequence modeling for linear recurrences. arXiv preprint arXiv:2604.21100, 2026a.arxiv.org/abs/2604.21100
- [44]Neehal Tumma, Philipp Nazari, Rohan Cherukuri, Daniela Rus, and T. Konstantin Rusch. Linquant: Functional invariances for low-precision linear recurrences, 2026b. URL https://arxiv.org/abs/2610.39418.
- [45]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
- [46]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Alurraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multitask language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024.arxiv.org/abs/2406.01574
- [47]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087–38099. PMLR, 2023.arxiv.org/abs/2211.10438
- [48]Zukang Xu, Yuxuan Yue, Xing Hu, Dawei Yang, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Jiangyong Yu, XUCHEN, and Sifan Zhou. Mambaquant: Quantizing the mamba family with variance aligned rotation methods. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=KI45uDnmzv.arxiv.org/abs/2501.13484
- [49]Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023.arxiv.org/abs/2312.06635
- [50]Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522, 2024.arxiv.org/abs/2406.06484
- [51]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In International Conference on Learning Representations, volume 2025, pp. 29687–29707, 2025.arxiv.org/abs/2412.06464
- [52]Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, and Ying Wei. Stepquant: When and where errors matter in delta-rule recurrent state quantization. arXiv preprint arXiv:2609.38169, 2026.arxiv.org/abs/2609.38169
- [53]Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1910.07467
- [54]Tao Zhang, Jianchao Tan, Pingwei Sun, Yanqi Yu, Zunhai Su, Zixu Jiang, Yuchen Xie, Xunliang Cai, and Ziqian Zeng. Damp: Decay-aware mixed-precision recurrent-state quantization. arXiv preprint arXiv:2608.27513, 2026.arxiv.org/abs/2608.27513










