betaFeedback

WorksComputer Science

AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement

Abstract

Existing LLM post-training quantization techniques determine precision before inference begins, yet the precision a model requires varies dynamically across prompts, layers, and inference phases. We present AutoEMatic, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime. A lightweight policy selects each layer's precision at every prefill and decode step from two runtime signals: the entropy of the next-token distribution and the magnitude of each layer's update to its hidden states. This runtime adaptation is made practical by AutoFP, a multi-scale floating-point format that derives 2-, 4-, and 6-bit precisions from a single NVFP4 base format. We realize AutoEMatic on two targets: GPU kernels that execute AutoFP on NVFP4 Tensor Cores, and an accelerator whose vector MAC datapath executes all AutoFP formats natively. Across reasoning and agentic benchmarks, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches group-scaled FP6 accuracy on most benchmarks, using 4.9 bits per weight versus 4.5 for NVFP4. On GB300 GPUs, AutoEMatic retains approximately 90-98% of NVFP4 decode throughput across the evaluated configurations. Furthermore, the AutoEMatic accelerator reduces latency by 1.6-2.3x and energy by 7-23% compared to an accelerator with NVFP4-only datapath at iso-accuracy.

Introduction

AI, driven by LLMs, is reaching an inflection point as the focus shifts from training to efficient inference at scale. Rising request volumes and long-horizon agentic workloads (Kariyappa & Suh, 2026) strain inference infrastructure in compute, memory bandwidth, and energy. Post-training quantization (PTQ) techniques improve inference efficiency by lowering the precision of weights, activations, or KV cache. Existing PTQ techniques (Lu et al., 2026; NVIDIA ModelOpt, 2025), however, employ static precision assignments, determined before the start of inference rather than adapting to the model’s evolving numerical demands during generation. These numerical demands vary dynamically with the prompt, between prefill and decode, and from one decoding step to the next. Therefore, a static assignment over-provisions precision where it is unnecessary and vice-versa. This motivates runtime precision adaptation: dynamically adjusting precision across prompts, layers and inference phases to match a model’s numerical demands.

Recent works have taken initial steps toward this goal. FlexQuant [5] switches layer-wise weight precision during decoding but stores separate quantized weight copies. Multi-scale quantization [12] avoids these copies by nesting several precisions within one integer bit layout whose width equals the highest supported precision. From this nested layout, lower precisions are obtained by truncating the least significant bits. DP-LLM (Kwon et al., 2026) extends this, switching each layer at decode time between two offline-calibrated precisions according to an input-dependent estimate of quantization error.

These approaches leave three challenges. First, their policies are not true runtime adaptations as they rely on offline calibration. This limits their ability to accurately track evolving precision demand during inference (Figure 1 (amber region)), yielding a suboptimal accuracy–precision tradeoff. Second, existing multi-scale formats use integer representations that suffer substantial accuracy degradation when lower precisions (sub-6-bit) are extracted from the nested representations, especially for activations and the KV cache (§ 5). Their mechanism does not directly extend to group-scaled floating-point formats such as NVFP4, which is used for low-precision inference on NVIDIA Tensor Cores. Moreover, extracting precisions from their nested representations incur irregular memory accesses and dequantization overheads [16], that sacrifice inference performance. Finally, they do not address batched inference, where one layer must serve requests of different precisions.

Figure 1. LiveCodeBench accuracy versus decode throughput for Qwen3.8-27B (batch size 1, one GB300 GPU).

Contributions. To address these limitations, we introduce AutoEMatic. At its core is a format-agnostic runtime policy that selects each layer’s precision across prefill and decode; AutoFP makes the policy practical by deriving all precisions from one NVFP4 base, and GPU kernels and an accelerator demonstrate AutoEMatic on existing and custom hardware. These four components form a full-stack co-design:

  • Runtime precision adaptation policy (§ 3.1): We identify two model-internal signals, a per-token and a per-layer measure of precision demand, and derive from their behavior a set of rules that assign layer-wise precisions.

  • Quantization format (§ 3.2): We introduce AutoFP, the first multi-scale floating-point (FP) format, which composes 2-, 4-, and 6-bit precisions from a shared NVFP4 base: the foundation from which other formats are composed. AutoFP6, the 6-bit E2M3 precision is obtained through mantissa refinement: by adding mantissa bits selectively to high quantization error base elements. The refinement metadata is stored separately from the base and applied on demand based on the runtime policy. Conversely, AutoFP2, the 2-bit precision is a quinary format, obtained by remapping the exponent bits of the base precision.

  • System implementation (§ 4): We integrate AutoEMatic into vLLM with kernels that execute AutoFP using NVFP4 Tensor Cores and efficiently serve concurrent requests.

  • Hardware accelerator (§ 6): We design an accelerator with an AutoFP-native mixed-precision datapath, which can process two AutoFP2 weights, one NVFP4 weight, or one AutoFP6 weight against a single streamed input activation.

To the best of our knowledge, AutoEMatic (Figure 1) is the first framework to jointly adapt weight, activation, and KV-cache precision at runtime, across prefill and decode, using an FP-based multi-scale format. Across reasoning and agentic benchmarks on models including Qwen3.8-27B, Gemma4-26B, and MiniMax-M2.7, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches FP6 (the group-scaled 6-bit counterpart to NVFP4) on most benchmarks, using 4.9 bits per weight versus 4.5 for NVFP4. AutoEMatic sustains this accuracy advantage while retaining 90–98% of NVFP4 throughput. On custom hardware, AutoEMatic reduces latency by 1.6–2.3× and energy by 7–23% compared to an NVFP4-only datapath at iso-accuracy.

Motivating Analyses

Precision Demand Changes During Inference

To understand how the precision across layers of a model change throughout inference, we set up an oracle that determines for each layer at every prefill and decode step, the lowest precision (WxAx-KVx) such that the KL-divergence between the quantized and full-precision model’s next-token distributions are within 10−310^{-3} nats, maintaining iso-generation. Following a full-precision generation, each layer is quantized sequentially, accounting for cumulative quantization effects. The oracle chooses between the following candidates: NVFP4, group-scaled 2-bit (E1M0) and 6-bit (E2M3) counterparts to NVFP4, and standard FP8 (E4M3).

Decode Variations. We apply this analysis to responses from DS-Qwen-14B on LiveCodeBench (LCB) and Qwen3.8-27B on τ\tau-bench, with up to 10K tokens per turn. Figure 2(a) aggregates the oracle precisions assignments across layers within the early, middle, and late thirds of each response. The distribution varies across decoding stages and models, reflecting underlying layer-wise precision variations from one decode step to the next (§C.1, Appx.).

Precision distribution, layer-wise precision assignments, and underprovisioning rates

Figure 2. (a) Precision distribution averaged across layers in the early, middle, and late generation stages (DS-Qwen-14B, LCB; Qwen3.8-27B, τ\tau-bench). (b) Layer-wise precision assignments for initial prefill and additional refills (Qwen3.8-27B, τ\tau-bench). (c) Rate of underprovisioning for ModelOpt AutoQuantize and FlexQuant (DS-Qwen-14B, LCB).

Prefill Variations. In agentic workloads, returns from tool calls extend the context and trigger additional prefills (Lu et al., 2026), termed refills. Figure 2(b) shows Qwen3.8-27B’s layer-wise oracle precision assignments over the initial prefill and additional refills. tial prefill and refills. Demand changes sharply between prefill stages: refill 3 uses 2-bit for nearly all layers, while others demonstrate a broader precision distribution (§C).

Therefore, precision demand depends jointly on the model, prompt, layer, and inference stage, and can change with any of these factors. The challenge of runtime-precision adaptation is to identify these dynamic variations without access to the full-precision reference used by the oracle.

Limitation of Static Mixed-Precision Assignment

We further ask whether a static assignment calibrated in-distribution (calibrated and evaluated on the same dataset) can closely track the oracle’s precision assignments. We examine two schemes: ModelOpt AutoQuantize (NVIDIA ModelOpt, 2025), which fixes per-layer precisions using calibrated sensitivities, and FlexQuant [5], which follows a static schedule of lowering precision during inference. Both use 100 LiveCodeBench calibration samples under a 6-bit budget, matching the oracle’s achieved average precision, and are evaluated on the remaining 780 samples. Figure 2(c) reports underprovisioning: the fraction of decode steps where the oracle requires higher precision for a layer than the static policy assigns. AutoQuantize under-provisions 26.9% on average and FlexQuant 37.6%. In-distribution calibration tailors to the aggregate precision demands of the calibration samples. However, the resulting fixed assignments or schedules do not respond to fine-grained variations across prompts and decode steps, leading to under-provisioning even on held-out samples from the same dataset, which directly impacts accuracy (§5).

Non-uniform precision requirement within groups

Raising the precision (refining) of a tensor, for example from 4 to 6 bits, is conventionally applied uniformly [18]. We instead ask whether the error of a group-scaled format such as NVFP4 is concentrated in a few elements of each group, so that refining those elements alone suffices. We partition each 16-element group into four 4-element subgroups (SG=4SG = 4; §I (Appx.)) and sweep NMSE thresholds from 0.3 to 0.001, and map the elements of every subgroup above the threshold to 6-bit (E2M3). For Qwen3-8B and DS-Qwen-14B (Figure 3), refining ~ 50% of weight subgroups achieves ~ 90% of the NMSE reduction obtained by moving from NVFP4 to FP6g^{g} uniformly. Activations exhibit a similar trend. AutoFP leverages this to reduce metadata overhead when constructing higher precisions from NVFP4.

Figure 3. Refining a subset of elements achieves NMSE comparable to 6-bit. Y-axis is NMSE, such that NVFP4 is 1 and FP6g^{g} is 0, vs. fraction of subgroups refined to 6-bit.

AutoEMatic Quantization

Runtime Precision Adaptation Policy

Precision-Adaptation Signals. AutoEMatic determines precision at runtime from two complementary model-internal signals. The first is a global token-generation uncertainty signal GE\mathcal{G}_{\mathcal{E}}, measured as the normalized Shannon entropy of the top-KK tokens’ output logits (§D.1). When the model is uncertain about the next token (high entropy), small quantization induced perturbations in the output logits can change which token it generates [6 a]. Such steps warrant higher precision and vice-versa. However, a global signal alone cannot determine layer-wise precision variations. The second signal is based on the layer structure of LLMs. LLM layers have a residual connection that carries its input directly to its output, with the attention and MLP blocks adding their contributions along the way [27]. A layer whose output hidden states Hℓ(t)\mathbf{H}_{\ell}(t) are close in representation to its input Hℓ−1(t)\mathbf{H}_{\ell-1}(t) contributes little beyond copying its input to the output; we term it a low-activity layer, and its weights, activations, and KV cache can tolerate lower precision. This local layer-update activity LA\mathcal{L}_{A} is measured at every step tt, for every layer ℓ\ell as,

LAℓ(t)=∥Hℓ(t)−Hℓ−1(t)∥2∥Hℓ−1(t)∥2+ε,(1)\mathcal{L}_{A}^{\ell}(t) = \frac{\left\lVert\mathbf{H}_{\ell}(t) - \mathbf{H}_{\ell-1}(t) \right\rVert_{2}}{\left\lVert\mathbf{H}_{\ell-1}(t) \right\rVert_{2} + \varepsilon}, \tag*{(1)}

where ε\varepsilon ensures numerical stability. The two signals combine into one precision-demand metric for prefill or decode (P/D), MP/Dℓ(t)=λP/DLAℓ(t)+(1−λP/D)GE(t)\mathcal{M}_{P/D}^{\ell}(t) = \lambda_{P/D}\mathcal{L}_{A}^{\ell}(t) + (1-\lambda_{P/D})\mathcal{G}_{\mathcal{E}}(t), with λP/D∈[0,1]\lambda_{P/D} \in[0,1] set separately for each phase2. Rather than calibrated thresholds, AutoEMatic tracks the running mean μ\mu and standard deviation σ\sigma of MP/Dℓ\mathcal{M}_{P/D}^{\ell} and establishes a set of rules, that vary across inference stages: sustained increases (decreases) of the metric over a fixed number of preceding layers relative to μ\mu and σ\sigma raise (lower) the precision of the it, with α>β\alpha> \beta. That is, AutoEMatic lowers precision more conservatively. At low-entropy, thresholds are reversed.

next layer. Intuitively, these rules observe sustained trends in the metric as evidence that precision should change, rather than reacting to isolated fluctuations. Each layer executes at WxAxW_xA_x-KVxKV_x with x∈P={2,4,6}x \in\mathcal{P} = \{2, 4, 6\} chosen by the rule. 2-bit activations and KV cache degrade accuracy substantially, therefore, the 2-bit level applies to weights only (W2A4-KV4). P\mathcal{P} denotes nominal precision levels; the policy is agnostic to the choice of levels or quantization format.

Prefill policy. Prefill precedes token generation, so no token entropy exists and we set λP=1\lambda_P = 1 (MP=LA\mathcal{M}_P = \mathcal{L}_A). We observe that, across models and datasets, MP\mathcal{M}_P shows a single large spike3 at an early layer, at a position largely invariant across prompts and refills (Figure 4(a)). Consistent with early-exit work (Shen & Zou, [23]), the spike marks the layer by which the model has integrated the prompt context, so the layers up to and including it are the most sensitive and must execute at maxP=6\mathcal{P} = 6 bits. Because the spike position differs across models, AutoEMatic detects it online as MPℓ>μ+3σ\mathcal{M}^{\ell}_P > \mu+ 3\sigma. The subsequent layers follow the rule, nPn_P layers below (above) μ\mu lower (raise) the precision of the next layer by one level in P\mathcal{P}, saturating at its bounds.

Figure 4. Layer-wise behavior of MP\mathcal{M}_{P} and MD\mathcal{M}_{D}. (a) Prefill MP\mathcal{M}_{P} of Qwen3.8-27B on a 7B-bench task. (b) Decode MD\mathcal{M}_{D} of GPT-OSS-20B on RULER-16K, with high- and low-E\mathcal{E} steps.

Decode policy. During decode, we set λD=0.7\lambda_D = 0.7 incorporating generation entropy into the metric (Figure 4(b)). Since token entropy varies little between consecutive steps (Ramachandran et al., [21]) and the GE\mathcal{G}_{\mathcal{E}} of the current step is known only after its token is generated, MD\mathcal{M}_D reuses GE\mathcal{G}_{\mathcal{E}} from the preceding step. Every decode step starts layer 0 at 4 bits and applies the rules in Table 1. Since layer tolerance to quantization varies with generation entropy, we separate decoding into high-entropy (GE≥τ\mathcal{G}_{\mathcal{E}} \ge\tau) and low-entropy (GE<τ\mathcal{G}_{\mathcal{E}} < \tau) steps by a threshold τ\tau. At high-entropy steps, the rule requires α\alpha consecutive low-activity layers to lower precision, but only β\beta consecutive high-activity layers to raise it, with α>β\alpha> \beta. That is, AutoEMatic lowers precision more conservatively. At low-entropy, thresholds are reversed.

PolicyPrefillDecode
MetricMP=LA\mathcal{M}_P = \mathcal{L}_AHigh entropyLow entropy
MD=0.7LA+0.3GE\mathcal{M}_D = 0.7\mathcal{L}_A + 0.3\mathcal{G}_\mathcal{E}MD=0.7LA+0.3GE\mathcal{M}_D = 0.7\mathcal{L}_A + 0.3\mathcal{G}_\mathcal{E}
Initial precision6-bit4-bit4-bit
Rules activateAfter spikeLayer 0Layer 0
Raise precisionnPn_P layers >μ> \muβ\beta layers >μ> \muα\alpha layers >μ> \mu
Lower precisionnPn_P layers <μ< \muα\alpha layers <μ< \muβ\beta layers <μ< \mu

Table 1. Runtime precision adaptation policy.

AutoFP Quantization Format

Storing one quantized copy per level in P\mathcal{P} would cost 13.5 bits per weight. AutoFP instead stores one representation, the NVFP4 base with its group scales and additional metadata, and derives the 2- and 6-bit levels from it on demand.

AutoFP6. Each additional mantissa bit adds local precision around every NVFP4 value. However, a floating-point value cannot be refined by ‘appending’ mantissa bits. Since NVFP4 rounds to nearest, the full-precision value may lie below the E2M1 value, whereas appended mantissa bits can only select values above it, and not the nearest E2M3 value in the neighborhood that minimizes quantization error (e.g., 1.4 is stored as 1.5 in E2M1; appended bits can then reach only 1.5–1.875, never 1.375).

Therefore, instead of storing the additional metadata as mantissa bits, AutoFP encodes them as additive signed offsets. Because the E2M1 grid is a subset of the E2M3 grid, every E2M1 value XqX_q is also an E2M3 value. Refining XqX_q only requires storing the direction and number of E2M3 grid steps to move from it. Since E2M3 adds three additional values around every XqX_q, a 2-bit two’s-complement signed offset d∈{−2,−1,0,+1}d \in\{-2,-1,0,+1\} offers sufficient resolution. The step size around XqX_q with exponent ee is 2e−32^{e-3}, and the refinement term is δ=d⋅2e−3\delta= d \cdot2^{e-3} (the mantissa equivalent value of the signed offset) and the refined value is thus, Xq+δX_q + \delta. The refinement is thus additive. Since the step 2e−32^{e-3} is dependent on the exponent of XqX_q, decoding dd exactly would require a per-element exponent computation (§7). AutoFP therefore decodes dd with a fixed step 2−32^{-3}, δ=d⋅2−3∈{−14,−18,0,+18}\delta= d \cdot2^{-3} \in\left\{-\frac{1}{4},-\frac{1}{8},0,+\frac{1}{8}\right\}. The fixed step is E2M3-like since it reproduces the E2M3 grid for values <2< 2 and applies a step finer than E2M3 for larger values.

To reduce the metadata overhead of storing signed offsets, AutoFP6 refines only the subgroups with the largest NMSE reduction (§2.3). Each 16-element group carries a 4-bit refinement mask MXM_X with one bit per 4-element subgroup, set to 1 for a refined subgroup and 0 otherwise, and the signed offsets are stored only for the subgroups with mask bit of 1 (Figure 5). The NVFP4 base is left intact, the signed offsets are stored separately and applied on demand for AutoFP6 layers. Since the refinement is additive, every refined operand then splits as X^=Xq+MX⊙Xr\hat{X} = X_q + M_X \odot X_r for X∈{W,A,K,V}X \in\{W, A, K, V\}, where Xr=δX_r = \delta. By linearity, a refined GEMM is, W^A^=WqAq+(MW⊙Wr)Aq+Wq(MA⊙Ar)+(MW⊙Wr)(MA⊙Ar)\hat{W}\hat{A} = W_qA_q + (M_W \odot W_r)A_q + W_q(M_A \odot A_r) + (M_W \odot W_r)(M_A \odot A_r).

Mantissa refinement of NVFP4 to AutoFP6

Figure 5. Mantissa refinement of NVFP4 to AutoFP6.

Two choices make this hardware-friendly. First, AutoEMatic imposes N ⁣: ⁣4N\!:\!4 structured sparse (Ramachandran et al., [20]) refinement within each group, such that NWN_W weight and NA=4−NWN_A = 4 - N_W activation subgroups are refined per group. Second, it keeps MWM_W and MAM_A disjoint. MWM_W selects NWN_W subgroups for refinement (along KK-dimension) and is shared across the rows of WW, and MAM_A, shared across the columns of AA. This ensures disjointness holds for every row–column pair in the GEMM. Disjointness eliminates the second-order product of refinements, which contributes negligibly to accuracy (§7), reducing the number of additional operations required for a refined layer. MAM_A is selected at runtime and MWM_W is selected offline under a refinement budget ρ\rho, the fraction of weight subgroups refined, either by NMSE or by a refinement-aware GPTQ calibration that assigns refinement for subgroups with the largest Hessian-weighted error (§E).

AutoFP2. Truncating the stored NVFP4 to 2 bits yields an E1M0 format representing ±1,±2\pm1,\pm2, which cannot represent zero and causes significant accuracy degradation. AutoFP2 instead is a quinary format, Q∈{−2,−1,0,1,2}Q \in\{-2,-1,0,1,2\}, derived from the NVFP4 code by retaining the sign bit ss and remapping the two exponent bits e1e0e_1e_0 to a 2-bit magnitude in {0,1,2}\{0,1,2\} on demand via {e1⋅e0,e1⊕e0}\{e_1 \cdot e_0, e_1 \oplus e_0\}. Nominally, representing AutoFP2 requires 3-bits. However, we show in § 6, the accelerator packs two AutoFP2 digits into 5 bits, 2.5 bits per digit. AutoFP2 reuses the NVFP4 scales which normalize to [−6,6][-6,6]; since QQ reaches only ±2\pm2, the scale is multiplied by a constant cc (§ 5). As AutoFP2 is derived from NVFP4 on demand, it saves no memory bandwidth, and its main benefit is in prefill throughput and energy (§ 6).

System Implementation

We implement kernels that executes AutoFP6 natively on NVFP4 Tensor Cores from a single packed model: E2M1 base values with E4M3 group scales sW/As_{W/A}, signed offsets dd stored as one byte per refined subgroup, and refinement masks MW/AM_{W/A} stored as 4 bits per group along the KK-dimension (§D.3, Appx.). Since Tensor Cores do not support sub-4-bit formats, AutoFP2 layers are emulated as NVFP4, which preserves the accuracy of the precision decision but not the performance of AutoFP2 execution.

Refinement as an Extended-K NVFP4 Operation

A GEMM accumulates products along its reduction dimension KK, so a sum of GEMMs with the same output shape equals one GEMM over operands concatenated along KK (see Appx.). AutoEMatic executes the refined GEMM of § 3.2 in this manner. Since MWM_W and MAM_A are disjoint, the two refinement terms can be merged into a single GEMM of up to length-KK as, WqrAqrW_{qr}A_{qr} (Figure 6). Each of these operands are constructed as follows: for subgroups that refine weights, WqrW_{qr} holds the refinement term δW\delta_W and AqrA_{qr} holds the activation base values AqA_q in the corresponding position, such that WqrAqrW_{qr}A_{qr} realizes (MW⊙Wr)Aq(M_W \odot W_r)A_q. Similarly, for subgroups with activation-refinement, WqrW_{qr} holds the weight base values WqW_q and AqrA_{qr} holds δA\delta_A, realizing Wq(MA⊙Ar)W_q(M_A \odot A_r). The reduction dimension is up to length-KK because it depends on the number of subgroups refined. A refined layer thus executes as a single NVFP4 GEMM of reduction length up to 2K2K as, [Wq∣Wqr]⋅[Aq∣Aqr][W_q \mid W_{qr}] \cdot[A_q \mid A_{qr}], which we term the extended-K GEMM. The GEMMs use Blackwell’s tcgen05.mma instruction, which consumes only E2M1 values with E4M3 group scales. The refinement δ\delta is not representable in E2M1 as is, whereas its scaled version 4δ∈{−1,−0.5,0,+0.5}4\delta\in\{-1,-0.5,0,+0.5\} is. The refined subgroups of WqrW_{qr} and AqrA_{qr} therefore hold 4δ4\delta, and the additional factor of four is absorbed into the scale of WqrW_{qr}, which is set to sW/4s_W/4. Each refined subgroup then contributes (sW/4⋅4δW)(sAAq)=sWsAδWAq(s_W/4 \cdot4\delta_W)(s_AA_q)=s_Ws_A\delta_WA_q or (sW/4⋅Wq)(sA⋅4δA)=sWsAWqδA(s_W/4\cdot W_q)(s_A\cdot4\delta_A)=s_Ws_AW_q\delta_A. The kernel fuses the construction of both extended operands with the MMA. Attention follows the same construction and runs on Flash-Infer’s FP8 attention kernel. As in ThinKV ([21]), AutoEMatic uses two page types of equal byte size: a base page holds TT unrefined tokens with only the base values, and a refinement page holds T/2T/2 refined tokens with base values and signed offsets in NVFP4. This enables one paged allocator to serve both types.

Figure 6. Extended-KK GEMM for one 16-element group and three requests (batch size of 3).

Serving Mixed-Precision Batches

Requests in a batch may select different precisions for the same layer. The weight operands do not vary with that choice: WqW_q is shared by every request, and WrW_r with its mask MWM_W is fixed offline. Only the activation masks MAM_A and their signed offsets dd differ across requests. Rather than running a separate GEMM for each precision decision, AutoEMatic forms the union of masks UMAU_{M_A}, the per-group OR of the activation masks of all requests in the batch, and serves the whole batch with one extended-KK GEMM against a single WqrW_{qr} built from it. As shown in Figure 6, WqrW_{qr} holds 4δW4\delta_W in its weight-refined subgroups, WqW_q in every subgroup of UMAU_{M_A}, and zero elsewhere. Each request then constructs its own AqrA_{qr}: AqA_q in the weight-refined subgroups,

4δA4\delta_A in the subgroups it refines.

Accuracy and GPU Performance

Experimental Setup

Models, Datasets, and Baselines. We evaluate Qwen3-8B, Qwen3.8-27B, DeepSeek-R1-Llama-70B, Gemma-4-26B-A4B, and MiniMax-M2.7 on reasoning (LongBench v2 (Bai et al., 2025), AIME 2024 [11], LiveCodeBench (Jain et al., 2025)) and agentic (τ2\tau^2-bench (Barres et al., 2025), LongMemEval (Wu et al., 2024), SWE-bench Verified [15]) workloads. For WxAx-KVx quantization, we compare against uniformly quantized NVFP4 and FP6g^{g}, AutoQuantize (NVIDIA ModelOpt, [13]), which selects between NVFP4 or FP6g^{g} per layer under a bit budget, and DP-LLM (Kwon et al., 2026), which switches among 3- to 6-bit of an Any-Precision format (Park et al., [17]). Since DP-LLM adapts only weights, we extend its policy to activations and KV cache (DP-LLM†^{\dagger}) for a fair comparison. AutoQuantize, DP-LLM†^{\dagger}, and AutoEMatic’s refinement-aware GPTQ for weights use 256 samples from WikiText-2 for calibration. All experiments run in vLLM (Kwon et al., 2023) on NVIDIA GB300 GPUs with a 32K-token generation limit at temperature 0.5.

Hyperparameters. We set P={2,4,6}\mathcal{P} = \{2, 4, 6\}; rules np=2n_p = 2 and α=4\alpha= 4, β=2\beta= 2; entropy threshold τ=0.10\tau= 0.10, GE\mathcal{G}_{\mathcal{E}} is computed over the top-K=10K = 10 logits; subgroup size SG=4SG = 4; AutoFP2 scale multiplier c=2c = 2; activation NMSE threshold 0.003; weight refinement budget ρ=0.2\rho= 0.2; KV page size T=64T = 64. Hyperparameters are ablated in Appx.

Storage and Execution Precision. Storage precision is the packed model size in average bits per weight including metadata. For AutoQuantize, it is the target bit budget plus 0.5-bits for scale, 6-bits for DP-LLM†^{\dagger} and 4+0.5+2ρ=4.94 + 0.5 + 2\rho= 4.9 bits for AutoEMatic at ρ=0.2\rho= 0.2 (since the masks are shared across rows for weights, it is negligible). Execution precision is the nominal bitwidth at which layers execute averaged over layers and inference steps.

Main Accuracy Results

Comparison with Static and Runtime Baselines. AutoEMatic operates without a bit budget: its execution precision follows the dynamic decisions of the runtime policy. For a fair comparison, AutoQuantize and DP-LLM†^{\dagger} are calibrated to the budget nearest AutoEMatic’s achieved average. As shown in Table 2, across models and datasets, at 4.5–5.1 bits of average execution precision, accuracy of AutoEMatic exceeds AutoQuantize by up to 10% and DP-LLM†^{\dagger} by up to 66.9% under a similar budget. AutoQuantize has a high underprovision rate which directly impacts its accuracy. AutoEMatic’s advantage is particularly pronounced in agentic tasks, whose repeated prefills and long decodes impose dynamic precision demands, that DP-LLM†^{\dagger}’s offline-calibrated runtime-policy cannot reliably track. DP-LLM†^{\dagger} is further penalized by its nested integer format: the sub-6-bit precisions extracted from the shared representation incur substantial error. Overall, through runtime adaptation driven by model-internal signals and the AutoFP format, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches FP6g^{g}, with a storage precision of 4.9 bits.

ModelMethod (budget)LB v2AIMELCBτ2\tau^2-benchLME
Qwen3-8BBF1635.456.766.530.742.8
NVFP433.350.049.217.526.0
FP6g^{g}35.456.761.430.739.9
AutoQuantize (4.5)35.450.060.020.231.9
DP-LLM†^{\dagger} (4.5)21.6 (4.8)16.7 (4.5)26.6 (4.6)10.6 (4.5)18.5 (4.5)
AutoEMatic35.4 (4.6)56.7 (4.7)60.5 (4.8)27.2 (4.6)38.2 (5.1)
Qwen3.8-27BBF1656.993.376.374.682.8
NVFP448.786.770.650.070.6
FP6g^{g}56.986.776.371.782.8
AutoQuantize (4.8)56.986.774.962.081.2
DP-LLM†^{\dagger} (4.8)25.4 (4.7)56.7 (4.8)51.4 (4.5)25.6 (4.6)15.9 (4.7)
AutoEMatic56.9 (4.5)86.7 (4.6)76.3 (4.6)71.7 (5.1)82.8 (4.8)
Gemma-4-26BBF1646.786.785.761.472.6
NVFP440.263.375.346.563.6
FP6g^{g}45.580.083.160.570.1
AutoQuantize (5.0)41.080.080.158.668.1
DP-LLM†^{\dagger} (5.0)35.6 (4.8)50.0 (5.4)68.6 (5.4)11.5 (4.9)16.0 (4.9)
AutoEMatic45.5 (5.0)80.0 (5.0)83.1 (5.0)60.5 (5.1)70.1 (4.9)

Table 2. Accuracy comparison of AutoEMatic and baselines across reasoning and agentic workloads. Execution precision appears in parentheses next to accuracy. LB v2: LongBench v2, LCB: LiveCodeBench, LME: LongMemEval.

Large-Scale Agentic Analysis. On MiniMax-M2.7, a 230B-parameter agentic coding model, evaluated on SWE-bench Verified, AutoEMatic matches FP6g^{g} accuracy and remains within 0.5% of baseline (at FP8) at an average execution precision of 4.2 bits in prefill and 4.5 in decode, against 6 bits for FP6g^{g} (Table 3). Figure 7 demonstrates AutoEMatic’s runtime adaptation across prefill and decode.

Layer-wise precision assignment by AutoEMatic on MiniMax-M2.7

Figure 7. Layer-wise precision assignment by AutoEMatic on MiniMax-M2.7.

MethodExec. PrecisionAcc. (%)
PrefillDecode
FP88871.0
NVFP44467.8
FP6g^{g}6670.5
AutoEMatic4.24.570.5

Table 3. Accuracy comparison of MiniMax-M2.7 on SWE-bench Verified.

GPU Performance Analysis

Figure 8 reports prefill time-to-first-token (TTFT) and decode throughput for DS-Llama-70B and MiniMax-M2.7 on 4× GB300 GPUs, sweeping input lengths of 1K–128K tokens for prefill and batch sizes of 1–128 for decode with an 8K-token input and a 16K-token output. We compare against methods that execute on Tensor Cores: NVFP4 through the native CUTLASS W4A4 path, FP6g^{g} through a fused kernel that dequantizes E2M3 to FP8, and AutoQuantize uses either based on NVFP4/FP6g^{g} layers. AutoEMatic’s runs use precision assignments from LongBench v2.

Decode throughput and prefill TTFT analysis

Figure 8. Decode throughput and prefill TTFT analysis on 4× GB300 GPUs for DS-LLaMA-70B and MiniMax-M2.7.

Prefill. AutoEMatic reduces TTFT by up to 1.4× over FP6g^{g} and 1.25× over AutoQuantize across input lengths. Specifically, despite AutoEMatic’s runtime precision adaptation enabling superior accuracy compared to NVFP4, the TTFT is within 10% of the NVFP4 baseline (small performance penalty). Two factors keep this performance penalty small: (a) when all requests in a batch choose 4-bit for a layer, they execute at NVFP4 throughput and (b) when any request chooses AutoFP6 it executes as a single extended-KK GEMM on Tensor Cores, instead of separate GEMMs (§7).

Decode. AutoEMatic improves decode throughput by 1.5–2.5× over the full-precision baseline across batch sizes, and attains 90–98% of NVFP4 throughput, i.e., near-NVFP4 throughput at FP6g^{g} accuracy.

Accelerator Architecture

Architecture Overview

On GPUs, AutoFP2 is emulated and AutoFP6 requires an extension of the reduction dimension. An AutoFP-native datapath targets both limitations of existing hardware. We extend a representative accelerator MAGNet (Venkatesan et al., [24]; Hooper et al. (2025)) as proof-of-concept. The output-stationary local-weight-stationary (OS-LWS) design (Figure 9(a)) has 256 PEs and a global SRAM (L2) that loads data from DRAM (modeled). Each PE (Figure 9(b)) has mixed-precision vector MAC (vMAC) that computes one VV-element NVFP4/AutoFP6 or two VV-element AutoFP2 dot products per cycle. A PPU processes completed outputs (Figure 9(c)). Signals mode_nvfp4 and refine_en select AutoFP2, NVFP4, or AutoFP6 modes of operation.

Figure 9. AutoEMatic accelerator architecture.

Weight Encoder and Packed AutoFP2 Format

Weights are stored in DRAM as NVFP4 (§ 3.2), but inside the PEs every weight is held in AutoFP2, so that the vMAC can execute all modes with uniformity. Two requirements follow. First, for the vMAC to process AutoFP2 at twice the throughput of NVFP4, each weight load from the buffers must deliver two AutoFP2 weights. Second, in NVFP4 mode a single AutoFP2 digit cannot represent an NVFP4 value. We introduce a packed AutoFP2 format that satisfies both by encoding a pair of AutoFP2 digits (Q1,Q0Q_1,Q_0) in 5 bits (2.5 bits per digit). In the packed format (Figure 10), an escape bit EE precedes two 2-bit fields F1F_1 and F0F_0. A nonzero AutoFP2 digit is one of ±1 and ±2. When both digits are nonzero (E=0E = 0), F1F_1 and F0F_0 hold them in E1M0. Otherwise (E=1E = 1), F1F_1 records which digits are zero and F0F_0 holds the nonzero digit, if any. The weight encoder forms the AutoFP2 packed format as weights enter the PE and writes them to the weight buffer. The packing overhead is amortized by the weight stationary dataflow. In AutoFP2 mode (mode_nvfp4=0), it applies the exponent remapping of § 3.2 to two weights from adjacent rows (different KK dimensions) that share an input activation and packs the two digits. In NVFP4/AutoFP6 modes (mode_nvfp4=1), the pair represents one NVFP4 weight as Wq=c0Q0+c1Q1W_q = c_0Q_0 + c_1Q_1 with fixed coefficients. We set W‾q=(5Q0+2Q1)/2\overline{W}_q = (5Q_0 + 2Q_1)/2, such that their linear combinations span the maximum NVFP4 magnitude 6 (derivation in Appx.). For e.g., 0.5=(5⋅1+2⋅(−2))/20.5 = (5\cdot1 + 2\cdot(-2))/2. Each AutoFP2 digit is a Boolean function of the E2M1 bits (§D.7).

Figure 10. AutoFP2 representation.

vMAC Datapath Design

Each PE is a V=32V = 32 element wide vMAC (Figure 11) organized as two V/2=16V/2 = 16 base lane blocks each with dedicated set of refinement lanes and pipelined into multiply and accumulate stages. The vMAC accepts weights in the packed AutoFP2 format. The refinement data has a separate input interface carrying the masks and signed offsets of the refined subgroups. Each base lane kk comprises an even (F0,kF_{0,k}) and an odd (F1,kF_{1,k}) AutoFP2 sub-lane that shares one streamed NVFP4 activation. For energy-efficiency, all computations use fixed-point arithmetic (§D.8).

Figure 11. AutoFP vector MAC datapath.

Base Lanes. In the multiply stage, the escape bit EE of each packed pair decodes the two digits. When E=1E = 1, F1F_1 identifies which digit is zero, and the corresponding sub-lane is bypassed: its product is set to zero. For a nonzero digit, F1F_1 or F0F_0 holds an E1M0 code, ±1\pm1 or ±2\pm2, so each sub-lane forms AkFj,kA_kF_{j,k}, j∈{0,1}j \in\{0, 1\}, with simple shifters. Within a block, the even and odd sub-lanes feed separate adder trees in the adder stage, A=∑k=015AkF0,kA = \sum_{k=0}^{15} A_kF_{0,k} and B=∑k=015AkF1,kB = \sum_{k=0}^{15} A_kF_{1,k} for lanes 0–15, and CC, DD for lanes 16–31. Every execution mode differs only in how the tree outputs are combined. In NVFP4 mode, the AA, BB correspond to the partial sum accumulated from the same weight Wq,kW_{q,k}, therefore the mode-dependent reduction unit computes P0=(5A+2B)/2P_0 = (5A + 2B)/2 (implemented in fixed-point) to form the corresponding NVFP4 partial sum, and similarly for P1P_1, as shown in Figure 11. The base lanes perform the same operation in AutoFP6 mode. In AutoFP2 mode, within a block the two sub-lanes process corresponding weights from different rows (so that they can share one input activation). The weights of the same row are therefore spread across the even and odd sub-lanes of both blocks, so the mode-dependent reduction unit sums across blocks,

P0=A+CP_0 = A + C and P1=B+DP_1 = B + D.

Sparse Refinement Lanes. The refinement datapath adds V/4=8V/4 = 8 refinement lanes per block as two SG=4SG = 4-lane slots (in §7 we establish this as sufficient). In AutoFP2/NVFP4 mode (refine_en=0), the refinement lanes are gated for energy efficiency. Because MWM_W and MAM_A are disjoint, a refined weight/activation sub-group occupies exactly one slot. The collector supplies each slot with the refinement mask, four 2-bit signed-offsets dd, and a bit identifying weight/activation refinement. Each lane has two 4:1 muxes that pick the corresponding base-weight and activation subgroups using the mask; a 2:1 selector chooses the corresponding XqX_q: AqA_q for weight refinement, or WqW_q for activation refinement (note the similarity with construction of WqrW_{qr}, AqrA_{qr}). Each lane forms R=2XqdR = 2X_qd in fixed-point arithmetic. Each refinement lanes’ contributions are accumulated as R=∑RR = \sum R. It is accumulated with the base lanes partial sums as, Oj=4Pj+RO_j = 4P_j + R.

Scaling and accumulation. Each OjO_j is multiplied by the scale factor of its corresponding block. In AutoFP2 mode, each OjO_j is a 32-element dot product corresponding to two different weight rows with the same activations. In NVFP4 and AutoFP6 mode, each OjO_j corresponds to a 16-element dot product along the same row (same KK dimension). After scaling, O0O_0 and O1O_1 are summed into one 64-bit accumulator in NVFP4/AutoFP6, while in AutoFP2 mode O0,O1O_0,O_1 are held in the two 32-bit halves of the accumulator.

Shared Post-Processing Unit

The PPU (Figure 9(c)) quantizes completed PE outputs into the next layer’s input activations. If a layer requires refinement, it also computes per-subgroup NMSE and selects subgroups with NMSE above τ\tau excluding those in MWM_W and assigns signed offsets dd. The PPU also performs non-linear operations, and computes the metric MP/DM_{P/D}.

Accelerator Evaluation

Implementation. The accelerator is implemented in SystemC, with RTL generated using Siemens Catapult HLS. We use a 2MB global SRAM, 896B weight, 1.5KB activation, and 2KB accumulation buffers. We perform synthesis in TSMC 3nm at a frequency of 1 GHz. Synopsys VCS is used for simulation and PrimeTime PX for power estimation.

Area Analysis. Figure 12(a) breaks down the area of the AutoEMatic accelerator. The 2MB global SRAM occupies 75% of the area. The PE array, with all 256 PEs, takes 23.8%. The per-PE weight encoders account for only 0.4% and the vMAC 3.6% of the total area; within a vMAC the refinement lanes (12.3%) and the mode-dependent reduction logic for NVFP4- and AutoFP2-mode (6.7%), result in a net 1.2× overhead over an NVFP4-only vMAC. The accelerator is only 1.1% more area than the NVFP4 baseline.

Figure 12. Area breakdown of (a) the AutoEMatic accelerator and (b) its vMAC.

Iso-Accuracy Performance Comparison. We compare against single-format AutoFP2 and NVFP4 baselines under iso-accuracy with AutoEMatic by replaying its layer-wise precision traces, as recorded on LiveCodeBench. Layers above a baseline’s native precision execute with an extended reduction dimension, accumulating into the same partial sums; the NVFP4 design, for instance, executes a refined layer analogous to the extended-KK GEMM (§ 4). Figure 13(a) reports end-to-end latency normalized to the NVFP4 design at batch sizes 1 and 8. AutoEMatic is 1.6–2.3× faster than NVFP4 for two reasons: AutoFP2 layers run at twice the NVFP4 throughput, which benefits compute-bound prefill most (1.3–1.8× faster), and refined layers execute natively in the sparse refinement lanes without the additional pass an extended-KK reduction requires. Across evaluated models, on average, AutoEMatic achieves a compute density of 5.46 TOPS/mm2^2, compared to 3.23 TOPS/mm2^2 for NVFP4 and 6.16 TOPS/mm2^2 for AutoFP2.

Figure 13. Normalized latency and energy comparison with single-format baselines, under iso-accuracy.

Iso-Accuracy Energy Comparison. Figure 13(b) reports energy for the same traces, normalized to NVFP4 and broken down into different components, each including static and dynamic energy. AutoEMatic consumes 0.77–0.93× the energy of NVFP4, as (a) all modes use AutoFP2 lanes and (b) lower runtime. Its vMAC datapath averages 15fJ/op, against 22fJ/op for NVFP4 and 10fJ/op for AutoFP2.

Discussion and Ablations4

[Algorithm+Quantization] Component Ablations. Table 4 isolates the runtime policy and the quantization format. Applying AutoEMatic’s runtime policy to separately quantized AutoFP2, NVFP4, and FP6 copies (row 3), attains 66.2%, but needs 13.5 stored bits per weight. Applied to the integer multi-scale Any-Precision format (row 4), attains only 43.8%. This suggests that a favorable accuracy–efficiency tradeoff requires both: the runtime policy and an FP-based multi-scale format. Combined (row 7), AutoEMatic preserves 65.7%, within 0.6% of the separate-copy upper bound, at 4.9 bits, executing 40% layers at 6-bit, 45% at 4-bit and 15% at 2-bit. Selecting the refined weight sub-groups by NMSE alone, with a threshold of 0.003 at the same budget (Table 4, row 5), attains 64.8%; refinement-aware GPTQ (row 6), which selects them by their reduction in the Hessian-weighted error raises accuracy to 65.6%.

ComponentBitsAcc.2 / 4 / 6 (%)
1. BF161668.4–
2. NVFP4 (W-A-KV)4.551.70 / 100 / 0
3. + AutoEMatic runtime policy w/ P={2,4,6}P = \{2, 4, 6\} copies13.566.220 / 40 / 40
4. AnyPrecision LLM on PP + AutoEMatic runtime policy643.88 / 42 / 50
5. AutoFP64.964.80 / 0 / 100
6. + Refinement-aware GPTQ4.965.60 / 0 / 100
7. AutoEMatic format + runtime policy, P={2,4,6}P = \{2, 4, 6\}4.965.615 / 45 / 40

Table 4. Component ablations with DS-LLaMA-70B on LCB.

[Quantization] Refinement Decoding. Figure 14(a) compares decoding signed offsets with the fixed step 2−32^{-3} against exact, per-element exponent computation 2e−32^{e-3} on DS-LLaMA-70B and Qwen3-8B on LCB. The fixed-step has similar accuracy as the exact exponent computation, while achieving up to 28% higher decode throughput. This is because the fixed-step does not require loading the base value when constructing the extended-KK GEMM operands.

Figure 14. Ablations. (a) Fixed vs. exponent-scaled decoding of dd. (b) Underprovision rate of AutoEMatic. (c) Prefill precision at spike. (d) Refined subgroups per group. (e) Accuracy vs. throughput pareto. (f) Kernel ablations. (g) For one decode step: MD\mathcal{M}_D across layers and the precision assigned by the oracle, an instantaneous rule, and AutoEMatic.

[Algorithm] AutoEMatic Runtime Policy vs. Oracle. Figure 14(b) compares per-layer underprovision rate of AutoEMatic’s precision decisions against the oracle of §2{}^{\S}2 restricted to the precision set P\mathcal{P}. AutoEMatic underprovisions only 7.7% of layer executions on average, and its 2/4/6-bit distribution of 15/45/40% closely tracks the oracle’s 20/41/39%. Using LΛ\mathcal{L}_{\Lambda} alone has a higher rate of underprovisioning, suggesting the combined metric MΛ\mathcal{M}_{\Lambda} as the optimal choice. We omit Gε\mathcal{G}_{\varepsilon} from this comparison because it is a global signal shared across layers.

[Algorithm] Rule-based vs. Instantaneous Adaptation. Figure 14(g) traces one decode step of DS-Qwen-14B layer by layer: MD\mathcal{M}_{D} (top) and the precision assigned by the oracle of §2.1{}^{\S}2.1, by an instantaneous policy that switches with every rise and fall in MD\mathcal{M}_{D}, and by AutoEMatic’s rule-based policy. Evident from the oracle’s own behavior, the rules are not ad hoc, oracle precision does not rise (drop) immediately after an increase (decrease) in the metric, but only once4: See Appx. for additional discussions and ablations. the change persists over several layers. The instantaneous policy switches on blips and drops to 2 bits at layers where the oracle requires 6. AutoEMatic’s rules are built inspired by this behavior of the oracle and change precision only once the model has provided sufficient evidence.

[Algorithm] Prefill Precision at the Spike. Figure 14(c) varies only how the spiking layer in prefill is executed, keeping the decode policy fixed. Running prefill entirely in BF16 gives 71.7%, the same as AutoEMatic, which executes the spike layers at 6 bits. Lowering the spike layer to NVFP4 costs 3.8% and to AutoFP2 15.9%.

[Quantization] Subgroups Refined Per Group. Figure 14(d) shows the distribution of weight+activation subgroups refined per group, NW+NAN_W + N_A, for DS-LLaMA-70B on LCB and MiniMax-M2.7 on SWE-bench. 22–31% of groups need no refinement (0:4), and only 1.1% of DS-LLaMA-70B groups and none in MiniMax-M2.7 exceed 2:4, supporting the choice of 8 refinement lanes.

[System] Kernel Ablations. Figure 14(e) plots accuracy against decode throughput normalized to NVFP4 for DS-LLaMA-70B on LCB at batch 512. AutoEMatic lies on the Pareto frontier: 1.3× faster than FP6, and 1.1× faster than uniform AutoFP6. AutoEMatic’s superior performance-accuracy tradeoff stems from the runtime policy: it assigns 4 bits to most layers, which then run at NVFP4 throughput, and 6 bits only where accuracy requires it, which still runs as a single extended-KK GEMM on the same NVFP4 Tensor Cores. Figure 14(f) ablates AutoEMatic’s two kernel mechanisms under different batch sizes. Without the extended-KK GEMM, executing the base and refinement GEMMs separately and summing their outputs lowers throughput by 1.5× at batch size 1; without the union of masks, splitting mixed-precision batches into per-precision sub-batches lowers throughput by 1.6× at batch size 512.

[Accelerator] Precision Mix Sweep. Figure 15 evaluates different layer-wise precision distributions on DS-LLaMA-70B against an NVFP4-only datapath. AutoEMatic’s runtime assignment (15/45/40) lies on the Pareto frontiers of accuracy versus area efficiency and accuracy versus energy efficiency among the evaluated configurations of different precision distributions, achieving 65.6% accuracy with only an 8% performance/area penalty and 9% higher energy/op.

Figure 15. DS-LLaMA-70B LCB accuracy vs. performance per area and energy per op, normalized to an NVFP4-only datapath.

Related Work

Multi-scale quantization. Any-Precision LLM ([17]), Matryoshka Quantization ([12]), Any-BCQ ([16]), and RRQ ([10]) store several precisions within one integer representation, as nested bit planes or successive residual stages, from which the selected precision is reassembled before computation.

Runtime precision adaptation. FlexQuant ([5], DP-LLM (Kwon et al., 2026), MoBiQuant [25], and QuickSilver (Khanna et al., 2025) adapt weight precision during decoding through predetermined schedules, calibrated error thresholds, or trained routers, and Mix-Quant (Lu et al., 2026) assigns NVFP4 to prefill and BF16 to decoding. For the KV cache, PM-KVQ [6] progressively lowers precision and ThinKV [21] assigns precision by reasoning importance. Appx. provides an extended discussion.

Conclusion

AutoEMatic shows that adapting precision to the evolving numerical demands of LLM inference improves its accuracy–efficiency tradeoff. AutoFP switches precision at runtime through mantissa refinement and exponent remapping of a shared NVFP4 base, without requiring redundant weight copies. Future work will explore achieving memory benefits with AutoFP2 and extending AutoFP to higher precisions through joint exponent and mantissa refinement.

Extended Related Work

Static post-training quantization. Most PTQ methods choose a precision offline and keep it fixed. GPTQ (Frantar et al., 2023) quantizes a weight matrix one column at a time and uses approximate inverse-Hessian information to update the remaining weights, compensating the quantization error. AWQ [4] identifies salient weight channels from activation statistics and protects them by per-channel scaling before quantization.

Group-scaled floating-point formats and their extensions. MX formats [22, 14] share a power-of-two E8M0 scale across a group of 32 FP8, FP6, FP4, or INT8 elements, and NVFP4 [2] stores E2M1 elements in groups of 16 with an E4M3 group scale and a per-tensor FP32 scale. MX+ (Lee et al., 2025) observes that the group’s largest element does not need its exponent field given the shared scale and repurposes those bits as extra mantissa for that element. M²XFP (Hu et al., 2026) adds two metadata bits per subgroup, which serve as extra mantissa bits for the subgroup’s largest activation and as a scale multiplier for weights. AdaMX [9] chooses per group between FP4 and INT4 grids combined with group-maximum mantissa extension or scale refinement at a constant bit width. These extensions are fixed after quantization and refine one element or one scale per group. AutoEMatic is generally applicable to any group-scaled floating-point format.

Multi-scale quantization. Multi-scale methods store several precisions in one representation so that precision can change without separate checkpoints. Any-Precision LLM [17] quantizes a seed model at the lowest bit width with non-uniform clustering, then adds one bit at a time by splitting each cluster, storing the codes as bit planes so that lower-bit codes are prefixes of higher-bit codes. Matryoshka Quantization [12] co-trains a single INT8 model so that slicing its most significant bits yields usable INT4 and INT2 models. AnyBCQ [16] builds 2- to 4-bit models on binary-coded quantization by freezing existing bit planes and adding residual planes with re-optimized scales, and RRQ [10] represents weights as a 2-bit base plus successive 2-bit quantized residuals, so that 4-, 6-, and 8-bit precisions are sums of residual stages.

Runtime precision adaptation. Mix-Quant (Lu et al.,

  1. applies NVFP4 to prefill and keeps BF16 for decoding, fixing each phase’s precision. FlexQuant Liu et al., [5] stores both INT8 and INT4 weights and, during decoding, replaces INT8 layers with INT4 in an order fixed offline by KL-divergence analysis whenever a moving average of output entropy falls below a threshold. MoBiQuant Wang et al., [25] decomposes weights into recursive 2-bit residual slices and uses a trained token router to decide how many slices each token activates. QuickSilver Khanna et al., 2025 computes, at a layer, the entropy of each token’s hidden representation and assigns 8-, 4-, or 2-bit precision by thresholds.

KV-cache compression. KIVI Liu et al., [7] quantizes the key cache per channel and the value cache per token to 2 bits without tuning, and KVQuant Hooper et al., 2024 combines per-channel pre-RoPE key quantization, sensitivity-weighted non-uniform datatypes, and a dense-and-sparse split that keeps outliers separately. For long chain-of-thought reasoning, PM-KVQ Liu et al., [6] lowers the bit width of the KV cache progressively as the context grows and allocates higher bit widths to more sensitive blocks, and ThinKV Ramachandran et al., [21] classifies chain-of-thought segments into thought types from attention sparsity, assigning token precision by thought importance and evicting tokens of less critical thoughts.

Mixed-precision accelerators. MicroScopiQ Ramachandran et al., [19] retains outlier weights at higher precision and offsets their additional bits by pruning the least important weights, executing on multi-precision integer processing elements with a network-on-chip that routes the high-precision outliers. AdaMX Luo et al., [9] shares one multiplier datapath across its group formats and applies up to three per-group scalar correction terms through an auxiliary path. MixFP4 Zou et al., [28] selects one of two FP4 element formats, E2M1 or E1M2, per group, signaled by the sign bit of the E4M3 group scale, and decodes both to a common internal E2M2 representation.

Low-Precision Floating-Point Formats

We write EeMmE e M m for a signed floating-point format with one sign bit, ee exponent bits, and mm mantissa bits. Table 5 summarizes them.

FormatS/E/MMagnitudesBits/elem.
E2M1 (NVFP4)1/2/1{0,0.5,1,1.5,2,3,4,6}\{0, 0.5, 1, 1.5, 2, 3, 4, 6\}4.5
E2M3 (FP6g)1/2/30–7.5 in steps of 14\frac{1}{4}6.5
E1M01/1/0{1,2}\{1, 2\}2.5
E0M11/0/1{0,1}\{0, 1\}2.5
Quinary digit (AutoFP2)–{0,1,2}\{0, 1, 2\}3.0
E4M3 (scale)1/4/32−92^{-9} – 448–

Table 5. Low-precision floating-point formats.

NVFP4

NVFP4 Alvarez et al., [2] stores each element as an E2M1 value (exponent bias 1) with magnitudes {0,0.5,1,1.5,2,3,4,6}\{0, 0.5, 1, 1.5, 2, 3, 4, 6\} under two levels of scaling. A tensor XX is split into groups XjX_j of 16 contiguous elements along the reduction dimension; each group has an E4M3 group scale BjB_j, and the tensor has an FP32 tensor scale TT:

T=amax⁡(X)6⋅448,Bj=QE4M3(amax⁡(Xj)6T),(2)T = \frac{\operatorname{amax}(X)}{6 \cdot448}, \qquad B_j = Q_{\mathrm{E4M3}}\left(\frac{\operatorname{amax}(X_j)}{6T}\right), \tag*{(2)}
qi=QE2M1(xiTBj),x^i=TBjqi,i∈Xj,(3)q_i = Q_{\mathrm{E2M1}}\left(\frac{x_i}{TB_j}\right), \qquad\hat{x}_i = TB_jq_i,\quad i \in X_j, \tag*{(3)}

where QFQ_F rounds to the nearest value of format FF (ties to even). The group scale maps each group’s absolute maximum onto 6, the largest E2M1 value, and the tensor scale maps the largest group scale onto 448, the largest E4M3 value. With one 8-bit scale per 16 elements, NVFP4 costs 4+8/16=4.54 + 8/16 = 4.5 bits per element.

FP6g

FP6g, the group-scaled 6-bit counterpart of NVFP4 used as a baseline throughout the paper, keeps the group size and scaling hierarchy of NVFP4 and replaces E2M1 by E2M3 (exponent bias 1). The two additional mantissa bits give 32 magnitudes, in steps of 18\frac{1}{8} on [0,2)[0, 2), 14\frac{1}{4} on [2,4)[2, 4), and 12\frac{1}{2} on [4,7.5][4, 7.5], and lower the worst-case relative rounding error of normal values from 1/41/4 to 1/161/16. The scales follow Eqs. (2) and (3) with the E2M3 maximum 7.5 in place of 6, so FP6g costs 6+8/16=6.56 + 8/16 = 6.5 bits per element. Table 6 lists the positive magnitudes of both element formats.

E\ME2M1 (NVFP4)E2M3 (FP6g)
01000001010011100101110111
0000.500.1250.250.3750.50.6250.750.875
0111.511.1251.251.3751.51.6251.751.875
102322.252.52.7533.253.53.75
114644.555.566.577.5

Table 6. Positive magnitudes of E2M1 (NVFP4) and E2M3 (FP6g).

Two-Bit Formats and the AutoFP2 Digit

With one bit for the sign, a 2-bit floating-point format spends its other bit on either the exponent or the mantissa. E1M0 spends it on the exponent: giving the magnitudes {1,2}\{1, 2\} and the codebook {±1,±2}\{\pm1, \pm2\}, which cannot represent zero, so small values are forced to ±1\pm1. E0M1 spends it on the mantissa: both codes are subnormal, giving the ternary codebook {−1,0,+1}\{-1, 0, +1\} with a redundant −0-0, which represents zero but only one nonzero magnitude. AutoFP2’s quinary digit {−2,−1,0,1,2}\{-2, -1, 0, 1, 2\} keeps both by remapping the two exponent bits of the NVFP4 base (§ 3.2).

Sharing NVFP4 Scales with AutoFP2

A tensor stored in NVFP4 that executes at AutoFP2 reuses the NVFP4 scales TT and BjB_j. These normalize each group to [−6,6][-6, 6], whereas AutoFP2 digits reach only ±2\pm2, so decoding with TBjTB_j alone would shrink each group by up to 3×3\times. The scale is therefore multiplied by a scale multiple cc:

x^i=cTBjqi,qi∈{−2,−1,0,1,2}.(4)\hat{x}_i = cTB_jq_i,\qquad q_i \in\{-2,-1,0,1,2\}. \tag*{(4)}

Setting c=3c = 3 matches the two ranges exactly, with reconstruction levels {0,±3,±6}\{0, \pm3, \pm6\} in NVFP4 units. We instead use c=2c = 2, with levels {0,±2,±4}\{0, \pm2, \pm4\}, for two reasons. First, a factor of two is hardware-friendly. Second, the AutoFP2 digit 1 absorbs the four E2M1 magnitudes {1,1.5,2,3}\{1, 1.5, 2, 3\}, which hold most group-normalized values. Reconstructing this digit at 2 rather than 3 places it in the middle of that range and halves its worst-case error from 2 to 1 NVFP4 units.

Worked Example of Mantissa Refinement

Table 7 applies mantissa refinement to one 16-element NVFP4 group with scale s=TBj=0.25s = TB_j = 0.25.

Subgroupx/sx/sNVFP4 base qqCodeErr.AutoFP6 refinement δ\deltaCodeq+δq+\deltaErr.
SG01.001.000100––––
SG0−0.50−0.510010––––
SG03.003.001010––––
SG06.006.001110––––
SG11.131.000100.13+18+\frac{1}{8}011.1250.005
SG12.242.001000.24+18+\frac{1}{8}012.1250.115
SG1−0.87−1.010100.13+18+\frac{1}{8}01−0.8750.005
SG11.371.500110.13−18-\frac{1}{8}111.3750.005
SG22.763.001010.24−14-\frac{1}{4}102.750.010
SG2−0.62−0.510010.12−18-\frac{1}{8}11−0.6250.005
SG24.244.001100.24+18+\frac{1}{8}014.1250.115
SG2−2.02−2.011000.02000−2.00.020
SG30.520.500010.02––––
SG3−1.98−2.011000.02––––
SG30.000.000000––––
SG32.953.001010.05––––

Table 7. Mantissa refinement of one NVFP4 group (s=0.25s = 0.25). Values are in NVFP4 units; codes are sign — exponent — mantissa. Green rows are the refined subgroups (mask 0110).

Additional Motivating Analyses

Layer-Wise Precision Across Decode Steps

Figure 16 shows the oracle per-layer precision at individual decode steps.

Layer-wise precision across decode steps

Figure 16. Layer-wise precision across decode steps. Per-layer precision at ten decode steps of one DS-Qwen-14B LiveCodeBench response (top) and one Qwen3.8-27B τ\tau-bench response (bottom).

Layer-Wise Precision Across Prefill and Refills

Figure 17 repeats the prefill and refill analysis of Figure 2(b) for an additional model and workload, Gemma-4-26B on

Figure 17. Oracle layer-wise precision for the initial prefill and three refills of Gemma-4-26B on three LongMemEval samples (S1–S3).

LongMemEval, showing the oracle layer-wise precision of the initial prefill and three refills for three samples.

Precision Demand by Generation Entropy

Figure 18 splits the decode steps of the oracle of §2 by GE\mathcal{G}_{\mathcal{E}} at τ=0.10\tau= 0.10. High-entropy steps consistently require higher precision compared to low-entropy steps.

Oracle precision demand in high- (H) and low-entropy (L) decode steps

Figure 18. Oracle precision demand in high- (H) and low-entropy (L) decode steps.

Layer-Wise Behavior of MP and MD Across Models and Datasets

Figure 30 extends Figure 4 to Gemma-4-26B, Qwen3.8-27B, Qwen3-8B, and DS-LLaMA-70B on LiveCodeBench, GSM8K, and AIME. Figure 19 shows MP\mathcal{M}_P of the initial prefill and three refills of one τ2\tau^2-bench task for Gemma-4-26B and Qwen3.8-27B.

Figure 19. Prefill MP\mathcal{M}_{P} for the initial prefill and three refills of one τ2\tau^{2}-bench task for (a) Gemma-4-26B and (b) Qwen3.8-27B.

Figure 30. Layer-wise MP\mathcal{M}_{P} and MD\mathcal{M}_{D} across models and datasets. For each model (rows) and dataset (column pairs), prefill MP\mathcal{M}_{P} of one representative sample and decode MD\mathcal{M}_{D} of the same sample at the highest- and lowest-GE\mathcal{G}_{\mathcal{E}} step.

Subgroup Refinement on Additional Models

Figure 20 repeats the experiment of Figure 3 on Qwen3.8-27B and Gemma-4-26B with GSM8K, for three layers each.

Additional AutoEMatic Details

Signals Used for Runtime Precision Adaptation

Figure 21 illustrates the two model-internal signals of § 3.1.

Entropy Signal Across Decode Steps

Figure 22 traces GE\mathcal{G}_{\mathcal{E}} over a 16,383-step AIME generation.

Memory Layout

Figure 23 shows the memory layout of AutoFP, with the NVFP4 base stored separately from the refinement metadata.

Extended-KK GEMM

Figure 24 illustrates the identity behind the extended-KK GEMM of § 4.1 on a numeric example: two GEMMs with the same output shape sum to one GEMM over operands concatenated along KK.

Construction of the Extended-KK Operands

Figure 25 shows how WqrW_{qr} and AqrA_{qr} are constructed.

Figure 20. Refining a subset of elements achieves NMSE comparable to 6-bit. Y-axis is NMSE, such that NVFP4 is 1 and FP6g is 0, vs. fraction of subgroups refined to 6-bit, for Qwen3.8-27B and Gemma-4-26B on GSM8K.

Signals used by AutoEMatic for runtime precision adaptation

Figure 21. Signals used by AutoEMatic for runtime precision adaptation. ❶ Local layer-update activity LA\mathcal{L}_{A}. ❷ Global token-generation uncertainty GE\mathcal{G}_{\mathcal{E}}.

Numerical Example of the Extended-KK GEMM

Figure 26 shows a worked example of the GEMM [Wq∣Wqr][Aq∣Aqr][W_q \mid W_{qr}][A_q \mid A_{qr}]:

W^A^T=WqAqT+sW4WqrAqrT=[Wq∣Wqr][Aq∣Aqr]T=9.25+14(0.5⋅2+3⋅(−0.5))=9.125.(5)\widehat{W}\widehat{A}^{\mathsf{T}} = W_q A_q^{\mathsf{T}} + \frac{s_W}{4} W_{qr} A_{qr}^{\mathsf{T}} = [W_q \mid W_{qr}][A_q \mid A_{qr}]^{\mathsf{T}} = 9.25 + \frac{1}{4}(0.5 \cdot2 + 3 \cdot(-0.5)) = 9.125. \tag*{(5)}

NVFP4-to-AutoFP2 Decomposition

The weight encoder decomposes each NVFP4 weight into two AutoFP2 digits, Wq=c0Q0+c1Q1W_q = c_0Q_0 + c_1Q_1 with Q0,Q1∈{−2,…,2}Q_0, Q_1 \in\{-2,\ldots,2\} (§ 6). With c1=1c_1 = 1, to span the maximum NVFP4 magnitude 6 requires 2c0+2≥62c_0 + 2 \ge6. c0=2.5c_0 = 2.5 satisfies this such that c0Q0c_0Q_0 provides coarse-grained steps to different NVFP4 binades and c1Q1c_1Q_1 provides integer corrections [−2,2][-2,2], such that their linear combination spans every NVFP4 E2M1 value. Multiplying numerator and denominator by 2 gives integer coefficients such that Wq=(5Q0+2Q1)/2W_q = (5Q_0 + 2Q_1)/2 and the resulting reconstruction can be realized using only shifts and adds. Table 8 lists the digit pair of every positive E2M1 magnitude; negative weights negate both digits.

For each digit QiQ_i, the encoder produces three bits from the E2M1 bits {s,e1,e0,m}\{s,e_1,e_0,m\} of the weight: a valid bit viv_i, which is 1 if Qi≠0Q_i \ne0 and 0 if Qi=0Q_i = 0; a sign bit sis_i; and an exponent bit eq,ie_{q,i} with ∣Qi∣=2eq,i\lvert Q_i\rvert= 2^{e_{q,i}}, so that {si,eq,i}\{s_i,e_{q,i}\} is the E1M0 code of a nonzero digit:

Figure 22. Global signal GEG_{\mathcal{E}} over a 16K-token AIME generation of Qwen3.8-27B. The dashed line marks τ=0.10\tau= 0.10.

Figure 24. A sum of GEMMs as one GEMM over an extended reduction dimension.

AutoEMatic memory layout

Figure 23. AutoEMatic memory layout.

Figure 25. Construction of Wqr\mathbf{W}_{qr} and Aqr\mathbf{A}_{qr} for one 16-element group and one request. The weight mask places the weight offsets 4δW4\delta_W in Wqr\mathbf{W}_{qr} and copies the base AqA_q into Aqr\mathbf{A}_{qr}; the activation mask copies the base WqW_q into Wqr\mathbf{W}_{qr} and places the activation offsets 4δA4\delta_A in Aqr\mathbf{A}_{qr}; all other positions are zero.

v0=m∨(e1∧e0),v1=e1∨e0∨m,eq,0=e1∧(e0∨m),eq,1=¬e0∧(e1∨m),s0=s,s1=s⊕m⊕(e1∧e0),(6)\begin{aligned} v_0 &= m \lor(e_1 \land e_0), & v_1 &= e_1 \lor e_0 \lor m,\\ e_{q,0} &= e_1 \land(e_0 \lor m), & e_{q,1} &= \lnot e_0 \land(e_1 \lor m),\\ s_0 &= s, & s_1 &= s \oplus m \oplus(e_1 \land e_0), \tag*{(6)} \end{aligned}

where ∨\lor, ∧\land, ¬\lnot, and ⊕\oplus denote OR, AND, NOT, and XOR. The valid bits then select the AutoFP2 encoding of § 3.2: if both digits are nonzero (v0=v1=1v_0 = v_1 = 1), the escape bit is E=0E = 0 and the fields F1F_1 and F0F_0 hold the two E1M0 codes; otherwise E=1E = 1, F1F_1 records which digits are zero, and F0F_0 holds the E1M0 code of the nonzero digit, if any. The valid bits are consumed by this packing and are not stored.

Fixed-Point Arithmetic of the vMAC

All computation in the vMAC (§ 6.3) is fixed-point. Each input activation enters the lanes as the integer Ak=2Aq(k)A_k = 2A_q(k), since doubling an E2M1 value always yields an integer ((Hayashi et al., 2026)). AutoFP2 digits are already integers, whereas in NVFP4 mode the packed digit pair represents twice the NVFP4 weight, 2Wq=5Q0+2Q12W_q = 5Q_0 + 2Q_1 (§ 6.2). In the multiply stage each sub-lane forms AkFj,kA_kF_{j,k}, j∈{0,1}j \in\{0,1\}, and within each block the even and odd sub-lanes feed separate adder trees, A=∑k=015AkF0,kA = \sum_{k=0}^{15} A_kF_{0,k} and B=∑k=015AkF1,kB = \sum_{k=0}^{15} A_kF_{1,k} for lanes 0–15, and CC, DD for lanes 16–31.

In NVFP4 mode, AA and BB are partial sums over the two digits of the same weights Wq,kW_{q,k}, so by distributivity the mode-dependent reduction unit evaluates P0=5A+2B=∑kAk(5F0,k+2F1,k)=∑k(2Aq)(2Wq)P_0 = 5A + 2B = \sum_k A_k(5F_{0,k} + 2F_{1,k}) = \sum_k (2A_q)(2W_q), which is four times the dot product of block 0 in fixed-point; the division by two in (5A+2B)/2(5A + 2B)/2 of § 6.3 is deferred to the scaling stage. Similarly for P1P_1. In AutoFP2 mode, the even and odd sub-lanes of a block hold weights from two different rows, so the partial sums are summed across blocks as P0=2(A+C)P_0 = 2(A+C) and P1=2(B+D)P_1 = 2(B+D), two independent 32-element dot products; the factor of 2 keeps them at four times the dot product as well.

Each refinement lane forms R=2XqdR = 2X_qd in fixed-point. Since δ=d⋅2−3\delta= d \cdot2^{-3}, the floating-point equivalent is Xqδ=R/16X_q\delta= R/16, so RR is sixteen times the product. The lane contributions are accumulated as R=∑RR = \sum R, and the base sums PjP_j are shifted left by two bits to align with them, giving Oj=4Pj+RO_j = 4P_j + R at sixteen times the dot product. For example, with Aq=1.5A_q = 1.5, Wq=3W_q = 3, and dW=−1d_W = -1: P=(2⋅1.5)(2⋅3)=18P = (2\cdot1.5)(2\cdot3) = 18, R=−3R = -3, and O=69=16⋅1.5(3−1/8)O = 69 = 16\cdot1.5(3 - 1/8). In the scaling stage, each OjO_j is multiplied by the scale factor of its block and divided by 16, which recovers the floating-point partial sum.

Accelerator Floorplan

Figure 27 shows the AutoEMatic accelerator floorplan.

Post-layout floorplan of AutoEMatic

Figure 27. Post-layout floorplan of AutoEMatic.

Exact refined dot product, base GEMM, refinement GEMM, and one extended-K GEMM

Figure 26. (1) The exact product of the refined operands W^\widehat{\mathbf{W}} and A^\widehat{\mathbf{A}}. (2) The base GEMM on Wq\mathbf{W}_q and Aq\mathbf{A}_q plus a refinement GEMM on the constructed Wqr\mathbf{W}_{qr} and Aqr\mathbf{A}_{qr} at scale sW/4s_W/4. (3) A single GEMM over the concatenated operands.

WqW_q00.511.52346
Q0Q_001010222
Q1Q_10−2-21−1-12−2-2−1-11

Table 8. NVFP4-to-AutoFP2 decomposition.

Refinement-Aware GPTQ Calibration

Algorithm 1 lists AutoEMatic’s offline weight calibration, which extends GPTQ (Frantar et al., 2023) to an NVFP4 base with NN:4 structured sparse refinement. We follow the NVFP4 notation of §B.1: TT is the FP32 tensor scale, and Bn,jB_{n,j} is the E4M3 group scale of row nn in group jj, whose 16 columns we denote JJ. The calibration stores E2M1 codes qn,iq_{n,i} and, on refined subgroups, 2-bit signed offsets dn,id_{n,i}, decoded with the fixed step of §3.2 as δn,i=dn,i2−3∈D={−14,−18,0,+18}\delta_{n,i}=d_{n,i}2^{-3}\in D=\{-\frac{1}{4},-\frac{1}{8},0,+\frac{1}{8}\}; the dequantized weight is TBn,jqn,iTB_{n,j}q_{n,i} without refinement (NVFP4) and TBn,j(qn,i+δn,i)TB_{n,j}(q_{n,i}+\delta_{n,i}) with it (AutoFP6). A refinement is set to 0 if it would move the element farther from the original weight.

Each group has four 4-element subgroups, and Mℓ\mathcal{M}_{\ell} denotes the masks that refine ℓ\ell of them; the weight mask MW\mathbf{M}_W of §3.2 holds one such mask mm per group. For a mask mm, W^(m)\widehat{\mathbf{W}}(m) is the dequantized group that uses the refined values W^r\widehat{\mathbf{W}}^r on the selected subgroups and the NVFP4 values W^4\widehat{\mathbf{W}}^4 elsewhere. Its diagonal-Hessian-weighted error on group jj is

Ej(m)=∑i∈JHii∥W:,i−W^:,i(m)∥22.(7)\mathcal{E}_j(m)=\sum_{i\in J} H_{ii}\left\lVert\mathbf{W}_{:,i}-\widehat{\mathbf{W}}_{:,i}(m)\right\rVert_2^2. \tag*{(7)}

The algorithm differs from GPTQ in three ways. (1) Budget allocation. Before the GPTQ pass, a greedy allocation (lines 5–12) distributes ρK/4\rho K/4 refined subgroups across groups, ranked by their reduction in Ej\mathcal{E}_j on the original weights. Each group’s mask, which refines NWN_W of its four subgroups, is shared across all output rows; the remaining NA=4−NWN_A=4-N_W subgroups are left for activation refinement at runtime. (2) Group-wise quantization. The 16-element group, rather than a single column, is the unit of quantization. Its group scales Bn,jB_{n,j} are computed from the error-compensated weights when the group is reached, while the tensor scale TT stays fixed. (3) Group-wise error propagation. The quantization error of a whole group is propagated to the later columns at once (lines 22–23), rather than one column at a time as in GPTQ.

GPU Implementation Details

AutoEMatic is implemented in vLLM 0.15 (Kwon et al., 2023) as a quantization plugin, with a custom attention backend and CUDA kernels for Blackwell GPUs.

Accelerator Simulation Details

AutoEMatic performance is measured using a scheduling model calibrated against cycle-accurate Synopsys VCS simulation. Performance data is gathered from these measurements, retaining on-chip data-movement and backpressure costs. Off-chip DRAM transfer time is modeled as transferred bytes divided by an assumed sustained bandwidth of 1 TB/s and added to the execution time.

Additional Accuracy Results

Table 9 reports the accuracy of AutoEMatic and the baselines on DS-Llama-70B, DS-Qwen-32B, and Llama-3.1-8B-Instruct on the reasoning benchmarks.

ModelMethod (budget)LB v2AIMELCBGSM8K
DS-LLaMA-70BBF1641.066.768.294.5
DS-LLaMA-70BNVFP429.853.351.790.6
DS-LLaMA-70BFP6g41.066.766.294.5
DS-LLaMA-70BAutoQuantize (5.3)38.766.755.794.5
DS-LLaMA-70BFlexQuant (5.3)27.850.048.686.5
DS-LLaMA-70BDP-LLM (weight-only) (5.3)21.0 (5.1)43.3 (5.3)43.3 (5.3)81.6 (5.1)
DS-LLaMA-70BDP-LLM† (5.3)19.0 (5.1)36.7 (5.3)39.5 (5.3)77.6 (5.1)
DS-LLaMA-70BAutoEMatic41.0 (5.4)66.7 (5.1)65.6 (4.5)94.5 (4.9)
DS-Qwen-32BBF1649.873.354.292.5
DS-Qwen-32BNVFP443.560.050.587.6
DS-Qwen-32BFP6g49.870.052.690.5
DS-Qwen-32BAutoQuantize (4.8)49.866.751.289.3
DS-Qwen-32BFlexQuant (4.8)40.556.746.981.5
DS-Qwen-32BDP-LLM (weight-only) (4.8)36.6 (4.8)50.0 (4.8)45.0 (4.8)76.5 (4.5)
DS-Qwen-32BDP-LLM† (4.8)30.6 (4.8)36.7 (4.8)40.7 (4.8)71.8 (4.5)
DS-Qwen-32BAutoEMatic49.8 (4.6)70.0 (4.8)52.4 (4.6)90.5 (4.4)
Llama-3.1-8B-InstructBF1630.5––82.8
Llama-3.1-8B-InstructNVFP427.5––75.6
Llama-3.1-8B-InstructFP6g30.5––80.5
Llama-3.1-8B-InstructAutoQuantize (4.5)30.5––78.9
Llama-3.1-8B-InstructFlexQuant (4.5)25.9––71.6
Llama-3.1-8B-InstructDP-LLM (weight-only) (4.5)24.6 (4.4)––63.8 (4.3)
Llama-3.1-8B-InstructDP-LLM† (4.5)22.8 (4.4)––59.6 (4.3)
Llama-3.1-8B-InstructAutoEMatic30.5 (4.4)––80.5 (4.3)

Table 9. Accuracy of AutoEMatic and baselines on DS-Llama-70B, DS-Qwen-32B, and Llama-3.1-8B-Instruct. Cells report accuracy (%); execution precision appears in parentheses. LB v2: LongBench v2, LCB: LiveCodeBench. Llama-3.1-8B-Instruct is not evaluated on AIME and LCB, where its BF16 accuracy is too low for a meaningful comparison.

Additional Ablations

AutoFP2 scale multiple cc. Figure 28(a) sweeps the scale multiple cc of §B.4: c=2c=2 gives an NMSE of 0.09, within

Figure 28. Additional ablations on DS-LLaMA-70B (LCB). (a) Scale multiple cc of AutoFP2: mean layer-output NMSE over all 80 layers at W2A4. (b) Subgroup size SGSG with 20% of all weight subgroups refined. (c) Weight refinement budget ρ\rho, with storage precision. (d) Activation NMSE threshold, with the mean number of activation subgroups per group exceeding it. (e) λD\lambda_D sweep. (f) Entropy threshold τ\tau. (g) Trend lengths npn_p, α\alpha, and β\beta. (h) Overhead of computing GE\mathcal{G}_{\mathcal{E}} and LA\mathcal{L}_{\mathcal{A}} as a share of decode TPOT. Stars mark the chosen settings.

1.15×\times of nearest rounding, whereas c=1c = 1 clips the large magnitudes and c=3c = 3 places the merged E2M1 codes {1,1.5,2,3}\{1, 1.5, 2, 3\} at 3, raising the NMSE to 0.38 and 0.15.

Impact of second-order refinement. Figure 31 measures the second-order refinement product WrAr\mathbf{W}_r\mathbf{A}_r that the extended-KK GEMM omits (§ 4.1). Its energy is 1.4×10−61.4 \times10^{-6} of the per-layer output, and consequently the AutoFP6 output NMSE is 0.0041 with and without this term.

Figure 31. Second-order refinement term. Per-layer output NMSE of AutoFP6 with and without the second-order product WrArW_{r}A_{r} (top) and the energy of that product relative to the layer output (bottom) on DS-LLaMA-70B.

Extension of KK. The reduction extension of a refined GEMM depends on the # of subgroups refined. On Qwen3.8-27B τ2\tau^2-bench (Figure 32), the extension grows from 0.49K0.49K at batch 1 to 0.81K0.81K at batch 32 with average refined requests per layer rising from 0.2 to 7.2.

Figure 32. Extension of the reduction dimension of refined GEMMs on Qwen3.8-27B τ2\tau^{2}-bench vs. decode batch size, with the mean number of refined tokens per layer.

Extension to 8 bits. Widening the signed offsets from 2 to 4 bits extends AutoEMatic to AutoFP8 (Table 10). With every layer statically refined to AutoFP8 and refinement-aware GPTQ, DS-LLaMA-70B attains 67.9% on LCB at 5.3 stored bits per weight. The runtime policy with P={2,4,6,8}\mathcal{P} = \{2, 4, 6, 8\} attains the same 67.9%, within 0.5% of BF16, at 5.3 bits while executing 30% of layers at 2-bit, 35% at 4-bit, 15% at 6-bit, and 20% at 8-bit.

ComponentBitsAcc.2 / 4 / 6 / 8 (%)
BF1616.868.4
AutoEMatic format + runtime policy, P={2,4,6}\mathcal{P} = \{2, 4, 6\}4.965.615 / 45 / 40 / 0
AutoFP8 + Refinement-aware GPTQ5.367.90 / 0 / 0 / 100
AutoEMatic format + runtime policy, P={2,4,6,8}\mathcal{P} = \{2, 4, 6, 8\}5.367.930 / 35 / 15 / 20

Table 10. Extension to 8 bits with DS-LLaMA-70B on LCB; rows 1 and 7 of Table 4 are repeated for reference.

Subgroup size. Figure 28(b) varies SGSG with 20% of all weight subgroups refined. Finer subgroups raise accuracy only from 65.6% at SG=4SG = 4 to 66.1% at SG=1SG = 1, however, finer subgroups require more metadata per group and are not as performant. Coarser subgroups lose 2.4–3.1%, as a refined subgroup then spends signed offsets on elements that do not need them.

Weight refinement budget. Figure 28(c) sweeps ρ\rho, the fraction of all weight subgroups refined, over {10, 20, 40, 80}%, with storage of 4.5+2ρ4.5 + 2\rho bits per weight. Accuracy peaks at ρ=20%\rho= 20\% (65.6% at 4.9 bits) and falls on both sides: at 10% too few weight subgroups are refined, while at 40%

and 80% the weight mask occupies subgroups that the activations would otherwise refine at runtime (NA=4−NWN_{A} = 4 - N_{W}), so additional weight bits displace activation refinement and accuracy drops to 60.6%.

Activation refinement threshold. Figure 28(d) sweeps the NMSE threshold above which an activation subgroup is refined, over {0.001, 0.003, 0.01, 0.03}, together with the mean number of subgroups per group that exceed it. Lowering the threshold below 0.003 refines 2.7 rather than 1.6 subgroups per group for 0.3% more accuracy, while raising it to 0.01 and 0.03 leaves most groups unrefined.

λD\lambda_{D}. Figure 28(e) sweeps λD\lambda_{D} over {0.1, 0.3, 0.5, 0.7}.

Algorithm 1 Refinement-aware GPTQ calibration of W∈RN×KW \in\mathbb{R}^{N \times K}, given calibration inputs X∈RM×KX \in\mathbb{R}^{M \times K}, group size G=16G = 16, refinement budget ρ\rho, and damping ω\omega. Returns the E2M1 codes qn,iq_{n,i}, the refinement terms δn,i\delta_{n,i} (stored as 2-bit signed offsets) on refined subgroups, the scales TT and Bn,jB_{n,j}, and the masks mj⋆m_j^\star.

  1. H←X⊤X/MH \leftarrow X^\top X/M

  2. Hω←H+ωmean⁡(diag⁡H)IH_\omega\leftarrow H + \omega\operatorname{mean}(\operatorname{diag} H) I

  3. U←Cholesky⁡(Hω−1)⊤U \leftarrow\operatorname{Cholesky}(H_\omega^{-1})^\top

  4. T←amax⁡(W)/(6⋅448)T \leftarrow\operatorname{amax}(W)/(6 \cdot448) \hfill fixed

  5. for j=0j = 0 to K/G−1K/G - 1 do

  6. Lj(ℓ)←min⁡m∈MℓEj(m)L_j(\ell) \leftarrow\min_{m \in\mathcal{M}_\ell} \mathcal{E}_j(m) for ℓ∈{0,1,2}\ell\in\{0,1,2\}

  7. end for

  8. ℓj←0\ell_j \leftarrow0 for all jj

  9. for b=1b = 1 to ρK/4\rho K/4 do

  10. j⋆←arg⁡max⁡j:ℓj<2Lj(ℓj)−Lj(ℓj+1)j^\star\leftarrow\arg\max_{j:\ell_j<2} L_j(\ell_j) - L_j(\ell_j+1)

  11. ℓj⋆←ℓj⋆+1\ell_{j^\star} \leftarrow\ell_{j^\star}+1

  12. end for

  13. for j=0j = 0 to K/G−1K/G - 1 do

  14. J←jG:(j+1)GJ \leftarrow jG:(j+1)G

  15. Bn,j←QE4M3(amax⁡(Wn,J)/6T)B_{n,j} \leftarrow Q_{\mathrm{E4M3}}(\operatorname{amax}(W_{n,J})/6T) for all nn

  16. qn,i←QE2M1(Wn,i/TBn,j)q_{n,i} \leftarrow Q_{\mathrm{E2M1}}(W_{n,i}/TB_{n,j}) for all nn, i∈Ji \in J

  17. δn,i←arg⁡min⁡δ∈D∣Wn,i/TBn,j−qn,i−δ∣\delta_{n,i} \leftarrow\arg\min_{\delta\in\mathcal{D}} \left|W_{n,i}/TB_{n,j}-q_{n,i}-\delta\right|

  18. W^n,i4←TBn,jqn,i\widehat{W}^{4}_{n,i} \leftarrow TB_{n,j}q_{n,i}

  19. W^n,ir←TBn,j(qn,i+δn,i)\widehat{W}^{r}_{n,i} \leftarrow TB_{n,j}(q_{n,i}+\delta_{n,i})

  20. mj⋆←arg⁡min⁡m∈MℓjEj(m)m_j^\star\leftarrow\arg\min_{m \in\mathcal{M}_{\ell_j}} \mathcal{E}_j(m)

  21. W^:,J←W^(mj⋆)\widehat{W}_{:,J} \leftarrow\widehat{W}(m_j^\star)

  22. E←(W:,J−W^:,J)UJ,J−1E \leftarrow(W_{:,J}-\widehat{W}_{:,J})U_{J,J}^{-1}

  23. W:,(j+1)G:←W:,(j+1)G:−EUJ,(j+1)G:W_{:,(j+1)G:} \leftarrow W_{:,(j+1)G:}-EU_{J,(j+1)G:}

  24. end for

Entropy threshold. Figure 28(f) sweeps τ\tau, which separates the high- and low-entropy decode regimes, over {0.1,…,0.5}\{0.1,\ldots,0.5\}. Accuracy is highest at τ=0.1\tau=0.1 (65.6%) and declines monotonically to 62.6% at τ≥0.4\tau\geq0.4, as more steps with several plausible continuations are treated as low-entropy and allowed to lower precision aggressively.

Trend lengths. Figure 28(g) sweeps npn_p, α\alpha, and β\beta over {1,2,4,8}\{1,2,4,8\}.

Signal overhead. Figure 28(h) reports the cost of computing the signals as a share of decode TPOT. GE\mathcal{G}_{\mathcal{E}} is one top-KK selection and entropy over the logits per step, below 1.3% at every batch size; LA\mathcal{L}_{\mathcal{A}} is a per-layer reduction of the residual update, 2.4 μ\mus per layer, whose share falls from 2.0% at batch 1 to 1.1% at batch 128 as it amortizes over rows. Together the signals add 2.3–2.8% of TPOT.

Online serving. Figure 29 serves DS-LLaMA-70B in vLLM under continuous batching with Poisson request arrivals at increasing rates, using LongBench v2 prompts of about 8K tokens with 1–2K output tokens.

Online serving under continuous batching

Figure 29. Online serving under continuous batching. Median TPOT vs. achieved request rate for DS-LLaMA-70B on 4× GB300 (TP=4) with Poisson arrivals of LongBench v2 requests.

Accelerator latency and energy at iso-accuracy with NVFP4. The main comparison (§ 6) holds accuracy at AutoEMatic’s level. Figure 33 instead holds it at NVFP4’s. This enables the NVFP4 datapath to run all layers at its native precision. To achieve iso-accuracy with NVFP4, we tune AutoEMatic assignments on MiniMax-M2.7 such that AutoEMatic assigns 30% of layers to AutoFP2, 60% to NVFP4, and 10% to AutoFP6 and matches the accuracy of the NVFP4. As shown in Figure 33, AutoEMatic still reduces end-to-end latency by 12–14%, with prefill 1.7–2.0× faster at batch sizes 1 and 8, at 13% higher energy, due to the additional components in the mixed-precision AutoEMatic datapath.

Figure 33. Accelerator latency and energy at iso-accuracy with NVFP4.

DP-LLM and FlexQuant performance. Figure 34 reports the prefill latency of DP-LLM and FlexQuant on DS-LLaMA-70B. At batch size 1 they decode at 5.9 and 7.3 tokens/s, respectively, against 13 tokens/s for NVFP4 and 120 for AutoEMatic in Figure 8.

Figure 34. TTFT of the runtime-adaptive baselines DP-LLM and FlexQuant on DS-LLaMA-70B vs. input sequence length.

Instantaneous vs. trend-based changes. Table 11 compares making precision decisions instantaneously versus AutoEMatic’s trend-based policy. The trend-based policy reaches both lower average execution precision (4.6 vs. 4.7 bits) and higher accuracy (95.5% vs. 92.5%). Instantaneous decisions react to layer-to-layer noise in the metric, the trend rule acts only on sustained changes, which filters this noise.

PolicyBitsAcc.2 / 4 / 6 (%)
Instantaneous (np=α=β=1n_p=\alpha=\beta=1)4.792.50 / 66 / 34
AutoEMatic (np=2,α=4,β=2n_p=2,\alpha=4,\beta=2)4.695.50 / 69 / 31

Table 11. Instantaneous vs. trend-based precision changes with Qwen3-8B on GSM8K.

Limitations

As it stands, while AutoFP supports the precision set P={2,4,6}P = \{2, 4, 6\}, it cannot take advantage of AutoFP2’s memory bandwidth benefits. Using AutoFP2 as the stored base would require reconstructing NVFP4 for most layer executions, introducing overhead that can outweigh the bandwidth savings during decode. Furthermore, AutoFP2’s coarse representation requires larger corrections to recover NVFP4 values, which the current signed-offset encoding does not directly support. Future work will work on remedying this.

AI Usage Statement

Portions of this paper were refined with the assistance of a large language model (LLM), specifically OpenAI GPT 6-Sol, used to polish writing and help reduce verbosity to meet page limit. The LLM also assisted in writing the python code for making several figures for the data the authors independently collected. All technical content, methods, and results were conceived entirely by the authors, without influence from any AI tool. The AI was however used to run parameter sweeps, conduct design space exploration, suggest optimizations (which the authors thoroughly verified) and also launch and monitor accuracy and performance experiments.

References

  1. [2]Alvarez, E. et al. Introducing NVFP4 for efficient and accurate low-precision inference. NVIDIA Technical Blog, June 2025. Accessed September 19, 2026.
  2. [4]Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pp. 87–100, 2024.DOI
  3. [5]Liu, F., Wang, Z., Xia, J., Zhao, J., Zhao, S., Li, J., Liu, J., Jiang, L., and Guan, H. Flexquant: A flexible and efficient dynamic precision switching framework for llm quantization. arXiv preprint arXiv:2506.12024, 2025.arxiv.org/abs/2506.12024
  4. [6]Liu, T., Li, S., Yang, J., Zhao, T., Zhou, F., Song, X., Dai, G., Yan, S., Yang, H., and Wang, Y. PM-KVQ: Progressive mixed-precision KV cache quantization for long-CoT LLMs. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2505.18610.
  5. [7]Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In International Conference on Machine Learning, 2024.DOI
  6. [9]Luo, J., Jiang, X., Wen, T.-H., Qi, R., Chu, M., Wu, H., Kielian, G., Laurie, B., Zhang, Q., Cheng, Q., Sylvester, D., and Saligane, M. Heterogeneity-aware microscaling for efficient low-bit LLM inference, 2026a. URL https://arxiv.org/abs/2608.03867.
  7. [10]Luo, Y., Dong, B., Cheng, W., and Shen, H. Recurrent residual quantization: A progressive multi-precision representation for LLMs, 2026b. URL https://arxiv.org/abs/2608.04048.
  8. [11]MAA. Aime 2024 problems. https://artofproblemsolving.com/wiki/index.php/2024_AIME_I_Problems, 2024. Accessed: 2025-08-30.
  9. [12]Nair, P., Datta, P., Dean, J., Jain, P., and Kusupati, A. Matryoshka quantization. arXiv preprint arXiv:2502.06786, 2025.arxiv.org/abs/2502.06786
  10. [13]NVIDIA ModelOpt. NVIDIA Model Optimizer–AutoQuantize. GitHub repository, 2025. URL https://github.com/NVIDIA/Model-Optimizer.
  11. [14]Open Compute Project. OCP microscaling formats (MX) specification. Technical Report Version 1.0, Open Compute Project Foundation, 2023.
  12. [15]OpenAI. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/, August 2024. Accessed 2026-10-12.
  13. [16]Park, G., Bae, J., Kwon, B., Kim, B., Kwon, S. J., and Lee, D. Anybcq: Hardware efficient flexible binary-coded quantization for multi-precision llms. In International Conference on Learning Representations, volume 2026, pp. 125106–125122, 2026.arxiv.org/abs/2510.10467
  14. [17]Park, Y., Hyun, J., Cho, S., Sim, B., and Lee, J. W. Any-precision llm: Low-cost deployment of multiple, different-sized llms. arXiv preprint arXiv:2402.10517, 2024.arxiv.org/abs/2402.10517
  15. [18]Ramachandran, A., Kundu, S., and Krishna, T. Clamp-vit: Contrastive data-free learning for adaptive post-training quantization of vits. In European Conference on Computer Vision, pp. 307–325. Springer, 2024.arxiv.org/abs/2407.05266
  16. [19]Ramachandran, A., Kundu, S., and Krishna, T. Microscopiq: Accelerating foundational models through outlier-aware microscaling quantization. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pp. 1193–1209, 2025a.arxiv.org/abs/2411.05282
  17. [20]Ramachandran, A., Kundu, S., Raha, A., Kundu, S., Mathaikutty, D. K., and Krishna, T. Accelerating llm inference with flexible n: M sparsity via a fully digital compute-in-memory accelerator. In 2025 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1–7. IEEE, 2025b.
  18. [21]Ramachandran, A., Naseem, M., Sakr, C., Venkatesan, R., Khailany, B., and Krishna, T. Thinkv: Thought-adaptive kv cache compression for efficient reasoning models. In International Conference on Learning Representations, volume 2026, pp. 110072–110106, 2026.arxiv.org/abs/2510.01290
  19. [22]Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.arxiv.org/abs/2310.10537
  20. [23]Shen, Y. and Zou, A. River-llm: Large language model seamless exit based on kv share. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 37633–37645, 2026.
  21. [24]Venkatesan, R., Shao, Y. S., Wang, M., Clemons, J., Dai, S., Fojtik, M., Keller, B., Klinefelter, A., Pinckney, N., Raina, P., et al. Magnet: A modular accelerator generator for neural networks. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–8. IEEE, 2019.DOI
  22. [25]Wang, D., Kim, J., Han, S., Gudovskiy, D., Nakata, Y., Okuno, T., Peong, K., Jeon, K. E., Ko, J. H., Chen, Y., et al. Mobiquant: Mixture-of-bits quantization for token-adaptive any-precision llm. arXiv preprint arXiv:2602.20191, 2026.arxiv.org/abs/2602.20191
  23. [27]Zhao, Z., Wang, M., Yu, B., Liu, B., Han, X., and Zhao, X. DynamicPTQ: Mitigating activation quantization collapse via residual-stream dynamics, 2026. URL https://arxiv.org/abs/2606.12487.
  24. [28]Zou, J., Chen, Y., Wu, R., and Chen, X. MixFP4: Enhancing NVFP4 with adaptive FP4/INT4 block representations. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, 2026. URL https://arxiv.org/abs/2605.31035.

Paper details

vlnc:W_w7r9eesy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

OpenAI GPT 6-Sol

View statement
Looking for related papers…
Contents