WorksComputer Science
AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement
Abstract
Existing LLM post-training quantization techniques determine precision before inference begins, yet the precision a model requires varies dynamically across prompts, layers, and inference phases. We present AutoEMatic, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime. A lightweight policy selects each layer's precision at every prefill and decode step from two runtime signals: the entropy of the next-token distribution and the magnitude of each layer's update to its hidden states. This runtime adaptation is made practical by AutoFP, a multi-scale floating-point format that derives 2-, 4-, and 6-bit precisions from a single NVFP4 base format. We realize AutoEMatic on two targets: GPU kernels that execute AutoFP on NVFP4 Tensor Cores, and an accelerator whose vector MAC datapath executes all AutoFP formats natively. Across reasoning and agentic benchmarks, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches group-scaled FP6 accuracy on most benchmarks, using 4.9 bits per weight versus 4.5 for NVFP4. On GB300 GPUs, AutoEMatic retains approximately 90-98% of NVFP4 decode throughput across the evaluated configurations. Furthermore, the AutoEMatic accelerator reduces latency by 1.6-2.3x and energy by 7-23% compared to an accelerator with NVFP4-only datapath at iso-accuracy.
Introduction
AI, driven by LLMs, is reaching an inflection point as the focus shifts from training to efficient inference at scale. Rising request volumes and long-horizon agentic workloads (Kariyappa & Suh, 2026) strain inference infrastructure in compute, memory bandwidth, and energy. Post-training quantization (PTQ) techniques improve inference efficiency by lowering the precision of weights, activations, or KV cache. Existing PTQ techniques (Lu et al., 2026; NVIDIA ModelOpt, 2025), however, employ static precision assignments, determined before the start of inference rather than adapting to the model’s evolving numerical demands during generation. These numerical demands vary dynamically with the prompt, between prefill and decode, and from one decoding step to the next. Therefore, a static assignment over-provisions precision where it is unnecessary and vice-versa. This motivates runtime precision adaptation: dynamically adjusting precision across prompts, layers and inference phases to match a model’s numerical demands.
Recent works have taken initial steps toward this goal. FlexQuant [5] switches layer-wise weight precision during decoding but stores separate quantized weight copies. Multi-scale quantization [12] avoids these copies by nesting several precisions within one integer bit layout whose width equals the highest supported precision. From this nested layout, lower precisions are obtained by truncating the least significant bits. DP-LLM (Kwon et al., 2026) extends this, switching each layer at decode time between two offline-calibrated precisions according to an input-dependent estimate of quantization error.
These approaches leave three challenges. First, their policies are not true runtime adaptations as they rely on offline calibration. This limits their ability to accurately track evolving precision demand during inference (Figure 1 (amber region)), yielding a suboptimal accuracy–precision tradeoff. Second, existing multi-scale formats use integer representations that suffer substantial accuracy degradation when lower precisions (sub-6-bit) are extracted from the nested representations, especially for activations and the KV cache (§ 5). Their mechanism does not directly extend to group-scaled floating-point formats such as NVFP4, which is used for low-precision inference on NVIDIA Tensor Cores. Moreover, extracting precisions from their nested representations incur irregular memory accesses and dequantization overheads [16], that sacrifice inference performance. Finally, they do not address batched inference, where one layer must serve requests of different precisions.
Figure 1. LiveCodeBench accuracy versus decode throughput for Qwen3.8-27B (batch size 1, one GB300 GPU).
Contributions. To address these limitations, we introduce AutoEMatic. At its core is a format-agnostic runtime policy that selects each layer’s precision across prefill and decode; AutoFP makes the policy practical by deriving all precisions from one NVFP4 base, and GPU kernels and an accelerator demonstrate AutoEMatic on existing and custom hardware. These four components form a full-stack co-design:
Runtime precision adaptation policy (§ 3.1): We identify two model-internal signals, a per-token and a per-layer measure of precision demand, and derive from their behavior a set of rules that assign layer-wise precisions.
Quantization format (§ 3.2): We introduce AutoFP, the first multi-scale floating-point (FP) format, which composes 2-, 4-, and 6-bit precisions from a shared NVFP4 base: the foundation from which other formats are composed. AutoFP6, the 6-bit E2M3 precision is obtained through mantissa refinement: by adding mantissa bits selectively to high quantization error base elements. The refinement metadata is stored separately from the base and applied on demand based on the runtime policy. Conversely, AutoFP2, the 2-bit precision is a quinary format, obtained by remapping the exponent bits of the base precision.
System implementation (§ 4): We integrate AutoEMatic into vLLM with kernels that execute AutoFP using NVFP4 Tensor Cores and efficiently serve concurrent requests.
Hardware accelerator (§ 6): We design an accelerator with an AutoFP-native mixed-precision datapath, which can process two AutoFP2 weights, one NVFP4 weight, or one AutoFP6 weight against a single streamed input activation.
To the best of our knowledge, AutoEMatic (Figure 1) is the first framework to jointly adapt weight, activation, and KV-cache precision at runtime, across prefill and decode, using an FP-based multi-scale format. Across reasoning and agentic benchmarks on models including Qwen3.8-27B, Gemma4-26B, and MiniMax-M2.7, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches FP6 (the group-scaled 6-bit counterpart to NVFP4) on most benchmarks, using 4.9 bits per weight versus 4.5 for NVFP4. AutoEMatic sustains this accuracy advantage while retaining 90–98% of NVFP4 throughput. On custom hardware, AutoEMatic reduces latency by 1.6–2.3× and energy by 7–23% compared to an NVFP4-only datapath at iso-accuracy.
Motivating Analyses
Precision Demand Changes During Inference
To understand how the precision across layers of a model change throughout inference, we set up an oracle that determines for each layer at every prefill and decode step, the lowest precision (WxAx-KVx) such that the KL-divergence between the quantized and full-precision model’s next-token distributions are within nats, maintaining iso-generation. Following a full-precision generation, each layer is quantized sequentially, accounting for cumulative quantization effects. The oracle chooses between the following candidates: NVFP4, group-scaled 2-bit (E1M0) and 6-bit (E2M3) counterparts to NVFP4, and standard FP8 (E4M3).
Decode Variations. We apply this analysis to responses from DS-Qwen-14B on LiveCodeBench (LCB) and Qwen3.8-27B on -bench, with up to 10K tokens per turn. Figure 2(a) aggregates the oracle precisions assignments across layers within the early, middle, and late thirds of each response. The distribution varies across decoding stages and models, reflecting underlying layer-wise precision variations from one decode step to the next (§C.1, Appx.).

Figure 2. (a) Precision distribution averaged across layers in the early, middle, and late generation stages (DS-Qwen-14B, LCB; Qwen3.8-27B, -bench). (b) Layer-wise precision assignments for initial prefill and additional refills (Qwen3.8-27B, -bench). (c) Rate of underprovisioning for ModelOpt AutoQuantize and FlexQuant (DS-Qwen-14B, LCB).
Prefill Variations. In agentic workloads, returns from tool calls extend the context and trigger additional prefills (Lu et al., 2026), termed refills. Figure 2(b) shows Qwen3.8-27B’s layer-wise oracle precision assignments over the initial prefill and additional refills. tial prefill and refills. Demand changes sharply between prefill stages: refill 3 uses 2-bit for nearly all layers, while others demonstrate a broader precision distribution (§C).
Therefore, precision demand depends jointly on the model, prompt, layer, and inference stage, and can change with any of these factors. The challenge of runtime-precision adaptation is to identify these dynamic variations without access to the full-precision reference used by the oracle.
Limitation of Static Mixed-Precision Assignment
We further ask whether a static assignment calibrated in-distribution (calibrated and evaluated on the same dataset) can closely track the oracle’s precision assignments. We examine two schemes: ModelOpt AutoQuantize (NVIDIA ModelOpt, 2025), which fixes per-layer precisions using calibrated sensitivities, and FlexQuant [5], which follows a static schedule of lowering precision during inference. Both use 100 LiveCodeBench calibration samples under a 6-bit budget, matching the oracle’s achieved average precision, and are evaluated on the remaining 780 samples. Figure 2(c) reports underprovisioning: the fraction of decode steps where the oracle requires higher precision for a layer than the static policy assigns. AutoQuantize under-provisions 26.9% on average and FlexQuant 37.6%. In-distribution calibration tailors to the aggregate precision demands of the calibration samples. However, the resulting fixed assignments or schedules do not respond to fine-grained variations across prompts and decode steps, leading to under-provisioning even on held-out samples from the same dataset, which directly impacts accuracy (§5).
Non-uniform precision requirement within groups
Raising the precision (refining) of a tensor, for example from 4 to 6 bits, is conventionally applied uniformly [18]. We instead ask whether the error of a group-scaled format such as NVFP4 is concentrated in a few elements of each group, so that refining those elements alone suffices. We partition each 16-element group into four 4-element subgroups (; §I (Appx.)) and sweep NMSE thresholds from 0.3 to 0.001, and map the elements of every subgroup above the threshold to 6-bit (E2M3). For Qwen3-8B and DS-Qwen-14B (Figure 3), refining ~ 50% of weight subgroups achieves ~ 90% of the NMSE reduction obtained by moving from NVFP4 to FP6 uniformly. Activations exhibit a similar trend. AutoFP leverages this to reduce metadata overhead when constructing higher precisions from NVFP4.
Figure 3. Refining a subset of elements achieves NMSE comparable to 6-bit. Y-axis is NMSE, such that NVFP4 is 1 and FP6 is 0, vs. fraction of subgroups refined to 6-bit.
AutoEMatic Quantization
Runtime Precision Adaptation Policy
Precision-Adaptation Signals. AutoEMatic determines precision at runtime from two complementary model-internal signals. The first is a global token-generation uncertainty signal , measured as the normalized Shannon entropy of the top- tokens’ output logits (§D.1). When the model is uncertain about the next token (high entropy), small quantization induced perturbations in the output logits can change which token it generates [6 a]. Such steps warrant higher precision and vice-versa. However, a global signal alone cannot determine layer-wise precision variations. The second signal is based on the layer structure of LLMs. LLM layers have a residual connection that carries its input directly to its output, with the attention and MLP blocks adding their contributions along the way [27]. A layer whose output hidden states are close in representation to its input contributes little beyond copying its input to the output; we term it a low-activity layer, and its weights, activations, and KV cache can tolerate lower precision. This local layer-update activity is measured at every step , for every layer as,
where ensures numerical stability. The two signals combine into one precision-demand metric for prefill or decode (P/D), , with set separately for each phase2. Rather than calibrated thresholds, AutoEMatic tracks the running mean and standard deviation of and establishes a set of rules, that vary across inference stages: sustained increases (decreases) of the metric over a fixed number of preceding layers relative to and raise (lower) the precision of the it, with . That is, AutoEMatic lowers precision more conservatively. At low-entropy, thresholds are reversed.
next layer. Intuitively, these rules observe sustained trends in the metric as evidence that precision should change, rather than reacting to isolated fluctuations. Each layer executes at - with chosen by the rule. 2-bit activations and KV cache degrade accuracy substantially, therefore, the 2-bit level applies to weights only (W2A4-KV4). denotes nominal precision levels; the policy is agnostic to the choice of levels or quantization format.
Prefill policy. Prefill precedes token generation, so no token entropy exists and we set (). We observe that, across models and datasets, shows a single large spike3 at an early layer, at a position largely invariant across prompts and refills (Figure 4(a)). Consistent with early-exit work (Shen & Zou, [23]), the spike marks the layer by which the model has integrated the prompt context, so the layers up to and including it are the most sensitive and must execute at max bits. Because the spike position differs across models, AutoEMatic detects it online as . The subsequent layers follow the rule, layers below (above) lower (raise) the precision of the next layer by one level in , saturating at its bounds.
Figure 4. Layer-wise behavior of and . (a) Prefill of Qwen3.8-27B on a 7B-bench task. (b) Decode of GPT-OSS-20B on RULER-16K, with high- and low- steps.
Decode policy. During decode, we set incorporating generation entropy into the metric (Figure 4(b)). Since token entropy varies little between consecutive steps (Ramachandran et al., [21]) and the of the current step is known only after its token is generated, reuses from the preceding step. Every decode step starts layer 0 at 4 bits and applies the rules in Table 1. Since layer tolerance to quantization varies with generation entropy, we separate decoding into high-entropy () and low-entropy () steps by a threshold . At high-entropy steps, the rule requires consecutive low-activity layers to lower precision, but only consecutive high-activity layers to raise it, with . That is, AutoEMatic lowers precision more conservatively. At low-entropy, thresholds are reversed.
| Policy | Prefill | Decode | |
| Metric | High entropy | Low entropy | |
| Initial precision | 6-bit | 4-bit | 4-bit |
| Rules activate | After spike | Layer 0 | Layer 0 |
| Raise precision | layers | layers | layers |
| Lower precision | layers | layers | layers |
Table 1. Runtime precision adaptation policy.
AutoFP Quantization Format
Storing one quantized copy per level in would cost 13.5 bits per weight. AutoFP instead stores one representation, the NVFP4 base with its group scales and additional metadata, and derives the 2- and 6-bit levels from it on demand.
AutoFP6. Each additional mantissa bit adds local precision around every NVFP4 value. However, a floating-point value cannot be refined by ‘appending’ mantissa bits. Since NVFP4 rounds to nearest, the full-precision value may lie below the E2M1 value, whereas appended mantissa bits can only select values above it, and not the nearest E2M3 value in the neighborhood that minimizes quantization error (e.g., 1.4 is stored as 1.5 in E2M1; appended bits can then reach only 1.5–1.875, never 1.375).
Therefore, instead of storing the additional metadata as mantissa bits, AutoFP encodes them as additive signed offsets. Because the E2M1 grid is a subset of the E2M3 grid, every E2M1 value is also an E2M3 value. Refining only requires storing the direction and number of E2M3 grid steps to move from it. Since E2M3 adds three additional values around every , a 2-bit two’s-complement signed offset offers sufficient resolution. The step size around with exponent is , and the refinement term is (the mantissa equivalent value of the signed offset) and the refined value is thus, . The refinement is thus additive. Since the step is dependent on the exponent of , decoding exactly would require a per-element exponent computation (§7). AutoFP therefore decodes with a fixed step , . The fixed step is E2M3-like since it reproduces the E2M3 grid for values and applies a step finer than E2M3 for larger values.
To reduce the metadata overhead of storing signed offsets, AutoFP6 refines only the subgroups with the largest NMSE reduction (§2.3). Each 16-element group carries a 4-bit refinement mask with one bit per 4-element subgroup, set to 1 for a refined subgroup and 0 otherwise, and the signed offsets are stored only for the subgroups with mask bit of 1 (Figure 5). The NVFP4 base is left intact, the signed offsets are stored separately and applied on demand for AutoFP6 layers. Since the refinement is additive, every refined operand then splits as for , where . By linearity, a refined GEMM is, .

Figure 5. Mantissa refinement of NVFP4 to AutoFP6.
Two choices make this hardware-friendly. First, AutoEMatic imposes structured sparse (Ramachandran et al., [20]) refinement within each group, such that weight and activation subgroups are refined per group. Second, it keeps and disjoint. selects subgroups for refinement (along -dimension) and is shared across the rows of , and , shared across the columns of . This ensures disjointness holds for every row–column pair in the GEMM. Disjointness eliminates the second-order product of refinements, which contributes negligibly to accuracy (§7), reducing the number of additional operations required for a refined layer. is selected at runtime and is selected offline under a refinement budget , the fraction of weight subgroups refined, either by NMSE or by a refinement-aware GPTQ calibration that assigns refinement for subgroups with the largest Hessian-weighted error (§E).
AutoFP2. Truncating the stored NVFP4 to 2 bits yields an E1M0 format representing , which cannot represent zero and causes significant accuracy degradation. AutoFP2 instead is a quinary format, , derived from the NVFP4 code by retaining the sign bit and remapping the two exponent bits to a 2-bit magnitude in on demand via . Nominally, representing AutoFP2 requires 3-bits. However, we show in § 6, the accelerator packs two AutoFP2 digits into 5 bits, 2.5 bits per digit. AutoFP2 reuses the NVFP4 scales which normalize to ; since reaches only , the scale is multiplied by a constant (§ 5). As AutoFP2 is derived from NVFP4 on demand, it saves no memory bandwidth, and its main benefit is in prefill throughput and energy (§ 6).
System Implementation
We implement kernels that executes AutoFP6 natively on NVFP4 Tensor Cores from a single packed model: E2M1 base values with E4M3 group scales , signed offsets stored as one byte per refined subgroup, and refinement masks stored as 4 bits per group along the -dimension (§D.3, Appx.). Since Tensor Cores do not support sub-4-bit formats, AutoFP2 layers are emulated as NVFP4, which preserves the accuracy of the precision decision but not the performance of AutoFP2 execution.
Refinement as an Extended-K NVFP4 Operation
A GEMM accumulates products along its reduction dimension , so a sum of GEMMs with the same output shape equals one GEMM over operands concatenated along (see Appx.). AutoEMatic executes the refined GEMM of § 3.2 in this manner. Since and are disjoint, the two refinement terms can be merged into a single GEMM of up to length- as, (Figure 6). Each of these operands are constructed as follows: for subgroups that refine weights, holds the refinement term and holds the activation base values in the corresponding position, such that realizes . Similarly, for subgroups with activation-refinement, holds the weight base values and holds , realizing . The reduction dimension is up to length- because it depends on the number of subgroups refined. A refined layer thus executes as a single NVFP4 GEMM of reduction length up to as, , which we term the extended-K GEMM. The GEMMs use Blackwell’s tcgen05.mma instruction, which consumes only E2M1 values with E4M3 group scales. The refinement is not representable in E2M1 as is, whereas its scaled version is. The refined subgroups of and therefore hold , and the additional factor of four is absorbed into the scale of , which is set to . Each refined subgroup then contributes or . The kernel fuses the construction of both extended operands with the MMA. Attention follows the same construction and runs on Flash-Infer’s FP8 attention kernel. As in ThinKV ([21]), AutoEMatic uses two page types of equal byte size: a base page holds unrefined tokens with only the base values, and a refinement page holds refined tokens with base values and signed offsets in NVFP4. This enables one paged allocator to serve both types.
Figure 6. Extended- GEMM for one 16-element group and three requests (batch size of 3).
Serving Mixed-Precision Batches
Requests in a batch may select different precisions for the same layer. The weight operands do not vary with that choice: is shared by every request, and with its mask is fixed offline. Only the activation masks and their signed offsets differ across requests. Rather than running a separate GEMM for each precision decision, AutoEMatic forms the union of masks , the per-group OR of the activation masks of all requests in the batch, and serves the whole batch with one extended- GEMM against a single built from it. As shown in Figure 6, holds in its weight-refined subgroups, in every subgroup of , and zero elsewhere. Each request then constructs its own : in the weight-refined subgroups,
in the subgroups it refines.
Accuracy and GPU Performance
Experimental Setup
Models, Datasets, and Baselines. We evaluate Qwen3-8B, Qwen3.8-27B, DeepSeek-R1-Llama-70B, Gemma-4-26B-A4B, and MiniMax-M2.7 on reasoning (LongBench v2 (Bai et al., 2025), AIME 2024 [11], LiveCodeBench (Jain et al., 2025)) and agentic (-bench (Barres et al., 2025), LongMemEval (Wu et al., 2024), SWE-bench Verified [15]) workloads. For WxAx-KVx quantization, we compare against uniformly quantized NVFP4 and FP6, AutoQuantize (NVIDIA ModelOpt, [13]), which selects between NVFP4 or FP6 per layer under a bit budget, and DP-LLM (Kwon et al., 2026), which switches among 3- to 6-bit of an Any-Precision format (Park et al., [17]). Since DP-LLM adapts only weights, we extend its policy to activations and KV cache (DP-LLM) for a fair comparison. AutoQuantize, DP-LLM, and AutoEMatic’s refinement-aware GPTQ for weights use 256 samples from WikiText-2 for calibration. All experiments run in vLLM (Kwon et al., 2023) on NVIDIA GB300 GPUs with a 32K-token generation limit at temperature 0.5.
Hyperparameters. We set ; rules and , ; entropy threshold , is computed over the top- logits; subgroup size ; AutoFP2 scale multiplier ; activation NMSE threshold 0.003; weight refinement budget ; KV page size . Hyperparameters are ablated in Appx.
Storage and Execution Precision. Storage precision is the packed model size in average bits per weight including metadata. For AutoQuantize, it is the target bit budget plus 0.5-bits for scale, 6-bits for DP-LLM and bits for AutoEMatic at (since the masks are shared across rows for weights, it is negligible). Execution precision is the nominal bitwidth at which layers execute averaged over layers and inference steps.
Main Accuracy Results
Comparison with Static and Runtime Baselines. AutoEMatic operates without a bit budget: its execution precision follows the dynamic decisions of the runtime policy. For a fair comparison, AutoQuantize and DP-LLM are calibrated to the budget nearest AutoEMatic’s achieved average. As shown in Table 2, across models and datasets, at 4.5–5.1 bits of average execution precision, accuracy of AutoEMatic exceeds AutoQuantize by up to 10% and DP-LLM by up to 66.9% under a similar budget. AutoQuantize has a high underprovision rate which directly impacts its accuracy. AutoEMatic’s advantage is particularly pronounced in agentic tasks, whose repeated prefills and long decodes impose dynamic precision demands, that DP-LLM’s offline-calibrated runtime-policy cannot reliably track. DP-LLM is further penalized by its nested integer format: the sub-6-bit precisions extracted from the shared representation incur substantial error. Overall, through runtime adaptation driven by model-internal signals and the AutoFP format, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches FP6, with a storage precision of 4.9 bits.
| Model | Method (budget) | LB v2 | AIME | LCB | -bench | LME |
| Qwen3-8B | BF16 | 35.4 | 56.7 | 66.5 | 30.7 | 42.8 |
| NVFP4 | 33.3 | 50.0 | 49.2 | 17.5 | 26.0 | |
| FP6 | 35.4 | 56.7 | 61.4 | 30.7 | 39.9 | |
| AutoQuantize (4.5) | 35.4 | 50.0 | 60.0 | 20.2 | 31.9 | |
| DP-LLM (4.5) | 21.6 (4.8) | 16.7 (4.5) | 26.6 (4.6) | 10.6 (4.5) | 18.5 (4.5) | |
| AutoEMatic | 35.4 (4.6) | 56.7 (4.7) | 60.5 (4.8) | 27.2 (4.6) | 38.2 (5.1) | |
| Qwen3.8-27B | BF16 | 56.9 | 93.3 | 76.3 | 74.6 | 82.8 |
| NVFP4 | 48.7 | 86.7 | 70.6 | 50.0 | 70.6 | |
| FP6 | 56.9 | 86.7 | 76.3 | 71.7 | 82.8 | |
| AutoQuantize (4.8) | 56.9 | 86.7 | 74.9 | 62.0 | 81.2 | |
| DP-LLM (4.8) | 25.4 (4.7) | 56.7 (4.8) | 51.4 (4.5) | 25.6 (4.6) | 15.9 (4.7) | |
| AutoEMatic | 56.9 (4.5) | 86.7 (4.6) | 76.3 (4.6) | 71.7 (5.1) | 82.8 (4.8) | |
| Gemma-4-26B | BF16 | 46.7 | 86.7 | 85.7 | 61.4 | 72.6 |
| NVFP4 | 40.2 | 63.3 | 75.3 | 46.5 | 63.6 | |
| FP6 | 45.5 | 80.0 | 83.1 | 60.5 | 70.1 | |
| AutoQuantize (5.0) | 41.0 | 80.0 | 80.1 | 58.6 | 68.1 | |
| DP-LLM (5.0) | 35.6 (4.8) | 50.0 (5.4) | 68.6 (5.4) | 11.5 (4.9) | 16.0 (4.9) | |
| AutoEMatic | 45.5 (5.0) | 80.0 (5.0) | 83.1 (5.0) | 60.5 (5.1) | 70.1 (4.9) |
Table 2. Accuracy comparison of AutoEMatic and baselines across reasoning and agentic workloads. Execution precision appears in parentheses next to accuracy. LB v2: LongBench v2, LCB: LiveCodeBench, LME: LongMemEval.
Large-Scale Agentic Analysis. On MiniMax-M2.7, a 230B-parameter agentic coding model, evaluated on SWE-bench Verified, AutoEMatic matches FP6 accuracy and remains within 0.5% of baseline (at FP8) at an average execution precision of 4.2 bits in prefill and 4.5 in decode, against 6 bits for FP6 (Table 3). Figure 7 demonstrates AutoEMatic’s runtime adaptation across prefill and decode.

Figure 7. Layer-wise precision assignment by AutoEMatic on MiniMax-M2.7.
| Method | Exec. Precision | Acc. (%) | |
| Prefill | Decode | ||
| FP8 | 8 | 8 | 71.0 |
| NVFP4 | 4 | 4 | 67.8 |
| FP6 | 6 | 6 | 70.5 |
| AutoEMatic | 4.2 | 4.5 | 70.5 |
Table 3. Accuracy comparison of MiniMax-M2.7 on SWE-bench Verified.
GPU Performance Analysis
Figure 8 reports prefill time-to-first-token (TTFT) and decode throughput for DS-Llama-70B and MiniMax-M2.7 on 4× GB300 GPUs, sweeping input lengths of 1K–128K tokens for prefill and batch sizes of 1–128 for decode with an 8K-token input and a 16K-token output. We compare against methods that execute on Tensor Cores: NVFP4 through the native CUTLASS W4A4 path, FP6 through a fused kernel that dequantizes E2M3 to FP8, and AutoQuantize uses either based on NVFP4/FP6 layers. AutoEMatic’s runs use precision assignments from LongBench v2.

Figure 8. Decode throughput and prefill TTFT analysis on 4× GB300 GPUs for DS-LLaMA-70B and MiniMax-M2.7.
Prefill. AutoEMatic reduces TTFT by up to 1.4× over FP6 and 1.25× over AutoQuantize across input lengths. Specifically, despite AutoEMatic’s runtime precision adaptation enabling superior accuracy compared to NVFP4, the TTFT is within 10% of the NVFP4 baseline (small performance penalty). Two factors keep this performance penalty small: (a) when all requests in a batch choose 4-bit for a layer, they execute at NVFP4 throughput and (b) when any request chooses AutoFP6 it executes as a single extended- GEMM on Tensor Cores, instead of separate GEMMs (§7).
Decode. AutoEMatic improves decode throughput by 1.5–2.5× over the full-precision baseline across batch sizes, and attains 90–98% of NVFP4 throughput, i.e., near-NVFP4 throughput at FP6 accuracy.
Accelerator Architecture
Architecture Overview
On GPUs, AutoFP2 is emulated and AutoFP6 requires an extension of the reduction dimension. An AutoFP-native datapath targets both limitations of existing hardware. We extend a representative accelerator MAGNet (Venkatesan et al., [24]; Hooper et al. (2025)) as proof-of-concept. The output-stationary local-weight-stationary (OS-LWS) design (Figure 9(a)) has 256 PEs and a global SRAM (L2) that loads data from DRAM (modeled). Each PE (Figure 9(b)) has mixed-precision vector MAC (vMAC) that computes one -element NVFP4/AutoFP6 or two -element AutoFP2 dot products per cycle. A PPU processes completed outputs (Figure 9(c)). Signals mode_nvfp4 and refine_en select AutoFP2, NVFP4, or AutoFP6 modes of operation.
Figure 9. AutoEMatic accelerator architecture.
Weight Encoder and Packed AutoFP2 Format
Weights are stored in DRAM as NVFP4 (§ 3.2), but inside the PEs every weight is held in AutoFP2, so that the vMAC can execute all modes with uniformity. Two requirements follow. First, for the vMAC to process AutoFP2 at twice the throughput of NVFP4, each weight load from the buffers must deliver two AutoFP2 weights. Second, in NVFP4 mode a single AutoFP2 digit cannot represent an NVFP4 value. We introduce a packed AutoFP2 format that satisfies both by encoding a pair of AutoFP2 digits () in 5 bits (2.5 bits per digit). In the packed format (Figure 10), an escape bit precedes two 2-bit fields and . A nonzero AutoFP2 digit is one of ±1 and ±2. When both digits are nonzero (), and hold them in E1M0. Otherwise (), records which digits are zero and holds the nonzero digit, if any. The weight encoder forms the AutoFP2 packed format as weights enter the PE and writes them to the weight buffer. The packing overhead is amortized by the weight stationary dataflow. In AutoFP2 mode (mode_nvfp4=0), it applies the exponent remapping of § 3.2 to two weights from adjacent rows (different dimensions) that share an input activation and packs the two digits. In NVFP4/AutoFP6 modes (mode_nvfp4=1), the pair represents one NVFP4 weight as with fixed coefficients. We set , such that their linear combinations span the maximum NVFP4 magnitude 6 (derivation in Appx.). For e.g., . Each AutoFP2 digit is a Boolean function of the E2M1 bits (§D.7).
Figure 10. AutoFP2 representation.
vMAC Datapath Design
Each PE is a element wide vMAC (Figure 11) organized as two base lane blocks each with dedicated set of refinement lanes and pipelined into multiply and accumulate stages. The vMAC accepts weights in the packed AutoFP2 format. The refinement data has a separate input interface carrying the masks and signed offsets of the refined subgroups. Each base lane comprises an even () and an odd () AutoFP2 sub-lane that shares one streamed NVFP4 activation. For energy-efficiency, all computations use fixed-point arithmetic (§D.8).
Figure 11. AutoFP vector MAC datapath.
Base Lanes. In the multiply stage, the escape bit of each packed pair decodes the two digits. When , identifies which digit is zero, and the corresponding sub-lane is bypassed: its product is set to zero. For a nonzero digit, or holds an E1M0 code, or , so each sub-lane forms , , with simple shifters. Within a block, the even and odd sub-lanes feed separate adder trees in the adder stage, and for lanes 0–15, and , for lanes 16–31. Every execution mode differs only in how the tree outputs are combined. In NVFP4 mode, the , correspond to the partial sum accumulated from the same weight , therefore the mode-dependent reduction unit computes (implemented in fixed-point) to form the corresponding NVFP4 partial sum, and similarly for , as shown in Figure 11. The base lanes perform the same operation in AutoFP6 mode. In AutoFP2 mode, within a block the two sub-lanes process corresponding weights from different rows (so that they can share one input activation). The weights of the same row are therefore spread across the even and odd sub-lanes of both blocks, so the mode-dependent reduction unit sums across blocks,
and .
Sparse Refinement Lanes. The refinement datapath adds refinement lanes per block as two -lane slots (in §7 we establish this as sufficient). In AutoFP2/NVFP4 mode (refine_en=0), the refinement lanes are gated for energy efficiency. Because and are disjoint, a refined weight/activation sub-group occupies exactly one slot. The collector supplies each slot with the refinement mask, four 2-bit signed-offsets , and a bit identifying weight/activation refinement. Each lane has two 4:1 muxes that pick the corresponding base-weight and activation subgroups using the mask; a 2:1 selector chooses the corresponding : for weight refinement, or for activation refinement (note the similarity with construction of , ). Each lane forms in fixed-point arithmetic. Each refinement lanes’ contributions are accumulated as . It is accumulated with the base lanes partial sums as, .
Scaling and accumulation. Each is multiplied by the scale factor of its corresponding block. In AutoFP2 mode, each is a 32-element dot product corresponding to two different weight rows with the same activations. In NVFP4 and AutoFP6 mode, each corresponds to a 16-element dot product along the same row (same dimension). After scaling, and are summed into one 64-bit accumulator in NVFP4/AutoFP6, while in AutoFP2 mode are held in the two 32-bit halves of the accumulator.
Shared Post-Processing Unit
The PPU (Figure 9(c)) quantizes completed PE outputs into the next layer’s input activations. If a layer requires refinement, it also computes per-subgroup NMSE and selects subgroups with NMSE above excluding those in and assigns signed offsets . The PPU also performs non-linear operations, and computes the metric .
Accelerator Evaluation
Implementation. The accelerator is implemented in SystemC, with RTL generated using Siemens Catapult HLS. We use a 2MB global SRAM, 896B weight, 1.5KB activation, and 2KB accumulation buffers. We perform synthesis in TSMC 3nm at a frequency of 1 GHz. Synopsys VCS is used for simulation and PrimeTime PX for power estimation.
Area Analysis. Figure 12(a) breaks down the area of the AutoEMatic accelerator. The 2MB global SRAM occupies 75% of the area. The PE array, with all 256 PEs, takes 23.8%. The per-PE weight encoders account for only 0.4% and the vMAC 3.6% of the total area; within a vMAC the refinement lanes (12.3%) and the mode-dependent reduction logic for NVFP4- and AutoFP2-mode (6.7%), result in a net 1.2× overhead over an NVFP4-only vMAC. The accelerator is only 1.1% more area than the NVFP4 baseline.
Figure 12. Area breakdown of (a) the AutoEMatic accelerator and (b) its vMAC.
Iso-Accuracy Performance Comparison. We compare against single-format AutoFP2 and NVFP4 baselines under iso-accuracy with AutoEMatic by replaying its layer-wise precision traces, as recorded on LiveCodeBench. Layers above a baseline’s native precision execute with an extended reduction dimension, accumulating into the same partial sums; the NVFP4 design, for instance, executes a refined layer analogous to the extended- GEMM (§ 4). Figure 13(a) reports end-to-end latency normalized to the NVFP4 design at batch sizes 1 and 8. AutoEMatic is 1.6–2.3× faster than NVFP4 for two reasons: AutoFP2 layers run at twice the NVFP4 throughput, which benefits compute-bound prefill most (1.3–1.8× faster), and refined layers execute natively in the sparse refinement lanes without the additional pass an extended- reduction requires. Across evaluated models, on average, AutoEMatic achieves a compute density of 5.46 TOPS/mm, compared to 3.23 TOPS/mm for NVFP4 and 6.16 TOPS/mm for AutoFP2.
Figure 13. Normalized latency and energy comparison with single-format baselines, under iso-accuracy.
Iso-Accuracy Energy Comparison. Figure 13(b) reports energy for the same traces, normalized to NVFP4 and broken down into different components, each including static and dynamic energy. AutoEMatic consumes 0.77–0.93× the energy of NVFP4, as (a) all modes use AutoFP2 lanes and (b) lower runtime. Its vMAC datapath averages 15fJ/op, against 22fJ/op for NVFP4 and 10fJ/op for AutoFP2.
Discussion and Ablations4
[Algorithm+Quantization] Component Ablations. Table 4 isolates the runtime policy and the quantization format. Applying AutoEMatic’s runtime policy to separately quantized AutoFP2, NVFP4, and FP6 copies (row 3), attains 66.2%, but needs 13.5 stored bits per weight. Applied to the integer multi-scale Any-Precision format (row 4), attains only 43.8%. This suggests that a favorable accuracy–efficiency tradeoff requires both: the runtime policy and an FP-based multi-scale format. Combined (row 7), AutoEMatic preserves 65.7%, within 0.6% of the separate-copy upper bound, at 4.9 bits, executing 40% layers at 6-bit, 45% at 4-bit and 15% at 2-bit. Selecting the refined weight sub-groups by NMSE alone, with a threshold of 0.003 at the same budget (Table 4, row 5), attains 64.8%; refinement-aware GPTQ (row 6), which selects them by their reduction in the Hessian-weighted error raises accuracy to 65.6%.
| Component | Bits | Acc. | 2 / 4 / 6 (%) |
| 1. BF16 | 16 | 68.4 | – |
| 2. NVFP4 (W-A-KV) | 4.5 | 51.7 | 0 / 100 / 0 |
| 3. + AutoEMatic runtime policy w/ copies | 13.5 | 66.2 | 20 / 40 / 40 |
| 4. AnyPrecision LLM on + AutoEMatic runtime policy | 6 | 43.8 | 8 / 42 / 50 |
| 5. AutoFP6 | 4.9 | 64.8 | 0 / 0 / 100 |
| 6. + Refinement-aware GPTQ | 4.9 | 65.6 | 0 / 0 / 100 |
| 7. AutoEMatic format + runtime policy, | 4.9 | 65.6 | 15 / 45 / 40 |
Table 4. Component ablations with DS-LLaMA-70B on LCB.
[Quantization] Refinement Decoding. Figure 14(a) compares decoding signed offsets with the fixed step against exact, per-element exponent computation on DS-LLaMA-70B and Qwen3-8B on LCB. The fixed-step has similar accuracy as the exact exponent computation, while achieving up to 28% higher decode throughput. This is because the fixed-step does not require loading the base value when constructing the extended- GEMM operands.
Figure 14. Ablations. (a) Fixed vs. exponent-scaled decoding of . (b) Underprovision rate of AutoEMatic. (c) Prefill precision at spike. (d) Refined subgroups per group. (e) Accuracy vs. throughput pareto. (f) Kernel ablations. (g) For one decode step: across layers and the precision assigned by the oracle, an instantaneous rule, and AutoEMatic.
[Algorithm] AutoEMatic Runtime Policy vs. Oracle. Figure 14(b) compares per-layer underprovision rate of AutoEMatic’s precision decisions against the oracle of restricted to the precision set . AutoEMatic underprovisions only 7.7% of layer executions on average, and its 2/4/6-bit distribution of 15/45/40% closely tracks the oracle’s 20/41/39%. Using alone has a higher rate of underprovisioning, suggesting the combined metric as the optimal choice. We omit from this comparison because it is a global signal shared across layers.
[Algorithm] Rule-based vs. Instantaneous Adaptation. Figure 14(g) traces one decode step of DS-Qwen-14B layer by layer: (top) and the precision assigned by the oracle of , by an instantaneous policy that switches with every rise and fall in , and by AutoEMatic’s rule-based policy. Evident from the oracle’s own behavior, the rules are not ad hoc, oracle precision does not rise (drop) immediately after an increase (decrease) in the metric, but only once4: See Appx. for additional discussions and ablations. the change persists over several layers. The instantaneous policy switches on blips and drops to 2 bits at layers where the oracle requires 6. AutoEMatic’s rules are built inspired by this behavior of the oracle and change precision only once the model has provided sufficient evidence.
[Algorithm] Prefill Precision at the Spike. Figure 14(c) varies only how the spiking layer in prefill is executed, keeping the decode policy fixed. Running prefill entirely in BF16 gives 71.7%, the same as AutoEMatic, which executes the spike layers at 6 bits. Lowering the spike layer to NVFP4 costs 3.8% and to AutoFP2 15.9%.
[Quantization] Subgroups Refined Per Group. Figure 14(d) shows the distribution of weight+activation subgroups refined per group, , for DS-LLaMA-70B on LCB and MiniMax-M2.7 on SWE-bench. 22–31% of groups need no refinement (0:4), and only 1.1% of DS-LLaMA-70B groups and none in MiniMax-M2.7 exceed 2:4, supporting the choice of 8 refinement lanes.
[System] Kernel Ablations. Figure 14(e) plots accuracy against decode throughput normalized to NVFP4 for DS-LLaMA-70B on LCB at batch 512. AutoEMatic lies on the Pareto frontier: 1.3× faster than FP6, and 1.1× faster than uniform AutoFP6. AutoEMatic’s superior performance-accuracy tradeoff stems from the runtime policy: it assigns 4 bits to most layers, which then run at NVFP4 throughput, and 6 bits only where accuracy requires it, which still runs as a single extended- GEMM on the same NVFP4 Tensor Cores. Figure 14(f) ablates AutoEMatic’s two kernel mechanisms under different batch sizes. Without the extended- GEMM, executing the base and refinement GEMMs separately and summing their outputs lowers throughput by 1.5× at batch size 1; without the union of masks, splitting mixed-precision batches into per-precision sub-batches lowers throughput by 1.6× at batch size 512.
[Accelerator] Precision Mix Sweep. Figure 15 evaluates different layer-wise precision distributions on DS-LLaMA-70B against an NVFP4-only datapath. AutoEMatic’s runtime assignment (15/45/40) lies on the Pareto frontiers of accuracy versus area efficiency and accuracy versus energy efficiency among the evaluated configurations of different precision distributions, achieving 65.6% accuracy with only an 8% performance/area penalty and 9% higher energy/op.
Figure 15. DS-LLaMA-70B LCB accuracy vs. performance per area and energy per op, normalized to an NVFP4-only datapath.
Related Work
Multi-scale quantization. Any-Precision LLM ([17]), Matryoshka Quantization ([12]), Any-BCQ ([16]), and RRQ ([10]) store several precisions within one integer representation, as nested bit planes or successive residual stages, from which the selected precision is reassembled before computation.
Runtime precision adaptation. FlexQuant ([5], DP-LLM (Kwon et al., 2026), MoBiQuant [25], and QuickSilver (Khanna et al., 2025) adapt weight precision during decoding through predetermined schedules, calibrated error thresholds, or trained routers, and Mix-Quant (Lu et al., 2026) assigns NVFP4 to prefill and BF16 to decoding. For the KV cache, PM-KVQ [6] progressively lowers precision and ThinKV [21] assigns precision by reasoning importance. Appx. provides an extended discussion.
Conclusion
AutoEMatic shows that adapting precision to the evolving numerical demands of LLM inference improves its accuracy–efficiency tradeoff. AutoFP switches precision at runtime through mantissa refinement and exponent remapping of a shared NVFP4 base, without requiring redundant weight copies. Future work will explore achieving memory benefits with AutoFP2 and extending AutoFP to higher precisions through joint exponent and mantissa refinement.
Extended Related Work
Static post-training quantization. Most PTQ methods choose a precision offline and keep it fixed. GPTQ (Frantar et al., 2023) quantizes a weight matrix one column at a time and uses approximate inverse-Hessian information to update the remaining weights, compensating the quantization error. AWQ [4] identifies salient weight channels from activation statistics and protects them by per-channel scaling before quantization.
Group-scaled floating-point formats and their extensions. MX formats [22, 14] share a power-of-two E8M0 scale across a group of 32 FP8, FP6, FP4, or INT8 elements, and NVFP4 [2] stores E2M1 elements in groups of 16 with an E4M3 group scale and a per-tensor FP32 scale. MX+ (Lee et al., 2025) observes that the group’s largest element does not need its exponent field given the shared scale and repurposes those bits as extra mantissa for that element. M²XFP (Hu et al., 2026) adds two metadata bits per subgroup, which serve as extra mantissa bits for the subgroup’s largest activation and as a scale multiplier for weights. AdaMX [9] chooses per group between FP4 and INT4 grids combined with group-maximum mantissa extension or scale refinement at a constant bit width. These extensions are fixed after quantization and refine one element or one scale per group. AutoEMatic is generally applicable to any group-scaled floating-point format.
Multi-scale quantization. Multi-scale methods store several precisions in one representation so that precision can change without separate checkpoints. Any-Precision LLM [17] quantizes a seed model at the lowest bit width with non-uniform clustering, then adds one bit at a time by splitting each cluster, storing the codes as bit planes so that lower-bit codes are prefixes of higher-bit codes. Matryoshka Quantization [12] co-trains a single INT8 model so that slicing its most significant bits yields usable INT4 and INT2 models. AnyBCQ [16] builds 2- to 4-bit models on binary-coded quantization by freezing existing bit planes and adding residual planes with re-optimized scales, and RRQ [10] represents weights as a 2-bit base plus successive 2-bit quantized residuals, so that 4-, 6-, and 8-bit precisions are sums of residual stages.
Runtime precision adaptation. Mix-Quant (Lu et al.,
applies NVFP4 to prefill and keeps BF16 for decoding, fixing each phase’s precision. FlexQuant Liu et al., [5] stores both INT8 and INT4 weights and, during decoding, replaces INT8 layers with INT4 in an order fixed offline by KL-divergence analysis whenever a moving average of output entropy falls below a threshold. MoBiQuant Wang et al., [25] decomposes weights into recursive 2-bit residual slices and uses a trained token router to decide how many slices each token activates. QuickSilver Khanna et al., 2025 computes, at a layer, the entropy of each token’s hidden representation and assigns 8-, 4-, or 2-bit precision by thresholds.
KV-cache compression. KIVI Liu et al., [7] quantizes the key cache per channel and the value cache per token to 2 bits without tuning, and KVQuant Hooper et al., 2024 combines per-channel pre-RoPE key quantization, sensitivity-weighted non-uniform datatypes, and a dense-and-sparse split that keeps outliers separately. For long chain-of-thought reasoning, PM-KVQ Liu et al., [6] lowers the bit width of the KV cache progressively as the context grows and allocates higher bit widths to more sensitive blocks, and ThinKV Ramachandran et al., [21] classifies chain-of-thought segments into thought types from attention sparsity, assigning token precision by thought importance and evicting tokens of less critical thoughts.
Mixed-precision accelerators. MicroScopiQ Ramachandran et al., [19] retains outlier weights at higher precision and offsets their additional bits by pruning the least important weights, executing on multi-precision integer processing elements with a network-on-chip that routes the high-precision outliers. AdaMX Luo et al., [9] shares one multiplier datapath across its group formats and applies up to three per-group scalar correction terms through an auxiliary path. MixFP4 Zou et al., [28] selects one of two FP4 element formats, E2M1 or E1M2, per group, signaled by the sign bit of the E4M3 group scale, and decodes both to a common internal E2M2 representation.
Low-Precision Floating-Point Formats
We write for a signed floating-point format with one sign bit, exponent bits, and mantissa bits. Table 5 summarizes them.
| Format | S/E/M | Magnitudes | Bits/elem. |
| E2M1 (NVFP4) | 1/2/1 | 4.5 | |
| E2M3 (FP6g) | 1/2/3 | 0–7.5 in steps of | 6.5 |
| E1M0 | 1/1/0 | 2.5 | |
| E0M1 | 1/0/1 | 2.5 | |
| Quinary digit (AutoFP2) | – | 3.0 | |
| E4M3 (scale) | 1/4/3 | – 448 | – |
Table 5. Low-precision floating-point formats.
NVFP4
NVFP4 Alvarez et al., [2] stores each element as an E2M1 value (exponent bias 1) with magnitudes under two levels of scaling. A tensor is split into groups of 16 contiguous elements along the reduction dimension; each group has an E4M3 group scale , and the tensor has an FP32 tensor scale :
where rounds to the nearest value of format (ties to even). The group scale maps each group’s absolute maximum onto 6, the largest E2M1 value, and the tensor scale maps the largest group scale onto 448, the largest E4M3 value. With one 8-bit scale per 16 elements, NVFP4 costs bits per element.
FP6g
FP6g, the group-scaled 6-bit counterpart of NVFP4 used as a baseline throughout the paper, keeps the group size and scaling hierarchy of NVFP4 and replaces E2M1 by E2M3 (exponent bias 1). The two additional mantissa bits give 32 magnitudes, in steps of on , on , and on , and lower the worst-case relative rounding error of normal values from to . The scales follow Eqs. (2) and (3) with the E2M3 maximum 7.5 in place of 6, so FP6g costs bits per element. Table 6 lists the positive magnitudes of both element formats.
| E\M | E2M1 (NVFP4) | E2M3 (FP6g) | ||||||||
| 0 | 1 | 000 | 001 | 010 | 011 | 100 | 101 | 110 | 111 | |
| 00 | 0 | 0.5 | 0 | 0.125 | 0.25 | 0.375 | 0.5 | 0.625 | 0.75 | 0.875 |
| 01 | 1 | 1.5 | 1 | 1.125 | 1.25 | 1.375 | 1.5 | 1.625 | 1.75 | 1.875 |
| 10 | 2 | 3 | 2 | 2.25 | 2.5 | 2.75 | 3 | 3.25 | 3.5 | 3.75 |
| 11 | 4 | 6 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 |
Table 6. Positive magnitudes of E2M1 (NVFP4) and E2M3 (FP6g).
Two-Bit Formats and the AutoFP2 Digit
With one bit for the sign, a 2-bit floating-point format spends its other bit on either the exponent or the mantissa. E1M0 spends it on the exponent: giving the magnitudes and the codebook , which cannot represent zero, so small values are forced to . E0M1 spends it on the mantissa: both codes are subnormal, giving the ternary codebook with a redundant , which represents zero but only one nonzero magnitude. AutoFP2’s quinary digit keeps both by remapping the two exponent bits of the NVFP4 base (§ 3.2).
Sharing NVFP4 Scales with AutoFP2
A tensor stored in NVFP4 that executes at AutoFP2 reuses the NVFP4 scales and . These normalize each group to , whereas AutoFP2 digits reach only , so decoding with alone would shrink each group by up to . The scale is therefore multiplied by a scale multiple :
Setting matches the two ranges exactly, with reconstruction levels in NVFP4 units. We instead use , with levels , for two reasons. First, a factor of two is hardware-friendly. Second, the AutoFP2 digit 1 absorbs the four E2M1 magnitudes , which hold most group-normalized values. Reconstructing this digit at 2 rather than 3 places it in the middle of that range and halves its worst-case error from 2 to 1 NVFP4 units.
Worked Example of Mantissa Refinement
Table 7 applies mantissa refinement to one 16-element NVFP4 group with scale .
| Subgroup | NVFP4 base | Code | Err. | AutoFP6 refinement | Code | Err. | ||
| SG0 | 1.00 | 1.0 | 0010 | 0 | – | – | – | – |
| SG0 | −0.50 | −0.5 | 1001 | 0 | – | – | – | – |
| SG0 | 3.00 | 3.0 | 0101 | 0 | – | – | – | – |
| SG0 | 6.00 | 6.0 | 0111 | 0 | – | – | – | – |
| SG1 | 1.13 | 1.0 | 0010 | 0.13 | 01 | 1.125 | 0.005 | |
| SG1 | 2.24 | 2.0 | 0100 | 0.24 | 01 | 2.125 | 0.115 | |
| SG1 | −0.87 | −1.0 | 1010 | 0.13 | 01 | −0.875 | 0.005 | |
| SG1 | 1.37 | 1.5 | 0011 | 0.13 | 11 | 1.375 | 0.005 | |
| SG2 | 2.76 | 3.0 | 0101 | 0.24 | 10 | 2.75 | 0.010 | |
| SG2 | −0.62 | −0.5 | 1001 | 0.12 | 11 | −0.625 | 0.005 | |
| SG2 | 4.24 | 4.0 | 0110 | 0.24 | 01 | 4.125 | 0.115 | |
| SG2 | −2.02 | −2.0 | 1100 | 0.02 | 0 | 00 | −2.0 | 0.020 |
| SG3 | 0.52 | 0.5 | 0001 | 0.02 | – | – | – | – |
| SG3 | −1.98 | −2.0 | 1100 | 0.02 | – | – | – | – |
| SG3 | 0.00 | 0.0 | 0000 | 0 | – | – | – | – |
| SG3 | 2.95 | 3.0 | 0101 | 0.05 | – | – | – | – |
Table 7. Mantissa refinement of one NVFP4 group (). Values are in NVFP4 units; codes are sign — exponent — mantissa. Green rows are the refined subgroups (mask 0110).
Additional Motivating Analyses
Layer-Wise Precision Across Decode Steps
Figure 16 shows the oracle per-layer precision at individual decode steps.

Figure 16. Layer-wise precision across decode steps. Per-layer precision at ten decode steps of one DS-Qwen-14B LiveCodeBench response (top) and one Qwen3.8-27B -bench response (bottom).
Layer-Wise Precision Across Prefill and Refills
Figure 17 repeats the prefill and refill analysis of Figure 2(b) for an additional model and workload, Gemma-4-26B on
Figure 17. Oracle layer-wise precision for the initial prefill and three refills of Gemma-4-26B on three LongMemEval samples (S1–S3).
LongMemEval, showing the oracle layer-wise precision of the initial prefill and three refills for three samples.
Precision Demand by Generation Entropy
Figure 18 splits the decode steps of the oracle of §2 by at . High-entropy steps consistently require higher precision compared to low-entropy steps.

Figure 18. Oracle precision demand in high- (H) and low-entropy (L) decode steps.
Layer-Wise Behavior of MP and MD Across Models and Datasets
Figure 30 extends Figure 4 to Gemma-4-26B, Qwen3.8-27B, Qwen3-8B, and DS-LLaMA-70B on LiveCodeBench, GSM8K, and AIME. Figure 19 shows of the initial prefill and three refills of one -bench task for Gemma-4-26B and Qwen3.8-27B.
Figure 19. Prefill for the initial prefill and three refills of one -bench task for (a) Gemma-4-26B and (b) Qwen3.8-27B.
Figure 30. Layer-wise and across models and datasets. For each model (rows) and dataset (column pairs), prefill of one representative sample and decode of the same sample at the highest- and lowest- step.
Subgroup Refinement on Additional Models
Figure 20 repeats the experiment of Figure 3 on Qwen3.8-27B and Gemma-4-26B with GSM8K, for three layers each.
Additional AutoEMatic Details
Signals Used for Runtime Precision Adaptation
Figure 21 illustrates the two model-internal signals of § 3.1.
Entropy Signal Across Decode Steps
Figure 22 traces over a 16,383-step AIME generation.
Memory Layout
Figure 23 shows the memory layout of AutoFP, with the NVFP4 base stored separately from the refinement metadata.
Extended- GEMM
Figure 24 illustrates the identity behind the extended- GEMM of § 4.1 on a numeric example: two GEMMs with the same output shape sum to one GEMM over operands concatenated along .
Construction of the Extended- Operands
Figure 25 shows how and are constructed.
Figure 20. Refining a subset of elements achieves NMSE comparable to 6-bit. Y-axis is NMSE, such that NVFP4 is 1 and FP6g is 0, vs. fraction of subgroups refined to 6-bit, for Qwen3.8-27B and Gemma-4-26B on GSM8K.

Figure 21. Signals used by AutoEMatic for runtime precision adaptation. ❶ Local layer-update activity . ❷ Global token-generation uncertainty .
Numerical Example of the Extended- GEMM
Figure 26 shows a worked example of the GEMM :
NVFP4-to-AutoFP2 Decomposition
The weight encoder decomposes each NVFP4 weight into two AutoFP2 digits, with (§ 6). With , to span the maximum NVFP4 magnitude 6 requires . satisfies this such that provides coarse-grained steps to different NVFP4 binades and provides integer corrections , such that their linear combination spans every NVFP4 E2M1 value. Multiplying numerator and denominator by 2 gives integer coefficients such that and the resulting reconstruction can be realized using only shifts and adds. Table 8 lists the digit pair of every positive E2M1 magnitude; negative weights negate both digits.
For each digit , the encoder produces three bits from the E2M1 bits of the weight: a valid bit , which is 1 if and 0 if ; a sign bit ; and an exponent bit with , so that is the E1M0 code of a nonzero digit:
Figure 22. Global signal over a 16K-token AIME generation of Qwen3.8-27B. The dashed line marks .
Figure 24. A sum of GEMMs as one GEMM over an extended reduction dimension.

Figure 23. AutoEMatic memory layout.
Figure 25. Construction of and for one 16-element group and one request. The weight mask places the weight offsets in and copies the base into ; the activation mask copies the base into and places the activation offsets in ; all other positions are zero.
where , , , and denote OR, AND, NOT, and XOR. The valid bits then select the AutoFP2 encoding of § 3.2: if both digits are nonzero (), the escape bit is and the fields and hold the two E1M0 codes; otherwise , records which digits are zero, and holds the E1M0 code of the nonzero digit, if any. The valid bits are consumed by this packing and are not stored.
Fixed-Point Arithmetic of the vMAC
All computation in the vMAC (§ 6.3) is fixed-point. Each input activation enters the lanes as the integer , since doubling an E2M1 value always yields an integer ((Hayashi et al., 2026)). AutoFP2 digits are already integers, whereas in NVFP4 mode the packed digit pair represents twice the NVFP4 weight, (§ 6.2). In the multiply stage each sub-lane forms , , and within each block the even and odd sub-lanes feed separate adder trees, and for lanes 0–15, and , for lanes 16–31.
In NVFP4 mode, and are partial sums over the two digits of the same weights , so by distributivity the mode-dependent reduction unit evaluates , which is four times the dot product of block 0 in fixed-point; the division by two in of § 6.3 is deferred to the scaling stage. Similarly for . In AutoFP2 mode, the even and odd sub-lanes of a block hold weights from two different rows, so the partial sums are summed across blocks as and , two independent 32-element dot products; the factor of 2 keeps them at four times the dot product as well.
Each refinement lane forms in fixed-point. Since , the floating-point equivalent is , so is sixteen times the product. The lane contributions are accumulated as , and the base sums are shifted left by two bits to align with them, giving at sixteen times the dot product. For example, with , , and : , , and . In the scaling stage, each is multiplied by the scale factor of its block and divided by 16, which recovers the floating-point partial sum.
Accelerator Floorplan
Figure 27 shows the AutoEMatic accelerator floorplan.

Figure 27. Post-layout floorplan of AutoEMatic.

Figure 26. (1) The exact product of the refined operands and . (2) The base GEMM on and plus a refinement GEMM on the constructed and at scale . (3) A single GEMM over the concatenated operands.
| 0 | 0.5 | 1 | 1.5 | 2 | 3 | 4 | 6 | |
| 0 | 1 | 0 | 1 | 0 | 2 | 2 | 2 | |
| 0 | 1 | 2 | 1 |
Table 8. NVFP4-to-AutoFP2 decomposition.
Refinement-Aware GPTQ Calibration
Algorithm 1 lists AutoEMatic’s offline weight calibration, which extends GPTQ (Frantar et al., 2023) to an NVFP4 base with :4 structured sparse refinement. We follow the NVFP4 notation of §B.1: is the FP32 tensor scale, and is the E4M3 group scale of row in group , whose 16 columns we denote . The calibration stores E2M1 codes and, on refined subgroups, 2-bit signed offsets , decoded with the fixed step of §3.2 as ; the dequantized weight is without refinement (NVFP4) and with it (AutoFP6). A refinement is set to 0 if it would move the element farther from the original weight.
Each group has four 4-element subgroups, and denotes the masks that refine of them; the weight mask of §3.2 holds one such mask per group. For a mask , is the dequantized group that uses the refined values on the selected subgroups and the NVFP4 values elsewhere. Its diagonal-Hessian-weighted error on group is
The algorithm differs from GPTQ in three ways. (1) Budget allocation. Before the GPTQ pass, a greedy allocation (lines 5–12) distributes refined subgroups across groups, ranked by their reduction in on the original weights. Each group’s mask, which refines of its four subgroups, is shared across all output rows; the remaining subgroups are left for activation refinement at runtime. (2) Group-wise quantization. The 16-element group, rather than a single column, is the unit of quantization. Its group scales are computed from the error-compensated weights when the group is reached, while the tensor scale stays fixed. (3) Group-wise error propagation. The quantization error of a whole group is propagated to the later columns at once (lines 22–23), rather than one column at a time as in GPTQ.
GPU Implementation Details
AutoEMatic is implemented in vLLM 0.15 (Kwon et al., 2023) as a quantization plugin, with a custom attention backend and CUDA kernels for Blackwell GPUs.
Accelerator Simulation Details
AutoEMatic performance is measured using a scheduling model calibrated against cycle-accurate Synopsys VCS simulation. Performance data is gathered from these measurements, retaining on-chip data-movement and backpressure costs. Off-chip DRAM transfer time is modeled as transferred bytes divided by an assumed sustained bandwidth of 1 TB/s and added to the execution time.
Additional Accuracy Results
Table 9 reports the accuracy of AutoEMatic and the baselines on DS-Llama-70B, DS-Qwen-32B, and Llama-3.1-8B-Instruct on the reasoning benchmarks.
| Model | Method (budget) | LB v2 | AIME | LCB | GSM8K |
| DS-LLaMA-70B | BF16 | 41.0 | 66.7 | 68.2 | 94.5 |
| DS-LLaMA-70B | NVFP4 | 29.8 | 53.3 | 51.7 | 90.6 |
| DS-LLaMA-70B | FP6g | 41.0 | 66.7 | 66.2 | 94.5 |
| DS-LLaMA-70B | AutoQuantize (5.3) | 38.7 | 66.7 | 55.7 | 94.5 |
| DS-LLaMA-70B | FlexQuant (5.3) | 27.8 | 50.0 | 48.6 | 86.5 |
| DS-LLaMA-70B | DP-LLM (weight-only) (5.3) | 21.0 (5.1) | 43.3 (5.3) | 43.3 (5.3) | 81.6 (5.1) |
| DS-LLaMA-70B | DP-LLM† (5.3) | 19.0 (5.1) | 36.7 (5.3) | 39.5 (5.3) | 77.6 (5.1) |
| DS-LLaMA-70B | AutoEMatic | 41.0 (5.4) | 66.7 (5.1) | 65.6 (4.5) | 94.5 (4.9) |
| DS-Qwen-32B | BF16 | 49.8 | 73.3 | 54.2 | 92.5 |
| DS-Qwen-32B | NVFP4 | 43.5 | 60.0 | 50.5 | 87.6 |
| DS-Qwen-32B | FP6g | 49.8 | 70.0 | 52.6 | 90.5 |
| DS-Qwen-32B | AutoQuantize (4.8) | 49.8 | 66.7 | 51.2 | 89.3 |
| DS-Qwen-32B | FlexQuant (4.8) | 40.5 | 56.7 | 46.9 | 81.5 |
| DS-Qwen-32B | DP-LLM (weight-only) (4.8) | 36.6 (4.8) | 50.0 (4.8) | 45.0 (4.8) | 76.5 (4.5) |
| DS-Qwen-32B | DP-LLM† (4.8) | 30.6 (4.8) | 36.7 (4.8) | 40.7 (4.8) | 71.8 (4.5) |
| DS-Qwen-32B | AutoEMatic | 49.8 (4.6) | 70.0 (4.8) | 52.4 (4.6) | 90.5 (4.4) |
| Llama-3.1-8B-Instruct | BF16 | 30.5 | – | – | 82.8 |
| Llama-3.1-8B-Instruct | NVFP4 | 27.5 | – | – | 75.6 |
| Llama-3.1-8B-Instruct | FP6g | 30.5 | – | – | 80.5 |
| Llama-3.1-8B-Instruct | AutoQuantize (4.5) | 30.5 | – | – | 78.9 |
| Llama-3.1-8B-Instruct | FlexQuant (4.5) | 25.9 | – | – | 71.6 |
| Llama-3.1-8B-Instruct | DP-LLM (weight-only) (4.5) | 24.6 (4.4) | – | – | 63.8 (4.3) |
| Llama-3.1-8B-Instruct | DP-LLM† (4.5) | 22.8 (4.4) | – | – | 59.6 (4.3) |
| Llama-3.1-8B-Instruct | AutoEMatic | 30.5 (4.4) | – | – | 80.5 (4.3) |
Table 9. Accuracy of AutoEMatic and baselines on DS-Llama-70B, DS-Qwen-32B, and Llama-3.1-8B-Instruct. Cells report accuracy (%); execution precision appears in parentheses. LB v2: LongBench v2, LCB: LiveCodeBench. Llama-3.1-8B-Instruct is not evaluated on AIME and LCB, where its BF16 accuracy is too low for a meaningful comparison.
Additional Ablations
AutoFP2 scale multiple . Figure 28(a) sweeps the scale multiple of §B.4: gives an NMSE of 0.09, within
Figure 28. Additional ablations on DS-LLaMA-70B (LCB). (a) Scale multiple of AutoFP2: mean layer-output NMSE over all 80 layers at W2A4. (b) Subgroup size with 20% of all weight subgroups refined. (c) Weight refinement budget , with storage precision. (d) Activation NMSE threshold, with the mean number of activation subgroups per group exceeding it. (e) sweep. (f) Entropy threshold . (g) Trend lengths , , and . (h) Overhead of computing and as a share of decode TPOT. Stars mark the chosen settings.
1.15 of nearest rounding, whereas clips the large magnitudes and places the merged E2M1 codes at 3, raising the NMSE to 0.38 and 0.15.
Impact of second-order refinement. Figure 31 measures the second-order refinement product that the extended- GEMM omits (§ 4.1). Its energy is of the per-layer output, and consequently the AutoFP6 output NMSE is 0.0041 with and without this term.
Figure 31. Second-order refinement term. Per-layer output NMSE of AutoFP6 with and without the second-order product (top) and the energy of that product relative to the layer output (bottom) on DS-LLaMA-70B.
Extension of . The reduction extension of a refined GEMM depends on the # of subgroups refined. On Qwen3.8-27B -bench (Figure 32), the extension grows from at batch 1 to at batch 32 with average refined requests per layer rising from 0.2 to 7.2.
Figure 32. Extension of the reduction dimension of refined GEMMs on Qwen3.8-27B -bench vs. decode batch size, with the mean number of refined tokens per layer.
Extension to 8 bits. Widening the signed offsets from 2 to 4 bits extends AutoEMatic to AutoFP8 (Table 10). With every layer statically refined to AutoFP8 and refinement-aware GPTQ, DS-LLaMA-70B attains 67.9% on LCB at 5.3 stored bits per weight. The runtime policy with attains the same 67.9%, within 0.5% of BF16, at 5.3 bits while executing 30% of layers at 2-bit, 35% at 4-bit, 15% at 6-bit, and 20% at 8-bit.
| Component | Bits | Acc. | 2 / 4 / 6 / 8 (%) |
| BF16 | 16.8 | 68.4 | |
| AutoEMatic format + runtime policy, | 4.9 | 65.6 | 15 / 45 / 40 / 0 |
| AutoFP8 + Refinement-aware GPTQ | 5.3 | 67.9 | 0 / 0 / 0 / 100 |
| AutoEMatic format + runtime policy, | 5.3 | 67.9 | 30 / 35 / 15 / 20 |
Table 10. Extension to 8 bits with DS-LLaMA-70B on LCB; rows 1 and 7 of Table 4 are repeated for reference.
Subgroup size. Figure 28(b) varies with 20% of all weight subgroups refined. Finer subgroups raise accuracy only from 65.6% at to 66.1% at , however, finer subgroups require more metadata per group and are not as performant. Coarser subgroups lose 2.4–3.1%, as a refined subgroup then spends signed offsets on elements that do not need them.
Weight refinement budget. Figure 28(c) sweeps , the fraction of all weight subgroups refined, over {10, 20, 40, 80}%, with storage of bits per weight. Accuracy peaks at (65.6% at 4.9 bits) and falls on both sides: at 10% too few weight subgroups are refined, while at 40%
and 80% the weight mask occupies subgroups that the activations would otherwise refine at runtime (), so additional weight bits displace activation refinement and accuracy drops to 60.6%.
Activation refinement threshold. Figure 28(d) sweeps the NMSE threshold above which an activation subgroup is refined, over {0.001, 0.003, 0.01, 0.03}, together with the mean number of subgroups per group that exceed it. Lowering the threshold below 0.003 refines 2.7 rather than 1.6 subgroups per group for 0.3% more accuracy, while raising it to 0.01 and 0.03 leaves most groups unrefined.
. Figure 28(e) sweeps over {0.1, 0.3, 0.5, 0.7}.
Algorithm 1 Refinement-aware GPTQ calibration of , given calibration inputs , group size , refinement budget , and damping . Returns the E2M1 codes , the refinement terms (stored as 2-bit signed offsets) on refined subgroups, the scales and , and the masks .
\hfill fixed
for to do
for
end for
for all
for to do
end for
for to do
for all
for all ,
end for
Entropy threshold. Figure 28(f) sweeps , which separates the high- and low-entropy decode regimes, over . Accuracy is highest at (65.6%) and declines monotonically to 62.6% at , as more steps with several plausible continuations are treated as low-entropy and allowed to lower precision aggressively.
Trend lengths. Figure 28(g) sweeps , , and over .
Signal overhead. Figure 28(h) reports the cost of computing the signals as a share of decode TPOT. is one top- selection and entropy over the logits per step, below 1.3% at every batch size; is a per-layer reduction of the residual update, 2.4 s per layer, whose share falls from 2.0% at batch 1 to 1.1% at batch 128 as it amortizes over rows. Together the signals add 2.3–2.8% of TPOT.
Online serving. Figure 29 serves DS-LLaMA-70B in vLLM under continuous batching with Poisson request arrivals at increasing rates, using LongBench v2 prompts of about 8K tokens with 1–2K output tokens.

Figure 29. Online serving under continuous batching. Median TPOT vs. achieved request rate for DS-LLaMA-70B on 4× GB300 (TP=4) with Poisson arrivals of LongBench v2 requests.
Accelerator latency and energy at iso-accuracy with NVFP4. The main comparison (§ 6) holds accuracy at AutoEMatic’s level. Figure 33 instead holds it at NVFP4’s. This enables the NVFP4 datapath to run all layers at its native precision. To achieve iso-accuracy with NVFP4, we tune AutoEMatic assignments on MiniMax-M2.7 such that AutoEMatic assigns 30% of layers to AutoFP2, 60% to NVFP4, and 10% to AutoFP6 and matches the accuracy of the NVFP4. As shown in Figure 33, AutoEMatic still reduces end-to-end latency by 12–14%, with prefill 1.7–2.0× faster at batch sizes 1 and 8, at 13% higher energy, due to the additional components in the mixed-precision AutoEMatic datapath.
Figure 33. Accelerator latency and energy at iso-accuracy with NVFP4.
DP-LLM and FlexQuant performance. Figure 34 reports the prefill latency of DP-LLM and FlexQuant on DS-LLaMA-70B. At batch size 1 they decode at 5.9 and 7.3 tokens/s, respectively, against 13 tokens/s for NVFP4 and 120 for AutoEMatic in Figure 8.
Figure 34. TTFT of the runtime-adaptive baselines DP-LLM and FlexQuant on DS-LLaMA-70B vs. input sequence length.
Instantaneous vs. trend-based changes. Table 11 compares making precision decisions instantaneously versus AutoEMatic’s trend-based policy. The trend-based policy reaches both lower average execution precision (4.6 vs. 4.7 bits) and higher accuracy (95.5% vs. 92.5%). Instantaneous decisions react to layer-to-layer noise in the metric, the trend rule acts only on sustained changes, which filters this noise.
| Policy | Bits | Acc. | 2 / 4 / 6 (%) |
| Instantaneous () | 4.7 | 92.5 | 0 / 66 / 34 |
| AutoEMatic () | 4.6 | 95.5 | 0 / 69 / 31 |
Table 11. Instantaneous vs. trend-based precision changes with Qwen3-8B on GSM8K.
Limitations
As it stands, while AutoFP supports the precision set , it cannot take advantage of AutoFP2’s memory bandwidth benefits. Using AutoFP2 as the stored base would require reconstructing NVFP4 for most layer executions, introducing overhead that can outweigh the bandwidth savings during decode. Furthermore, AutoFP2’s coarse representation requires larger corrections to recover NVFP4 values, which the current signed-offset encoding does not directly support. Future work will work on remedying this.
AI Usage Statement
Portions of this paper were refined with the assistance of a large language model (LLM), specifically OpenAI GPT 6-Sol, used to polish writing and help reduce verbosity to meet page limit. The LLM also assisted in writing the python code for making several figures for the data the authors independently collected. All technical content, methods, and results were conceived entirely by the authors, without influence from any AI tool. The AI was however used to run parameter sweeps, conduct design space exploration, suggest optimizations (which the authors thoroughly verified) and also launch and monitor accuracy and performance experiments.
References
- [2]Alvarez, E. et al. Introducing NVFP4 for efficient and accurate low-precision inference. NVIDIA Technical Blog, June 2025. Accessed September 19, 2026.
- [4]Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pp. 87–100, 2024.DOI
- [5]Liu, F., Wang, Z., Xia, J., Zhao, J., Zhao, S., Li, J., Liu, J., Jiang, L., and Guan, H. Flexquant: A flexible and efficient dynamic precision switching framework for llm quantization. arXiv preprint arXiv:2506.12024, 2025.arxiv.org/abs/2506.12024
- [6]Liu, T., Li, S., Yang, J., Zhao, T., Zhou, F., Song, X., Dai, G., Yan, S., Yang, H., and Wang, Y. PM-KVQ: Progressive mixed-precision KV cache quantization for long-CoT LLMs. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2505.18610.
- [7]Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In International Conference on Machine Learning, 2024.DOI
- [9]Luo, J., Jiang, X., Wen, T.-H., Qi, R., Chu, M., Wu, H., Kielian, G., Laurie, B., Zhang, Q., Cheng, Q., Sylvester, D., and Saligane, M. Heterogeneity-aware microscaling for efficient low-bit LLM inference, 2026a. URL https://arxiv.org/abs/2608.03867.
- [10]Luo, Y., Dong, B., Cheng, W., and Shen, H. Recurrent residual quantization: A progressive multi-precision representation for LLMs, 2026b. URL https://arxiv.org/abs/2608.04048.
- [11]MAA. Aime 2024 problems. https://artofproblemsolving.com/wiki/index.php/2024_AIME_I_Problems, 2024. Accessed: 2025-08-30.
- [12]Nair, P., Datta, P., Dean, J., Jain, P., and Kusupati, A. Matryoshka quantization. arXiv preprint arXiv:2502.06786, 2025.arxiv.org/abs/2502.06786
- [13]NVIDIA ModelOpt. NVIDIA Model Optimizer–AutoQuantize. GitHub repository, 2025. URL https://github.com/NVIDIA/Model-Optimizer.
- [14]Open Compute Project. OCP microscaling formats (MX) specification. Technical Report Version 1.0, Open Compute Project Foundation, 2023.
- [15]OpenAI. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/, August 2024. Accessed 2026-10-12.
- [16]Park, G., Bae, J., Kwon, B., Kim, B., Kwon, S. J., and Lee, D. Anybcq: Hardware efficient flexible binary-coded quantization for multi-precision llms. In International Conference on Learning Representations, volume 2026, pp. 125106–125122, 2026.arxiv.org/abs/2510.10467
- [17]Park, Y., Hyun, J., Cho, S., Sim, B., and Lee, J. W. Any-precision llm: Low-cost deployment of multiple, different-sized llms. arXiv preprint arXiv:2402.10517, 2024.arxiv.org/abs/2402.10517
- [18]Ramachandran, A., Kundu, S., and Krishna, T. Clamp-vit: Contrastive data-free learning for adaptive post-training quantization of vits. In European Conference on Computer Vision, pp. 307–325. Springer, 2024.arxiv.org/abs/2407.05266
- [19]Ramachandran, A., Kundu, S., and Krishna, T. Microscopiq: Accelerating foundational models through outlier-aware microscaling quantization. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pp. 1193–1209, 2025a.arxiv.org/abs/2411.05282
- [20]Ramachandran, A., Kundu, S., Raha, A., Kundu, S., Mathaikutty, D. K., and Krishna, T. Accelerating llm inference with flexible n: M sparsity via a fully digital compute-in-memory accelerator. In 2025 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1–7. IEEE, 2025b.
- [21]Ramachandran, A., Naseem, M., Sakr, C., Venkatesan, R., Khailany, B., and Krishna, T. Thinkv: Thought-adaptive kv cache compression for efficient reasoning models. In International Conference on Learning Representations, volume 2026, pp. 110072–110106, 2026.arxiv.org/abs/2510.01290
- [22]Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.arxiv.org/abs/2310.10537
- [23]Shen, Y. and Zou, A. River-llm: Large language model seamless exit based on kv share. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 37633–37645, 2026.
- [24]Venkatesan, R., Shao, Y. S., Wang, M., Clemons, J., Dai, S., Fojtik, M., Keller, B., Klinefelter, A., Pinckney, N., Raina, P., et al. Magnet: A modular accelerator generator for neural networks. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–8. IEEE, 2019.DOI
- [25]Wang, D., Kim, J., Han, S., Gudovskiy, D., Nakata, Y., Okuno, T., Peong, K., Jeon, K. E., Ko, J. H., Chen, Y., et al. Mobiquant: Mixture-of-bits quantization for token-adaptive any-precision llm. arXiv preprint arXiv:2602.20191, 2026.arxiv.org/abs/2602.20191
- [27]Zhao, Z., Wang, M., Yu, B., Liu, B., Han, X., and Zhao, X. DynamicPTQ: Mitigating activation quantization collapse via residual-stream dynamics, 2026. URL https://arxiv.org/abs/2606.12487.
- [28]Zou, J., Chen, Y., Wu, R., and Chen, X. MixFP4: Enhancing NVFP4 with adaptive FP4/INT4 block representations. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, 2026. URL https://arxiv.org/abs/2605.31035.






















