betaFeedback

WorksComputer Science

Integer-Exact Microscaling: Running MXFP4 Language Models on INT8 Matrix Units

Abstract

On hardware whose fastest matrix datapath is INT8×\timesINT8, such as the Neural Accelerators of Apple's M5 GPUs, 4-bit weights must become INT8 operands. Existing MXFP4 kernels already map FP4 (E2M1) values to integers by doubling them, but apply every 32-element block scale inside the reduction. We show when no per-block work is needed: a row of MXFP4 weights is exactly one INT8 vector times one power-of-two scale if its block exponents span at most three, so a linear layer becomes a single INT8 GEMM with one rescale per output. Enforcing the condition adds at most 1/16 of a row's largest rounding error to the blocks it changes, and real weights already satisfy it: 97.9-99.97% of rows across seven models, including 99.93% of the 95M rows of a 125B-parameter mixture-of-experts (MoE) model, Qwen3.8-Flash-Next. The resulting spread-constrained MXFP4 files stay standard MXFP4. With integer instead of power-of-two block multipliers, an integer-exact format (16-wide blocks, multipliers 1..10, 4.26 bits) runs as one exact INT8 GEMM and is more accurate than NVFP4 (4.5 bits) with its standard scale rule on five small models, within 0.06 perplexity of NVFP4 with scale search. On Qwen3.8-Flash-Next's routed experts with INT8 activations it is as close to the FP8 model as NVFP4 with scale search (KL divergence 0.0557 vs. 0.0553) and closer than MLX's MXFP4 and affine INT4; with every matrix quantized it solves 94.8% of GSM8K vs. 91.0% for MLX's MXFP4. Quantizing only the 74k routed expert matrices (96% of the parameters) to either of our formats with INT8 activations costs at most 0.2% perplexity. A complete W4A8 system on the M5 produces outputs bit-identical to the quantized reference; prefill is 1.34-1.74x faster than MLX on a dense model, 1.10-1.57x on two MoE models up to 35B parameters and 1.22-1.29x on Qwen3.8-Flash-Next, where shared non-GEMM work bounds the gain.

Introduction

Running large language models (LLMs) on personal hardware is attractive for privacy and cost, and it is made possible by 4-bit weight quantization: memory is limited, and token generation is bound by memory bandwidth, so bytes per weight dominate. Processing the prompt (prefill) is different. It is compute-bound, and its speed is set by how fast the hardware multiplies matrices. On hardware without 4-bit matrix units, deployed 4-bit formats are dequantized to 16-bit floating point inside the matrix multiplication (W4A16) (Hannun et al., [14]; Gerganov et al., [12]), so prefill runs at the 16-bit rate.

Apple’s M5 GPUs add a matrix engine, the Neural Accelerator, to every GPU core (Waschkowski et al., [44]). INT8×INT8\mathrm{INT8}\times\mathrm{INT8} runs on it at about twice the FP16 rate, but only when both operands are integers; every mixed fixed×integer combination is slower than plain FP16 (§2; consistent with measurements reported by MLX and community developers (MLX issue 4627, 2026; oMLX contributors, 2026; scratchy contributors, 2026)). W4A16 therefore cannot use the fastest datapath. A W4A8 design can, provided 4-bit weights become INT8 operands without changing the model. The same holds on other matrix units whose fast path is INT8 but that lack FP4, such as the INT8 tensor cores of NVIDIA GPUs before Blackwell, AMD’s and Intel’s integer matrix units and Arm’s SME; the M5 is our case study, and only the kernel design of § 4 is specific to it.

Turning 4-bit weights into INT8 operands is not new. W4A8 schemes such as QServe [22] and DGQ (Zhang et al., 2023) define formats with integer group scales so that the conversion stays in integer arithmetic; for MXFP4 [35, 41], llama.cpp and other kernels use the fact that doubled E2M1 values are integers up to 12 and multiply them against INT8 activations, applying each block’s E8M0 scale to that block’s partial sum (ggml-org, 2025; Georganas et al., 2025; vLLM contributors, 2026); and concurrent open-source work runs INT8-activation prefill on the M5 for affine INT4 weights with per-group corrections (oMLX contributors, 2026; scratchy contributors, 2026). Our observation is that for MXFP4 the per-block scales can usually be removed from the reduction altogether: power-of-two scales are the special case of integer scales that needs only shifts, so they fold into the INT8 operands, and the only obstacle is range, which a condition on the row’s block exponents settles exactly (Figure 1). A folded layer is one INT8 GEMM over the full reduction with a single rescale per output, which matrix units execute at full rate. Folding is not faster than well-engineered per-group corrections (§ 7); what it adds is exactness, standard MXFP4 files, and the FP4 codebook at the INT8 rate.

Figure 1. Left: deployed 4-bit formats are dequantized to FP16 and multiplied at the FP16 rate (W4A16). We rewrite MXFP4 weights exactly as INT8 operands and quantize activations to INT8, so the GEMM runs on the INT8 datapath, about 2× faster on Apple M5. The weights remain a valid MXFP4 file. Right: the rewrite is exact when a row’s block exponents lie within a window of four values below the largest; blocks below it (red) are coarsened, which adds at most 1/16 of the row’s largest rounding error (Proposition 2).

Contributions.

  • Per-row scale folding for standard MXFP4 (§ 3): a row of MXFP4 weights equals an INT8 vector times the power-of-two scale 2E−12^{E-1}, with EE its smallest block exponent, if and only if its block exponents span at most three (for rows whose top block reaches the E2M1 value 4 or 6, as every block does under the OCP rule). We give a quantizer that enforces this, prove that the elements it changes err by at most 1/16 of the largest grid error at the row’s top exponent, and explain with a null model why real weights almost always satisfy the condition already, extending a measurement of block-exponent ranges made for a custom accelerator [25] to an exactness condition.

  • An integer-exact FP4 format family (§ 3.5): the INT8-closed instance of two-level integer scaling [8, 20, 22] (Zhang et al. 2023) with the FP4 codebook. At 4.26 bits, 16-wide blocks with integer multipliers beat NVFP4’s standard scale rule (4.5 bits) on five small models, come within 0.06 perplexity of NVFP4 with scale search, are as close to a 125B MoE’s original model as NVFP4 with scale search when its experts are quantized (KL divergence), beat the 4-bit formats deployed on Macs, solve the most GSM8K problems on that model (94.8%), and still run as one exact INT8 GEMM.

  • Evidence across seven models up to a 125B MoE (§ 5, § 7): 97.9–99.97% of rows satisfy the condition, perplexity is unchanged by enforcing it, GPTQ rounding composes with it, per-token INT8 activations cost at most 0.3% perplexity and no measurable GSM8K accuracy on Qwen3.8-Flash-Next, and the deployed system matches MXFP4’s zero-shot accuracy at equal coverage.

  • A complete, exact W4A8 system on MLX § 4, § 6: fused dense prefill, grouped MoE prefill and decode on the standard MXFP4 bytes, faster than MLX in prefill on a dense model and three MoE models up to 125B parameters (and than llama.cpp on the first three), with batch-1 decode at 0.90–1.01×\times MLX’s speed on the smaller models and, on the 125B model, at MLX’s speed with MLX’s kernels reading the same SC-MXFP4 file (0.65×\times on our kernels for the 16-block format), and kernel outputs bit-identical to the quantized reference.

Background and motivation

MXFP4. An MXFP4 tensor stores each weight as w=2ebvw = 2^{e_b}v with v∈E2M1={0,±12,±1,±32,±2,±3,±4,±6}v \in\mathrm{E2M1} = \{0, \pm\frac{1}{2}, \pm1, \pm\frac{3}{2}, \pm2, \pm3, \pm4, \pm6\} and one 8-bit exponent (E8M0) ebe_b per block bb of 32 consecutive values along the reduction dimension, 4.25 bits per weight [35]. NVFP4 uses 16-value blocks with FP8 (E4M3) scales and is usually more accurate, but its scales are not powers of two.

The M5 Neural Accelerators. The M5 GPU’s matrix units are programmed through Metal 4 tensors and the matmul2d operation [2]. The shipped headers (macOS 26.6) accept half, bfloat, float, int8 and uint8 operands and a packed 4-bit integer type, but no FP8 or FP4 type (macOS 27 adds them; we have not measured them), and Apple does not publish per-type throughput. We measured it with minimal matmul2d kernels whose inputs are small integers, so that every operand type computes the same product (verified bit-exactly), timing all types interleaved after a clock warm-up (Appendix § 13). Figure 2 shows the result: INT8×\timesINT8 reaches 109–127 TOPS, 1.79–1.98×\times the FP16 rate, while FP16×\timesINT8 and FP16×\timesINT4 run at 0.68–0.96×\times FP16, because the hardware converts the integer operand. The native INT8×\timesINT4 path reaches 1.70–1.82×\times FP16, but its 4-bit operand holds −8…7-8 \ldots7 and cannot represent MXFP4 values. Weight-only quantization therefore gains nothing in compute-bound prefill on this hardware; the integer rate requires integer activations and integer weights. Similar INT8/FP16 ratios have since been reported by MLX and community developers on other M5 and M6 parts (MLX issue 4627, 2026; oMLX contributors, 2026; scratchy contributors, 2026).

Figure 2. matmul2d throughput on the M5 Max Neural Accelerators by operand type (left ×\times right operand) for M×N×KM \times N \times K shapes. INT8×\timesINT8 reaches 1.8–2.0×\times FP16; every mixed float×\timesinteger combination is slower than FP16.

Integer-exact MXFP4

The embedding

Doubling an E2M1 value gives an integer, 2v∈{0,±1,±2,±3,±4,±6,±8,±12}2v \in\{0, \pm1, \pm2, \pm3, \pm4, \pm6, \pm8, \pm12\}, a fact existing MXFP4 kernels use to form INT8 operands (ggml-org, 2025; Georganas et al., 2025). For a weight row nn with block exponents en,be_{n,b}, let En=min⁡ben,bE_n = \min_b e_{n,b}, the spread sn=max⁡ben,b−Ens_n = \max_b e_{n,b} - E_n, and

qn,k=2vn,k⋅2en,b(k)−En,wn,k=2En−1qn,k.(1)q_{n,k} = 2v_{n,k} \cdot2^{e_{n,b(k)}-E_n}, \qquad w_{n,k} = 2^{E_n-1}q_{n,k}. \tag*{(1)}

Proposition 1 (Exact embedding). qn,kq_{n,k} is an integer for every kk and 2En−1qn,⋅2^{E_n-1}q_{n,\cdot} reproduces the MXFP4 row exactly. If sn≤3s_n \le3, then ∣qn,k∣≤96\lvert q_{n,k}\rvert\le96, so the row is an INT8 vector times one scale. Conversely, if sn≥4s_n \ge4 and a block at the largest exponent contains ±4\pm4 or ±6\pm6 (as every block whose exponent follows the OCP rule does), some ∣qn,k∣≥128\lvert q_{n,k}\rvert\ge128.

The proof is in Appendix § 12. With per-token INT8 activations am,ka_{m,k} and scales σm\sigma_m, a linear layer becomes a single INT8 GEMM over the full reduction dimension with one rescale per output,

ym,n=σm2En−1∑kam,kqn,k,(2)y_{m,n} = \sigma_m 2^{E_n-1} \sum_k a_{m,k}q_{n,k}, \tag*{(2)}

and INT32 accumulation cannot overflow for K≤(231−1)/(127⋅96)≈176kK \le(2^{31} - 1)/(127 \cdot96) \approx176\mathrm{k}.

Spread-constrained quantization

We quantize to MXFP4 by choosing each block exponent by squared-error search over the OCP rule ⌊log⁡2max⁡k∣wk∣⌋−2\lfloor\log_2 \max_k \lvert w_k\rvert\rfloor- 2 and its two neighbours, then raising any exponent below max⁡ben,b−3\max_b e_{n,b} - 3 to that value. We call the result spread-constrained MXFP4 (SC-MXFP4). It is a valid MXFP4 tensor that any MXFP4 decoder reads unchanged, and by Proposition 1 it runs exactly on INT8 units. Raising an exponent only coarsens blocks whose values are small relative to the row:

Proposition 2 (Cost of the constraint). Let E∗=max⁡ben,bE^* = \max_b e_{n,b}. Every element of a block whose exponent is raised to E∗−3hasabsoluteroundingerroratmost$2E∗−4E^* - 3 has absolute rounding error at most \$2^{E^*-4}, which is 1/16 of 2E∗2^{E^*}, the largest error of round-to-nearest between the grid points of a block at exponent E∗E^*.

If exponent search gives the top block the exponent above its OCP value, that block’s own values stay below 4⋅2E∗4 \cdot2^{E^*} and err by at most 2E∗−12^{E^*-1}, so the coarsened blocks then err by at most 1/8 of the top block’s actual worst case.

Why real weights satisfy the condition

Under the OCP rule, sn≤3s_n \le3 whenever the largest block maximum of a row is less than 8× its smallest (and only if it is less than 16×), so the condition is about the dynamic range of block maxima within a row. For rows of i.i.d. Gaussian weights, maxima of 32 values concentrate tightly: over 50k simulated rows per width (K=1024K = 1024–6144; unit scale, exponents chosen by the same search as for real weights), the median of log⁡2(max⁡b/min⁡b)\log_2(\max_b / \min_b) is 1.1–1.5, its 99.99th percentile 2.0–2.3, and no row violates the condition. Rows of heavy-tailed i.i.d. weights (Student-tt, 4 degrees of freedom) violate it in 2.7–37% of cases. Real weight rows (97.9–99.97% compliant) therefore behave like light-tailed rows within each row, even though weight distributions pooled across rows are heavy-tailed; the few violations come from two structures. On Qwen3-4B (2.1% of rows violating), they concentrate in down_proj (9.0% of its rows; 82% of those caused by a near-zero block, i.e. dead intermediate channels) and gate_proj (3.6%; all but one caused by an outlier block, i.e. outlier input channels), and stay below 1.3% elsewhere. In both cases the blocks that the constraint coarsens are the small ones, which contribute little to the output. The constraint only touches the blocks that are small relative to their row, which is why it costs nothing measurable (§5).

Composing with GPTQ

Because the exponents are fixed before rounding, GPTQ (Frantar et al. [10]) can choose each element’s E2M1 value with the exponents held fixed, compensating rounding errors on later columns through the inverse Hessian of the layer’s inputs. The result stays on the spread-constrained MXFP4 grid, so it is still a standard MXFP4 file and still exact on INT8 units.

Beyond MXFP4: integer-exact microscaling formats

The embedding uses two properties: element values that are integers up to a fixed scale, and block scales whose ratios within a row are integers. Neither requires E2M1 or powers of two. Any format of the form

wn,k=βnmn,b(k)cn,k,c∈±C, m∈M⊂{1,2,…},(3)w_{n,k} = \beta_n m_{n,b(k)} c_{n,k}, \qquad c \in\pm\mathcal{C},\ m \in\mathcal{M} \subset\{1,2,\ldots\}, \tag*{(3)}

with an integer magnitude codebook C\mathcal{C}, a set M\mathcal{M} of small integer block multipliers and one float scale βn\beta_n per row, executes exactly as INT8 operands q=mcq = m c with one rescale per output, provided

max⁡C⋅max⁡M≤127\max C \cdot\max\mathcal{M} \le127. SC-MXFP4 is the instance C={0,1,2,3,4,6,8,12}C = \{0, 1, 2, 3, 4, 6, 8, 12\} (doubled E2M1), M={1,2,4,8}\mathcal{M} = \{1, 2, 4, 8\}, 32-wide blocks and βn\beta_n a power of two. The two features that make NVFP4 more accurate than MXFP4, 16-wide blocks and block scales finer than powers of two, are both available inside this constraint: with doubled E2M1 elements, M={1,…,10}\mathcal{M} = \{1, \ldots, 10\} gives ten block-scale levels per row instead of four. We quantize by trying, per row, eight values per octave below the largest βn\beta_n that represents the row maximum, βmax⁡2−j/8\beta_{\max}2^{-j/8} with βmax⁡=max⁡k∣wn,k∣/(max⁡Cmax⁡M)\beta_{\max} = \max_k |w_{n,k}|/(\max C \max\mathcal{M}) and j=0,…,7j = 0, \ldots, 7, taking per block the multiplier with the least squared error and per row the βn\beta_n with the least total error. Storage is 4+⌈log⁡2∣M∣⌉/B+32/K4 + \lceil\log_2 |\mathcal{M}|\rceil/B + 32/K bits per weight for block size BB; the kernels change only in that the unpack multiplies by mm instead of shifting. These formats are no longer MXFP4 files: they trade compatibility for accuracy. Table 1 compares them with the standard 4-bit formats (§ 5).

A W4A8 system on Apple M5

Weight layout. We store a lossless re-layout of SC-MXFP4: a 4-bit code per weight (sign and E2M1 index), the block shift en,b−En∈{0,…,3}e_{n,b} - E_n \in\{0,\ldots,3\} in 2 bits per 32 weights, and one FP32 scale per row: 4+2/32+32/K4 + 2/32 + 32/K bits per weight for reduction dimension KK (4.07 at K=4096K = 4096, 4.11 for the K=640K = 640 expert down projections of § 7; MXFP4: 4.25). Decoding is q=±LUT⁡[idx⁡]≪shift⁡q = \pm\operatorname{LUT}[\operatorname{idx}] \ll\operatorname{shift} with LUT⁡=(0,1,2,3,4,6,8,12)\operatorname{LUT} = (0, 1, 2, 3, 4, 6, 8, 12), packed as the nibbles of one 32-bit constant. The 4-bit codes are E2M1 encodings packed as in MLX’s MXFP4 tensors; only the E8M0 scales are re-encoded, offline when packing, as the 2-bit shifts and the row scale, from which each E8M0 byte is recovered exactly. SC-MXFP4 can therefore also be stored, and is read by MLX’s kernels, as a standard 4.25-bit MXFP4 tensor.

Dense prefill. Every token multiplies the same weights, so each layer’s weights are unpacked once per call into an INT8 buffer (a temporary of N⋅KN \cdot K bytes that MLX’s allocator recycles across layers) and multiplied by a plain INT8 matmul reading device memory. Unpacking inside the GEMM’s K-loop instead, as W4A8 kernels on NVIDIA GPUs do [22], re-unpacks the same tile for every row tile and runs at 0.39–0.55× FP16 on M5 (Appendix § 14). Activations are quantized per token, fused with the preceding RMSNorm; q/k/v and gate/up each run as one GEMM on stacked weights, with gate and up rows interleaved in pairs so that SwiGLU’s two inputs share a register tile; the scales, residual additions and SwiGLU are applied in the GEMM’s registers. Every non-

GEMM step reproduces MLX’s rounding, so the model’s logits are bit-identical to running each layer separately.

MoE prefill. One counting-sort kernel orders the token–expert assignments by expert, the INT8 activation rows are gathered in that order, and all experts run in one grouped INT8 GEMM over their contiguous rows; a combine kernel reads each token’s expert outputs back by position and weights them by the routing scores. Each expert sees only a few rows of a short prompt, so a separate unpack pass would move more data than the GEMM reads; the grouped kernel unpacks each weight tile into threadgroup memory instead, and switches to the two-pass scheme above, which unpacks every expert of the layer, once the average expert has 256 rows (decided on the host from the prompt length: Lk/E≥256Lk/E \ge256).

Decode. A GEMV reads the packed weights directly and computes the same integer accumulations; decode logits are bit-identical to running the prefill kernels one token at a time (checked on Qwen3-1.7B, OLMoE-1B-7B and Qwen3.5-35B-A3B).

Accuracy

The kernels are exact, so accuracy depends only on the quantization scheme: SC-MXFP4 weights and symmetric per-token INT8 activations in every transformer-block linear layer, with embeddings, norms, attention, routers and the output head in BF16. We evaluate it with fake quantization, which matches the deployed kernels to within 0.004 perplexity on these models (§ 6; within 0.4% on 8 windows of the 125B model, § 7), on Qwen3-0.6B/1.7B/4B [36], SmolLM3-3B [18], the MoE OLMoE-1B-7B [31] and the MoE Qwen3.5-35B-A3B [28] test perplexity, 2048-token windows, full test set).

The constraint is free. SC-MXFP4 and unconstrained MXFP4 (both with exponent search, W4A16) differ by at most 0.007 perplexity on the four models where both were run (Table 10 in Appendix § 17), including Qwen3-4B, where 2.1% of rows were adjusted, and the relative weight error changes by at most 0.01 on all seven models (Table 12 in Appendix § 19). Exponent search improves on the OCP rule on three of four models (e.g. 8.71 → 8.55 on SmolLM3).

Table 10: WikiText-2 perplexity ablations (same protocol as Table 2). The two W4A16 rows with exponent search show three decimals for the comparison in § 5. – = not run.

MethodQwen3 0.6BQwen3 1.7BQwen3 4BSmolLM3 3BOLMoE 1B-7B‡Qwen3.5 35B-A3B‡
BF16 weights, INT8 activations12.989.648.138.007.59–
MXFP4, OCP exponent rule, W4A1615.1910.818.548.71––
MXFP4, exponent search, W4A1615.19810.6978.5068.550––
SC-MXFP4 W4A1615.19510.7048.5068.5488.0366.001
SC-MXFP4 + GPTQ W4A16–10.22––––
SC-MXFP4 W4A815.5510.998.758.628.116.05
SC-MXFP4 W4A8 + Hadamard (128-blocks)17.4513.318.96–––
SC-MXFP4 W4A8 + Hadamard (32-blocks)–13.188.80–––
E2M1, 16-blocks, m≤10m \le10 W4A16–10.13––––

Table 10.

Table 12: Rows whose block exponents (MXFP4 with exponent search) span at most 3 over the whole row, and relative weight MSE (×103\times10^{3}) before and after enforcing the constraint, over all transformer-block linear layers. 1Every W4A8 matrix of the FP8 checkpoint (95.4M rows; 73,728 of the 74,028 matrices are routed experts), with the FP8 experts decoded exactly.

ModelMatricessn≤3s_n \le3Max sns_nMSE, MXFP4 →\to SC
Qwen3-0.6B19699.8%612.70 →\to 12.70
Qwen3-1.7B19699.7%612.72 →\to 12.72
Qwen3-4B25297.9%812.74 →\to 12.74
SmolLM3-3B25299.7%812.65 →\to 12.66
OLMoE-1B-7B (MoE)313699.9%812.53 →\to 12.53
Qwen3.5-35B-A3B (MoE)3097099.97%712.71 →\to 12.71
Qwen3.8-Flash-Next (MoE)17402899.93%912.83 →\to 12.84

Table 12.

INT8 activations are cheap. Per-token INT8 activations add +0.07 to +0.36 perplexity over the same weights in W4A16 (+0.8% on Qwen3.5-35B-A3B), about the same as INT8 activations on unquantized weights (+0.09 to +0.32): the two error sources do not compound noticeably.

Against other 4-bit formats. NVFP4 is the most accurate 4-bit format on every model, and MLX’s affine INT4 (group 64) is more accurate than SC-MXFP4 on three of six models; neither has power-of-two scales, so neither runs exactly on INT8 units. SC-MXFP4 W4A8 is 0.2–0.3 perplexity behind them on those models, in exchange for 1.37–1.90× faster linear layers. Under llama.cpp’s own perplexity protocol (Appendix § 20), its Q4_K_M (about 5 bits per weight) is more accurate than our formats, and the 16-block format of § 3.5 is more accurate than Q4_0 at the same 16-bit activations. GPTQ rounding [10] composes with the INT8 path without any change to format or kernels: on Qwen3-1.7B, calibrated on WikiText-2 train, it lowers W4A8 perplexity on the deployed kernels from 10.99 to 10.53 (10.52 in fake quantization, Table 2), and in W4A16 from 10.70 to 10.22, level with round-to-nearest NVFP4 (10.23). It does not improve downstream accuracy, however: the zero-shot mean drops from 61.9 to 60.6, with gains on LAMBADA (the task closest to the calibration text) and losses on ARC, the known overfitting of GPTQ to a narrow calibration set (we calibrated on WikiText-2 only; a broader corpus may avoid it). We therefore report round-to-nearest SC-MXFP4 as our main configuration.

MethodBitsINT8Qwen30.6B1.7B4BSmolLM3 3BOLMoE 1B-7B‡Qwen3.5 35B-A3B‡
BF161612.679.407.907.917.495.72
INT4 affine g64 (MLX) W4A164.50no15.8312.108.568.447.865.92
NVFP4 W4A164.50no14.4410.238.268.297.825.87
NVFP4 W4A16, scale search4.50no–10.08––––
NVFP4 W4A84.50no14.9110.518.478.367.925.93
NVFP4 W4A8, scale search4.50no14.6310.348.448.307.845.91
SC-MXFP4 W4A8 (std. MXFP4 file)4.07yes15.5510.998.758.628.116.05
+ GPTQ4.07yes–10.52––––
E2M1, 32-blocks, m≤8m \le8 W4A84.10yes–10.878.56–––
E2M1, 16-blocks, m≤10m \le10 W4A84.26yes14.6010.408.438.357.85–

Table 2. WikiText-2 perplexity. W4A16: 4-bit weights, 16-bit activations; W4A8: per-token INT8 activations. SC-MXFP4 W4A8 is what our kernels compute. All methods quantize the same matrices. ‡MoE: every expert is quantized, the router stays BF16. Bits at K=4096K = 4096 (the FP32 row scales of our formats add 32/K32/K). – = not run.

Integer-exact formats close the gap to NVFP4. Table 1 ranks formats by layer-output error on Qwen3-1.7B. The integer-exact format with doubled-E2M1 elements, 16-wide blocks and block multipliers m∈{1,…,10}m \in\{1, \ldots, 10\} (4.26 bits) has 31% less error than SC-MXFP4 and comes within 22% of NVFP4 (29% of NVFP4 with scale search); every integer-exact E2M1 variant is better than MLX’s affine INT4. With uniform INT4 elements instead, the structure of integer-scale W4A8 schemes such as QServe’s (here in a symmetric variant), the same block sizes and multipliers give 12.4–15.1 × 10−310^{-3} instead of 7.4–9.0 × 10−310^{-3}: E2M1’s non-uniform grid is part of the gain. Exponent search lowers MXFP4’s weight error but not its layer-output error here (10.59 vs. 9.86 for the OCP rule), although it lowers perplexity (Table 10); the spread constraint adds 0.2% to it (SC-MXFP4: 10.61). In perplexity with INT8 activations (Table 2) it reaches 14.60 on Qwen3-0.6B, 10.40 on Qwen3-1.7B, 8.43 on Qwen3-4B, 8.35 on SmolLM3 and 7.85 on the MoE OLMoE-1B-7B, against 15.55, 10.99, 8.75, 8.62 and 8.11 for SC-MXFP4 and 14.91, 10.51, 8.47, 8.36 and 7.92 for NVFP4 at 4.5 bits; on all five models it is also more accurate than MLX’s 4.5-bit affine format with 16-bit activations (on OLMoE by a margin of 0.01). Our format chooses scales by error search, whereas NVFP4’s standard rule sets each block scale from the block maximum; giving NVFP4 the same kind of search (each block’s E4M3 scale and its two neighbours on either side, least squared error) yields 14.63, 10.34, 8.44, 8.30 and 7.84, so the two formats are within 0.06 perplexity of each other (ours ahead on two of the five models), while ours uses 0.24 fewer bits per weight and, unlike NVFP4, runs exactly on INT8 units. A 4.10-bit variant (32-wide blocks, m≤8m \le8) matches MLX’s 4.5-bit affine format on Qwen3-4B (8.56 vs. 8.56) under INT8 rather than 16-bit activations.

Rotations. Block-wise Hadamard rotations, which MR-GPTQ [9] combines with GPTQ to improve MXFP4, increase the error of every E2M1-based format and of NVFP4 under round-to-nearest in our measurements (Table 1), and raise SC-MXFP4 W4A8 perplexity from 10.99 to 13.18 on Qwen3-1.7B and from 8.75 to 8.80 on Qwen3-4B; they help only the uniform INT4 grid. Per-token INT8 activations do not need outlier suppression here, and rotated weights are a poorer match for the E2M1 grid. We did not evaluate rotations combined with GPTQ, MR-GPTQ’s setting, under the spread constraint.

FormatBitsOutput error (×103\times10^{3})INT8-exact
NVFP4, scale search4.505.70no
NVFP4 (amax/6 rule)4.506.02no
E2M1, 16-blocks, m∈{1..10}m \in\{1..10\}4.267.36yes
E2M1, 16-blocks, m∈{1..8}m \in\{1..8\}4.197.97yes
E2M1, 32-blocks, m∈{1..8}m \in\{1..8\}4.108.88yes
E2M1, 32-blocks, m∈{1..10}m \in\{1..10\}4.138.99yes
MXFP4 (OCP rule)4.259.86no
INT4, 16-blocks, m∈{1..16}m \in\{1..16\} + rotation4.2610.52yes
MXFP4, exponent search4.2510.59no
SC-MXFP4 (standard MXFP4 file)4.0710.61yes
E2M1, 16-blocks, m∈{1,2,4,8}m \in\{1, 2, 4, 8\}4.1311.68yes
INT4, 16-blocks, m∈{1..16}m \in\{1..16\}4.2612.38yes
E2M1, 16-blocks, m∈{1..8}m \in\{1..8\} + rotation4.1912.42yes
INT4, 16-blocks, m∈{1..8}m \in\{1..8\}4.1913.03yes
E2M1, 32-blocks, m∈{1..10}m \in\{1..10\} + rotation4.1314.01yes
E2M1, 32-blocks, m∈{1,2,4,8}m \in\{1, 2, 4, 8\} (float row scale)4.0714.07yes
NVFP4 + rotation4.5014.17no
INT4 affine g64 (MLX)4.5014.27no
INT4, 32-blocks, m∈{1..16}m \in\{1..16\}4.1315.12yes
SC-MXFP4 (standard MXFP4 file) + rotation4.0719.45yes

Table 1. Integer-exact formats (Eq. (3)) vs. standard 4-bit formats on Qwen3-1.7B: layer-output error tr⁡(ΔWHΔW⊤)/tr⁡(WHW⊤)\operatorname{tr}(\Delta W H\Delta W^{\top})/\operatorname{tr}(W H W^{\top}) with HH from WikiText-2 train activations, every third layer, round-to-nearest. Bits include block codes and one FP32 scale per row (K=4096K = 4096). “+ rotation”: randomized Hadamard transform of each 32-wide input block. Scale search: each NVFP4 block’s E4M3 scale and its two neighbours on either side; exponent search: the OCP exponent and its two neighbours (least squared weight error).

Downstream tasks on the deployed system. Table 3 reports zero-shot accuracy (lm-evaluation-harness), with every W4A8 scoring pass running through our kernels, next to MLX’s MXFP4 W4A16 quantizing the same matrices. SC-MXFP4 W4A8 matches it: 61.9 vs. 61.8 on Qwen3-1.7B and 75.5 vs. 75.7 on Qwen3.5-35B-A3B, every task within one standard error except LAMBADA on Qwen3-1.7B, where W4A8 is higher. The integer-exact 16-block format (§3.5) scores 62.4 on Qwen3-1.7B, the highest of the 4-bit systems we measured, with significant gains on LAMBADA and HellaSwag and no ARC regression, unlike GPTQ. MLX’s default conversions also quantize embeddings and the output head and score 0.3–1.2 points lower; we therefore compare formats only at equal coverage.

ModelMethodARC-eARC-cHSPIQAWGLAMB.Mean
Qwen3-1.7BBF1673.445.066.475.563.663.264.5
MXFP4 W4A16 (default)70.543.562.974.663.349.060.6
MXFP4 W4A1672.544.763.974.362.852.361.8
W4A8, SC-MXFP4 (ours)73.243.663.573.662.355.261.9
W4A8, 16-block (ours)72.743.364.773.862.058.062.4
Qwen3.5-35B-A3BBF1685.062.475.784.076.375.676.5
MXFP4 W4A16 (default)83.262.875.183.375.672.375.4
MXFP4 W4A1684.663.275.483.674.673.075.7
W4A8, SC-MXFP4 (ours)84.363.875.183.473.872.775.5

Table 3. Zero-shot accuracy (%) of the deployed systems (Qwen3.5: first 1000 examples per task). MXFP4 and W4A8 quantize the same matrices; “default” is MLX’s conversion, which also quantizes embeddings and head. Normalized accuracy for ARC-c, HellaSwag and PIQA.

Speed

All timings are on an M5 Max (128 GB, macOS 26.6, AC power, High Power mode), on an otherwise idle GPU, with methods interleaved in one process and medians reported; runs disturbed by other GPU activity are rejected (Appendix § 16). Baselines are MLX 0.32 with MXFP4 and affine 4bit checkpoints and llama.cpp (b11706, built from source, Q4_0 and MXFP4_MOE; Q4_K_M and IQ4_XS in Appendix § 20). Both baselines already use the M5 matrix units, with FP16 operands.

Linear layers. On LLM-sized layers (512–4096 tokens, NN, KK up to 14336) W4A8, including activation quantization, unpack and scaling, is 1.37–1.90× faster than MLX’s quantized_matmul when both run from MLX and are timed alike (1.57–2.03× for our standalone kernels, timed differently) and 1.48–1.64× faster than the identical two-pass design with FP16 operands in the same harness, which isolates the effect of the operand type (Appendix § 15). Every output equals the exact integer reference.

End to end. Table 4 reports time to first token (the whole prompt in one forward pass) and decode throughput. On the dense Qwen3-1.7B, prefill is 1.34–1.74× faster than MLX MXFP4 over both clock regimes (Table 9) and 1.56–2.49× faster than llama.cpp. On the MoEs the gain grows with prompt length, because each expert sees only about 8L/E8L/E of an LL-token prompt’s rows and short prompts are bound by streaming expert weights for every method: 1.20–1.57× on OLMoE-1B-7B (64 experts) and 1.10–1.43× on Qwen3.5-35B-A3B (256 experts) over both clock regimes, where W4A8 covers 99.93% of the block-linear parameters and 86–96% of prefill FLOPs. The packed weights equal the fake-quantized reference for every matrix of all three models, every integer accumulation we checked is exact, and WikiText-2 perplexity on the kernels equals the reference (Qwen3.5: 6.052 vs. 6.051). Decode runs from the same 4.1-bit weights at 0.99–1.01× MLX’s speed on the dense model and OLMoE and at 0.90–0.91× on Qwen3.5-35B-A3B, whose batch-1 decode is bound by host-side graph construction and per-kernel latency in the routing for both systems; it is 1.03–1.22× faster than llama.cpp on the first two models and 0.96× on the third.

ModelTokensllama.cppMLX MXFP4W4A8vs. MLXvs. llama.cpp
Qwen3-1.7B (dense)51241.235.426.41.34×1.56×
2048175.0131.575.41.74×2.32×
81921020.2690.5428.01.61×2.38×
OLMoE-1B-7B (MoE)51241.438.431.91.20×1.30×
2048158.3122.587.81.40×1.80×
4096342.7251.3167.51.50×2.05×
Qwen3.5-35B-A3B (MoE)512140.3134.9122.71.10×1.14×
2048511.3386.0317.51.22×1.61×
81922388.21630.61140.21.43×2.09×
Decode, 512-token context (tokens/s)
Qwen3-1.7B2873533500.99×1.22×
OLMoE-1B-7B4034484420.99×1.09×
Qwen3.5-35B-A3B1371461310.90×0.96×

Table 4. Time to first token (ms) and decode throughput (tokens/s) on an M5 Max. Speedups, from unrounded medians, are W4A8 vs. MLX MXFP4 measured in the same session and vs. llama.cpp (Q4_0, its fastest 4-bit type here), whose prefill was measured in a separate session (the faster of two runs); every prefill ratio against it is at least 1.14×. MoE rows: the default configuration (§ 4). Prefill rows use the sustained clock regime (prompts following long ones; Table 9 also gives the burst regime). llama.cpp’s MXFP4_MOE type is 11–26% slower than its Q4_0 on the two MoEs, its Q4_K_M and IQ4_XS prefill at 0.82–0.96× Q4_0’s speed (Appendix § 20). Decode rows: all three systems back to back in one session per model.

Table 9: Prefill time to first token (ms, median of 7–10) by clock regime, with the speedup over MLX MXFP4 in the same run in parentheses (ratio of medians). MoE: the default configuration (W4A8) and the two expert-weight paths forced at every length. – = not measured.

RegimeSystem512204840968192
Qwen3-1.7B (dense)
burstMLX MXFP431.4120.4––
W4A821.8 (1.44)70.4 (1.71)––
sustainedMLX MXFP435.4131.5–690.5
W4A826.4 (1.34)75.4 (1.74)–428.0 (1.61)
OLMoE-1B-7B (MoE, 64 experts)
burstMLX MXFP437.6118.7242.4–
W4A831.3 (1.20)94.6 (1.25)165.6 (1.46)–
in-kernel30.5 (1.23)89.2 (1.33)178.3 (1.36)–
two-pass45.4 (0.83)87.9 (1.35)164.9 (1.47)–
sustainedMLX MXFP438.4122.5251.3556.5
W4A831.9 (1.20)87.8 (1.40)167.5 (1.50)355.3 (1.57)
in-kernel32.3 (1.19)92.1 (1.33)187.8 (1.34)413.6 (1.35)
two-pass45.5 (0.84)87.1 (1.41)164.7 (1.53)353.2 (1.58)
Qwen3.5-35B-A3B (MoE, 256 experts)
burstMLX MXFP4126.7370.1––
W4A8109.9 (1.15)300.5 (1.23)––
in-kernel109.8 (1.15)296.1 (1.25)––
two-pass198.8 (0.64)361.7 (1.02)––
sustainedMLX MXFP4134.9386.0–1630.6
W4A8122.7 (1.10)317.5 (1.22)–1140.2 (1.43)
in-kernel111.5 (1.21)306.0 (1.26)–1183.2 (1.38)
two-pass202.9 (0.66)367.2 (1.05)–1158.0 (1.41)

Table 9.

A more accurate INT8-exact format on the same kernels. The kernels never assume MXFP4: switching to 16-wide blocks with multipliers 1,…,101,\ldots,10 (4.26 bits) changes only the packer. On Qwen3-1.7B its packed weights equal the fake-quantized ones, perplexity on the kernels is 10.395 (SC-MXFP4 10.985, NF4 10.508), prefill and decode speed are unchanged and the zero-shot mean rises to 62.4 (Appendix § 18, Table 11). On the 125B MoE of §7 it keeps the prefill speed but, unable to use MLX’s decode kernels, decodes at 0.65× MLX’s speed.

bits/wPPLTasks meanPrefill TTFT (ms) 512Prefill TTFT (ms) 2048Prefill TTFT (ms) 8192Decode tok/s
MLX MXFP4 (W4A16)4.25–61.831.5120.3654.7368
NVFP4 (W4A8, fake quant)∗^{*}4.5010.51–––––
SC-MXFP4 (W4A8)4.0810.9961.922.070.4395.1359
16-block, m≤10m \le10 (W4A8)4.2610.4062.422.370.5390.3357

Table 11. SC-MXFP4 and the 16-wide, integer-multiplier format on the same INT8 kernels (Qwen3-1.7B, M5 Max, one session separate from Table 4’s; prefill at 512 and 2048 tokens in the burst regime, at 8192 in the sustained regime). Perplexity: WikiText-2, W4A8, measured on the kernels; NVFP4 is fake-quantized (it has no exact INT8 execution; ∗^{*}with scale search 10.34). Tasks: mean zero-shot accuracy over six lm-evaluation-harness tasks, on the kernels. bits/w: average over this model’s quantized matrices.

A 125B mixture of experts: Qwen3.8-Flash-Next

Qwen3.8-Flash-Next, released in August 2026 ([37]), has 125B language-model parameters: 48 layers of Gated DeltaNet and sparse attention (3:1), each with 512 routed experts of 2560 × 640 (top-10) and a shared expert, four gated residual streams, and a 51B-parameter n-gram embedding table that is only ever indexed. We start from the official FP8 checkpoint (173 GB; routed experts and the n-gram in FP8, the rest BF16) and run it with the model code of mlx-lm pull request 1788 (mlx-lm contributors, 2026). In BF16 the model does not fit in 128 GB, so perplexity is evaluated layer by layer: the hidden states of all 145 WikiText-2 windows are kept for every method, each layer is read once from the checkpoint, quantized once per format and applied to every method’s states. W4A8 covers every DeltaNet and attention projection and every routed and shared expert (98% of the parameters outside the n-gram table); the router, gates, mixers and n-gram table stay as in the checkpoint. All formats use round-to-nearest without calibration data.

Accuracy. Table 5. Per-token INT8 activations cost at most 0.3% for every format (0.0% for the integer-exact format), so the loss is the 4-bit weights, and almost all of it comes from the dense projections, 2–3% of the parameters: quantizing only the 73,728 routed expert matrices (96% of the parameters, where the INT8 speedup is) costs every FP4 format at most 0.4% with INT8 activations (ours at most 0.2%). At that scope the FP4 formats are within about ±1% of the FP8 model and plain MXFP4 even scores below it, so perplexity no longer separates them; the per-token KL divergence from the FP8 model does (32 windows; 16-bit / INT8 activations): 0.0534 / 0.0557 for the integer-exact 16-block format, 0.0539 / 0.0553 for NVFP4 with scale search, 0.0610 / 0.0628 for affine INT4 (the format of the concurrent INT8 prefill path (oMLX contributors, 2026)), 0.0796 / 0.0802 for SC-MXFP4 and 0.0931 / 0.0940 for MXFP4 with the OCP rule, the format with the lowest perplexity.

Formatbits/wW4A16W4A8vs. FP8on M5
NVFP4, E4M3 scale search4.504.9044.917+5.5%FP16
NVFP4, amax/6 rule4.504.9184.928+5.7%FP16
16-block, m≤10m \le10 (ours)4.274.9704.969+6.6%INT8, exact
Affine INT4, g64 (MLX 4-bit)4.504.9954.996+7.2%FP16 / INT8{sup}*`
SC-MXFP4 (ours, std. MXFP4)4.095.0975.107+9.6%INT8, exact
MXFP4, OCP rule (MLX)4.255.1995.212+11.8%FP16
Routed experts only (the scope of the public NVFP4 and FP8 releases)
NVFP4, E4M3 scale search4.504.6564.661−0.0%FP16
16-block, m≤10m \le10 (ours)4.284.6684.664+0.1%INT8, exact
SC-MXFP4 (ours, std. MXFP4)4.094.6664.671+0.2%INT8, exact
NVFP4, amax/6 rule4.504.6744.678+0.4%FP16
Affine INT4, g64 (MLX 4-bit)4.504.7544.761+2.1%FP16 / INT8{sup}*`
MXFP4, OCP rule (MLX)4.254.6214.619−0.9%FP16

Table 5. Qwen3.8-Flash-Next, WikiText-2 test perplexity (145 windows of 2048 tokens; FP8 checkpoint: 4.662). All methods quantize the same matrices (every projection and expert); NVIDIA’s NVFP4 release quantizes only the routed experts ([33]). bits/w: for this model’s matrices (our formats’ row scales add 32/K32/K, and the expert down projections have K=640K = 640). “vs. FP8”: the W4A8 column, from unrounded perplexities. All perplexities use the model’s reference DeltaNet recurrence. Last column: the format multiplies on M5 ({sup}*INT8 with a correction per 64-element group, as in oMLX ({citep}(oMLX contributors, 2026))).

The rate of tokens whose top-1 prediction changes orders them the same way, with the 16-block format and NVFP4 within 0.2 points of each other (7.7–7.9%). Quantizing the dense projections as well triples the divergence (0.177 for the 16-block format), which is why deployments keep them at higher precision. With every matrix quantized, the integer-exact 16-block format is more accurate than both 4-bit formats deployed on Macs at either activation precision (W4A8: 4.969 vs. 4.996 and 5.212), at fewer bits, and removes a third of SC-MXFP4’s loss; NVFP4 remains 1.1% ahead of it on this model (on the five small models the two were within 0.06). SC-MXFP4, a standard MXFP4 file, is more accurate than MXFP4 as MLX quantizes it, because exponent search matters more than the spread constraint. 99.929% of the model’s 95.4M weight rows satisfy the spread condition: 0.004–0.087% of expert rows violate it (the expert down projections have only 20 blocks per row), the dense projections 2–12.5%, almost always (97.5%) because of one outlier block. The packed weights equal the fake-quantized ones (0 of 298M checked differ), and over the first 8 windows perplexity on our kernels is 2.633 against 2.638 for the reference with BF16 rounding, and 2.511 against 2.521 for the 16-block format (all with the reference DeltaNet recurrence; with the model’s Metal kernel, 2.647 against 2.640 and 2.511 against 2.514). Downstream, with every matrix quantized, on all 1319 GSM8K test problems (5-shot as chat turns, greedy, prefill and decode on each system’s kernels, the model’s Metal DeltaNet kernel throughout), our W4A8 system solves 92.65±0.72% with SC-MXFP4 weights, MLX’s W4A16 kernels on the same SC-MXFP4 weights 92.80±0.71%, and MLX’s W4A16 kernels on its own MXFP4 quantization of the same matrices 90.98±0.79% (lenient answer extraction: 92.95, 93.40 and 92.42%). Paired, intra-8 activations change nothing measurable (17 vs. 19 problems solved by only one of the two, McNemar pp = 0.87; the same final answer on 1261 of 1319), whereas the weight quantizer does (SC-MXFP4 vs. MLX’s MXFP4, both W4A16: 52 vs. 28, pp = 0.010; lenient pp = 0.15). The integer-exact 16-block format on our kernels solves 94.84±0.61% (lenient 95.15%) more than SC-MXFP4 (52 vs. 23, pp = 0.001) and MLX’s MXFP4 (70 vs. 19, p<10−4p < 10^{-4}).

Speed. Table 6. In isolation our MoE blocks (router, routed and shared experts) at this model’s shapes, where an expert sees only 10, 40 or 160 rows of a 512-, 2048- or 8192-token prompt, are 1.33–1.65× faster than MLX’s (random weights, chained calls). End to end the gain is 1.22–1.29×: at 2048 tokens the whole difference (226 ms) is the experts’ difference (48 × 4.6 ms), and about 0.5 s of each prefill is work outside the W4A8 matrices that both systems share (the DeltaNet recurrence, attention, 96 residual-stream mixers, the n-gram lookup on the host), a larger share at 8192 tokens. MLX’s affine INT4 (68.6 GiB) is no faster than its MXFP4: in the same sessions its prefill is within about 2% and its decode within 1%. At batch 1 decode is different: every weight is read once for one multiply-add (over 1.3 GB of expert weights per token), so a step is bound by streaming weights and by per-kernel latency, and INT8 arithmetic cannot help. With our W4A8 GEMVs, decode runs at 36.3 tokens/s; in a profile of that step (27.5 ms, separate run) the expert GEMVs and routing take 11.6 ms, the dense GEMVs 7.9 ms and everything else 8.1 ms. MLX’s MXFP4 GEMV, which reads our packed bytes unchanged (with one derived E8M0 byte per 32 weights) and is 1.6–2.2× faster than ours in isolation (ours also quantizes its input inside every threadgroup), raises this to 37.9 when it replaces ours for the dense projections, and with MLX’s kernels for every matrix (MLX’s model on our SC-MXFP4 file, as in the GSM8K comparison above) decode is as fast as with MLX’s own weights (42.4 and 42.5 against 42.4 and 42.4 tokens/s, two sessions each). The SC-MXFP4 row of Table 6 therefore decodes with MLX’s kernels on the same file, keeping one E8M0 byte per 32 weights next to our block codes and giving up bit-identical prefill and decode; our prefill then shortens every request. The 16-block format cannot use MLX’s kernels: it keeps the prefill gain (1.21–1.25×, within 5% of SC-MXFP4) but decodes at 27.4 tokens/s on ours, unlike on Qwen3-1.7B (Appendix § 18), because with 16-wide blocks our expert GEMV, the largest part of the step, applies twice as many block multipliers (profiled in one session: 22.6 against 13.0 ms per step with SC-MXFP4, the rest of the step 16.8 against 18.3 ms); we have not optimized this path, and its prefill gain outweighs the decode loss only for answers up to 16 tokens after a 2048-token prompt and 63 after an 8192-token prompt.

51220488192decode
MLX MXFP4 (W4A16)3261056479942.4
W4A8, SC-MXFP4 (ours)254830394342.4*
W4A8, 16-block (ours)266844398227.4
SC-MXFP4 vs. MLX1.29×1.27×1.22×1.00×*
16-block vs. MLX1.23×1.25×1.21×0.65×

Table 6. Qwen3.8-Flash-Next on an M5 Max: time to first token (ms) and batch-1 decode (tokens/s) of our W4A8 kernels with SC-MXFP4 (58.9 GiB) and with the 16-block format (61.6 GiB), and of MLX MXFP4 W4A16 on exactly the same matrices (65.0 GiB). The systems do not fit in memory together, so each was loaded in its own session (two sessions each, alternating; medians agree within 5%). *Decode with MLX’s W4A16 kernels on the same SC-MXFP4 file (two later sessions, 42.4 and 42.5, in which MLX MXFP4 decoded at 42.4); with our W4A8 GEMVs, 37.6 (0.89×).

Against concurrent INT8 kernels. oMLX’s INT8-activation kernels (oMLX contributors, 2026) (release 0.7.1rc1, the same MLX version) multiply MLX’s affine INT4 weights against INT8 activations and correct every 64-element group. At this model’s shapes (random weights, chained calls), our dense projections run at 0.89–1.11× their speed at 512 tokens and 1.07–1.24× at 2048–8192 tokens; for the routed experts their gate/up kernel, which decodes weights directly into the matrix unit’s register fragments, is faster than our grouped GEMMs, which stage them in threadgroup memory, although their down projection stays in W4A16 (ours: 0.94, 0.86 and 0.76× their speed at 512, 2048 and 8192 tokens). The per-group correction therefore costs well-engineered kernels little; what the integer-exact formats add is accuracy (with INT8 activations affine INT4 loses +2.1% on the routed experts, ours at most +0.2%, Table 5), exactness, and compatibility with standard MXFP4 files, and register-level decoding is an optimization our kernels can adopt. We did not run oMLX end to end: its checkpoint for this model keeps the n-gram table in memory (30 GB at 4 bits), which with the weights would take about 120 GB of the 128 GB.

Related work

W4A8 and integer scales. VS-Quant [8] introduced a coarse float scale times small integer per-vector scales, the structure of llama.cpp’s k-quants and of IQ4_KS (a float row scale, 7-bit integer block scales and an INT8 codebook at 4.25 bits) [12, 20]; its block scales times its codebook exceed the INT8 range, so its kernels scale each block’s partial sum. QServe [22] and DGQ (Zhang et al., 2023) bound INT4 codes times integer group scales so that the weights are INT8 operands of one GEMM; OQQ (Zhang et al., 2024) and Integer Scale [21] apply amplified integer group scales to partial sums. Our format family (§3.5) is the INT8-closed instance of this structure with the FP4 codebook instead of INT4, which is where its accuracy comes from (Table 1).

FP4 weights on integer hardware. That doubled E2M1 values are integers is used by llama.cpp’s MXFP4 and NVFP4 kernels (an INT8 lookup table, dot products against INT8 activations), by a CPU W4A8 MXFP4 kernel in ML-SpecQD and by an open VLLM GEMV for AMD GPUs that reads MXFP4 checkpoints unchanged; all apply the E8M0 scale to each 32-element block’s partial sum (ggml-org, 2025; Georganas et al., 2025; vLLM contributors, 2026); in the other direction, FP64 GEMMs have been emulated on FP4 tensor cores with integer limbs held in E2M1 [19]. ZeroQuant-FP4 constrains group scales to powers of two so that casting FP4 to FP8 is a shift [46], shared microexponents confine sub-block exponents to a window [40], and AdaMX reports that over 90% of output channels have block exponents within a range of three, which its accelerator exploits with a per-channel bias [25]. We give the exact row-level condition under which every per-block scale folds into INT8 operands, the quantizer and bound that enforce it, and the consequence that a standard MXFP4 layer is one INT8 GEMM with one rescale per output.

FP4 formats and accuracy. MX formats [35, 41] and NVFP4 target native FP4 hardware. MR-GPTQ [9] combines block rotations with GPTQ to bring MXFP4 close to NVFP4, and Four Over Six [6] chooses NVFP4 block scales adaptively on existing hardware; MXFP [15], NxFP [24], HiFloat4 [26], MixFP4 (Zou et al., 2026), IF4 [7] and AdaMX [25] refine block scales or element formats for new hardware; INT v.s. FP [5] and ARCQuant [27] compare formats under direct casting. Like the round-to-nearest ablations of MR-GPTQ and INT v.s. FP, we find that block rotations do not help E2M1 formats or NVFP4 without GPTQ, and extend this to W4A8; we did not test rotations combined with GPTQ. Rotations [3, 23], smoothing (Xiao et al., 2023) and GPTQ [10] remain compatible with SC-MXFP4. MoE-specific quantization [17, 16] and calibration-free methods [32] improve the weights themselves and are orthogonal to how they are executed.

LLM inference on Apple Silicon. MLX [14] and llama.cpp’s Metal backend [12] dequantize 4-bit weights to 16 bits inside their kernels. BaseRT [44] routes prefill GEMMs through the M5 Neural Accelerators with 4- and 8-bit weights but does not report operand types or activation quantization, and its engine is distributed as a binary [38]. Concurrently with this work, the open-source projects oMLX (oMLX contributors, 2026) and scratchy (scratchy contributors, 2026) added INT8-activation prefill on the M5 for MLX’s affine INT4 weights, applying the affine correction at every 64-element group; oMLX routes the expert gate and up projections of Qwen3.8-Flash-Next this way and leaves the down projection in W4A16, and an open MLX proposal stages dequantized weights as INT8 [29]. Ours differs in the weight format (standard MXFP4 and integer-exact formats that need no per-group correction), in covering every projection including the expert down projection and decode, and in exactness: our outputs are bit-identical to the quantized reference, and we measure the accuracy of the deployed W4A8 path. On Apple’s CPU matrix units, INT8 outer products run at 2×2 \times the rate of FP16 with FP32 accumulation [39]. Benazir & Lin (2025) identify dequantization as a limiter of quantized inference on Apple Silicon. Systems that stream weights of models larger than memory from flash [1, 45] target decode and are complementary.

Limitations and conclusion

Measurements are from one M5 Max. On Qwen3.8-Flash-Next the integer-exact format is 1.1% behind NVFP4 in perplexity, all results use round-to-nearest without calibration (GPTQ composes with SC-MXFP4, § 5), and the 125B model is quantized from its FP8 release rather than from BF16. Under llama.cpp’s perplexity protocol (Appendix § 20) its Q4_K_M (about 5 bits per weight) is more accurate than our formats, and SC-MXFP4 also trails its other 4-bit types on the two MoEs; we did not evaluate block rotations combined with GPTQ. End-to-end gains are bounded by the work outside the quantized matrices, which is large in hybrid models such as Qwen3.8-Flash-Next (1.22–1.29× end to end versus 1.33–1.65× for its expert layers), and our batch-1 decode kernels are at or below MLX’s speed: decode is bound by memory and per-kernel latency, where INT8 arithmetic does not help, so on that model SC-MXFP4 decodes with MLX’s kernels and the 16-block format, which cannot use them, at 0.65× MLX’s speed. The model’s Metal kernel for the DeltaNet recurrence is less precise than its reference implementation, which we use for the perplexity tables (it changes 8-window perplexity by −0.3 to +0.6%, Appendix E). The integer-exact formats are no longer standard MXFP4 files. INT8-activation prefill on the M5 has been developed concurrently in open-source projects for affine INT4 weights (oMLX contributors, 2026; scratchy contributors, 2026), and at the kernel level our design is not faster than theirs (§ 7): the advantage of folding the scales is exactness and the formats it enables, not speed over carefully engineered per-group corrections.

On hardware whose fastest datapath is INT8×INT8, the standard MXFP4 format does not have to be dequantized to 16 bits, nor its block scales applied inside the reduction: with a constraint that real weights already satisfy, its rows are exact INT8 vectors with one scale. This turns MXFP4 models into W4A8 models with the same accuracy and makes their prefill 1.1–1.7× faster on Apple’s M5; the exactness and the formats apply to any INT8 matrix unit without FP4. Relaxing power-of-two scales to small integer multipliers keeps this exactness and reaches NVFP4’s accuracy (on a 125B MoE’s experts, the same divergence from the original model as NVFP4 with scale search, at 4.28 instead of 4.5 bits, and the best GSM8K accuracy we measured), so on integer hardware the choice of 4-bit format need not trade accuracy for prefill speed.

AI USE STATEMENT

Anthropic’s Claude, used as a coding agent under the authors’ direction, wrote and ran the kernels, benchmark harnesses, and quantization and evaluation code, searched and summarized related work, and drafted the text of this paper. Argon (Gemini 4) gave feedback on a draft of the paper. The authors directed the research, reviewed the code, results and text, and take full responsibility for the content. All reported numbers come from the scripts in the code release.

Reproducibility Statement

The code repository, https://github.com/shinyoungkang1/mxfp4-int8-m5, contains every kernel, benchmark harness, quantizer and evaluation script, and the scripts that run each experiment; the measurement protocol is in Appendix E. All models and datasets are public.

Proofs

Proposition 1. 2vn,k2v_{n,k} is an integer and en,b(k)−Ene_{n,b(k)}-E_n is a non-negative integer, so qn,kq_{n,k} is an integer, and 2En−1qn,k=2en,b(k)vn,k2^{E_n-1}q_{n,k}=2^{e_{n,b(k)}}v_{n,k} term by term. Since ∣2v∣≤12\lvert2v\rvert\le12 and en,b(k)−En≤sne_{n,b(k)}-E_n\le s_n, ∣qn,k∣≤12⋅2sn≤96\lvert q_{n,k}\rvert\le12\cdot2^{s_n}\le96 for sn≤3s_n\le3. If sn≥4s_n\ge4 and a block at exponent En+snE_n+s_n contains v=±4v=\pm4 or ±6\pm6, then ∣q∣≥8⋅2sn≥128>127\lvert q\rvert\ge8\cdot2^{s_n}\ge128>127. Under the OCP rule a block’s maximum lies in [4⋅2eb,8⋅2eb)[4\cdot2^{e_b},8\cdot2^{e_b}) and rounds to ±4\pm4 or ±6\pm6.

Proposition 2. Let block bb have exponent eb≤E∗−4e_b\le E^*-4 before the constraint and L=E∗−3L=E^*-3 after, and let eO=⌊log⁡2max⁡k∣wk∣⌋−2e_O=\lfloor\log_2\max_k\lvert w_k\rvert\rfloor-2 be its OCP exponent, so that ∣w∣<8⋅2eO\lvert w\rvert<8\cdot2^{e_O}. Exponent search picks eb∈{eO−1,eO,eO+1}e_b\in\{e_O-1,e_O,e_O+1\}, so eO≤L−1e_O\le L-1 except when eb=L−1e_b=L-1 and eO=Le_O=L. If eO≤L−1e_O\le L-1, every value satisfies ∣w∣<8⋅2L−1=4⋅2L\lvert w\rvert<8\cdot2^{L-1}=4\cdot2^L; for ∣w∣\lvert w\rvert in [0,4⋅2L][0,4\cdot2^L] (the grid is symmetric) the scaled E2M1 grid {0,12,1,32,2,3,4}⋅2L\{0,\frac{1}{2},1,\frac{3}{2},2,3,4\}\cdot2^L has spacing at most 2L2^L, so round-to-nearest at exponent LL errs by at most 2L−1=2E∗−42^{L-1}=2^{E^*-4}. In the remaining case the search preferred exponent L−1L-1 to LL although the block maximum lies in [4⋅2L,8⋅2L)[4\cdot2^L,8\cdot2^L). The two grids coincide on [2L,3⋅2L][2^L,3\cdot2^L]; below 2L2^L the finer grid of L−1L-1 lowers an element’s squared error by at most (2L−2)2=22L/16(2^{L-2})^2=2^{2L}/16, and above 3⋅2L3\cdot2^L it clips to 3⋅2L3\cdot2^L, which never lowers the squared error: on (3,72]⋅2L(3,\frac{7}{2}]\cdot2^L both grids round to 3⋅2L3\cdot2^L; on (72,5]⋅2L(\frac{7}{2},5]\cdot2^L, where exponent LL rounds to 4⋅2L4\cdot2^L, clipping raises it by (2∣w∣−7⋅2L)2L(2\lvert w\rvert-7\cdot2^L)2^L; on (5,8)⋅2L(5,8)\cdot2^L, where LL rounds or saturates to 6⋅2L6\cdot2^L, by 3⋅2L(2∣w∣−9⋅2L)>3⋅22L3\cdot2^L(2\lvert w\rvert-9\cdot2^L)>3\cdot2^{2L}. At most 31 elements can gain, so L−1L-1 wins only if every element above 4⋅2L4\cdot2^L has ∣w∣≤14332⋅2L\lvert w\rvert\le\frac{143}{32}\cdot2^L; at exponent LL such an element rounds to 4⋅2L4\cdot2^L with error at most 1532⋅2L<2L−1\frac{15}{32}\cdot2^L<2^{L-1}, and the other elements are below 4⋅2L4\cdot2^L, covered above. In a block at exponent E∗E^*, the grid spacing between 4 and 6 is 2⋅2E∗2\cdot2^{E^*}, so round-to-nearest can err by up to 2E∗2^{E^*} between grid points (values above 6⋅2E∗6\cdot2^{E^*} saturate and can err more).

Hardware characterization

Table 7: matmul2d throughput (TFLOPS or TOPS, median of 15 interleaved repetitions) on the M5 Max, AC power, High Power mode; speedup vs. FP16 in parentheses (median of per-iteration ratios). One matmul2d call per 128×64128\times64 output tile, 4 simdgroups, grouped tile order, weights N×KN\times K. All outputs bit-exact.

M×N×KM\times N\times KFP16INT8×\timesINT8INT8×\timesINT4FP16×\timesINT8FP16×\timesINT4
512 ×\times 4096 ×\times 409654.9108.7 (1.97)95.5 (1.74)45.9 (0.84)48.9 (0.89)
1024 ×\times 4096 ×\times 409655.6110.2 (1.98)94.9 (1.70)50.3 (0.90)49.5 (0.89)
4096 ×\times 4096 ×\times 409664.4126.7 (1.97)116.3 (1.81)58.5 (0.91)60.8 (0.94)
2048 ×\times 8192 ×\times 819259.6118.9 (1.98)108.1 (1.81)53.2 (0.90)55.8 (0.93)
4096 ×\times 14336 ×\times 409665.4127.4 (1.95)117.8 (1.80)60.5 (0.92)62.8 (0.96)
4096 ×\times 4096 ×\times 1433663.9114.2 (1.79)111.5 (1.74)43.5 (0.68)50.7 (0.79)
8192 ×\times 8192 ×\times 409665.6127.3 (1.94)119.3 (1.82)60.1 (0.92)62.8 (0.96)

Table 7.

Storing weights K×NK\times N instead of N×KN\times K (reduction dimension contiguous) is about 2×2\times slower for every operand type. FP16 output is no faster than FP32 output under sustained load. On battery with Low Power Mode, sustained throughput drops to about a third (FP16 ≈\approx 23 TFLOPS) with unchanged INT8/FP16 ratios (1.85–2.04×), so the ratio is not a power-state artifact.

Kernel details

Fused vs. two-pass. A single kernel that unpacks each 64×12864 \times128 weight tile (64 output channels by 128 reduction elements) into threadgroup memory inside the K-loop and calls matmul2d on it runs at 0.39–0.55× FP16 on LLM-sized layers, although it is still 1.2–1.3× faster than the identical kernel with FP16 tiles: every M-tile re-unpacks the same weights, and the scalar unpack competes with the matrix unit. Unpacking each layer once into an INT8 buffer (about 1.5 bytes moved per weight; 0.03 ms for 4096×40964096 \times4096, 0.15 ms for 14336×409614336 \times4096) and running a plain INT8 matmul2d from device memory is 1.36–1.51× faster than FP16 with 16-bit weights. In Table 8 this path also includes the activation quantization (0.04 ms at 2048×40962048 \times4096) and the scaling of the outputs (0.12 ms at 2048×40962048 \times4096 outputs), both linear in the number of tokens; the W8A8 bound of Table 8 includes the activation quantization but neither unpack nor scaling, which is why the gap between them grows with MM.

M×N×KM \times N \times KMLX FP16matmul2d FP16MLX MXFP4W4A16 fusedW4A16 2-passW4A8 (ours)W8A8†vs MLX MXFP4 mixedvs MLX MXFP4 alike
512×4096×4096512 \times4096 \times40960.3210.3000.3460.9340.3610.2200.1681.57×1.48×
1024×4096×40961024 \times4096 \times40960.6770.5830.7161.8240.6660.4230.3191.69×1.69×
2048×4096×40962048 \times4096 \times40961.3601.1781.4463.4431.2420.8380.6281.72×1.80×
4096×4096×40964096 \times4096 \times40962.9572.2733.2126.3302.4201.5811.2592.03×1.86×
2048×14336×40962048 \times14336 \times40964.9544.0005.35811.1984.3482.7031.9921.98×1.90×
2048×4096×143362048 \times4096 \times143366.7184.0715.18913.4554.3142.6962.3741.92×1.37×

Table 8. Linear-layer latency (ms, median) on M5 Max; all W4 variants except MLX read the same SC-MXFP4 weights. Our kernels are timed with GPU timestamps: the elapsed time of one command buffer that runs a pipeline’s kernels back to back, divided by the number of runs it contains, so gaps between a pipeline’s kernels count: matmul2d FP16 is a plain FP16 GEMM; W4A16 fused unpacks to FP16 tiles inside the K-loop; W4A16 2-pass unpacks to an FP16 buffer and runs an FP16 GEMM with the row scales; W4A8 quantizes the activations, unpacks to an INT8 buffer and runs an INT8 GEMM, scaling in registers or in a separate pass, whichever is faster; †W8A8 quantizes the activations and runs an INT8 GEMM with INT32 output from 8-bit weights (no unpack, no scaling), a bound on the achievable speed. MLX’s kernels are timed by wall clock in a separate process (10 calls per evaluation), which includes some host overhead, so the “mixed” vs-MLX column mixes two timing methods. “Alike”: our MLX-integrated kernels (the ones used end to end) and MLX’s, called from MLX and timed alike by wall clock in one process, interleaved; at K=14336K = 14336 the MLX-integrated W4A8 is slower than the standalone one (3.90 vs. 2.70 ms). Like for like are this column and the comparisons against matmul2d FP16 and W4A16 2-pass.

Epilogues. Scales applied through the tensor API’s cooperative destination tensor in the GEMM’s registers are about 10% slower than an INT32 output plus a separate scaling pass on single layers, but in the fused model the in-register epilogue (scale; scale and residual; SwiGLU on gate/up rows interleaved in pairs) is 1.04–1.12× faster at 2048–8192 tokens, where the INT32 round trip of the 12288-wide gate/up output dominates.

MoE. One counting-sort kernel groups token–expert assignments by expert (replacing two argsorts and gathers; identical logits) and returns each assignment’s position in expert order; the INT8 activation rows are gathered into that order, one per assignment, and the combine kernel reads each token’s top-kk output rows back by position, weights them by the routing scores and adds the residual, fused with the next RMSNorm and activation quantization. The grouped GEMM is one dispatch over all experts; each threadgroup finds its expert and row tile from a prefix sum of per-expert tile counts. With in-kernel unpack each threadgroup unpacks its expert’s 64×6464 \times64 weight tile per K-chunk of 64 into threadgroup memory (K-chunk 64 beats 128 at every length: half the threadgroup memory, twice the occupancy). The two-pass variant unpacks all of a layer’s experts into an INT8 temporary of E⋅N⋅KE \cdot N \cdot K bytes (0.27 GB for OLMoE’s stacked gate/up projections, 0.54 GB for Qwen3.5-35B-A3B’s). Two-pass is 0.83× MLX at 512 tokens on OLMoE and 0.66× on Qwen3.5-35B-A3B, because unpacking every expert costs more than the few-row GEMMs; from about 256 rows per expert the two variants are even.

Decode. The GEMV turns each 4-bit code cc (sign and E2M1 index) into the half-precision number 2v⋅2−152v \cdot2^{-15} with one bit operation whose 16-bit result is reinterpreted as a half, ((c&7)≪9)∣((c&8)≪12)((c \& 7) \ll9) \mid((c \& 8) \ll12) (index 1 becomes the subnormal 2−152^{-15}), multiplies it by the INT8 activations held as floats and sums each 32-block exactly in FP32 (a multiple of 2−152^{-15} below 2). It multiplies the block sum by the block multiplier (2en,b∈En2^{e_n,b} \in E_n, or mm for the integer-multiplier formats), accumulates a lane’s blocks in FP32, which stays below 2242^{24} units of 2−152^{-15}, converts the lane sum exactly to the integer ∑kam,nqn,k\sum_k a_{m,n} q_{n,k} (sign 2152^{15} acc), reduces across lanes in INT32 and applies the same epilogues. A row’s reduction is split over the 32 lanes of a SIMD group, 32 codes per lane per step, so a lane accumulates K/32K/32 products of at most 127 · 120 (16-block format) or 127 · 96 (SC-MXFP4) units of 2−152^{-15}, which stays below 2242^{24} for KK up to 35k or 44k (448 products per lane at K=14336K = 14336); for MoE models it gathers the selected experts’ rows itself. Folding the activation quantization into the GEMV prologue made decode 0.61–0.74× slower, because every threadgroup recomputes the row statistics.

Linear-layer latency

Measurement protocol

All measurements are on one Apple M5 Max (128 GB, macOS 26.6.2) on AC power in High Power mode. Kernel microbenchmarks use GPU timestamps; every repetition times all compared kernels back to back after a 1 s clock warm-up, and speedups are medians of per-repetition ratios. End-to-end measurements use wall clock in one process, with 1 s warm-up, two warm-up passes per configuration and rotated interleaving of methods and lengths; medians of 7–15 repetitions. The GPU runs in two clock regimes: isolated prompts on an idle GPU (burst) are faster than prompts following long ones (sustained), and MLX’s kernels slow down more than ours under sustained load, so we report both where they differ (e.g. dense prefill at 2048 tokens: 1.71× burst, 1.74× sustained). Every timing run waits until GPU utilization is at most 6%, and is rejected if utilization stays above 15% after it or if MLX MXFP4 deviates from its reference time by more than 7–10% (MLX’s kernels are the same in every run, so a slower MLX time flags interference; the check can only remove runs whose baseline was slowed); runs taken while another application used the GPU were discarded. Across a configuration’s repetitions, the range of prefill times is a median 11% (ours) and 13% (MLX) of the median. MLX baselines are timed in the same process as ours; the layer-level MLX comparison (Appendix D) is timed by wall clock in a separate process. Table 9 lists every prefill measurement behind the ranges in the text. The default MoE configuration unpacks expert weights inside the grouped GEMM below 256 rows per expert and in a separate pass from there on; where a forced variant runs the same kernels as the default (OLMoE: in-kernel at 512 tokens, two-pass from 2048; Qwen3.5: in-kernel up to 2048 tokens, two-pass at 8192), the medians differ by up to 10%, the run-to-run spread of these measurements.

Qwen3.8-Flash-Next. The model code is mlx-lm pull request 1788 at a pinned commit. Weights are read layer by layer from memory-mapped safetensors of the official FP8 checkpoint (E4M3 decoded exactly with a 256-entry table, times the per-128 × 128-tile scales). The 51B-parameter n-gram table is never loaded; the rows a batch addresses are gathered from the mapped FP8 shards on the host, for every system alike. Our system replaces only the W4A8 projections: the model’s own DeltaNet, attention and residual-mixing code runs with stub modules that return our GEMMs’ outputs, and the output projections’ inputs are quantized and multiplied by our kernels, so every operation between the GEMMs is the reference implementation’s. The layer-streamed perplexity keeps, for each method, the hidden states of all 145 windows (four residual streams, BF16, about 6 GB per method); a method’s weights are fake-quantized once per layer, experts 64 at a time. The perplexity tables use the model’s op-by-op reference recurrence for Gated DeltaNet; its Metal kernel, which both timed systems use, changes the perplexity of the first 8 windows by −0.3 to +0.6% and not the order of the formats (FP8: 2.326 vs. 2.325; fake-quantized SC-MXFP4 2.640 vs. 2.638, 16-block 2.514 vs. 2.521, MLX MXFP4 W4A16 2.756 vs. 2.764; on our kernels 2.647 vs. 2.633 and 2.511 vs. 2.511); the GSM8K runs use the Metal kernel. Packing the 125B model takes about 30 min for SC-MXFP4 and 52 min for the 16-block format. The three timed systems need 59, 62 and 65 GiB and are measured in separate processes, alternating, each session under the protocol above.

Accuracy ablations

GPTQ (calibrated on WikiText-2 train, non-sequential, exponents fixed) lowers Qwen3-1.7B W4A8 perplexity on the deployed kernels from 10.99 to 10.53 but lowers the zero-shot mean from 61.9 to 60.6 (ARC-e −4.1, ARC-c −3.3, HellaSwag and PIQA −1.1, LAMBADA +1.8), the known overfitting of GPTQ to a narrow calibration domain.

The integer-exact format on the kernels

The kernels never assume MXFP4: they read a weight as a 4-bit code times an integer block multiplier from a table, times one FP32 scale per row. Switching to 16-wide blocks with multipliers 1,…,101,\ldots,10 (4-bit block codes, 4.26 bits per weight; ∣q∣≤120\lvert q\rvert\le120) therefore changes only the packer.

On Qwen3-1.7B the packed weights equal the fake-quantized ones, all integer accumulations are exact, decode is bit-identical to prefill, and WikiText-2 perplexity on the kernels is 10.395 (reference 10.395) versus 10.985 for SC-MXFP4 and 10.508 for NVFP4 at the same INT8 activations, which removes 37% of SC-MXFP4’s perplexity loss. Speed is unchanged: in one session, prefill takes 22.3, 70.5 and 390 ms at 512, 2048 and 8192 tokens (SC-MXFP4: 22.0, 70.4, 395; MLX MXFP4: 31.5, 120.3, 655), i.e. 1.41–1.71× faster than MLX MXFP4, and decode runs at 357 vs. 359 tokens/s (Table 11). On the 125B MoE of § 7 the prefill gain stays (1.21–1.25× vs. 1.22–1.29× for SC-MXFP4) but decode, on our kernels, runs at 0.65× MLX’s speed (Table 6). Zero-shot accuracy on the kernels rises to a mean of 62.4 over six tasks (SC-MXFP4: 61.9; MLX MXFP4 W4A16 on the same matrices: 61.8; BF16: 64.5), with significant gains on LAMBADA (+2.8) and HellaSwag (+1.2) and no task worse by more than its standard error; the format needs no calibration data.

Spread statistics

Figure 3. Block-exponent spread per weight row. Spread above 3 (shaded) needs an exponent adjustment. Logarithmic axis; its bottom tick marks 0 (no rows).

For reference, on the six smaller models the relative weight MSE of NVFP4 is 8.99–9.02 × 10−310^{-3} and of MLX’s affine INT4 (group 64) 8.19–8.43 × 10−310^{-3}, against 12.5–12.7 × 10−310^{-3} for MXFP4 and SC-MXFP4 (Table 12). On Qwen3.5-35B-A3B, 1.1% of routed-expert gate/up rows are numerically dead (∣w∣≈10−38\lvert w\rvert\approx10^{-38}); their row scale is a float32 subnormal that the GPU flushes to zero, so they contribute exactly 0 instead of ∼10−37\sim10^{-37}, far below BF16 resolution.

Comparison with llama.cpp's 4-bit types

Table 13 compares llama.cpp’s 4-bit types with ours under one perplexity protocol. llama.cpp’s Q4_K_M (a k-quant that keeps some tensors at 6 bits) and IQ4_XS (non-uniform 4-bit levels with a 6-bit scale per 32 weights; IQ4_KS, the type closest to our format family, is only in the ik_llama.cpp fork) are quantized from the BF16 GGUFs without an importance matrix, like every format in this paper, and scored with llama-perplexity, which splits the text into disjoint 2048-token windows and scores only the second half of each. We score our fake-quantized formats the same way (same text, windows and scored tokens); our BF16 is within 0.11% of llama.cpp’s on all three models, so each format’s change against its own BF16 compares across the two. The coverage still differs: llama.cpp’s types also quantize the token embedding (4–6 bits; MXFP4_MOE: 8) and keep the output layer at 6–8 bits, Q4_K_M and IQ4_XS keep some attention and FFN matrices at 5–6 bits, and MXFP4_MOE keeps every matrix outside the routed experts at 8 bits, whereas ours quantize every attention and MLP matrix to 4 bits and keep the embedding and output layer in BF16.

With 16-bit activations, as llama.cpp’s Metal kernels compute, SC-MXFP4 loses slightly less than Q4_0 on the dense Qwen3-1.7B (13.1 vs. 14.9%) but more than every llama.cpp type on the two MoEs (6.9 and 4.9% against 1.7–4.3%), consistent with standard MXFP4’s power-of-two block scales being coarser than Q4_0’s 16-bit ones and with MXFP4_MOE keeping attention at 8 bits. The integer-exact 16-block format loses less than Q4_0 on both models where we ran it, less than IQ4_XS on Qwen3-1.7B and less than MXFP4_MOE on OLMoE, with every attention matrix at 4 bits; Q4_K_M, at about 5 bits per weight, is more accurate than our formats throughout. INT8 activations, which our kernels use, add 0.7–2.8 points. Q4_K_M and IQ4_XS prefill more slowly than Q4_0, so the prefill speedups over llama.cpp in Table 4, which use Q4_0, are the smallest over any of its 4-bit types we measured.

Table 13: llama.cpp’s 4-bit types and ours, scored the same way: llama-perplexity’s protocol on the WikiText-2 test set (disjoint 2048-token windows, second half of each scored; not comparable with Table 2), each against its own BF16. llama.cpp: b11706, Metal. Ours: fake-quantized as in Table 2 (W4A8 is what our kernels compute), about 4.1 (SC-MXFP4) and 4.3 (16-block) bits per weight in the quantized matrices, embedding and output layer in BF16. Speed: relative to Q4_0 in the same session, prefill at 512, 2048 and 8192 tokens (OLMoE: 4096; the faster of two batch sizes) and decode at a 512-token context; MXFP4_MOE’s speed is in Table 4. – = not run or not applicable.

Qwen3-1.7BOLMoE-1B-7BQwen3.5-35B-A3B
BF16 perplexity, llama.cpp / ours
BF168.436 / 8.4456.724 / 6.7275.131 / 5.133
llama.cpp: change against its BF16 (bits per weight of the file)
Q4_0+14.9% (4.90)+4.3% (4.54)+3.2% (4.55)
Q4_K_M+4.8% (5.15)+2.2% (4.87)+1.9% (4.89)
IQ4_XS+8.9% (4.73)+2.7% (4.34)+1.7% (4.37)
MXFP4_MOE–+4.2% (4.55)+1.8% (4.57)
Ours: change against our BF16
SC-MXFP4 W4A16+13.1%+6.9%+4.9%
SC-MXFP4 W4A8+15.9%+8.3%+5.6%
16-block, m≤10m \le10 W4A16+7.7%+3.8%–
16-block, m≤10m \le10 W4A8+10.4%+5.1%–
llama.cpp speed relative to Q4_0: prefill (range over lengths) / decode
Q4_K_M0.90–0.95 / 0.950.87–0.89 / 0.960.91–0.92 / 0.89
IQ4_XS0.82–0.96 / 0.980.84–0.90 / 1.000.92 / 0.92

Table 13. llama.cpp’s 4-bit types and ours, scored the same way: llama-perplexity’s protocol on the WikiText-2 test set (disjoint 2048-token windows, second half of each scored; not comparable with Table 2), each against its own BF16. llama.cpp: b11706, Metal. Ours: fake-quantized as in Table 2 (W4A8 is what our kernels compute), about 4.1 (SC-MXFP4) and 4.3 (16-block) bits per weight in the quantized matrices, embedding and output layer in BF16. Speed: relative to Q4_0 in the same session, prefill at 512, 2048 and 8192 tokens (OLMoE: 4096; the faster of two batch sizes) and decode at a 512-token context; MXFP4_MOE’s speed is in Table 4. – = not run or not applicable.

References

  1. [1]Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a Flash: Efficient large language model inference with limited memory, 2023. arXiv:2312.11514.arxiv.org/abs/2312.11514
  2. [2]Apple. Discover Metal 4 tensors and Metal Performance Primitives. WWDC25, 2025.
  3. [3]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems, 2024. arXiv:2404.00456.arxiv.org/abs/2404.00456
  4. [5]Mengzhao Chen, Meng Wu, Hui Jin, Zhihang Yuan, Jing Liu, Chaoyi Zhang, Yunshui Li, Jie Huang, Jin Ma, Zeyue Xue, Zhiheng Liu, Xingyan Bin, and Ping Luo. INT v.s. FP: A comprehensive study of fine-grained low-bit quantization formats, 2025. arXiv:2510.25602.arxiv.org/abs/2510.25602
  5. [6]Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, Keith Wyss, Mahdi Nazemi, Asit Mishra, Carlo del Mundo, Tijmen Blankenvoort, and Song Han. Four over six: More accurate NVFP4 quantization with adaptive block scaling, 2025. arXiv:2512.02010.arxiv.org/abs/2512.02010
  6. [7]Jack Cook, Hyemin S. Lee, Kathryn Le, Junxian Guo, Giovanni Traverso, Anantha P. Chandrakasan, and Song Han. Adaptive block-scaled data types, 2026. arXiv:2603.28765.arxiv.org/abs/2603.28765
  7. [8]Steve Dai, Rangharajan Venkatesan, Haoxing Ren, Brian Zimmer, William J. Dally, and Brucek Khailany. VS-Quant: Per-vector scaled quantization for accurate low-precision neural network inference. In Proceedings of Machine Learning and Systems, 2021. arXiv:2102.04503.arxiv.org/abs/2102.04503
  8. [9]Vage Egiazarian, Roberto L. Castro, Denis Kuznedelev, Andrei Panferov, Eldar Kurtic, Shubh ra Pandit, Alexandre Marques, Mark Kurtz, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Bridging the gap between promise and performance for microscaling FP4 quantization. In International Conference on Learning Representations, 2026. arXiv:2509.23202.
  9. [10]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.17323
  10. [12]Georgi Gerganov et al. llama.cpp: LLM inference in C/C++. https://github.com/ggml-org/llama.cpp, 2023.
  11. [14]Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. MLX: Efficient and flexible machine learning on Apple silicon. https://github.com/ml-explore/mlx, 2023.
  12. [15]Weiming Hu, Zihan Zhang, Haoyan Zhang, Chen Zhang, Cong Guo, Yu Feng, Tianchi Hu, Guanglin Li, Guipeng Hu, Junsong Wang, and Jingwen Leng. M2XFP: A metadata-augmented microscaling data format for efficient low-bit quantization, 2026. arXiv:2601.19213.arxiv.org/abs/2601.19213
  13. [16]Xing Hu, Zhixuan Chen, Dawei Yang, Zukang Xu, Chen Xu, Zhihang Yuan, Sifan Zhou, and Jiangyong Yu. MoEQuant: Enhancing quantization for mixture-of-experts large language models via expert-balanced sampling and affinity guidance, 2025. arXiv:2505.03804.arxiv.org/abs/2505.03804
  14. [17]Beichen Huang, Yueming Yuan, Zelei Shao, and Minjia Zhang. MiLo: Efficient quantized MoE inference with mixture of low-rank compensators. In Proceedings of Machine Learning and Systems, 2025. arXiv:2504.02658.arxiv.org/abs/2504.02658
  15. [18]Hugging Face. SmolLM3: smol, multilingual, long-context reasoner. https://huggingface.co/blog/smollm3, 2025.
  16. [19]Shun ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, and Takahiro Katagiri. DGEMM with Ozaki scheme I/II on FP4 tensor cores: A base-13 E2M1 limb representation, 2026. arXiv:2608.06812.arxiv.org/abs/2608.06812
  17. [20]Iwan Kawrakow. New SOTA quantization: 4.25 bpw IQ4_KS. https://github.com/ikawrakow/ik_llama.cpp/pull/83, 2024.
  18. [21]Qingyuan Li, Ran Meng, Yiduo Li, Bo Zhang, Yifan Lu, Yerui Sun, Lin Ma, and Yuchen Xie. Integer scale: A free lunch for faster fine-grained quantization of LLMs, 2024. arXiv:2405.14597.arxiv.org/abs/2405.14597
  19. [22]Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. In Machine Learning and Systems, 2025.
  20. [23]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantization with learned rotations. In International Conference on Learning Representations, 2025. arXiv:2405.16406.arxiv.org/abs/2405.16406
  21. [24]Yun-Chen Lo, Gu-Yeon Wei, and David Brooks. Nanoscaling floating-point (NxFP): NanoMantissa, adaptive microexponents, and code recycling for direct-cast compression of large language models, 2024. arXiv:2412.19821.arxiv.org/abs/2412.19821
  22. [25]Junyi Luo, Xinting Jiang, Tai-Hao Wen, Ruichen Qi, Minxing Chu, Hongyi Wu, Gregory Kielian, Ben Laurie, Qirui Zhang, Quan Cheng, Dennis Sylvester, and Mehdi Saligane. Heterogeneity-aware microscaling for efficient low-bit LLM inference, 2026a. arXiv:2608.03867.arxiv.org/abs/2608.03867
  23. [26]Yuanyong Luo, Jing Huang, Yu Cheng, Ziwei Yu, Kaihua Tang, Xinda Ma, Xin Wang, Anping Tong, Guipeng Hu, Yun Xu, Mehran Taghian, Peng Wu, Guanglin Li, Yunke Peng, Tianchi Hu, Minqi Chen, Michael Bi Mi, Hu Liu, Xiping Zhou, Junsong Wang, Qiang Lin, and Heng Liao. HiFloat4 format for language model inference, 2026b. arXiv:2602.11287.arxiv.org/abs/2602.11287
  24. [27]Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, and Xindian Ma. AR-CQuant: Boosting NVFP4 quantization with augmented residual channels for LLMs, 2026. arXiv:2601.07475.DOI
  25. [28]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. arXiv:1609.07843.arxiv.org/abs/1609.07843
  26. [29]MLX issue 4627. qmm: stage dequantized weights as int8/fp8 instead of fp16 to use the 8-bit matrix lanes. https://github.com/ml-explore/mlx/issues/4627, 2026.
  27. [31]Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Wei-jia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannah Hajishirzi. OLMoE: Open mixture-of-experts language models. In International Conference on Learning Representations, 2025. arXiv:2409.02060.arxiv.org/abs/2409.02060
  28. [32]Lorenz K. Müller, Philippe Bich, Jiawei Zhuang, Ahmet Çelik, Luca Benfenati, and Lukas Cavigelli. SINQ: Sinkhorn-normalized quantization for calibration-free low-precision LLM weights, 2025. arXiv:2509.22944.arxiv.org/abs/2509.22944
  29. [33]NVIDIA. Qwen3.8-Flash-Next-NVFP4 model card. https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4, 2026.
  30. [35]Open Compute Project. OCP microscaling formats (MX) specification, version 1.0. Technical report, Open Compute Project, 2023.
  31. [36]Qwen Team. Qwen3 technical report, 2025. arXiv:2505.09388.
  32. [37]Qwen Team. Qwen3.8-Flash-Next. https://huggingface.co/Qwen/Qwen3.8-Flash-Next, 2026.
  33. [38]Prabod Rathnayaka, Fabian Waschkowski, and Lukas Wesemann. BaseRT: Best-in-class LLM inference on Apple silicon via native Metal, 2026. arXiv:2607.00501.
  34. [39]Stefan Remke and Alexander Breuer. Hello SME! Generating fast matrix multiplication kernels using the Scalable Matrix Extension, 2024. arXiv:2409.18779.
  35. [40]Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, Lei Shao, Gaurav Kolhe, Dimitry Melts, Jasmine Klar, Renee L’Heureux, Matt Perry, Doug Burger, Eric Chung, Zhaoxia Deng, Sam Nagshineh, Jongsoo Park, and Maxim Naumov. With shared microexperts, a little shifting goes a long way. In International Symposium on Computer Architecture, 2023a. arXiv:2302.08007.
  36. [41]Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Rodriguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, and Eric Chung. Microscaling data formats for deep learning, 2023b. arXiv:2310.10537.arxiv.org/abs/2310.10537
  37. [44]Fabian Waschkowski, Prabod Rathnayaka, and Lukas Wesemann. BaseRT: Advancing best-in-class LLM inference with Apple M5 Neural Accelerators, 2026. arXiv:2607.19438.
  38. [45]Dan Woods. Flash-MoE: running a 397B-parameter mixture-of-experts model from SSD on Apple silicon. https://github.com/danveloper/flash-moe, 2026.
  39. [46]Xiaoxia Wu, Zhewei Yao, and Yuxiong He. ZeroQuant-FP: A leap forward in LLMs post-training W4A8 quantization using floating-point formats, 2023. arXiv:2307.09782.

Paper details

vlnc:W_q7mb36sz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Anthropic’s Claude, Gemini 4 Argon (bare)

View statement
Looking for related papers…
Contents