betaFeedback

WorksComputer Science

Beyond 4 bits: Data-Centric Post-training Quantization and Recovery

Abstract

Post-training quantization (PTQ) compresses the weights of a trained model with limited compute and data. Down to 4 bits it is near-lossless, but at 2 bits scalar PTQ configurations fall to chance-level accuracy. We study three choices usually treated as implementation details that matter little at 4 bits but decide 2-bit quality: the quantization representation, the calibration data, and the recovery fine-tuning data, across six dense and mixture-of experts models (ranging from 3.83.8 to 3232B parameters). Representation is the largest factor and vector codebooks restore usable models where scalar PTQ pipelines collapse. Most of the remaining loss in accuracy is on generative tasks, and the amount of performance drop is model-dependent. Furthermore, at a short recovery budget, calibration and recovery data are coupled. The best recovery data depends on the calibration data, so the two must be chosen jointly. The resulting recipe involves a vector codebook representation, a small task-aligned calibration mixture, and an inexpensive codebook-only recovery stage. Including codebooks, it uses about 2.32.3 effective bits per weight and retains 8989 to 97%97\% of full-precision question-answering (QA) accuracy where scalar PTQ is at chance, and, including recovery, uses 1.41.4 to 30×30\times less compute than the strongest quantization-aware-training (QAT) baseline.

Introduction

Post-training quantization (PTQ) is standard for serving large language models under tight memory budgets. At 4 bits (W4A16: 4-bit weights, 16-bit activations), it costs only a few accuracy points [21, 38, 50, 8, 42]. At 2 bits, scalar PTQ falls to chance-level accuracy [20, 11, 40 Figure 1a], and quantized reasoning models can overthink and loop [44]. Richer codebooks [54, 40, 20, 51] or learned quantization [41, 19, 16] trade compute for quality, yet 2-bit models still lag their 4-bit counterparts.

Three choices usually treated as implementation details govern 2-bit quality: the quantization representation, the calibration data that shape quantization, and the recovery data used to fine-tune the quantized model. They matter little at 4 bits but become consequential below 4 bits, where calibration and recovery data also interact, so tuning them separately can be misleading. We study 2-bit quantization and recovery across six dense and mixture-of-experts (MoE) models (3.8 to 32B parameters) and propose an actionable recipe (§ 5) for low-budget 2-bit quantization. Our primary observations are:

  • Representation dominates at 2 bits. Every scalar PTQ pipeline we test falls to random chance-level, while vector PTQ at ∼2.3 effective bits keeps 89–97% of a full-precision model’s question-answering accuracy without training, using 39–304× less compute than trained scalar methods (Table 4).

  • Loss of reasoning ability is model-dependent. Question-answering retention is similar across models, but HumanEval (codegen) retention spans 45% (Phi-4-mini, Qwen3-8B) to 81% (Qwen3-MoE), tracking the model and its scale more than the quantizer (Figure 4).

ModelBF16GSQBitDist.VPTQYAQAModelBF16GSQBitDist.VPTQYAQA
Phi-4-mini71.052.653.057.6{sup}N63.3Phi-4-mini73.217.918.429.5{sup}N37.4
Llama-3.1-8B73.157.060.363.3{sup}N65.7Llama-3.1-8B42.918.116.216.126.6
Qwen3-MoE72.629.455.964.8{sup}S66.7Qwen3-MoE61.80.045.753.0{sup}S62.4
OLMoE-1B-7B68.329.649.163.0{sup}S64.0OLMoE-1B-7B32.40.00.915.2{sup}S20.6

Table 4. Cross-paradigm comparison at 2 bits (P-fp16; OLMoE row P-bf16). Best AvgQA (a) and AvgGen (b) per model. VPTQ and YAQA select the best recipe per metric from their recipe grids. GSQ: layer-wise scalar PTQ; BitDistiller: scalar QAT; VPTQ, YAQA: vector PTQ. Shaded bold marks the best per model. N Native (unrotated) VPTQ, P-bf16, where it beats rotated VPTQ (Appendix Table 27). SExperts share one codebook per layer and projection type (2.09/2.14 effective bits

Figure 4. 2-bit retention of QA and chat-template HumanEval by model. Models are ordered by total parameters (hatched: MoE). Best no-recovery VPTQ/YAQA checkpoint per metric on our recipe grid within the 2.5-bit budget (YAQA on the MoEs). The dotted Llama-3.1-8B bar is completion-style HumanEval (base model, no chat template); completion-prompt AvgGen is in Table 33.

  • Calibration and recovery data interact. Choosing them jointly improves Phi-4-mini’s accuracy on generation tasks from 11.1% (web calibration) to 45.7% (math calibration, web recovery); the best recovery data flips with calibration on all three dense models we study (Table 1 and Table 2).

CalibrationRecoveryPhi-4-mini AvgQAPhi-4-mini GSM8KPhi-4-mini HEvalPhi-4-mini AvgGenLlama-3.1-8B AvgQALlama-3.1-8B GSM8KLlama-3.1-8B HEvalLlama-3.1-8B AvgGen
webnone63.115.56.711.165.111.67.39.5
webweb63.324.213.618.964.16.610.88.7
webmath62.833.814.224.063.79.911.210.5
mathnone58.353.220.136.762.033.421.327.4
mathweb60.056.934.645.762.427.417.122.2
mathmath58.760.124.642.361.428.214.621.4
BF1670.881.762.872.373.250.237.844.0

Table 1. The best recovery data depends on the calibration data (2-bit YAQA, P-bf16; mean of three recovery seeds, std in Table 21). Calibration: data used to compute the Hessians when quantizing. Recovery: data used for the short codebook-only fine-tuning after quantization (3,200 examples); none is the unrecovered base. The web data we use is FineWeb-Edu; and the math dataset is OpenR1-Math. Shaded bold: best AvgGen per model; red: recovery lowers AvgGen below its base. Follow-up evaluations on Qwen3-8B are in Table 23 in the Appendix.

Modelweb basemath baseAvgGenGSM8KHumanEvalmath vs. web
Phi-4-mini+5.1 [2.9, 7.4]−3.4 [−6.9, 0.2]−8.5 [−12.6, −4.3]−6.4−10.6+22.5
Llama-3.1-8B+1.8 [0.8, 3.6]−0.8 [−2.7, 1.2]−2.7 [−5.0, −0.7]−2.5−2.9+12.2
Qwen3-8B*+3.6 [1.1, 5.8]−5.8 [−9.0, −2.7]−9.4 [−13.3, −5.3]−3.8−15.1+16.5

Table 2. Switching recovery data helps one calibration base and not the other (change in AvgGen points, P-bf16; 95% bootstrap intervals over recovery seeds and test items). Math vs. web recovery: AvgGen after math recovery minus after web recovery, on each base. Interaction: math-base minus web-base column (up to rounding); negative means the best recovery data flips with the calibration data. Calibration: math minus web calibration, averaged over both recovery sources. * the held-out model our hypothesis is confirmed on.

The proposed recipe is a vector representation with a task-aligned calibration mixture and a short codebook-only recovery stage. With recovery, it narrows the gap to 4-bit models while using 1.4 to 30×30 \times less compute than the strongest quantization-aware-training (QAT) baseline; a cheap per-model check handles model dependence (§ 5). Upon acceptance, we will release our code, Hessian mixtures, and quantized models.

Related Work

Scalar PTQ. GPTQ ([21]) sets the near-lossless 4-bit weight-only baseline via approximate second-order error minimization; AWQ ([38]) and SmoothQuant ([57]) rescale salient channels, while QuaRot ([8]) and SpinQuant ([42]) rotate weights before rounding. The scalar PTQ configurations we evaluate collapse at 2 bits without any further quantization-aware training or recovery (§ 4.4).

Vector and ultra-low-bit quantization. Codebooks give better rate-distortion below 3 bits. QuIP ([11]) adds incoherence processing to adaptive scalar rounding; QuIP# ([51]) pairs it with lattice codebooks and fine-tuning. AQLM, GPTVQ, VPTQ, and QTIP ([20, 54, 40, 52]) use additive, hierarchical, or trellis-coded codebooks. YAQA ([53]) adds a Fisher-weighted rounding objective. Non-uniform ([32]) and 1-bit ([28, 61, 59]) methods also target lower precision to go even beyond 2-bits.

Trained scalar quantization. OmniQuant ([50]) learns clipping thresholds block-wise, and GSQ ([16]) learns scalar-grid assignments through a Gumbel-Softmax relaxation; we use GSQ as an intermediate-cost scalar baseline between PTQ and QAT.

Calibration data. PTQ usually calibrates on a few hundred C4 or WikiText-2 sequences ([21, 38, 50, 20]). Calibration data affect 4-bit models ([55]) and pruning ([9, 30]), and self-generated text can replace curated corpora ([56]). We find that calibration data effects 2-bit PTQ methods much more than on 4-bit methods and this effect is consistent across representations and recipe grids (Appendix § 10.§ 6).

Evaluating quantized LLMs. Quantization costs generation, reasoning, and knowledge-intensive tasks more than perplexity or multiple-choice QA as suggested by [29, 36, 31, 34, 39], and robustness depends on model size and token density [33, 47] (Dettmers & Zettlemoyer 2023). We confirm this gap at 2 bits and show that the degradation of generation quality depends on calibration, recovery data and model-scale.

Recovery for quantized models. Distillation-based recovery [26] narrows the quantization gap without full QAT. LLM-QAT and BitDistiller [41, 19] retrain all weights; LoRA and QLoRA [27, 18] add high-precision parameters; closest to ours, EfficientQAT [13] trains all weights block-wise, then quantization parameters end to end; ParetoQ [43] studies extremely low-bit QAT. AQLM and QuIP# fine-tune codebooks or continuous parameters after quantization. PV-Tuning [45] additionally also learns discrete assignments to the codebook vectors. In this work we use codebook-only recovery and show that its best data flips with the calibration data, so the two must be chosen jointly (§ 4.2).

Background and Method

We study post-training quantization followed by recovery fine-tuning at the same format and bitwidth.

Scalar and Vector Quantization

Scalar quantization maps each weight to one of 2b2^b levels, x^=s(q−z)\hat{x}=s(q-z), with integer code qq, per-group scale ss, and optional zero-point zz. Error-aware PTQ, such as GPTQ [21], minimizes output error on a calibration distribution XX with input Hessian HX=Ex∼X[xx⊤]H_X=\mathbb{E}_{x\sim X}[xx^\top]:

min⁡W^∈QEx∼X∥(W−W^)x∥22=min⁡W^∈QTr⁡[(W−W^)HX(W−W^)⊤](1)\min_{\hat{W}\in Q}\mathbb{E}_{x\sim X}\left\|(W-\hat{W})x\right\|_2^2=\min_{\hat{W}\in Q}\operatorname{Tr}\left[(W-\hat{W})H_X(W-\hat{W})^\top\right] \tag*{(1)}

where QQ is the set of representable weights. Vector quantization maps each dd-dimensional sub-vector wiw_i to an entry of a learned codebook C={ck}k=1KC=\{c_k\}_{k=1}^{K}, w^i=cai\hat{w}_i=c_{a_i}, using weight correlations that scalar grids miss. VPTQ [40] solves Equation (1) over codebooks. Structured-curvature methods such as KronQ [35] and YAQA [53] use Kronecker-factored full-model KL curvature and YAQA rounds onto trellis-coded sets.

Calibration mixtures. The objective depends on the data only through HXH_X. A mixture with weights αj≥0\alpha_j\ge0, ∑jαj=1\sum_j\alpha_j=1, over domains with Hessians HjH_j has Hα=∑jαjHjH_\alpha=\sum_j\alpha_jH_j would allow one to approximate the Hessian for different data distributions, cheaply. We sweep mixtures by reweighting precomputed Hessians without re-collecting Hessians on actual dataset mixtures; each mixture still needs its own quantization pass. For YAQA we mix the Kronecker input and output factors separately, approximating re-estimation on different dataset mixtures.

Recovery Fine-Tuning

Recovery freezes the quantized backbone and trains only quantization parameters: the codebook CC with assignments fixed, or the scales ss (and zz). QAT backpropagates through the quantizer with a straight-through estimator [41] and keeps optimizer state for all weights (≈14P\approx14P bytes for PP parameters). Recovery keeps the bitwidth and needs only a teacher and student (≈2P+bP/8\approx2P+bP/8 bytes). We distill from the full-precision model with L=αLCE+(1−α)τ2KL⁡(σ(zt/τ)∥σ(zs/τ))\mathcal{L}=\alpha\mathcal{L}_{\mathrm{CE}}+(1-\alpha)\tau^2\operatorname{KL}\left(\sigma(z_t/\tau)\mathbin\Vert\sigma(z_s/\tau)\right), where ztz_t and zsz_s are teacher and student logits, σ\sigma is the softmax, τ\tau the temperature, and α∈(0,1)\alpha\in(0,1) unless noted. Our recovery pipeline includes codebook-only recovery. In the appendix we cover alternative losses and logit truncation (Appendix § 10.7), a scale-only recovery scalar control (Appendix § 10.7), and finetuning with low-rank adapters [27, 18, 58, 37], which add half-precision parameters (Appendix § 10.11).

Experiments and Results

Experimental setup

Models and quantizers. We evaluate Phi-4-mini [7] (3.8B, instruct), Llama-3.1-8B [23] (base), and Qwen3-30B-A3B [60] (reasoning MoE, ∼\sim3B active). Qwen3-8B (reasoning) checks the calibration and recovery choices; OLMoE-1B-7B [46] (instruct MoE) and Qwen3-32B (reasoning, dense) extend the comparisons to smaller MoEs and large-scale dense models, respectively. The scalar baselines we use are SpinQuant [42] and QuaRot [8] at group size 128. Furthermore, we observe that learned rotations do not help at 2 bits (Appendix § 10.§ 3). The vector-quantization baselines we study are VPTQ [40] (2162^{16}-entry codebook over 8-dimensional vectors) and YAQA [53], a Fisher-weighted trellis quantizer, both in the same incoherence-rotated frame; we report native (unrotated) VPTQ where it is stronger. We also compare vector quantization against GSQ [16] (scalar PTQ with learned assignments) and BitDistiller [19] (scalar QAT). In our experiments we quantize attention and MLP projections, including every MoE expert. Routers, embeddings, norms and the language modeling head stay in half precision. Our ablation on the router precision in Appendix § 10.10, suggests that the router is pretty robust to quantization. Counting indices, codebooks, and scales, every method uses 2.09–2.33 effective bits per weight. In the MoEs, VPTQ shares codebooks across experts, while YAQA operates with a single large codebook for all weights. We evaluate dequantized weights, report packed-size storage estimates, and count dominant model passes, amortizing Hessian collection over the calibration sweep. Configurations and protocols are in Appendices § 10.§ 3, § 10.§ 2.

Calibration and evaluation. For calibration data we primarily focus on web, instruction, math (OpenR1-Math), code, and their mixtures. We report an AvgQA metric which averages over seven QA tasks: MMLU (5-shot) [24], WinoGrande [49], and LAMBADA [48], scored by accuracy, and ARC-Easy, ARC-Challenge [14], HellaSwag [62], and PIQA [10], scored by length-normalized accuracy (all 0-shot except MMLU). We report an AvgGen metric which averages GSM8K (5-shot, strict match) [15] and HumanEval (0-shot, pass@1) [12]; we also report IFEval (0-shot, prompt-strict) [63]. Results use bfloat16 evaluation (P-bf16) unless a table is marked P-fp16 (float16); the two differ only marginally in our analyses.

Calibration and recovery data are not separable

The best recovery data depends on the calibration data. We quantize Phi-4-mini and Llama-3.1-8B with web or math calibration data, then apply short codebook-only recovery (3,200 examples) on web or math data (Figure 2, Table 1). Math recovery helps the web-calibrated base but not the math-calibrated one, giving a negative interaction on both models with 95% intervals excluding zero (Table 2). Calibration is the larger factor and math calibration raises generation under either recovery source. Furthermore, whether to recover or not is model-dependent. Every recovery stage helps the instruction-tuned Phi-4-mini. However for the pretrained base Llama-3.1-8B, math calibration and no recovery procedure works best. The interaction sign also holds on a logit scale and these results cover one domain pair at a short budget (Table 8).

Figure 2. Calibration and recovery data are not separable at 2 bits. 2-bit YAQA bases are calibrated on web or math text (OpenR1-Math), then unrecovered (hollow markers) or given short codebook-only recovery budget on web or math text (3,200 examples). The shared yy-axis is AvgGen as a percentage of each model’s BF16 AvgGen; labels give absolute AvgGen. Means ±\pm std over three recovery seeds (P-bf16). Circles mark the best cell. Boxes give the interaction contrast and its 95% hierarchical-bootstrap interval. Detailed results are in Table 1.

ModelCalibration recipeAvgQA baseAvgQA +rec.GSM8K baseGSM8K +rec.HumanEval baseHumanEval +rec.IFEval baseIFEval +rec.
Phi-4-miniweb63.162.915.542.214.028.029.021.8
Phi-4-minimath58.360.453.262.534.147.631.227.7
Phi-4-minimath+web (0.5/0.5)62.563.238.853.517.733.533.133.1
Phi-4-miniweb+math (0.7/0.3)62.863.339.953.318.332.337.329.4
Phi-4-minimw64: math+web (0.6/0.4)61.763.244.058.317.139.032.229.9
Phi-4-miniBF1670.881.775.055.8
Llama-3.1-8Bmath+web (0.5/0.5)65.664.822.721.214.014.67.85.9
Llama-3.1-8Bweb+math (0.7/0.3)65.564.818.418.010.417.76.57.8
Llama-3.1-8Ball-4 (equal four-way)65.164.419.916.818.322.68.38.9
Llama-3.1-8BBF1673.250.237.89.4
Qwen3-MoEweb66.067.873.876.075.076.823.0§22.0
Qwen3-MoEBF1672.690.092.728.3

Table 8. Codebook recovery is model-dependent. Each row is a 2-bit YAQA model quantized with the listed calibration data. base: no recovery; +rec.: after long codebook-only recovery whose recovery data is the same mixture as the calibration data (forward KL, BF16 teacher). HumanEval uses each instruction-tuned model’s chat template (Phi-4-mini, Qwen3-MoE) and completion prompts for the base Llama-3.1-8B. Teal/red: change beyond two paired standard errors. P-bf16 except the Qwen3-MoE AvgQA, GSM8K, and IFEval cells (P-fp16). §P-bf16, mean of three evaluations.

Prospective confirmation. Before collecting Qwen3-8B data, we pre-registered its 2×22 \times2 design, analysis, and three predictions (Appendix A.9): a negative interaction, web recovery at least as good as math recovery on the math-calibrated base, and a larger interaction on HumanEval than on GSM8K. All of these predictions hold (Table 2) on this model. The interaction also stays negative on held-out tasks, with fresh Phi-4-mini calibration samples, and with disjoint recovery-data samples (Figure 3; Appendices A.9 and Table 25). On prompt-independent tasks alone (MBPP, MATH-500), the interaction effect is clear only on Phi-4-mini (Appendix A.9).

Figure 3. The interaction stays negative across robustness checks. Interaction contrast with 95% intervals; left of zero, the best recovery data flips with the calibration data. Circles: bootstrap over recovery seeds and test items, on the main benchmarks or on held-out tasks (HumanEval+ reuses HumanEval prompts). Squares: intervals over three independently re-quantized bases per arm (Welch) or three disjoint recovery-data samples (tt).

Table 25: Interaction across disjoint recovery-data samples (Phi-4-mini, original bases, P-bf16; GSM8K / HumanEval per cell, interaction on AvgGen). The original row is the 3,200-example sample of Table 1 (mean over three seeds). The 20k and 40k rows are samples drawn through the same shuffle buffer after skipping 20k/40k source rows (seed 42). They share no rows with the original.

Sampleweb→webweb \to webweb→mathweb \to mathmath→webmath \to webmath→mathmath \to mathInteraction
original24.2 / 13.633.8 / 14.256.9 / 34.660.1 / 24.6−8.5-8.5
+20k rows23.2 / 13.433.6 / 12.256.3 / 35.460.1 / 25.0−7.9-7.9
+40k rows25.1 / 14.634.4 / 14.658.4 / 31.761.5 / 27.4−5.3-5.3

Table 25.

What the interaction does not imply. It is not a rule to “add the domain absent from calibration”: on Phi-4-mini every short recovery improves over its base, and calibration changes only which recovery source helps more.

Scalar PTQ collapses in the tested pipelines

Scalar PTQ has a sharp 2-bit cliff. At 4 bits the best scalar method stays close to BF16; at 2 bits every tested scalar pipeline falls to the random QA floor with 0.0 accuracy on generation, on all three models (Table 3 and curves in Appendix A.15).

ModelBF16 QABF16 GenScalar 4-bit QAScalar 4-bit GenScalar 2-bit QAScalar 2-bit GenVector 2-bit QAVector 2-bit Gen
Phi-4-mini71.073.269.3Q^{Q}65.8S^{S}29.3S^{S}0.063.3Y^{Y}12.2Y^{Y}
Llama-3.1-8B73.142.971.8S^{S}38.0Q^{Q}29.0Q^{Q}0.065.1Y^{Y}9.6Y^{Y}
Qwen3-30B-A3B72.661.870.3S^{S}66.2S^{S}29.4S^{S}0.066.0Y^{Y}54.0Y^{Y}

Table 3. Quantization ladder (web calibration, no recovery; P-fp16). Best QA and generation per precision; superscripts mark the winning method: SpinQuant (SS), QuaRot (QQ; expert-wise on the MoE), YAQA (YY), VPTQ (VV). Red: collapse to the random floor; shaded: 2-bit vector pipelines.

Vector codebooks revive 2-bit models. Vector pipelines raise 2-bit QA into the mid-60s from the 20s floor on all three families, and YAQA stays ahead in a matched-bit control (Table 13).

Method (bits/weight)web AvgQAweb AvgGenmath AvgQAmath AvgGen
VPTQ (2.00 / 2.33)51.51.241.06.5
VPTQ, model-wide (2.25 / 2.26)58.44.347.511.8
YAQA (2.25 / 2.27)63.312.258.537.4

Table 13. Vector-quantizer rate control (Phi-4-mini).

Comparisons with trained scalar quantization

On question answering, the vector pipelines match or beat GSQ and BitDistiller on every model; on generation, YAQA beats both (Table 4). Within our recipe grid and bit budget, YAQA leads on every model. GSQ collapses to the random-accuracy floor on the MoEs.

Calibration chiefly controls generation

With the quantizer fixed, calibration data mainly moves generation: across recipes, AvgGen varies by 13.5–26.7 points per model, while AvgQA varies by only 3.9–7.5 (Table 5). At 4 bits, the same sweep moves scalar models by at most 1.6 QA and 7.9 generation points (Table 17). Math calibration gives the best generation on the dense models, a targeted mixture on Qwen3-MoE, and broad web-heavy data the best QA.

RecipePhi-4-mini QAPhi-4-mini GenLlama-3.1 QALlama-3.1 GenQwen3-MoE QAQwen3-MoE Gen
Web63.312.265.19.666.054.0
Math58.537.462.126.664.561.8
Code55.918.760.822.662.848.9
Math+web .5/.562.624.765.717.866.756.5
Web+math .7/.363.027.765.515.165.856.2
All-4 .25 each58.819.965.220.166.059.3
MWI .4/.4/.259.517.162.712.266.562.4
Δ7.526.74.917.03.913.5

Table 5. Calibration mainly changes generation (2-bit YAQA, P-fp16; bold: best; Δ: range over the full sweep).

Model4-bit scalar (QA / Gen spread)2-bit YAQA (QA / Gen)
ModelSpinQuantQuaRotKronQ2-bit YAQA (QA / Gen)
Phi-4-mini1.4 / 2.90.5 / 6.51.3 / 5.07.5 / 26.7
Llama-3.1-8B1.6 / 4.41.0 / 2.60.6 / 7.34.9 / 17.0

Table 17. Spread across calibration domains, 4-bit scalar versus 2-bit vector. Max minus min over five calibration domains (4-bit) or the full recipe sweep (2-bit YAQA). AvgQA is the seven-task mean. AvgGen is the GSM8K/HumanEval mean.

Generation retention is model-dependent. QA retention is similar across models, but HumanEval retention ranges from under half to over 80% (Figure 4). Within Qwen3 the dense 32B model retains far more generation than the dense 8B one, so model scale is necessary for strong retention. Furthermore, the precision of the MoE router barely changes accuracy (Appendix § 10.10).

Codebook-only recovery is model-dependent

Table 8 pairs each 2-bit base with long codebook-only recovery on the same recipe data. On Phi-4-mini, recovery changes QA little but roughly doubles HumanEval. On Qwen3-MoE it changes both little; on Llama-3.1-8B base model, QA decreases in all three long runs and generation has no consistent gain. IFEval falls on most Phi-4-mini recipes under completion prompts and is mostly, but not uniformly, preserved under the chat template (Appendix Table 35). Furthermore recovery does not help 2-bit models which hit the random floor. Scale-only recovery on a collapsed scalar 2-bit model leaves it near the floor (Appendix § 10.7).

Under the deployment prompt, short recovery need not cost instruction following. With the chat template, web recovery of the math-calibrated Phi-4-mini base raises GSM8K from 55.0 to 58.9, HumanEval from 34.1 to 43.9, and IFEval from 40.3 to 44.5 (four recovery seeds); math recovery raises GSM8K further (61.5) but lowers IFEval to 37.9 (Table 36). The best recovery data still flips with calibration and math recovery gives the highest AvgGen from the web base and web recovery from the math base.

Calib.RecoveryGSM8KHumanEvalIFEval
Phi-4-mini (YAQA)
BF1681.473.869.7
webnone13.015.243.1
webweb24.418.943.8
webmath36.521.344.2
mathnone55.034.140.3
mathweb58.943.944.5
mathmath61.537.237.9
Qwen3-30B-A3B (YAQA, web calibration)
BF1691.192.782.6
webnone58.075.072.6
webweb62.476.874.7

Table 36. Recovered checkpoints under the chat template (bfloat16; GSM8K 5-shot, humaneval instruct pass@1, IFEval prompt-strict).

Recovery data matter more than distillation hyperparameters. The type of knowledge distillation loss, the teacher precision, temperature, and top-kk truncation change AvgQA little. A 4-bit teacher matches a BF16 one, and dropping the teacher keeps most generation gain (Appendix § 10.7). With fixed web recovery data, generation gains 6.3 points by 3,200 examples and then plateaus, while QA rises slowly with more data (Figure 5). Recovery with low-rank-adaptor is in Appendix § 10.11.

Figure 5. Recovery gains saturate early. Gain over the unrecovered base as the recovery budget grows (Phi-4-mini, math-calibrated 2-bit YAQA, codebook-only recovery on web data; Table 22). Generation gains 6.3 points by 3,200 examples and then plateaus; QA rises slowly.

Overthinking under quantization

We reproduce the overthinking failure mode of Lotfi et al. [44] on Qwen3-30B-A3B, a reasoning model with an explicit thinking mode at 2-bits. On MATH-500 [25] levels 4–5 with greedy decoding, we compare the BF16 ceiling with the best 4-bit checkpoint and the expert-shared 2-bit VPTQ web/mixed 0.7/0.3 checkpoint of Table 10 in the Appendix. Accuracy is exact match on the final answer. “Markers/1k” counts hesitation and branch-forking tokens (e.g., “Wait”, “But”, “Alternatively”) per 1,000 generated tokens. The loop score is the repeated 4-gram fraction. Table 6 shows that 2-bit quantization amplifies overthinking without collapse. Accuracy falls (0.675 → 0.45), and the hesitation-marker rate, loop score, and length-cap hit rate rise. A small repetition penalty reduces looping and improves 2-bit accuracy (Table 7). Figure 15 in the appendix shows the effect of a repetition penalty.

Table 6: Overthinking under 2-bit quantization (Qwen3-30B-A3B, MATH-500, greedy). 2-bit amplifies hesitation and looping and moderately lowers accuracy without collapsing. “Cap-hit” is the length-cap hit rate. “R-then-A” is the reached-then-abandoned rate among wrong answers. We use 40 problems, and the 2-bit model is expert-shared VPTQ (2.09 bits).

PrecisionAccuracyMean CoTCap-hitMarkers/1kLoop (4-gram)R-then-A
BF160.67549280.2515.50.390.69
4-bit0.65052650.2512.40.410.50
2-bit0.45059000.4523.70.670.41

Table 6.

Table 7: Decoding penalties on the 2-bit model (Qwen3-30B-A3B, MATH-500, 2-bit). A repetition penalty reduces looping and chain-of-thought length and improves accuracy. The same penalty also helps BF16.

Rep. penaltyMarker penaltyAccuracyMean CoTCap-hitMarkers/1kLoop (4-gram)
1.000.42558840.47523.00.66
1.020.47551560.37510.20.62
1.060.50047260.3502.20.59
1.200.57550520.30015.00.40
1.300.62547990.25013.00.30

Table 7.

ModelCalibrationAvgQAGSM8KHumanEvalHumanEvalc
Qwen3-MoEweb64.869.525.664.6 (70%)
Qwen3-MoEinstruction64.867.519.567.7 (73%)
Qwen3-MoEcode63.866.939.076.2 (82%)
Qwen3-MoEmath59.872.217.759.8 (64%)
Qwen3-MoEall-4 mixture63.775.118.376.2 (82%)
Qwen3-MoEweb/math 0.7/0.366.076.322.667.1 (72%)
Qwen3-MoEmath+web 0.5/0.565.074.424.464.6 (70%)
Qwen3-MoEweb/mixed 0.7/0.367.275.634.872.6 (78%)
OLMoE-1B-7Bweb62.711.87.97.3 (21%)
OLMoE-1B-7Bmath+web63.018.711.612.2 (35%)

Table 10.

Figure 15. A repetition penalty helps BF16 and 2-bit alike (Qwen3-30B-A3B, 40 MATH-500 level-4/5 problems, greedy, same penalty for both models; the 2-bit model is expert-shared VPTQ). The penalty improves both precisions and lowers the 4-gram loop score for both. The matched-decoding gap remains (0.15 at 1.3).

A Practical Recipe for Accurate 2-Bit Models

Table 9 and the highlighted box below give the recipe. The representation default holds across six models, including two MoEs; the calibration defaults come from the three primary model families. Whether and how to recover depends on the model and the calibration data, so the recipe adds a cheap per-model check. Screening kk calibrated bases costs kk quantization passes, 2k2k short recoveries (≈100 PFLOP each for a 4B-parameter model and 3k3k evaluations, far less than QAT.

DecisionRecommendation
Representationvector codebook (YAQA).
Calibrationtask-aligned Hessian mixture
Recoveryshort codebook-only KD, or none for base models.
Recovery datachoose jointly with calibration.
KD settingsdefaults (τ=1\tau=1).
Teacher4-bit (matches BF16) or none.
MoE routerskeep in half precision.
ReportingQA and generation separately with repetition penalty applied.

Table 9. 2-bit decision guide (§ 4; KD and teacher rows: Phi-4-mini).

Recipe: five steps to an accurate 2-bit model

  1. Use a vector codebook (e.g., YAQA); scalar PTQ collapses at 2 bits.

  2. Calibrate on target-domain data; mix in broad web text when QA matters.

  3. Choose recovery data together with calibration: for each calibrated base, compare no, web, and target-domain recovery.

  4. Keep recovery short and optional: codebook-only on 3,200 examples; keep it only if it clears a preset margin without losing QA or instruction following.

  5. Evaluate as deployed: Report QA and generation separately against a matched full-precision reference. Evaluate quantized model at higher repetition penalty to avoid overthinking behavior.

Conclusion and Future work

Conclusion At 2 bits, quality depends heavily on representation, calibration data, and recovery. A vector codebook restores most collapsed accuracy and a task-aligned calibration mixture and short codebook-only recovery stage narrow the generation gap at much less compute than QAT. The data choices are not separable at a short recovery budget: calibration domain drives generation, and the best recovery data depends on it. Across three models and a held-out composite, math recovery helps a web-calibrated base but not a math-calibrated one (clearest on Phi-4-mini). Tuning the two data choices separately can therefore select the wrong recovery data, and whether to recover must be validated per model. Most remaining loss is in generation, whose measurement depends on prompt format and numerical precision.

Limitations and Future Work. We measure base and data-sample variability only on Phi-4-mini, and replicate bases share one rotation and one recovery seed. The interaction covers one domain pair and one quantizer (YAQA) at a short budget; we did not re-run the 2×22 \times2 design at long budgets. We leave these studies for future work.

Use of AI Tools

All ideas presented in this paper are original to the authors or built upon the related work cited herein. AI-based tools were used only in an assistive capacity: to sharpen the clarity of the authors’ initial drafts, to prepare initial drafts, to help typeset the paper in LaTeX, to rephrase sections, to improve the presentation of figures and tables, to correct grammatical errors and make sentences more salient, and to act as a coding assistant that implements the authors’ ideas. All technical content, claims, and conclusions were conceived, verified, and approved by the authors.

Acknowledgments

We are deeply grateful to Hari VK and Andrey Rybalchenko (Microsoft Research, Cambridge) for their extensive work to make application of incoherence rotations automatic and robust. This work allowed us to reliably test the effect of incoherent rotations on a large number of dense and mixture-of-experts transformer models with different architectures. The work underpins our rotated scalar baselines and the incoherence-rotated frame in which our vector quantizers operate on the MoE models, and it was essential for extending this study from dense models to Qwen3-30B-A3B and OLMoE-1B-7B.

References

Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025.

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240, 2024.

Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, and Shiwei Liu. Is c4 dataset optimal for pruning? an investigation of calibration data for llm pruning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18089–18099, 2024.

Additional Experimental Details and Results

In this section we provide additional details about out pipeline, evaluations and ablation studies on different knobs of the 2-bit quantization pipeline.

Evaluation protocols and result provenance

Our results use four protocols. All of them run with lm-evaluation-harness 0.4.12, transformers 4.53.3, and greedy decoding. They use 5-shot MMLU and GSM8K (strict match) and 0-shot HumanEval (pass@1) and IFEval (prompt-strict). Most controlled comparisons use one protocol. Table 4 mixes labeled native-VPTQ P-bf16 entries into a P-fp16 comparison.

P-fp16 (completion prompts, float16) covers the main sweeps in Table 3, Table 4, and Table 5. It also covers the Qwen3-MoE row of Table 8, the calibration sweeps, the recovery budget curve, and the Core6 seed table.

P-fp32 repeats P-fp16 in float32 for the two Phi-4-mini checkpoints affected by float16 decoding (Appendix § 10.8).

P-bf16 is the canonical controlled protocol. It uses completion prompts, bfloat16, per-item logs, and a finite-logit probe. It covers the Phi-4-mini and Llama-3.1-8B 2×22 \times2 design (Figure 2 and Table 21) and the Phi-4-mini and Llama-3.1-8B rows of Table 8. It also covers teacher-free recovery, learned SpinQuant, calibration-side controls, and the dense-32B and OLMoE evaluations.

Calib.RecoveryAvgQAGSM8KHEvalAvgGenIFEval
Phi-4-mini (three recovery seeds)
BF1670.881.762.872.355.8
webnone63.115.56.711.129.0
webweb63.3 ± 0.124.2 ± 0.313.6 ± 0.918.9 ± 0.529.8 ± 1.3
webmath62.8 ± 0.133.8 ± 0.714.2 ± 0.924.0 ± 0.423.4 ± 1.6
mathnone58.353.220.136.731.2
mathweb60.0 ± 0.156.9 ± 0.634.6 ± 0.745.7 ± 0.131.3 ± 0.6
mathmath58.7 ± 0.160.1 ± 0.524.6 ± 2.542.3 ± 1.027.0 ± 0.5
Llama-3.1-8B (seeds 42, 1, 2)
BF1673.250.237.844.09.4
webnone65.111.67.39.59.2
webweb64.1 ± 0.16.6 ± 0.210.8 ± 0.78.7 ± 0.44.7 ± 0.2
webmath63.7 ± 0.19.9 ± 0.211.2 ± 0.410.5 ± 0.24.3 ± 0.1
mathnone62.033.421.327.49.1
mathweb62.4 ± 0.127.4 ± 0.817.1 ± 0.622.2 ± 0.53.9 ± 0.3
mathmath61.4 ± 0.128.2 ± 0.514.6 ± 0.621.4 ± 0.44.6 ± 0.2

Table 21. Single-protocol bfloat16 evaluation of the 2 × 2 design. Mean ± sample std over three recovery seeds per cell. Every row uses bfloat16 weights and activations and the seven-task AvgQA (with MMLU). none rows are the no-recovery 2-bit YAQA bases. The BF16 row averages two independent evaluations. See Table 2 for interaction statistics. “Math” is OpenR1-Math.

P-chat covers chat-template tables. It uses each model’s chat template with thinking disabled and at most 1,024 generated tokens. It runs in bfloat16 unless marked.

evaluation details

Table 10: MoE VPTQ with expert-shared codebooks. One codebook per decoder layer and projection type, shared by all experts (2.09 / 2.14 effective bits; P-bf16). HumanEvalc uses the chat template. Parentheses give retention relative to BF16 under the same protocol. This protocol is bfloat16 for Qwen3-MoE (BF16 score 92.7) and float16 for OLMoE (BF16 score 34.8).

Evaluation and reproducibility. We score tasks with an lm-evaluation-harness-based split-task orchestrator ([22]). Multiple-choice tasks use log-likelihood scoring. MMLU and GSM8K use 5-shot prompting. GSM8K and HumanEval use greedy decoding with temperature 0 and no sampling. They are scored by strict match and pass@1. Compute accounting uses 128 scalar-calibration sequences and 128–2,048 vector-Hessian sequences per domain. Seed 42 is the default.

Per-method quantization configurations

Table 11 summarizes the quantization and recovery settings. Unless noted, methods target 2-bit weights at group size 128 and use the protocol of §A.2 with seed 42.

MethodParadigmEff. b/wTrained parametersCalibration / recovery data
QuaRotscalar PTQ (rotation)2.12none (GPTQ)128 sequences
SpinQuantscalar PTQ (rotation)2.12none (GPTQ)128 sequences
GSQtrained scalar2.12grid assignments0.1B-token, 10 epochs
BitDistillerfull QAT2.12all weights1-epoch mixture
VPTQvector PTQ2.09–2.33codebook (post-hoc)per-domain Hessians
YAQAvector PTQ2.26–2.31codebook (post-hoc)per-domain Hessians
PTQ+RFTcodebook recovery2.26–2.31codebook onlytask-aligned mixtures

Table 11. Per-method configuration summary. Paradigm, effective bits per quantized weight including codebooks, trained parameters, and calibration or recovery data. All methods use group size 128 and the protocol of §A.2.

QuaRot and SpinQuant (scalar PTQ). QuaRot and SpinQuant apply an incoherence rotation before scale-only ≈\approx2.12-bit GPTQ quantization (Appendix § 10.§ 2). QuaRot uses a fixed randomized Hadamard transform with activation ordering, Hessian-ordered columns, error feedback, and damping 0.1. SpinQuant uses residual-stream and per-head rotations (R1R_1, R2R_2) at randomized-Hadamard initialization, with experts rotated separately on Qwen3-MoE. Both methods consume 128 calibration sequences and add no trained weights.

Learned-rotation in Spinquant. We test learned SpinQuant rotations on the two dense primary models (Table 12). We optimize R1R_1 and R2R_2 with Cayley SGD for 100 steps on the same 128 calibration sequences from each domain. We use learning rates 1.5 and 0.25, then the same GPTQ quantization. Learned rotations leave all 2-bit models at the random-accuracy floor and barely change best-domain 4-bit AvgQA (Table 12). The scalar cliff is not from the unoptimized rotation. We could not fit the full-precision copy needed for Qwen3-30B-A3B, so its SpinQuant entries keep the Hadamard initialization.

ModelBitsFixed AvgQALearned, η=1.5\eta=1.5 AvgQALearned, η=1.5\eta=1.5 AvgGenLearned, η=0.25\eta=0.25 AvgQALearned, η=0.25\eta=0.25 AvgGen
Phi-4-mini469.268.4–69.363.2–68.968.2–69.762.9–70.4
Phi-4-mini229.328.8–29.30.028.7–29.40.0
Llama-3.1-8B471.870.4–71.636.5–40.570.9–71.937.1–41.8
Llama-3.1-8B229.828.7–29.40.029.3–29.70.0

Table 12. Learning SpinQuant’s rotations does not move the 2-bit floor. AvgQA (seven-task mean) and AvgGen (GSM8K/HumanEval mean) for Cayley-optimized R1/R2R_1/R_2 (100 steps, learning rate η\eta), shown as ranges over web, math, mixed, and code calibration. Fixed is the best-domain AvgQA of Hadamard-initialized SpinQuant used elsewhere (Table 37). Learned rows use bfloat16.

VPTQ (vector PTQ). VPTQ maps each 8-dimensional weight sub-vector to one of 65,536 (2162^{16}) codebook entries (2.00 index bits). Dense models use one 1 MiB FP16 codebook per linear layer (2.12–2.33 effective bits). MoEs use expert-shared codebooks (2.09–2.14). VPTQ uses channel-independent second-order optimization over a shared Hadamard frame with input Hessians collected by calibration domain. The rate control of §A.4 uses one model-wide 262,144-entry (2182^{18}) codebook, giving 2.25 index bits and 2.26 effective bits.

YAQA (vector PTQ). YAQA uses a trellis codebook with Fisher-weighted, Kronecker-factored curvature from the full-model KL objective and 2.25 index bits. Sign vectors and a shared lookup table bring the rate to 2.26–2.31 effective bits. YAQA also retains a fine-tuning-ready codebook under the same incoherence rotation, with curvature estimated from 128–2,048 sequences per calibration domain.

GSQ (trained scalar). GSQ learns discrete grid assignments through a Gumbel-softmax relaxation initialized with GPTQ (512 sequences). It optimizes one transformer block at a time with detached inputs. It runs 10 epochs over a 0.1B-token calibration set (48,828 sequences of length 2048) with group size 128 and a Lion optimizer.

Table 13: Vector-quantizer rate control (Phi-4-mini). Rows give index / effective bits per weight. Model-wide denotes one 2182^{18}-entry codebook shared by all linear layers, matching YAQA’s effective bits (P-bf16). Its HumanEval is 0 on both recipes. AvgQA is the seven-task mean; AvgGen averages GSM8K and HumanEval (pass@1). Bold marks best per column.

Table 14: AvgQA by vector quantizer and calibration domain. Bold marks the better method within each model/domain. S^{S}Expert-shared codebooks (2.09 effective bits, P-bf16).

ModelMethodWebInstructMathCodeMixture
Phi-4-miniVPTQ51.553.941.044.447.8
Phi-4-miniYAQA63.356.858.555.958.8
Llama-3.1-8BVPTQ57.659.845.753.454.5
Llama-3.1-8BYAQA65.163.962.160.864.3
Qwen3-30B-A3BVPTQS^{S}64.864.859.863.863.7
Qwen3-30B-A3BYAQA66.064.964.562.866.0

Table 14. AvgQA by vector quantizer and calibration domain.

BitDistiller (full QAT). BitDistiller trains a Hadamard-rotated 2-bit model with asymmetric activation clipping and CAKLD (β≈0.84\beta\approx0.84). We run one FSDP full-shard epoch across 8 GPUs. We use sequence length 1024, per-device batch 1, gradient accumulation 8 (effective batch 64), learning rate 2×10−52 \times10^{-5} with cosine decay, and 10% warmup. The recovery mixture has ∼\sim500k dense-model sequences and ∼\sim50k Qwen3-MoE sequences.

Codebook-only recovery (PTQ+RFT). Recovery freezes the quantized backbone and discrete assignments, then updates only the continuous codebook, leaving the effective bitwidth unchanged. Scalar scale-only recovery is a limited control (Appendix § 10.7). We do not test zero-point recovery. This stage distills from a BF16 or a ×4\times4 times smaller 4-bit teacher. The objective optimized is a temperature-1 forward-KL loss plus next-token cross-entropy. We optimize it with AdamW at learning rate 10−410^{-4} and a cosine schedule. Long Phi-4-mini runs (Table 8 and the main run) use 8 GPUs, per-device batch 1, gradient accumulation 2, sequence length 4,096 (65,536 tokens per optimizer step), 5,500 steps (≈\approx360M tokens), and 550 warmup steps on the recipe mixture. Llama uses the same layout for ≈\approx185M tokens. Qwen3-MoE uses sequence length 512, gradient accumulation 2, a device-sharded teacher, and ≈\approx1.2M tokens. Controlled short runs cover the 2×22 \times2 design, budget curve, seeds, and teacher-free control. They run on one GPU with per-device batch 1, gradient accumulation 8, sequence length 1,024 (8 sequences per step), and 40 warmup steps. The 51,200-example point keeps 8 sequences per step with 8-GPU data parallelism. Ablations and LoRA are in §A.7 and §A.11.

Per-domain vector-quantizer comparison

Table 14 compares VPTQ and YAQA by calibration domain. YAQA wins all completed dense-model comparisons. Expert-shared VPTQ stays close on Qwen3-MoE web, instruction, and code, and on Phi-4-mini math and code.

YAQA’s advantage is not a bit-rate advantage. On Phi-4-mini, VPTQ trails YAQA even when it spends more bits. With per-linear (2162^{16}, 8) codebooks, VPTQ uses 2.33 effective bits and YAQA uses 2.27. At matched bits, a model-wide 2182^{18}-entry VPTQ codebook (2.26 bits) narrows but does not close the web gap and trails further on math (Table 13).

Cross-paradigm 2-bit comparison

Table 15 compares trained scalar checkpoints with the strongest in-budget vector PTQ result on the same model in AvgQA. The vector checkpoint leads across models. BitDistiller performs full-model QAT, GSQ performs layer-wise discrete optimization, and VPTQ and YAQA are PTQ without training the weights.

ModelScalar baselineAvgQABest VQAvgQAΔ\Delta
Phi-4-miniGSQ52.6YAQA63.3+10.7
Phi-4-miniBitDistiller53.0YAQA63.3+10.3
Llama-3.1-8BGSQ57.0YAQA65.7+8.7
Llama-3.1-8BBitDistiller60.3YAQA65.7+5.4
Qwen3-MoEBitDistiller55.9YAQA66.7+10.8

Table 15. Vector PTQ versus trained scalar quantization at 2 bits. AvgQA (↑\uparrow). “Best VQ” selects the strongest task-complete VPTQ or YAQA checkpoint for each model within the 2.5-bit budget. The last column is Best VQ minus the scalar baseline.

Complete cross-model calibration sweep

Table 16 keeps recipes evaluated on all three main model families. The final row summarizes the task-complete spread. The per-domain scalar sweep measures calibration sensitivity at 4 bits. Table 17 reports the AvgQA and AvgGen spread across web, instruction, math, code, and mixed domains, alongside the 2-bit YAQA spread (Table 16 and Table 5). The 4-bit scalar spreads are smaller. The comparison crosses representations and recipe grids because 2-bit scalar models are at the random floor for every domain. It does not isolate a bitwidth-only effect (Table 17).

GroupRecipeCodeWebMathInstrPhi-4-mini QAPhi-4-mini GenLlama-3.1-8B QALlama-3.1-8B GenQwen3-30B-A3B QAQwen3-30B-A3B Gen
SingleWeb163.312.265.19.666.054.0
SingleInstruct156.810.763.911.864.949.8
SingleMath158.537.462.126.664.561.8
SingleCode155.918.760.822.662.848.9
Standard mixmath+web0.50.562.624.765.717.866.756.5
Standard mixweb+math0.70.363.027.765.515.165.856.2
Standard mixweb+instruct0.60.458.714.665.716.566.651.4
Standard mixmath+instruct0.60.457.318.264.617.664.253.8
Standard mixcode+web0.60.458.018.264.321.665.259.4
Standard mixcode+math0.60.456.219.461.623.563.953.7
Standard mixcode+math+web0.40.30.357.823.464.622.065.961.8
Standard mixall-40.250.250.250.2558.819.965.220.166.059.3
Targetedmwi4420.40.40.259.517.162.712.266.562.4
Δcomplete\Delta_{\mathrm{complete}}7.526.74.917.03.913.5

Table 16. Task-complete 2-bit YAQA calibration sweep. QA and generation means from complete per-task evaluations. Blank weight entries mean absent sources. Dashes mark cells with only aggregate AvgQA/HEval. Δcomplete\Delta_{\mathrm{complete}} excludes dashed result cells.

SourceGSM8K (test)HumanEvalMATH-500
web (FineWeb-Edu)0.00.00.0
math (OpenR1-Math-220k)0.00.64.8
code (Evol-Instruct-Code-80k)0.05.51.0
instruction (Alpaca-cleaned)0.02.40.6

Table 18. Benchmark overlap (13-gram) with the first 20,000 rows of each calibration/recovery source. Entries give the percentage of evaluation items sharing at least one 13-gram with the source text.

AxisSettingMMLUWGARC-eARC-cHSLMBPIQAGSM8KHEvalAvgQA
References
BF16 ceiling68.771.980.358.972.766.078.381.964.671.0
2-bit YAQA, no recovery51.763.673.648.561.759.873.744.98.561.8
Loss (soft cross-entropy uses teacher targets and equals forward KL up to the teacher entropy)
Forward KL53.766.175.050.463.960.575.149.619.563.5
Reverse KL53.665.774.750.463.860.174.950.220.163.3
Soft cross-entropy (teacher)53.765.875.050.563.960.574.949.720.163.5
Teacher precision
BF1653.766.175.050.463.960.575.149.619.563.5
4-bit matched53.766.574.650.463.861.275.148.718.963.6
Temperature
τ=0.5\tau= 0.553.466.475.050.763.860.374.848.419.563.5
τ=1.0\tau= 1.053.766.175.050.463.960.575.149.619.563.5
τ=2.0\tau= 2.053.766.174.850.363.560.174.950.219.563.3
Logit support
k=128k = 12853.665.775.050.463.960.775.149.420.163.5
k=512k = 51253.665.875.050.564.060.575.049.120.163.5
k=1024k = 102453.665.675.150.663.960.674.949.820.163.5
Full vocabulary53.766.175.050.463.960.575.149.619.563.5
Recovery data
Web53.766.175.050.463.960.575.149.619.563.5
Instruct53.266.775.050.263.959.875.149.414.663.4
Math53.665.373.248.763.360.074.258.410.462.6
Code53.466.073.049.463.462.074.553.321.363.1
Mixture53.766.174.850.364.060.374.855.017.763.4
Selected run (best QA/generation balance): mixture data53.766.174.850.364.060.374.855.017.763.4

Table 19. Codebook-only recovery ablations. Per-task accuracy on Phi-4-mini 2-bit YAQA. AvgQA is the mean of the seven QA tasks. GSM8K and HEval are reported separately.

Codebook-only recovery ablations

The codebook-only ablation uses the Phi-4-mini 2-bit YAQA mw64 checkpoint with math+web calibration. The reference configuration is forward KL, τ=1\tau= 1, full-vocabulary distillation, a BF16 teacher, and web recovery data. Each block varies one factor. Note that fixed-base reference differs from the web row of Table 8, which recovers the web-calibrated base.

Teacher-free recovery. We recovered the math-calibrated Phi-4-mini base of Appendix § 10.8 on the same 3,200 web examples with plain next-token cross-entropy and no teacher, keeping every other setting fixed (seed 42, bfloat16 evaluation). It reaches the same qualitative level as forward-KL distillation from a 4-bit teacher (three seeds, Table 21) and stays above the no-recovery base. In this single-seed Phi-4-mini setting, the teacher improves the result, but recovery data accounts for most of the effect.

Scale-only recovery of a scalar 2-bit model. We applied the same recovery recipe (forward KL, web data, 1,024-token context) to a scalar 2-bit Phi-4-mini model. This setup uses Hadamard rotation, GPTQ with activation ordering, group size 128, and scale-only recovery. We trained only the per-group scales, with the integer codes frozen (≈P/128\approx P/128 parameters). Recovery moves AvgQA only slightly off the floor, while GSM8K and HumanEval stay near zero. With fixed codes, rescaling does not rescue the scalar model. The same 3,200-example web-recovery recipe helps 2-bit codebook models of Phi-4-mini (Table 1 and Table 8). BitDistiller trains all weights and still trails no-recovery vector PTQ in generation (Figure 1b).

Figure 1. Phi-4-mini at 2 bits. (a) Scalar PTQ (SpinQuant, [42]) falls to the random floor on question-answering (QA) tasks as bitwidth decreases; a 2-bit vector representation (YAQA, [53]) stays higher. (b) Generation (AvgGen, the GSM8K/HumanEval mean) versus compute: trained scalar baselines (BitDistiller, [19]; GSQ, [16]) trail no-recovery vector PTQ, and short codebook-only recovery improves generation.

Controlled calibration–recovery interaction

To test additivity, we crossed web and math calibration bases with web and math recovery data on Phi-4-mini. All cells use codebook-only recovery with frozen assignments, 3,200 examples, context length 1,024, a 4-bit teacher, and seed 42. IFEval is held out from main-study AvgQA/AvgGen recipe selection. Core6 averages ARC-Easy, ARC-Challenge, PIQA, HellaSwag, WinoGrande, and LAMBADA and is not directly comparable to the paper’s seven-task AvgQA.

The recovery-data effect depends on the calibration base. Math recovery helps from the web base but not from the math base (Table 20; statistical summaries in Table 2). From the math base, math rather than web recovery trades higher GSM8K for lower HumanEval, Core6, and held-out IFEval (Table 20). Calibration and recovery data therefore cannot be tuned independently.

CalibrationRecoveryCore6GSM8KHumanEvalAvgGenIFEval
WebWeb65.723.514.018.828.1
WebMath65.035.114.024.626.6
MathWeb61.656.331.143.732.0
MathMath60.160.125.042.627.5

Table 20. Calibration and recovery interact at matched budget. AvgGen is the mean of GSM8K and HumanEval. The final column is held out from recipe selection. Core6 averages 6 QA tasks (excluding MMLU).

Quantity control. Holding the web base, FineWeb-Edu recovery source, context length, and all optimization settings fixed, increasing recovery data from 800 to 3,200 examples leaves Core6 nearly unchanged and improves AvgGen. The complete budget curve uses the math-calibrated base (Table 22).

ExamplesTokensPFLOPAvgQAAvgGen
0†0^{\dagger}0058.537.4
2000.2M658.838.5
8000.8M2559.141.8
3,2003.3M10160.343.7
12,80013.1M40361.044.2

Table 22. Recovery accuracy versus data budget (Phi-4-mini, 2-bit YAQA, math-calibrated base, FineWeb-Edu codebook-only recovery, seed 42, P-fp16). Tokens are examples ×1,024. PFLOP assumes 8N8N per token. †No-recovery base (Table 8).

Data-source control. At the same 3,200-example budget, with every other recovery setting fixed, replacing filtered FineWeb-Edu with unfiltered C4 leaves Core6 and held-out IFEval essentially unchanged but lowers both generation tasks and AvgGen. The generation benefit is not explained by recovery-set size alone. FineWeb-Edu and C4 differ in filtering, distribution, and collection history, so this control identifies a data-source effect rather than isolating filtering quality. Over the tested range, quality and quantity both matter for recovery data. The calibration-side analogue follows.

Calibration-side controls. We also varied the Hessian calibration set of the web-calibrated Phi-4-mini YAQA base (no recovery, bfloat16 evaluation, seven-task AvgQA). At a matched 256 sequences, filtered FineWeb-Edu and unfiltered C4 Hessians give similar QA and only small generation differences. Growing the FineWeb-Edu set from 256 to the 2,048 sequences of our web base mainly improves QA while leaving AvgGen unchanged. The source effect is small and generation-side (the HumanEval gap is three problems), so we treat it only as initial experimental evidence.

Generalization to a second model family. Repeating the 2 × 2 design on Llama-3.1-8B reproduces the same sign with smaller magnitude than Phi-4-mini. Filtered FineWeb-Edu recovery improves AvgGen while leaving Core6 essentially unchanged.

Recovery budget. All cells use 3,200 × 1,024 ≈ 3.3M recovery tokens. This is about 100 PFLOP at 8N per token on Phi-4-mini (N=3.84×109N = 3.84 \times10^9) and roughly 1% of the ∼360M-token main recovery run. The math-calibrated, web-recovered cell reaches the same range as the mw64 (math web 0.6 and 0.4) checkpoint under comparable protocols. The budget curve in Table 22 shows generation saturating near 3,200 examples.

Seed replicates of the 2 × 2 design. We repeat every cell of the 2 × 2 design on both models with two further recovery seeds (1 and 2) under the protocol of Table 20. On Llama-3.1-8B the sign is stable across recovery seeds. On Phi-4-mini every cell is similarly stable.

Held-out model and generation benchmarks

We verify our findings on interaction between calibration and recovery on the Qwen3-8B model in Table 23. We scored every 2 × 2 checkpoint and base on MBPP (3-shot), HumanEval+ (0-shot), and MATH-500 (4-shot, Minerva prompt). The three-task interaction is negative on all three models, with intervals excluding zero for Phi-4-mini and Qwen3-8B (Table 24). It is largest on HumanEval+, which reuses HumanEval’s prompts. The math-calibrated bases are much stronger than the web-calibrated ones, and on Llama-3.1-8B the no-recovery math base again beats both recovered cells.

Calib.RecoveryAvgQAGSM8KHEvalAvgGenIFEval
BF1670.987.561.674.525.0
webnone63.353.07.930.515.2
webweb65.1 ± 0.156.2 ± 0.414.6 ± 0.035.4 ± 0.216.2 ± 0.6
webmath64.8 ± 0.058.2 ± 0.519.9 ± 0.739.0 ± 0.216.8 ± 0.9
mathnone60.777.920.149.016.5
mathweb63.9 ± 0.178.8 ± 0.134.6 ± 0.456.7 ± 0.116.8 ± 0.9
mathmath62.3 ± 0.077.0 ± 0.224.8 ± 0.750.9 ± 0.415.0 ± 0.5

Table 23. Pre-registered Qwen3-8B 2 × 2 design (2-bit YAQA, P-bf16; mean ± std over three recovery seeds). none denotes the no-recovery base.

Table 24: Held-out interaction contrasts by benchmark (95% hierarchical bootstrap). HumanEval+ shares HumanEval’s prompts; MBPP and MATH-500 are independent. Bold marks intervals that exclude zero.

BenchmarkPhi-4-miniLlama-3.1-8BQwen3-8B
MBPP−5.6-5.6 [−9.7-9.7, −1.5-1.5]+0.9+0.9 [−3.0-3.0, 4.54.5]−6.8-6.8 [−10.8-10.8, −2.7-2.7]
HumanEval+−12.2-12.2 [−19.7-19.7, −4.7-4.7]−3.7-3.7 [−7.1-7.1, −0.4-0.4]−13.0-13.0 [−20.5-20.5, −6.1-6.1]
MATH-500−2.7-2.7 [−6.7-6.7, 1.11.1]−2.0-2.0 [−4.8-4.8, 0.80.8]+1.7+1.7 [−2.5-2.5, 5.95.9]
MBPP+MATH-500−4.2-4.2 [−7.2-7.2, −1.3-1.3]−0.6-0.6 [−2.9-2.9, 1.81.8]−2.6-2.6 [−5.6-5.6, 0.50.5]
All three−6.8-6.8 [−10.1-10.1, −3.6-3.6]−1.6-1.6 [−3.6-3.6, 0.30.3]−6.0-6.0 [−9.2-9.2, −3.0-3.0]

Table 24.

By benchmark (Table 24), the prompt-independent MBPP and MATH-500 contrast excludes zero only for Phi-4-mini. Llama’s MBPP contrast is positive, and MATH-500 shows no consistent effect.

MoE router precision

Router precision does not explain Qwen3-MoE’s strong 2-bit retention. Across two YAQA checkpoints, 2-bit routers remain essentially matched to fp16 routers (Table 29).

Checkpointfp168-bit4-bit2-bit
YAQA math+web68.568.568.568.6
YAQA MWI44268.268.168.068.0

Table 29. Qwen3-MoE router-precision ablation. Expert weights remain at their vector-quantized 2-bit setting; only router precision changes. The checkpoint labels are those in the router-ablation runs, scored by their QA average. Compare only within-row precision deltas.

LoRA-based recovery

LoRA [27] adapters raise attainable generation accuracy for codebook recovery but change the trainable and stored parameterization, so we report them separately. HumanEval uses adapters folded into materialized weights. Across recovery datasets, QA stays stable, while the best generation task depends on recovery domain, with the same pattern on Llama (Table 30). The best folded Phi checkpoint reaches 34.8 IFEval prompt-strict accuracy.

ModelRecovery dataMMLUWGARC-eARC-cHSLMBPIQAGSM8KAvgQAHEval
Phi-4-miniWeb55.865.776.351.065.561.575.345.364.417.7
Phi-4-miniInstruct55.665.375.551.065.760.775.550.764.230.5
Phi-4-miniMath54.864.376.350.065.160.574.463.363.619.5
Phi-4-miniCode55.164.673.750.465.461.875.052.563.732.9
Phi-4-miniMixture56.064.675.050.965.060.575.256.963.929.3
Llama-3.1-8BWeb49.769.976.648.873.469.078.723.366.613.4
Llama-3.1-8BMath49.769.277.448.373.369.278.233.766.514.6
Llama-3.1-8BCode48.769.176.848.673.768.478.226.166.226.2
Llama-3.1-8BMixture50.270.378.348.774.567.578.230.966.819.5

Table 30. LoRA + codebook recovery by recovery dataset. Per-task accuracy after folding the adapters into each materialized checkpoint. AvgQA averages the seven QA tasks. The Phi-4-mini aggregates cover all reported configurations.

Table 31 reports the remaining one-factor-at-a-time ablation axes. On Phi-4-mini, changing loss, temperature, logit support, or teacher precision leaves AvgQA nearly unchanged, while HumanEval varies more (Table 31).

AxisSettingPhi-4-mini AvgQAPhi-4-mini HEvalLlama-3.1-8B AvgQALlama-3.1-8B HEval
BaselineForward KL, τ=1\tau= 1, full vocab64.417.766.613.4
LossReverse KL64.121.366.515.2
LossCross-entropy64.419.566.614.0
Temperatureτ=0.5\tau= 0.564.116.566.314.0
Temperatureτ=2\tau= 264.018.9––
Logit supportk=128k = 12864.417.766.314.0
Logit supportk=512k = 51264.517.766.513.4
Logit supportk=1024k = 102464.516.566.513.4
Teacher4-bit matched64.315.966.313.4

Table 31. LoRA + codebook one-factor-at-a-time ablations. Aggregate results for the non-data axes. The baseline uses forward KL, τ=1\tau= 1, full-vocabulary logits, a BF16 teacher, and web recovery data. Dashes denote unavailable complete task aggregates.

Table 26: Interaction across independently quantized bases (Phi-4-mini, 2-bit YAQA, P-bf16). Each row is one quantized base. The original row is the base of Table 1 (mean over three recovery seeds), and replicates use seed 42. The recovery-data effect is math minus web recovery (AvgGen).

Calib.Base\multicolumn{3}{c}{web recovery}\multicolumn{3}{c}{math recovery}Effect
GSM8KHEvalAvgGenGSM8KHEvalAvgGen
weboriginal24.213.618.933.814.224.0+5.1+5.1
webreplicate 129.98.519.242.36.724.5+5.3+5.3
webreplicate 225.710.418.039.617.128.3+10.3+10.3
mathoriginal56.934.645.760.124.642.3−3.4-3.4
mathreplicate 157.333.545.461.027.444.2−1.2-1.2
mathreplicate 261.432.947.264.123.243.6−3.6-3.6

Table 26.

ModelMethodWebInstructMathCodeMixed
Phi-4-miniVPTQ-rotated51.4 / 1.553.7 / 7.540.9 / 6.444.6 / 11.447.6 / 7.8
Phi-4-miniVPTQ-native57.4 / 7.657.6 / 12.947.8 / 13.251.0 / 29.556.2 / 26.0
Phi-4-miniYAQA-rotated63.1 / 11.156.8 / 10.958.3 / 36.755.8 / 19.258.8† / 19.9†
Llama-3.1-8BVPTQ-rotated57.5 / 2.659.6 / 10.345.8 / 8.553.2 / 15.854.4 / 13.2
Llama-3.1-8BVPTQ-native63.3 / 5.562.4 / 11.457.8 / 16.057.6 / 19.762.0 / 17.1
Llama-3.1-8BYAQA-rotated65.1 / 9.563.7 / 12.362.0 / 27.460.7 / 22.964.2 / 18.5

Table 27. Native versus rotated VPTQ, and YAQA on the same calibration domain (AvgQA / AvgGen, P-bf16; †P-fp16). Effective bits are 2.33 / 2.27 for VPTQ (Phi-4-mini / Llama-3.1-8B) and 2.27 / 2.26 for YAQA.

MethodDomainAvgQAHEval
BitDistillerWeb56.11.8
BitDistillerInstruct35.90.0
BitDistillerMath38.00.0
BitDistillerCode32.40.6
BitDistillerMixture60.322.6
BitDistillerMixed-500k54.915.9

Table 28. Calibration-domain sensitivity of BitDistiller on Llama-3.1-8B. AvgQA and HumanEval for the five base-domain checkpoints plus the larger Mixed-500k recipe.

Qwen3-8B

We evaluate Qwen3-8B ([60]), a third dense model, at 2 bits with math+web calibration. YAQA beats VPTQ on aggregate and generation metrics, and on all evaluated tasks except ARC-Easy (Table 32). Appendix A.9 also reports web and math YAQA bases with crossed recovery.

MethodMMLUWGARC-eARC-cHSLMBPIQAGSM8KAvgQAHEval
BF1675.067.880.956.474.964.677.787.371.064.0
VPTQ56.762.475.746.259.652.073.064.460.82.4
YAQA63.765.774.148.365.455.673.268.663.77.9

Table 32. Qwen3-8B (dense) at 2 bits, math+web calibration. Entries are per-task accuracy. AvgQA averages the seven QA tasks. HumanEval reports pass@1. Bold marks the better of VPTQ and YAQA per column.

Separating sparsity from model scale with dense Qwen3-32B

As a size control for sparse Qwen3-30B-A3B, we quantize dense Qwen3-32B ([60]) with 2-bit VPTQ on web Hessians. The dense model retains nearly all QA and completion-style generation (Table 33). HumanEval retention rises with total size within the Qwen3 family. Qwen3-MoE is highest, dense Qwen3-32B is between it and Qwen3-8B, and OLMoE is close to the smaller dense models. IFEval retention is similar for the two ∼30B models. We also evaluate another MoE, the OLMoE ([46]) model; both the 2-bit vector variants recover most of the BF16 AvgQA score (Table 34).

Table 33: Generation retention by model size and sparsity. The completion-style column gives the best 2-bit AvgGen relative to BF16. The HumanEval column gives the range of 2-bit HumanEval retention under the chat template. Parameters are total / active (billions). {sup}*Single recipe (VPTQ, web). {sup}†Llama-3.1-8B is a base model without a chat template, so HumanEval uses completion-style bfloat16 evaluation. {sup}§`Math-calibrated base under P-bf16. The separate math+web checkpoint uses P-fp16.

ModelTypeParams (B)CompletionHumanEval
Phi-4-minidense3.851%19–45%
OLMoE-1B-7BMoE6.9 / 1.364%49%
Llama-3.1-8Bdense8.062%19–73%{sup}†`
Qwen3-8Bdense8.266%{sup}§`42–45%
Qwen3-32Bdense32.8100%{sup}*`68%{sup}*`
Qwen3-30B-A3BMoE30.5 / 3.3101%81%

Table 33.

Table 34: OLMoE-1B-7B-Instruct at 2 bits. The model has 64 non-fused experts. {sup}S`Expert-shared codebooks (2.14 effective bits, P-bf16).

ConfigurationAvgQAGSM8KHEvalPTB ppl
BF16 ceiling68.333.731.114.7
VPTQ{sup}S` web62.711.87.922.2
VPTQ{sup}S` web+math63.018.711.621.7
YAQA web+math64.026.015.219.3

Table 34.

Additional analyses

Figures 6–7 summarize two cross-cutting trends from our 2-bit sweeps. Figures 8–10 and Table 37 add four supporting analyses. First, QA and next-word prediction survive 2-bit vector PTQ, while generation is fragile. Second, generation retention is model-dependent (Figure 9). Third, vector PTQ improves the nominal quality–rate trade-off. Fourth, the scalar 2-bit cliff holds across scalar PTQ methods and different models.

ModelMethodBF164-bit2-bit
Phi-4-miniSpinQuant71.069.229.3
Phi-4-miniQuaRot71.069.329.8
Llama-3.1-8BSpinQuant73.171.829.8
Llama-3.1-8BQuaRot73.171.729.3
Qwen3-MoESpinQuant72.670.829.4
Qwen3-MoEQuaRot72.670.329.3

Table 37.

Figure 9. QA and generation are distinct axes. AvgQA versus AvgGen for every method and model at 2 bits. Points are the per-metric best entries of Table 4 (QA and generation may come from different recipes). Vector PTQ recovers QA; generation stays near BF16 only for Qwen3-MoE. Scalar 2-bit PTQ collapses on both axes.

Generation under the 2-bit cliff

Figure 12 reproduces the main-paper generation teaser on Phi-4-mini. AvgGen is the mean of GSM8K and HumanEval. SpinQuant collapses by 3-bit and drops to zero at 2-bit, while a vector codebook (YAQA, mw64 base) restores part of the gap. Figure 1(b) adds recovery. Figures 13 and 14 extend Figure 1(b) to Llama-3.1-8B and Qwen3-30B-A3B. Each 2-bit method is plotted at its Table 4 score against training compute, with the 4-bit and 2-bit scalar ladder for reference. Vector PTQ (YAQA/VPTQ) matches or beats GSQ and BitDistiller at less compute. Dense Llama loses more generation at 2 bits, while Qwen3-MoE stays near its under-measured completion-prompt reference and retains more under the chat template.

Figure 12. Generation cliff on Phi-4-mini. Scalar SpinQuant generation collapses by 3-bit and vanishes at 2-bit. A vector codebook restores part of the gap. AvgGen averages GSM8K and HumanEval.

Overthinking behavior at 2-bits: Additional Details

Figure 15 presents the effect of repetition penaty on accuracy and 4-gram lopp score at bf16 and 2-bit precisions.

Qualitative examples. Two representative MATH-500 generations from the 2-bit Qwen3-MoE checkpoint illustrate bounded solving and looping under default greedy decoding.

2-bit, bounded and correct (Level-4, gold = 5\sqrt{5})

“I need the radius of a right cylindrical tank filled at 20π20\pi m3^{3}/hr while the level rises 4 m/hr. Since V=πr2hV = \pi r^{2}h, dVdt=πr2dhdt\frac{dV}{dt} = \pi r^{2}\frac{dh}{dt}, so 20π=πr2⋅4⇒r2=5⇒r=520\pi= \pi r^{2} \cdot4 \Rightarrow r^{2} = 5 \Rightarrow r = \sqrt{5}.” The model emits a bounded 1,740-token chain and terminates with the correct 5\boxed{\sqrt{5}}.

2-bit, looping failure (Level-5, gold = 6−5i6 - 5i)

“I need to rotate zz around cc by π/4\pi/4: w=(z−c)eiπ/4+cw = (z-c)e^{i\pi/4}+c. Let me compute z−c…z-c\ldots” The chain reasons coherently at first. It then collapses into a degenerate repetition of one algebraic line (loop score 0.94). It fills all 8,192 tokens without ever emitting a boxed answer, so it is scored wrong. With a repetition penalty of 1.3 the same model solves this problem.

ModelConfigurationIFEval
Phi-4-mini-instructBF1656.6
2-bit YAQA (math+web)37.2
+ codebook recovery36.0
Llama-3.1-8BBF168.9
2-bit YAQA7.4
Qwen3-30B-A3BBF1628.3
2-bit YAQA24.4
OLMoE-1B-7B-instruct2-bit YAQA25.5

Table 35. IFEval prompt-strict accuracy (completion prompts).

Figure 6. Calibration composition primarily moves generation, not QA. Across 2-bit YAQA calibration recipes, QA varies modestly while generation moves more. The full sweep is in Table 5.

Figure 7. In recovery, data composition dominates; KD hyperparameters move it little across the tested grid. Loss type, temperature, top-k, and teacher precision have smaller effects, especially for generation.

Figure 8. Which capabilities survive 2-bit vector PTQ. Per-task retention of one representative balanced 2-bit vector checkpoint per model relative to BF16. QA and next-word prediction are largely retained, while generation (GSM8K, HumanEval) is fragile.

Table 37: The 2-bit scalar cliff is universal across rotation methods. Best-calibration-domain AvgQA at 16, 4, and 2 bits for two scalar PTQ methods (best calibration domain, so entries can exceed the web-only values of Table 3). All methods stay near BF16 at 4 bits and collapse at 2 bits. QuaRot is applied through expert-wise rotation on Qwen3-MoE.

Figure 10. The tested vector pipelines improve the nominal 2-bit quality–rate trade-off. Effective bits-per-weight versus AvgQA and AvgGen on Phi-4-mini. At ∼ 2 bits/weight, a vector codebook (YAQA and VPTQ) sits far above the scalar 2-bit floor but below scalar 4-bit quality.

Figure 11. The 2-bit scalar cliff on the other models (analogs of Figure 1(a)). Scalar PTQ (SpinQuant) AvgQA falls from near-lossless at 4 bits to the random-accuracy floor at 2 bits, while a 2-bit vector codebook (YAQA, web calibration) recovers much of the gap. AvgQA is the seven-task mean.

Llama-3.1-8B accuracy and generation versus training compute

Figure 13. Llama-3.1-8B accuracy and generation vs. training compute (analog of Figure 1(b)). Scalar = crimson, vector = teal. The xx-axis is training compute (log scale). PTQ+RFT is plotted at full-pipeline cost (YAQA plus recovery; the math+web long recovery of Table 8, P-bf16). VPTQ QA is native VPTQ, and BF16 references are P-fp16.

Figure 14. Qwen3-30B-A3B (MoE) accuracy and generation vs. training compute (analog of Figure 1(b)). The MoE stays near its under-measured completion-prompt reference. Under the chat template it retains more HumanEval than the smaller dense models. VPTQ uses expert-shared codebooks (2.09 effective bits). PTQ+RFT is plotted at full-pipeline cost (15 + 32 = 47 PFLOP).

References

  1. [7]Abouelenin, A., Ashfaq, A., Atkinson, A., Awadalla, H., Bach, N., Bao, J., Benhaim, A., Cai, M., Chaudhary, V., Chen, C., & al., . et . (2025). Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. In arXiv preprint arXiv:2503.01743. arXiv:2503.01743
  2. [8]Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., & Hensman, J. (2024). Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37, 100213–100240.
  3. [9]Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, and Shiwei Liu. Is c4 dataset optimal for pruning? an investigation of calibration data for llm pruning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18089–18099, 2024.arxiv.org/abs/2410.07461
  4. [10]Bisk, Y., Zellers, R., Gao, J., Choi, Y., & al., . et . (2020). Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34, 7432–7439.
  5. [11]Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 4396–4429. Curran Associates, Inc., 2023. doi: 10.52202/075280-0196. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/0df38cd13520747e1e64e5b123a78ef8-Paper-Conference.pdf.DOI
  6. [12]Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde De Oliveira Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., & al., . et . (2021). Evaluating large language models trained on code. In arXiv preprint.arXiv
  7. [13]Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models. In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 10081–10100, 2025.arxiv.org/abs/2407.11062
  8. [14]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. *arXiv preprint arXiv:1803.05457*, 2018.arxiv.org/abs/1803.05457
  9. [15]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., & al., . et . (2021). Training verifiers to solve math word problems. In arXiv preprint.arXiv
  10. [16]Dadgarnia, A., Tabesh, S., Nikdan, M., Helcig, M., Kurtic, E., Kleinegger, M., & Alistarh, D. (2026). Gsq: Highly-accurate low-precision scalar quantization for llms via gumbel-softmax sampling. In arXiv preprint.arXiv
  11. [18]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. *Advances in neural information processing systems*, 36:10088–10115, 2023.arxiv.org/abs/2305.14314
  12. [19]Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 102–116, 2024.arxiv.org/abs/2402.10631
  13. [20]Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., & Alistarh, D. (2024). Extreme compression of large language models via additive quantization. Proceedings of the 41st International Conference on Machine Learning.
  14. [21]Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). Optq: Accurate quantization for generative pre-trained transformers. The Eleventh International Conference on Learning Representations.
  15. [22]Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Goulding, Jeffrey Hsu, Kyle McDonnell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. *Zenodo*, 2021.DOI
  16. [23]Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., & al., . et . (2024). The llama 3 herd of models. In arXiv preprint.arXiv
  17. [24]Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. In arXiv preprint.arXiv
  18. [25]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. NeurIPS.
  19. [26]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arXiv
  20. [27]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.openreview.net/forum?id=nZeVKeeFYf9
  21. [28]Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: pushing the limit of post-training quantization for llms. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2402.04291
  22. [29]Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing llms: The truth is rarely pure and never simple. In International Conference on Learning Representations, volume 2024, pp. 36973–36993, 2024.arxiv.org/abs/2310.01382
  23. [30]Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, and Min Zhang. Beware of calibration data for pruning large language models. In International Conference on Learning Representations, volume 2025, pp. 18604–18620, 2025.arxiv.org/abs/2410.17711
  24. [31]Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. A comprehensive evaluation of quantization strategies for large language models. In Findings of the association for computational linguistics: ACL 2024, pp. 12186–12215, 2024.arxiv.org/abs/2402.16775
  25. [32]Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. SqueezeLLM: Dense-and-sparse quantization. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=0jpbpFia8m.openreview.net/forum?id=0jpbpFia8m
  26. [33]Tanishq Kumar, Zachary Ankner, Benjamin Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. In International Conference on Learning Representations, volume 2025, pp. 71833–71864, 2025.arxiv.org/abs/2411.04330
  27. [34]Kurtic, E., Marques, A. N., Pandit, S., Kurtz, M., & Alistarh, D. (2025). “give me bf16 or give me death”? accuracy-performance trade-offs in llm quantization. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 26872–26886.
  28. [35]Donghyun Lee, Yuhang Li, Ruokai Yin, and Priyadarshini Panda. Kronq: Llm quantization via kronnecker-factored hessian. arXiv preprint arXiv:2607.07964, 2026.arXiv
  29. [36]Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 28480–28524. PMLR, 2024a.arxiv.org/abs/2402.18158
  30. [37]Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models. In International Conference on Learning Representations, volume 2024, pp. 13409–13424, 2024b.arxiv.org/abs/2310.08659
  31. [38]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100, 2024.DOI
  32. [39]Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezhen Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models. arXiv preprint arXiv:2504.04823, 2025a.arXiv
  33. [40]Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 8181–8196, 2024a.arxiv.org/abs/2409.17066
  34. [41]Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. In *Findings of the association for computational linguistics: ACL 2024*, pp. 467–484, 2024b.arxiv.org/abs/2305.17888
  35. [42]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations. In *International Conference on Learning Representations*, volume 2025, pp. 92009–92032, 2025b.arxiv.org/abs/2405.16406
  36. [43]Liu, Z., Zhao, C., Huang, H., Chen, S., Zhang, J., Zhao, J., Roy, S., Jin, L., Xiong, Y., Shi, Y., & al., . et . (2026). Paretoq: Improving scaling laws in extremely low-bit llm quantization. Advances in Neural Information Processing Systems, 38, 91311–91336.
  37. [44]Lotfi, S., Kirichenko, P., Li, S., & Liu, Z. (2026). Quantized reasoning models think they need to think longer, but they do not. In NeurIPS 2026 Poster. https://neurips.cc/virtual/2026/poster/151668neurips.cc/virtual/2026/poster/151668
  38. [45]Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., & Richtarik, P. (2024). Pv-tuning: Beyond straight-through estimation for extreme llm compression. Advances in Neural Information Processing Systems, 37, 5074–5121.
  39. [46]Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Morrison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., & al., . et . (2025). Olmoe: Open mixture-of-experts language models. International Conference on Learning Representations, 2025, 62061–62121.
  40. [47]Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisong Zhang, Haitao Mi, and Dong Yu. Low-bit quantization favors undertrained llms. In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 32338–32348, 2025.arxiv.org/abs/2411.17691
  41. [48]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In *Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers)*, pp. 1525–1534, 2016.arxiv.org/abs/1606.06031
  42. [49]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. *Communications of the ACM*, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
  43. [50]Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Gao Peng, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In *International Conference on Learning Representations*, volume 2024, pp. 45472–45496, 2024.arxiv.org/abs/2308.13137
  44. [51]Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. *Proceedings of machine learning research*, 235:48630, 2024a.arxiv.org/abs/2402.04396
  45. [52]Albert Tseng, Qingyao Sun, David Hou, and Christopher De. Qtip: Quantization with trellises and incoherence processing. *Advances in Neural Information Processing Systems*, 37:59597–59620, 2024b.arxiv.org/abs/2406.11235
  46. [53]Albert Tseng, Zhaofeng Sun, and Christopher De Sa. Model-preserving adaptive rounding. In *International Conference on Machine Learning (ICML)*, 2026. YAQA: Yet Another Quantization Algorithm.arxiv.org/abs/2505.22988
  47. [54]Mart Van Baalen, Andrey Kuzmin, Ivan Koryakovskiy, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for llm quantization. arXiv preprint arXiv:2402.15319, 2024.arxiv.org/abs/2402.15319
  48. [55]Miles Williams and Nikolaos Aletras. On the impact of calibration data in post-training quantization and pruning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10100–10118, 2024.DOI
  49. [56]Miles Williams, George Chrysostomou, and Nikolaos Aletras. Self-calibration for language model quantization and pruning. In Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2025.DOI
  50. [57]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087–38099. PMLR, 2023.arxiv.org/abs/2211.10438
  51. [58]Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. In International Conference on Learning Representations, volume 2024, pp. 52401–52418, 2024a.arxiv.org/abs/2309.14717
  52. [59]Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. Advances in Neural Information Processing Systems, 37:66357–66382, 2024b.arxiv.org/abs/2402.11295
  53. [60]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., & al., . et . (2025). Qwen3 technical report. In arXiv preprint arXiv:2505.09388.arXiv
  54. [61]Zhihang Yuan, Yuzhang Shang, and Zhen Dong. Pb-llm: Partially binarized large language models. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.00034
  55. [62]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 4791–4800, 2019.arxiv.org/abs/1905.07830
  56. [63]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.arxiv.org/abs/2311.07911

Paper details

vlnc:W_5dqqj4fa
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI-based tools

View statement
Looking for related papers…
Contents