WorksComputer Science
ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning
Abstract
Reasoning language models generate thousands of tokens per answer, and generation speed is limited by memory bandwidth, so storing weights in two bits instead of bfloat16 can potentially make inference 8 cheaper. Two-bit methods are usually judged by perplexity and multiple-choice question-answering accuracy, both of which say little about whether a model can still reason through more difficult problems. We present ThinkBit, a strictly scalar two-bit framework built to preserve reasoning, in which every weight is an integer codeword times a bfloat16 scale shared by a group of weights. We first use a sensitivity metric to assign each layer a ternary, 2-bit, or 3-bit tier to allocate bitwidths optimally among layers. This is followed by a layerwise Gumbel-softmax-based scalar quantization algorithm that searches for optimal codewords block by block. Since a layerwise scheme does not capture how quantized layers interact with each other, we recalibrate scales, with the codewords frozen, using a knowledge-distillation-based loss against the teacher model. We then perform a short quantization-aware training (QAT) stage that lets the model mimic the base model's distribution by reassigning codewords. Finally, a paired reasoning test on training problems selects the final checkpoint between the recalibration stage and the QAT stage. On Qwen3-1.7B at matched size, our recalibrated checkpoint scores 46.7 on MATH-500 against 30.0 for EfficientQAT trained on the same data, and 73.0 after QAT on reasoning traces; at 0.52 GB it is smaller than every published 2-bit GGUF of the model yet more accurate on MATH-500. We show that this framework scales and up to 14B, a 3.74 GB model scores 94.8 on MATH-500, against 97.5 for the 29.5 GB bfloat16 model.
Introduction
Reasoning language models answer with long chains of thought, often tens of thousands of tokens long. Each generated token requires reading all of the model’s weights from memory, so generation speed is capped by memory bandwidth rather than compute. Storing weights in two bits instead of bf16 reduces weight memory and the memory traffic per token by up to . Above three bits per weight, scalar quantization formats, which store one integer per weight and one scale per group of weights, are already close to lossless. At two bits, however, the strongest methods go beyond this format. They use vector codebooks, rotations, zero-points, or distillation-based quantization-aware training (QAT) with small groups [5, 13, 15, 45, 48]. Scalar formats remain the easiest to deploy, because dequantizing a weight takes a single multiplication. However, it is still an open question whether a two-bit scalar format can preserve a model’s ability to reason through difficult problems.
Answering this question first requires fixing how two-bit models are evaluated. They are usually compared by perplexity and multiple-choice accuracy. These metrics barely separate configurations whose sampled reasoning accuracy differs by more than ten points (Figure 1b), because likelihood based metrics, score single-token predictions and never check whether a long chain of thought eventually reaches a correct answer. A two-bit framework for reasoning models should therefore be both designed for and evaluated on reasoning tasks.
Figure 1. Overview of our 2-bit quantization framework ThinkBit and importance of different stages
We design such a framework, ThinkBit, presented in Figure 1a with one primary constraint in mind. The constraint is that we want a strictly scalar quantization format. Every stored weight is an integer codeword multiplied by a bf16 scale shared by a group of 128 weights, with no vector codebooks, rotations, or zero-points. Our proposed framework has five stages. First, sensitivity-based bit allocation assigns a precision to each layer. Second, block-wise codeword search chooses the initial codewords. Third, scale recalibration repairs the quantized model while leaving every codeword unchanged. Fourth, for even more flexibility, QAT reassigns the small number of codewords that still affect the output. Finally, a paired reasoning test on MATH training problems decides whether to keep the QAT checkpoint (QAT) or the recalibrated checkpoint (Rec). We compare both checkpoints against a QAT-only baseline (QAT-only), which skips the bit allocation, codeword search and recalibration phases.
What “2-bit” means. The label “2-bit” does not by itself say how large a model is. Community quantizations portrayed as 2-bit usually keep the embeddings, the output head, and other sensitive tensors at higher precision, so their real storage cost is well above two bits per parameter. For example, Unsloth’s 2-bit releases [75] in the GGUF file format of llama.cpp [21] store 2.82–3.62 bits per parameter for Qwen3-1.7B and 2.42–3.12 for Qwen3-14B. Similarly, PrismML’s ternary Bonsai model [55], whose weights need about bits each, actually stores 2.15 bits per parameter for Qwen3-1.7B: every ternary weight sits in a 2-bit slot, with one FP16 scale per 128 weights. Hence, comparing such models by their nominal bit-width can hide differences in size of up to 50%. We therefore compare models by whole-model size in bytes, counting every stored tensor. By this measure, our 0.52 GB Qwen3-1.7B checkpoint stores 2.42 bits per parameter including the output head and embeddings, which is smaller than every published 2-bit GGUF of the same model.
Our work makes the following key contributions:
Two-bit models must be selected on reasoning tasks On Qwen3-1.7B, removing the codeword search, scale recalibration, both, or the budget allocation stage keeps the 7-task QA task average at 53.7–54.8, within a ±1.5-point margin. However, skipping these stages lowers MATH-500 accuracy from 43.0 after QAT stage to the range of 28.4–37.0 (Table 8), showing that selecting two-bit models on reasoning tasks is crucial since that is where most of the degradations are seen.
ThinkBit: A novel scalar two-bit framework built for reasoning. We propose a novel five stage framework (Figure 1a) (1) Allocate: Allocate precisions appropriately for each layer (2) Search: Search the integer codewords block by block (3) Recalibrate: Recalibrate the stored scales with the codewords frozen (4) QAT: Let quantization-aware training (QAT) reassign codewords and (5) Selection: Keep the better of the Stage-3 and Stage-4 checkpoints by a reasoning test. On Qwen3-1.7B our framework reaches 47.5 on sampled MATH-500 before QAT and 43.0 after, against 18.5–24.1 and 32.6–37.8 from tuned GPTQ initializations (Table 9).
Better two-bit reasoning per byte. At matched size (about 0.5 GB), our recalibrated checkpoint (Rec-h2) scores 46.7 on MATH-500, against 30.0 for EfficientQAT trained on the same data and tokens. At 0.52 GB, smaller than every published 2-bit GGUF of the model, the selected checkpoint scores higher than all of them on MATH-500 (Table 7).
| Qwen3-1.7B | GB | bits/param | 7-task | MMLU | GSM8K | HumanEval | BFCL NL | MATH-500 (think) greedy | MATH-500 (think) sampled |
| bf16 | 3.44 | 16 | 58.9 | 55.5 | 68.6 | 41.5 | 81.7 | 90.2 | 91.2 |
| bits per parameter | |||||||||
| PrismML 1-bit Bonsai° | 0.25 | 1.15 | 49.7 | 41.7 | 43.9 | 37.2 | 74.3 | 25.4 | 28.7 |
| PrismML Ternary Bonsai° | 0.46 | 2.15 | 55.0 | 48.6 | 60.0 | 48.2 | 74.7 | 42.8 | 46.4 |
| Ours: QAT+R-h2 (selected)§ | 0.52 | 2.42 | 53.8 | 47.0 | 58.7 | 25.6 | 73.0 | 68.6 | 73.0 |
| Ours: Rec-h2 | 0.52‡ | 2.42 | 51.1 | 45.2 | 51.8 | 22.6 | 73.0 | 39.0 | 46.7 |
| Ours: QAT-h2 | 0.52 | 2.42 | 54.0 | 48.0 | 54.1 | 20.1 | 72.0 | 38.8 | 42.0 |
| EfficientQAT (official), 2-bit head/emb.† | 0.54 | 2.53 | 51.9 | 45.8 | 43.7 | 22.0 | 58.4 | 31.6 | 30.0 |
| BitDistiller (official), 2-bit head/emb.† | 0.54 | 2.53 | 51.0 | 45.6 | 42.4 | 16.5 | 61.4 | 21.6 | 25.3 |
| 2.6–4 bits per parameter | |||||||||
| GGUF UD-IQ2_XXS [66] | 0.61 | 2.82 | 38.2 | 26.0 | 1.3 | 1.2 | 0.0 | 2.8 | 1.0 |
| GGUF UD-IQ2_M [65] | 0.71 | 3.30 | 48.6 | 46.4 | 22.9 | 15.2 | 42.8 | 14.2 | 24.6 |
| GGUF Q2_K [62] | 0.78 | 3.62 | 49.2 | 40.0 | 31.8 | 18.3 | 54.5 | 27.8 | 41.9 |
| >4 bits per parameter | |||||||||
| GGUF Q3_K_M [63] | 0.94 | 4.37 | 55.7 | 53.1 | 60.9 | 26.2 | 80.6 | 81.2 | 84.3 |
| Ours: QAT-hbf16 | 0.98 | 4.54 | 54.6 | 49.9 | 50.2 | 17.7 | 75.5 | 38.2 | 43.0 |
| Ours: Rec-hbf16 | 0.98* | 4.54 | 51.5 | 44.4 | 51.6 | 22.6 | 72.2 | 40.6 | 47.5 |
| EfficientQAT (official)† | 1.00 | 4.65 | 53.0 | 46.2 | 42.3 | 22.6 | 62.3 | 33.2 | 34.2 |
| BitDistiller (official)† | 1.00 | 4.65 | 51.7 | 45.9 | 41.4 | 20.7 | 63.8 | 25.8 | 27.8 |
| GGUF Q4_K_M [64] | 1.11 | 5.15 | 58.7 | 55.7 | 66.0 | 33.5 | 82.3 | 88.4 | 90.8 |
| On Qwen3-0.6B | |||||||||
| Qwen3-0.6B GGUF Q4_K_M [60] | 0.40 | 5.4¶ | 46.5 | 35.0 | 33.9 | 17.7 | 70.4 | 61.8 | 69.2 |
| Qwen3-0.6B GGUF Q8_0 [61] | 0.64 | 8.6¶ | 48.7 | 40.1 | 41.7 | 20.1 | 72.9 | 69.2 | 76.0 |
Table 7. At 0.52 GB, QAT+R-h2 reasons better than every Qwen3-1.7B file of up to 4 bits per parameter; Ternary Bonsai leads the non-reasoning columns. Same harness for all files; bits/param use the original 1.72B parameters. Rows grouped by bits per parameter; bold: best in the -bit group. §QAT on the reasoning mixture. ‡Second recalibration pass that quantizes the head and embedding. *Not separately re-packed. †Official code on our base mixture, same tokens. °Released files [55]. ¶Bits per Qwen3-0.6B parameter. BFCL NL: BFCL non-live.
| Qwen3-1.7B variant | WikiText-2 ↓ | 7-task ↑ | MMLU ↑ | GSM8K ↑ | HumanEval ↑ | MATH-500 (thinking) greedy ↑ | MATH-500 (thinking) sampled ↑ |
| bf16 | 16.67 | 58.9 | 55.5 | 68.6 | 41.5 | 90.2 | 91.2 |
| GSQ-hbf16 | 46.91 | 43.8 | 36.4 | 27.5 | 8.5 | 15.0 | 14.8 |
| Scale recalibration (codewords frozen) | |||||||
| RMSNorm gains only | 25.29 | 48.0 | 38.2 | 38.8 | 16.5 | 27.8 | 34.0 |
| scales only | 19.49 | 51.4 | 44.1 | 51.3 | 23.2 | 39.2 | 48.5 |
| scales + RMSNorm gains (Rec-hbf16) | 19.46 | 51.5 | 44.4 | 51.6 | 22.6 | 40.6 | 47.5 |
| Instead of QAT: re-round toward bf16, then briefly recalibrate scales | |||||||
| codewords with that can step (23%) | 26.51 | 44.5 | 35.0 | 13.0 | 7.9 | 6.4 | 5.5 |
| 6.4% with the largest movable | 20.23 | 49.6 | 41.2 | 43.6 | 14.6 | 32.2 | 35.4 |
| After QAT | |||||||
| QAT-hbf16: full framework (reference) | 16.54 | 54.6 | 49.9 | 50.2 | 17.7 | 39.0 | 43.0 |
| LW-QAT-hbf16: layer-wise start instead of GSQ | 16.53 | 53.7 | 48.7 | 47.9 | 24.4 | 33.0 | 35.0 |
| QAT-noRec-hbf16: without the scale recalibration (719M tokens) | 16.68 | 54.5 | 48.9 | 52.2 | 26.2 | 34.2 | 37.0 |
| QAT-only-hbf16: layer-wise start, without recalibration | 16.53 | 54.8 | 49.9 | 46.9 | 18.3 | 26.6 | 32.1 |
| QAT-only-h2: + untied 2-bit head/emb. | 17.44{sup}°` | 54.8 | 48.5 | 47.2 | 15.2 | 26.6 | 29.1 |
| codewords frozen () | 16.99{sup}°` | 53.6 | 48.7 | 46.9 | 14.0 | 24.4 | 30.1 |
| uniform 2-bit (2.125 bpw), GPTQ, recalibration | 16.74 | 54.4 | 49.2 | 45.8 | 18.3 | 25.4 | 28.4 |
| without Wikipedia and MMLU-aux data | 17.23 | 54.2 | 47.8 | 56.0 | 15.9 | 36.2 | 42.5 |
| also without Evol-Instruct code | 16.98 | 54.3 | 48.7 | 55.1 | 18.9 | 37.0 | 41.6 |
| revert MLP codeword reassignments | 18.99{sup}°` | 51.3 | 43.1 | 47.8 | 29.9 | 39.6 | 42.1 |
| revert attention codeword reassignments | 18.47{sup}°` | 53.5 | 48.9 | 45.9 | 12.2 | 34.6 | 39.4 |
| revert value-projection reassignments | 17.48{sup}°` | 54.7 | 49.9 | 46.5 | 17.1 | 35.6 | 38.8 |
Table 8. Removing any stage keeps the 7-task mean within 1.5 points but costs 6.0–14.6 points of sampled MATH-500. Qwen3-1.7B variants (bf16 head and embedding); paired tests in table 9. Shaded: full framework; bold: best quantized value. “Revert” restores the pre-QAT codewords in one layer class. °WikiText-2 from a separate pass (bf16: 16.72).
Methodology
ThinkBit turns a bf16 model into a model whose linear-layer weights are stored as small integers. Stages 1–2 work layer by layer on calibration data; Stages 3–4 recover or train the whole model against the bf16 model; Stage 5 only evaluates. Algorithm 2 in section A gives the full procedure and Table 10 in the appendix summarizes the notation. All models use the same experimental settings and we focus mainly on the Qwen3 [80] model family in our experiments.
| Symbol | Meaning |
| , , , | model; calibration and training sequences; selection problems |
| , | number of transformer blocks; calibration activations entering the current block |
| Format (§ 2.1) | |
| , , | bf16 and quantized weight of linear layer (); number of weights |
| , | group size (128); group of input column |
| , | integer codeword and bf16 group scale of weight |
| , , | tier (ternary, 2-bit, 3-bit); its grid; tiers allowed for layer |
| , | bits per weight of tier (eq. (2)); budget in decoder bits per weight |
| Stage 1: bit allocation (section 2.2) | |
| , | token of a sequence; bf16 next-token distribution |
| , , | layer input, layer output, and loss gradient at the output, at token |
| , | second moments of and (Kronecker factors of the Hessian) |
| , | rounding error ; its per-entry variance |
| , | sensitivity of layer (eq. (5)); rounding noise of tier (eq. (6)) |
| , | standard normal weights; rounding to the nearest point of |
| , , | price per bit; resulting average bits per weight; tier chosen by layer |
| Stage 2: codeword search (section 2.3) | |
| , , , | damped GPTQ Hessian; damping and its base factor; second moment of output channel |
| , , | upper Cholesky factor of ; scaled error of column ; scale shrink factor |
| , , , | starting codeword; candidate values; their offsets from ; logits |
| , , | Gumbel noise; its uniform source; logistic noise |
| , | relaxed one-hot vector; soft codeword |
| , | temperature; logit sharpening factor |
| , , , | ternary mask and sign; their logits |
| , , , | ReinMax sampling probabilities; one-hot sample; softmax Jacobian; ReinMax midpoint |
| , , , | bf16 and quantized residual-stream vectors; channel weights; output factor at the residual stream |
| Stages 3–4: recalibration and QAT (sections 2.3 and 2.5) | |
| , , | log-ratio of a scale; offset of an RMSNorm gain; the gain |
| , , | weights of the scale and gain penalties and of the hidden-state term |
| , , , | QAT latent; latent clipped to the grid; initialization fraction; clip margin () |
| , | output of transformer block at token ; blocks used for hidden-state distillation |
| Stage 5: selection (section 2.6) | |
| , , , | number of problems; accuracy of candidate on problem ; paired difference; random sign |
Table 10. Notation.
Weight format and bit accounting
Consider a linear layer with weight and weights. Each row is split into groups of consecutive input weights, and weight belongs to group . The stored weight is an integer codeword times one bf16 scale per group,
where the grid depends on the layer’s tier : ternary , 2-bit or 3-bit . Dequantization is one multiplication; there is no zero-point, codebook or rotation. A tier costs
bits per weight: the codeword plus its share of the 16-bit scale. For example, ternary costs bits, 2-bit 2.125 and 3-bit 3.125. The decoder bits per weight (bpw) of a model averages over the linear layers of the transformer blocks, weighted by ; bits per parameter also counts the embedding and output head and is the measure we use to compare models by size.
Allocate: How do we allocate precision appropriately?
This stage of ThinkBit aims at optimally allocating precision to layers depending upon how sensitive they are to precision. Layers differ in how much rounding them hurts the model, so the budget should go where it matters. KronQ [34] estimates this from second-order information. Let and be the input and output of layer at token position of a calibration sequence of tokens, and let be the language-model loss. KronQ approximates the Hessian of with respect to the layer’s weights (vectorized column by column) by a Kronecker product of an input factor and an output factor ,
where is the bf16 model’s next-token distribution and is the gradient of the loss at the layer’s output. The next tokens are sampled from the model itself, so approximates the true Fisher information rather than the empirical Fisher tied to particular data labels; for the cross-entropy loss the true Fisher equals the Gauss–Newton approximation of the Hessian. The Kronecker form assumes, as in K-FAC [49], that inputs and output gradients are independent and that gradients at different positions are uncorrelated; constant factors, such as the number of tokens, do not affect the allocation.
A rounding error changes the loss by a first-order term and a second-order term. Because the labels are sampled from the model, the gradient has zero mean, , so the first-order term vanishes in expectation and the loss rises, to second order, by
If the entries of are zero-mean and uncorrelated with variance , the expectation factorizes, . The sensitivity of layer is therefore
which needs only the squared norms of inputs and output gradients, collected in one forward and backward pass per calibration sequence.
The error variance depends on the tier. We summarize each tier by its rounding noise , the mean squared error when standard normal weights are rounded to with the best scale for their group,
where rounds to the nearest point of . In practice we measure with the scale search of Stage 2 rather than the exact minimum, which gives slightly larger values (0.190, 0.150 and 0.041 against 0.187, 0.146 and 0.038 for ternary, 2-bit and 3-bit) and changes none of the conclusions below. Because the weights have unit variance, is the error relative to the weight variance, so ; coarser grids have larger . The expected loss of putting layer in tier is then . Following KronQ [34], we rank layers by , i.e. we drop the per-layer weight-variance factor and treat all layers as having equal weight variance. The allocation minimizes the total under a budget of decoder bits per weight:
This is a multiple-choice knapsack problem, which we solve by Lagrangian relaxation (algorithm 1). Pricing each bit at splits the objective into independent per-layer terms, so for a fixed every layer simply takes
where is the set of tiers allowed for layer . A higher price moves layers to cheaper tiers, so the resulting average bits per weight does not increase as grows, and bisection finds the smallest price that meets the budget. Bits left over because tiers are discrete buy the upgrades with the largest loss reduction per bit.
Sensitive layers (large per weight) therefore receive more bits. One geometric consequence is worth noting: in the plane the 2-bit point lies above the straight line between the ternary and 3-bit points. Scaling the points by a layer’s sensitivity and size does not change this, so for every price , (8) prefers ternary or 3 bits to 2 bits for every layer and never chooses 2 bits on its own. In our runs only query and key projections are held at bits; for Qwen3-1.7B at the result is 75.3% ternary, 6.0% 2-bit and 18.7% 3-bit weights. The greedy step of algorithm 1 can still move a ternary layer up to 2 bits when leftover bits allow; without the query–key rule this happens to three layers of Qwen3-1.7B. Section D shows that holding query and key at bits is unnecessary on this model.
Search: How to best inititalize codewords with PTQ?
This stage of ThinkBit aims at initializing the codewords appropriately at the allocated precision using PTQ methods. Rounding each weight to its nearest codeword ignores how errors in different weights interact. GSQ [6] instead treats Algorithm 1 Bit allocation by Lagrangian relaxation (Stage 1)
Require: sensitivities , layer sizes , allowed tiers , tier costs and noise , budget no smaller than the cost of the cheapest allowed tiers
from eq. (8)
;
while do
repeat bisection on the price
if then else
until is negligible
for every layer
while some layer’s next allowed tier fits in the leftover bits do spend the slack
upgrade the layer maximizing
return
every codeword as a learnable distribution over candidate grid values with logits , optimizes all distributions and scales of one transformer block jointly, and finally keeps the most likely value of each. It needs a starting point, a differentiable relaxations of the discrete choice presented in Table 1, and an objective.
| Gumbel-softmax (eqs. (11) and (12)) | ReinMax (eq. (13)) | |
| Forward weight | soft blend of grid values, | hard grid value, |
| Gradient | exact gradient of the soft weight | second-order estimate for the hard weight |
| Temperature | annealed toward 0 | kept |
| Train–test gap | vanishes only as becomes one-hot | none: training uses grid weights |
| Used for | ternary and 3-bit layers | 2-bit layers |
Table 1. The two relaxations used by codeword search. Gumbel-softmax trains a smooth stand-in for the rounded weight with an exact gradient; ReinMax trains the rounded weight itself with an approximate gradient.
Layer-wise start. Every 2- and 3-bit layer is first quantized on its own with GPTQ [19], which minimizes the layer’s output error on calibration inputs,
by rounding one input column at a time and moving the not-yet-rounded columns to compensate. With the upper Cholesky factor of , after column is rounded its scaled error is subtracted as from every later column . The scale of each group is set when the group’s first column is reached, from the current (already compensated) weights of each row : , with the maximum over the group’s columns and the shrink factor that minimizes the group’s rounding error. For ternary this is . Following KronQ’s use of the output side, we also let the layer outputs set the damping : with the second moment of output channel and its mean over channels,
where is a base factor, is the mean diagonal entry and the clip bounds the factor to a fixed range. Because the ratios average to one, the factor is at most one (Jensen’s inequality) and falls as the output energy concentrates in fewer channels, so such layers receive less damping; in practice only the lower bound of the clip can bind. All other 2- and 3-bit layers keep the GPTQ codewords as their starting codewords . Query and key projections, at any tier, refine their GPTQ codewords for a fixed number of steps with the relaxations below, applied to eq. (9), keep the result, and skip the block search. Ternary layers skip GPTQ: they run the same layer-wise refinement from uninformative logits, with GPTQ’s scale rule, and the result is their starting point for the block search.
Gumbel-softmax relaxation (3-bit and ternary layers). For a 3-bit weight, the candidates are the starting codeword and its neighbors, with (restricted to the grid), and the logits start proportional to (plus small noise) so that is most likely value. Each forward pass draws Gumbel noise and uses the -weighted average of the candidates, where is the resulting soft one-hot vector [27]:
The Gumbel noise makes a random sample whose largest entry is distributed as softmax, and the gradient is taken for the drawn noise; the temperature controls how close is to one-hot and the sharpening factor scales the logits. Both are annealed over training, so the soft codeword starts as a blend of neighbors and ends near a single grid value. A ternary codeword factors into a mask and a sign, with and , and each factor is a binary relaxation (the binary concrete distribution [47]) with its own logit, or :
At the end the codeword is rounded to and .
ReinMax (2-bit layers). On the 2-bit grid , each forward pass instead draws a hard sample: with probabilities , a one-hot vector selects the codeword , where the candidates are now the four grid values, so the forward pass uses exactly the kind of weight that inference will use. A discrete sample has no gradient, so ReinMax [41] estimates . Writing and for the Jacobian of a softmax with output ,
centered to zero mean over . The straight-through Gumbel estimator would use only with from eq. (11), a first-order approximation; ReinMax evaluates the Jacobian at the midpoint between the soft probabilities and the drawn sample, as in Heun’s method for ordinary differential equations, which makes the estimate accurate to second order. It requires , so the annealed temperature is clamped at 1.
Block objective. The relaxations are trained on the output of each transformer block rather than each layer, so codewords can compensate for errors elsewhere in the block. Within a block, GSQ first fits the attention projections against the residual stream after attention and then the multi-layer perceptron (MLP) projections against the block output. Inputs are the activations of the already quantized preceding blocks, so later blocks also correct the errors of earlier ones, and the target is the bf16 block applied to the same inputs. With and the quantized and bf16 residual-stream vectors of width at token , over calibration tokens,
where is the gradient second moment of eq. (3) measured at that point of the residual stream, so channels whose errors move the loss more are weighted more. rescales the channel weights to mean 1 and bounds them so that no channel is ignored or dominates. Because the change of a layer’s output at token is , eq. (4) equals , a gradient-weighted output error; eq. (14) is a normalized, bounded version of this quantity with a diagonal , measured on the block’s output rather than on a single layer’s. Logits and scales are trained jointly for a fixed number of epochs per block, after which every codeword takes its most likely value.
Recalibrate: Is tuning scales and norms enough?
Fine-tuning only the higher-precision parameters, the scales and the RMSNorm gains, is cheap since they make up less than 1% of the model (one scale per 128 weights) and hence the optimizer state stays small. This stage of ThinkBit applies recalibration to scales and norm gains to measure how much accuracy it recovers on its own. Search fits each block in isolation, so small errors still add up across the model. Scale recalibration repairs them without touching the codewords, and thus without changing the stored integers. Every group scale is reparameterized as and every RMSNorm gain as , where and are the values after Stage 2 and the offsets and are bounded. Only the offsets and are trained, so the model keeps exactly the same format and size. They minimize the forward Kullback–Leibler (KL) divergence from the bf16 model’s next-token distribution to the quantized model’s [23], with small penalties that keep scales and gains near their starting values:
where averages over all scales or gains and weight the penalties. The penalties act on the offsets, and , not on the magnitudes of and : a scale or gain that keeps its Stage-2 value costs nothing, and a given relative change costs the same at any magnitude. They act as a trust region around the Stage-2 solution, whose codewords were chosen jointly with and stay frozen here. Unlike the block loss, the KL is measured on the whole model’s output, so a scale can trade a slightly larger local error for a smaller error downstream. Training keeps the step with the lowest held-out KL.
QAT: Does moving codewords help?
QAT lets codewords move, giving the model more freedom than recalibration. This stage of ThinkBit measures what that freedom adds, and whether it pays for its cost. Each weight gets a 32-bit floating-point (FP32) latent measured in codeword units. The forward pass rounds it onto the layer’s grid , and the backward pass treats the rounding as the identity (the straight-through estimator, STE [3]):
where sg stops the gradient and is just below , so inside the clip range. A codeword is reassigned when its latent crosses a rounding boundary, halfway between two grid values. If every latent started exactly on its codeword, small gradient steps would rarely cross a boundary. Latents therefore start partway toward the bf16 weight,
where is the bf16 weight and sets how far toward the latent starts ( in the framework). This leaves every codeword unchanged at step 0 (the offset stays below half a step) but places each latent on the side of its codeword where the bf16 weight lies, so training can move it across the boundary when the loss favors it. Scales and gains keep training. The loss adds a hidden-state distillation term to the forward KL of eq. (15):
where is the quantized model’s next-token distribution, and are the bf16 and quantized outputs of transformer block at token , is a subset of blocks that includes the last, and weights the term. QAT can also quantize the output head and the embedding to 2 bits with the same latents; for models whose head and embedding share weights (such as Qwen3-1.7B), it first unties them. As in Stage 3, training keeps the step with the lowest held-out KL.
Selection: Which checkpoint reasons better, recalibrated or QAT?
Stages 3 and 4 produce two candidates, the recalibrated checkpoint (Rec) and the QAT checkpoint (QAT), and likelihood metrics do not tell which reasons better (§ 7). The stage of ThinkBit, therefore compares them on held-out reasoning problems: level-3–5 MATH training problems, disjoint from MATH-500 (for example, ). Each candidate samples four answers per problem in thinking mode, and an answer counts only if the reasoning trace closes and the final answer is correct (strict scoring). Let and be the fractions of correct answers of candidates and on problem , and . The paired sign-flip test gives
estimated from 20,000 random sign vectors (counting the observed signs once, so ). The rule keeps the candidate with the higher mean accuracy if and otherwise the smaller checkpoint, or the recalibrated one when both have the same size.
Experimental setup
Models. We primarily study ThinkBit on the Qwen-3 family. Specifically, Qwen3-1.7B, Qwen3-8B and Qwen3-14B [80] (thinking mode, native tool calling; Qwen3-8B and Qwen3-14B have untied heads). For all models we use the same hyperparameters for ThinkBit.
Data. The quantization and training data mixture has three parts (i) each model’s own bf16 replies to prompts, namely Magicoder code instructions, GSM8K-train and MetaMathQA math problems, Hermes and Glaive tool calls, TriviaQA, Natural Questions and WebQuestions training questions, and MMLU-auxiliary questions without answers; (ii) the model’s own continuations of FineWeb-Edu and Wikipedia passages; and (iii) raw text from FineWeb-Edu, FineWeb, Wikipedia (WikiText articles removed), OpenWebMath, Evol-Instruct code, Hermes and Glaive multi-turn tool conversations and Python source (codeparrot-clean). Replies are sampled with each model’s chat template, which for Qwen3 includes the thinking block, and are capped at 4,096 tokens, so training never sees a complete long reasoning trace. Table 3 lists the data and amount used by each stage.
| Stage | Data | Amount |
| 1: KronQ sensitivity | subset of the Stage-2 calibration data | 2M tokens |
| 2: Codeword search (GSQ) | own replies, tool conversations, code and FineWeb-Edu | 57M tokens |
| 3: Scale recalibration | all sources (i)–(iii) presented in § 3 | 268M tokens |
| 4: QAT | all sources (i)–(iii) presented in § 3 | 537M tokens |
| 4*: QAT, reasoning data (+R) | (i)–(iii) plus complete long reasoning traces (§ 3) | 1.1–2.1B tokens |
| 5: Checkpoint selection | level-3–5 MATH training problems, disjoint from MATH-500 | 313 problems |
Table 3. Data used by each stage. *Longer QAT on the reasoning mixture, used for the selected checkpoints.
Reasoning data for the final QAT. The stage studies (§ 4 and § 7) use the mixture above. The selected checkpoints of § 5 and § 6 repeat Stage 4 for longer on a mixture that adds complete long reasoning traces: each model’s own bf16 thinking-mode answers to public math and code prompts and to hard competition problems (OpenMathReasoning and OpenCodeReasoning statements; truncated traces dropped), DeepSeek-R1 solutions from OpenMathReasoning, and PG-19 books and arXiv papers for long context; at 1.7B also OpenR1-Math-220k, Codeforces CoTs, OpenCodeReasoning, OpenCodeInstruct and Cosmopedia textbooks. Reasoning traces make up about 60% of the mixture. Every source passes the same decontamination against all evaluated benchmarks as the base mixture. At 1.7B, QAT+R-h2 starts from Rec-h2 (2.1B tokens, learning rate ); at 8B and 14B it starts from a further recalibration of Rec-h2 on the long-trace data (1.1B tokens, ); format and bytes are unchanged.
Checkpoint names. We name every checkpoint by the stages it went through (Table 2): a base name for the codeword start and the trained stages, a suffix for the precision of the head and embedding, and +R when Stage 4 (or 3) used the reasoning mixture. “Selected” marks the checkpoint kept by Stage 5. The same names are used in every table, caption and figure.
| Name | Checkpoint |
| GSQ | Stages 1–2: tiers allocated and codewords searched; no training |
| Rec | GSQ + Stage 3: scales and RMSNorm gains recalibrated, codewords frozen |
| QAT | Rec + Stage 4: QAT reassigns codewords (the full framework) |
| QAT-noRec | GSQ + Stage 4, without recalibration |
| LW-Rec, LW-QAT | as Rec and QAT, but starting from the layer-wise PTQ solution instead of codeword search |
| QAT-only | layer-wise start + Stage 4 only: no codeword search, no recalibration |
| -hbf16 | suffix: head and embedding kept in bf16 (tied at 1.7B) |
| -h2 | suffix: head and embedding quantized to 2 bits (untied at 1.7B) |
| +R | trained on the reasoning mixture of § 3, e.g. Rec+R-h2, QAT+R-h2 |
Table 2. Checkpoint names. At 1.7B, Rec-h2 uses a second recalibration pass that quantizes the head and embedding to 2-bits; Rec-h2 (1 pass) uses one.
Evaluation. Dense bf16 exports are scored with the Language Model Evaluation Harness (lm-eval) [20] and served by vLLM. The core suite contains ARC-e/c, MMLU, WinoGrande, HellaSwag, PIQA and LAMBADA; their mean log-likelihood accuracy is the 7-task mean. Generation covers math, code, instruction following and tool use on the Berkeley Function Calling Leaderboard (BFCL) [54]. The broad suite adds MMLU-Pro, TriviaQA, TruthfulQA, SciQ, OpenBookQA, CommonsenseQA, BoolQ, RACE, Beyond the Imitation Game Benchmark Hard (BBH), MATH-500, GSM8K chain-of-thought (GSM8K-CoT), Instruction Following Evaluation (IFEval), Mostly Basic Python Problems (MBPP) and HumanEval+. American Invitational Mathematics Examination (AIME) 2024/2025 and MATH-500 use the models’ chat/thinking format; We report thinking-mode MATH-500 in two ways. Greedy decodes each problem once and grades it with lm-eval. Sampled draws four answers per problem (temperature 0.6, top- 0.95, top- 20, up to 32k new tokens) and scores them strictly: an answer counts only if the reasoning trace closes with </think> and the final answer is correct. Sampled, strictly scored MATH-500 is our primary reasoning metric. AIME@16 denotes mean sampled pass@1 over 16 samples per problem.
Where does the quality come from?
We follow Qwen3-1.7B through Stages 3 and 4 (Table 4 and Figure 2) and then split reasoning accuracy into finishing a trace and solving the problem (Table 5).
Figure 2. Qwen3-1.7B training curves. (a) Scale recalibration (codewords frozen): held-out KL to bf16 falls from 0.641 to 0.282. (b) Long QAT from the recalibrated model: KL increases briefly while codewords start to move, then falls from 0.311 to 0.199 (QAT is scored on the text that follows its own training slice, so its starting KL differs from the end of (a)). (c) Share of integer codewords that differ from the recalibrated model.
| Qwen3-1.7B | WikiText-2 ↓ | 7-task | MMLU | GSM8K | HumanEval | MATH-500 |
| bf16 | 16.67 | 58.9 | 55.5 | 68.6 | 41.5 | 91.2 |
| Stage 2: GSQ-hbf16 | 46.91 | 43.8 | 36.4 | 27.5 | 8.5 | 14.8 |
| Stage 3: Rec-hbf16 | 19.46 | 51.5 | 44.4 | 51.6 | 22.6 | 47.5 |
| Stage 4: QAT-hbf16 | 16.54 | 54.6 | 49.9 | 50.2 | 17.7 | 43.0 |
Table 4. Recalibration recovers most of what codeword search loses; QAT further improves likelihood but not reasoning. Qwen3-1.7B after each stage (bf16 head and embedding); Bold: best quantized value.
| Model | Checkpoint | Strict acc. | Closed (%) | Acc. on closed |
| Qwen3-1.7B | bf16 | 91.2 | 99.2 | 91.9 |
| Rec-hbf16 | 47.5 | 88.0 | 53.9 | |
| QAT-hbf16 | 43.0 | 69.6 | 61.7 | |
| QAT-noRec-hbf16 | 37.0 | 57.2 | 64.5 | |
| LW-QAT-hbf16 | 35.0 | 68.2 | 51.3 | |
| QAT-only-hbf16 | 32.1 | 55.4 | 58.0 | |
| Qwen3-14B | bf16 | 97.5 | 99.9 | 97.6 |
| Rec-hbf16 | 92.8 | 97.3 | 95.4 | |
| LW-Rec-hbf16 | 89.7 | 97.5 | 91.9 | |
| QAT-h2 | 90.5 | 96.8 | 93.6 | |
| QAT-only-h2 | 89.6 | 96.4 | 92.9 |
Table 5. Two-bit models fail mainly by not finishing their reasoning; at 14B almost every trace finishes. Sampled MATH-500 split into the share of traces that close </think> and the accuracy on those. Bold: best quantized value per model.
Recalibration closes most of the likelihood gap. With the Stage 2 codewords frozen, recalibration lowers WikiText-2 perplexity from 46.91 to 19.46 (bf16: 16.67), raises the 7-task mean from 43.8 to 51.5 (bf16: 58.9) and sampled MATH-500 from 14.8 to 47.5 (Table 4). The scales do almost all of the work: held-out KL is 0.641 after GSQ, 0.282 with the scales trained, 0.418 with only the RMSNorm gains trained and 0.282 with both (Table 19 in Section H).
| Qwen3-1.7B, codewords frozen | Held-out KL | WikiText-2 |
| GSQ-hbf16 | 0.641 | 46.91 |
| + RMSNorm gains only | 0.418 | 25.29 |
| + scales only | 0.282 | 19.49 |
| + scales and RMSNorm gains (Rec-hbf16) | 0.282 | 19.46 |
Table 19. The scales carry recalibration: scales alone reach the same held-out KL as scales plus RMSNorm gains. Qwen3-1.7B, same data and schedule.
The scales fit the output, not the weights. Recalibration moves each scale by a median of 6% (log-offset in , Section 2.4), but not toward the scale that best fits the frozen codewords in weight space, for group : has the same sign as in 51% of groups, and the two correlate at 0.07. Relative weight error therefore improves slightly, from 0.386 to 0.393, while would lower it to 0.277. The objective explains why. Equation (15) scores the whole network’s output, so it weights each weight error by the activations it meets and lets the scales of one layer offset rounding errors in others. Block outputs show the same pattern (Table 20 in Section H). Given the same bf16 input, a recalibrated block is further from its bf16 output than before (mean relative error 0.094 vs. 0.079), and the hidden states of the recalibrated network are further from bf16 on average (0.28 vs. 0.21), yet next-token KL on WikiText-2 falls from 1.30 to 0.57. Recalibration does not make intermediate representations more faithful; it adjusts them so that the output distribution matches.
| Tier | median | same sign as (%) | corr. | GSQ | recalibrated | least-squares |
| ternary | 0.064 | 51.6 | 0.08 | 0.425 | 0.431 | 0.325 |
| 2-bit | 0.054 | 49.8 | 0.00 | 0.198 | 0.207 | 0.192 |
| 3-bit | 0.058 | 50.8 | 0.05 | 0.287 | 0.296 | 0.109 |
| all | 0.062 | 51.4 | 0.07 | 0.386 | 0.393 | 0.277 |
Table 20. Recalibration does not move scales toward the weight-space optimum, and weight error rises in every tier. Per-group scale changes on Qwen3-1.7B; corr.: correlation of with .
QAT reassigns few codewords. Long running QAT reassigns 6.4% of the codewords (Figure 2c), each by a single grid step toward its bf16 weight (Section J); initialization, not learning rate, decides how many move (§ 18). It brings WikiText-2 to 16.54, partly through Wikipedia in its training mix, the 7-task mean to 54.6 and MMLU from 44.4 to 49.9. GSM8K, HumanEval and sampled MATH-500 do not improve; they fall from 51.6, 22.6 and 47.5 to 50.2, 17.7 and 43.0 (Table 4). On a broader suite, reading comprehension and commonsense keep 80% or more of bf16’s score, while code and non-chat math do not recover (Section N).
Finishing reasoning traces versus solving the task. Strict scoring counts an answer only if it follows a closed </think> tag, so a two-bit model can fail by looping as well as by reasoning wrongly. Table 5 separates the two: bf16 Qwen3-1.7B closes 99.2% of its traces, the two-bit checkpoints of Table 5 55.4–88.0% and the 2-bit GGUFs 47–56%; budget forcing narrows but does not close this gap (Section K). Codeword search raises accuracy on closed traces, from 51.3 to 61.7 on Qwen3-1.7B, while the closing rate changes less (68.2 to 69.6; problem-matched tests in Table 23). QAT and recalibration act mainly on closing: QAT lowers it from 88.0 to 69.6 on Qwen3-1.7B, in the direction of its total effect, and recalibration raises it from 57.2 to 69.6 on Qwen3-1.7B. At 14B almost every trace closes (96.4–97.5%), so codeword search shows up only in closed-trace accuracy (95.4 vs. 91.9).
| Model | Contrast | closing rate | accuracy on closed | problems |
| Qwen3-1.7B | codeword search | +1.4 [−1.0, 3.8] | +10.4 [7.1, 13.6] | 414 |
| scale recalibration | +12.3 [9.8, 14.9] | +2.1 [−1.4, 5.8] | 379 | |
| QAT vs. Rec | −18.4 [−21.1, −15.6] | −0.5 [−3.5, 2.5] | 440 |
Table 23. Codeword search mainly raises accuracy on closed traces; recalibration and QAT mainly change how many traces close. Problem-matched sampled MATH-500 differences (95% CI); bold: p<0.01.
Scaling across model sizes
ThinkBit with every hyper-parameter fixed on Qwen3-1.7B, was applied unchanged to Qwen3-8B and Qwen3-14B. Table 6 compares each selected checkpoint, QAT+R-h2 (QAT on the reasoning mixture of § 3), with its bf16 model and with the smallest and largest published 2-bit GGUF files of the same model.
| Model | Checkpoint | GB | bits/param | 7-task | MMLU | GSM8K | HumanEval | MATH-500 | AIME@16 |
| Qwen3-1.7B | bf16 | 3.44 | 16 | 58.9 | 55.5 | 68.6 | 41.5 | 91.2 | 41.4 |
| bits per parameter | |||||||||
| QAT+R-h2 (selected)§ | 0.52 | 2.42 | 53.8 | 47.0 | 58.7 | 25.6 | 73.0 | 12.3 | |
| Rec+R-h2§ | 0.52 | 2.42 | 50.6 | 43.5 | 51.7 | 22.0 | 55.6 | 3.0 | |
| Rec-h2 | 0.52 | 2.42 | 51.1 | 45.2 | 51.8 | 22.6 | 46.7 | 1.0 | |
| 1-bit Bonsai | 0.25 | 1.15 | 49.7 | 41.7 | 43.9 | 37.2 | 28.7 | 0.7 | |
| Ternary Bonsai | 0.46 | 2.15 | 55.0 | 48.6 | 60.0 | 48.2 | 46.4 | 1.4 | |
| EfficientQAT, 2-bit head/emb.‡ | 0.54 | 2.53 | 51.9 | 45.8 | 43.7 | 22.0 | 30.0 | 0.6 | |
| BitDistiller, 2-bit head/emb.‡ | 0.54 | 2.53 | 51.0 | 45.6 | 42.4 | 16.5 | 25.3 | 0.1 | |
| 2.6–4 bits per parameter | |||||||||
| GGUF UD-IQ2_XXS [66] | 0.61 | 2.82 | 38.2 | 26.0 | 1.3 | 1.2 | 1.0 | 0.0 | |
| GGUF Q2_K [62] | 0.78 | 3.62 | 49.2 | 40.0 | 31.8 | 18.3 | 41.9 | 0.1 | |
| >4 bits per parameter | |||||||||
| EfficientQAT† | 1.00 | 4.65 | 53.0 | 46.2 | 42.3 | 22.6 | 34.2 | 1.0 | |
| BitDistiller† | 1.00 | 4.65 | 51.7 | 45.9 | 41.4 | 20.7 | 27.8 | 0.1 | |
| Qwen3-8B | bf16 | 16.4 | 16 | 70.7 | 73.0 | 87.9 | 63.4 | 96.9 | 71.2 |
| bits per parameter | |||||||||
| QAT+R-h2 (selected)§ | 2.08 | 2.03 | 65.8 | 64.8 | 84.5 | 47.0 | 92.2 | 46.9 | |
| Rec+R-h2§ | 2.08 | 2.03 | 65.1 | 63.8 | 83.2 | 56.1 | 89.3 | 38.4 | |
| Rec-h2 | 2.08 | 2.03 | 64.9 | 63.7 | 82.8 | 53.0 | 86.6 | 28.8 | |
| 1-bit Bonsai | 1.16 | 1.13 | 62.0 | 57.8 | 77.8 | 61.0 | 71.8 | 12.6 | |
| Ternary Bonsai | 2.18 | 2.13 | 66.5 | 64.4 | 82.8 | 62.2 | 83.7 | 24.4 | |
| GGUF UD-IQ2_XXS [74] | 2.61 | 2.54 | 60.4 | 59.2 | 47.4 | 24.4 | 35.2 | 1.0 | |
| 2.6–4 bits per parameter | |||||||||
| GGUF Q2_K [71] | 3.28 | 3.21 | 67.2 | 67.4 | 82.8 | 56.7 | 91.1 | 45.6 | |
| >4 bits per parameter | |||||||||
| EfficientQAT† | 4.35 | 4.25 | 66.5 | 62.4 | 72.5 | 33.5 | 71.9 | 10.3 | |
| BitDistiller† | 4.35 | 4.25 | 65.4 | 62.6 | 75.1 | 48.8 | 66.4 | 7.1 | |
| Qwen3-14B | bf16 | 29.5 | 16 | 74.3 | 77.1 | 88.2 | 57.3 | 97.5 | 73.6 |
| bits per parameter | |||||||||
| QAT+R-h2 (selected)§ | 3.74 | 2.03 | 69.9 | 70.5 | 87.3 | 59.8 | 94.8 | 57.7 | |
| Rec+R-h2§ | 3.74 | 2.03 | 69.5 | 69.7 | 88.2 | 59.1 | 94.4 | 53.5 | |
| Rec-h2 | 3.74 | 2.03 | 69.9 | 69.5 | 87.7 | 65.2 | 92.7 | 46.5 | |
| GGUF UD-IQ2_XXS [69] | 4.47 | 2.42 | 66.7 | 65.2 | 71.2 | 39.0 | 85.2 | 26.7 | |
| 2.6–4 bits per parameter | |||||||||
| GGUF Q2_K [67] | 5.75 | 3.12 | 71.6 | 73.8 | 87.3 | 54.9 | 95.0 | 63.0 | |
| EfficientQAT† | 6.65 | 3.60 | 71.2 | 68.7 | 77.5 | 45.1 | 85.4 | 28.4 | |
| BitDistiller†¶ | 6.65 | 3.60 | 69.8 | 68.9 | 83.2 | 56.1 | 84.9 | 24.7 |
Table 6. At every size, QAT+R-h2 is the most accurate checkpoint on MATH-500 and AIME@16 among files of at most 2.6 bits per parameter. MATH-500: sampled; AIME@16: mean pass@1 over 16 samples. Bonsai [55] and GGUF [75] files run in our harness. Rows grouped by bits per parameter; bold: best in the -bit group. §Trained on the reasoning mixture. †Official code on our base mixture, same tokens as Rec-h2, bf16 head and embedding. ‡Same with a 2-bit head and embedding. ¶Trained with 4,096-token sequences.
Larger models lose less. Two-bit weights cost the most reasoning at small scale. Sampled MATH-500 is 73.0 against bf16’s 91.2 at 1.7B, 92.2 against 96.9 at 8B and 94.8 against 97.5 at 14B, and the 7-task mean is 53.8 against 58.9, 65.8 against 70.7 and 69.9 against 74.3, while the checkpoints take 0.52, 2.08 and 3.74 GB instead of 3.44, 16.4 and 29.5 GB. AIME@16, the hardest benchmark, recovers more slowly: 12.3 against 41.4 at 1.7B, 46.9 against 71.2 at 8B and 57.7 against 73.6 at 14B.
Reasoning data for QAT. Running Stage 4 longer on a mixture with complete reasoning traces (§ 3) changes reasoning far more than any stage ablation. Against Rec-h2 of the same size, sampled MATH-500 rises from 46.7 to 73.0 at 1.7B (+26.3 [23.3, 29.4], , paired over problems), from 86.6 to 92.2 at 8B and from 92.7 to 94.8 at 14B, and AIME@16 from 1.0 to 12.3, 28.8 to 46.9 and 46.5 to 57.7. At 1.7B the gain is accuracy on finished traces (83.8 against 54.2) more than finishing them (87.1% against 86.0% of traces closed). The 7-task mean moves by at most 2.7 points, while HumanEval falls at 8B and 14B (53.0 to 47.0 and 65.2 to 59.8). On 500 MATH training problems, QAT+R-h2 scores above Rec+R-h2, recalibrated on the same mixture, at every size (63.1 against 45.0, ; 88.3 against 86.2, ; 92.7 against 91.9, ), so the selector (Stage 5) keeps QAT+R-h2 at 1.7B and 8B; at 14B the two files have the same size and the difference is not significant, and we keep QAT+R-h2, the checkpoint with the higher mean.
Against published two-bit models on huggingface. At every Qwen3 size the selected checkpoint is smaller than the smallest published 2-bit GGUF (UD-IQ2_XXS [66, 69, 74]) and more accurate on every task of Table 6; on sampled MATH-500 it scores 73.0 against 1.0 at 1.7B, 92.2 against 35.2 at 8B and 94.8 against 85.2 at 14B. Figure 3 shows the resulting Pareto dominance on reasoning tasks. UD-IQ2_XXS and the official EfficientQAT and BitDistiller files at every size, UD-IQ2_M [65, 73] and Q2_K at 1.7B and 8B, and Ternary Bonsai at 8B are all larger and less accurate than our checkpoint. Q2_K files [62, 67, 71] (0.78, 3.28 and 5.75 GB) score 41.9 against our 73.0 at 1.7B (+31.1 [27.9, 34.5], ), 91.1 against 92.2 at 8B (+1.1 [−0.5, 2.8], ) and 95.0 against 94.8 at 14B (−0.2 [−1.3, 1.0], ): our checkpoints match them at 63–67% of their size. Files of four or more bits per parameter (Q3_K_M and above), and Qwen3-0.6B at eight bits (76.0 at 0.64 GB), still score higher.
Figure 3. Accuracy against whole-model size for every file of each Qwen3 model scored in our harness: (a–c) sampled, strictly scored MATH-500; (d, e) AIME@16. Filled stars: QAT+R-h2 (selected); hollow stars: Rec-h2; diamonds: official EfficientQAT and BitDistiller trained on our base mixture (at 1.7B also with a 2-bit head and embedding); hexagons: PrismML Bonsai (ternary and 1-bit); lines join the model’s published GGUF files (Unsloth [75]; UD- prefix omitted) and bf16. The shaded corner holds every file that is both larger and less accurate than the selected checkpoint, i.e. Pareto-dominated by it; dominated files are drawn hollow. Grey triangles in (a): Qwen3-0.6B at 4 and 8 bits. Dotted lines: 2.6 and 4 bits per parameter (the size groups of the tables).
Against two-bit QAT methods and released low-bit models. The official EfficientQAT and BitDistiller, trained on our base mixture for the same 805M tokens as Rec-h2, store a bf16 head and embedding and per-group zero points, so
their files are 1.8–2.1 times larger than ours. They match or exceed Rec-h2 on the 7-task mean (53.0 and 51.7 against 51.1 at 1.7B, 66.5 and 65.4 against 64.9 at 8B, 71.2 and 69.8 against 69.9 at 14B) but not on reasoning: on sampled MATH-500 Rec-h2 is 12.5 and 18.9 points higher at 1.7B (46.7 against 34.2 and 27.8), 14.7 and 20.2 points higher at 8B (86.6 against 71.9 and 66.4; AIME@16 28.8 against 10.3 and 7.1) and 7.3 and 7.8 points higher at 14B (92.7 against 85.4 and 84.9; AIME@16 46.5 against 28.4 and 24.7), all paired over problems. Quantizing their head and embedding to 2 bits as well (0.54 GB, byte-matched at 1.7B) lowers MATH-500 to 30.0 and 25.3. PrismML’s Bonsai models [55] quantize every linear layer, the embedding and the head. Their training data and pipeline however are not published, so these rows compare released checkpoints. Ternary Bonsai keeps more of the non-reasoning scores (at 0.46 GB, 7-task 55.0 and HumanEval 48.2 at 1.7B), while QAT+R-h2 leads on sampled MATH-500 by 26.6 points at 1.7B (73.0 against 46.4) and 8.5 at 8B (92.2 against 83.7; both ) and on AIME@16 (12.3 against 1.4, 46.9 against 24.4).
Scaling to 14B. At Qwen3-14B, the QAT and QAT-only checkpoints (QAT-h2, QAT-only-h2) are within 0.4 points on the 7-task mean (70.6 against 70.2) and MMLU (70.2 against 69.9); on sampled MATH-500 they score 90.5 and 89.6, and on AIME@16 40.9 and 37.2 (tables 17 and 27). At equal bytes, the recalibrated checkpoint (Rec-h2) scores higher than QAT-h2 on the selector (87.5 against 83.3) and on MATH-500 (92.7 against 90.5). After recalibration, GSQ codewords score higher than layer-wise-start codewords (Rec-hbf16 against LW-Rec-hbf16: 92.8 against 89.7 on MATH-500, 48.2 against 36.7 on AIME@16).
Model size and inference
Training checkpoints store codewords in 4-bit containers; we re-pack them at their true widths and decode them again to verify the stored bytes (section M). With a 2-bit head and embedding, the Qwen3-1.7B QAT and recalibrated checkpoints (QAT-h2, Rec-h2) each measure 0.520 GB (2.42 bits per original parameter; 2.046 per quantized weight with scales), and the Qwen3-14B QAT-only and recalibrated checkpoints (QAT-only-h2, Rec-h2) 3.74 GB; all four decode bit-identically and load in stock transformers. With a bf16 tied embedding the 1.7B checkpoint (QAT-hbf16) is 0.977 GB, 64% of it embedding. Served by vLLM directly from the 4-bit containers on one H100, the 8B and 14B checkpoints take 5.68 and 9.31 GiB of weight memory instead of 15.27 and 27.52 GiB for bf16, and decode 212 and 150 tokens/s at batch 1 instead of 146 and 86 (section M).
Equal bytes, same harness. Table 7 and section O compare our Qwen3-1.7B checkpoints with PrismML’s Bonsai releases [55], the official EfficientQAT and BitDistiller trained on our base mixture, and Unsloth’s llama.cpp quantizations [21, 75] of the same model, calibrated with an importance matrix, decoded to bf16 and scored with the same harness. At 0.52 GB, our recalibrated and QAT checkpoints (Rec-h2, QAT-h2) are smaller than every 2-bit GGUF [62, 65, 66]. The recalibrated checkpoint selected under the base mixture (Rec-h2), from a second recalibration pass that quantizes the head and embedding, beats Q2_K [62] on strict sampled MATH-500 (by less under lenient scoring, 47.3 vs. 44.8) because more of its traces finish (86% vs. 56%); on traces both finish, Q2_K is more accurate, and with a forced answer the two tie (section K). Our QAT checkpoint (QAT-h2) is better on most greedy columns. QAT+R-h2, in the same 0.52 GB, raises sampled MATH-500 to 73.0 and greedy MATH-500 to 68.6, against 41.9 and 27.8 for Q2_K (+31.1 [27.9, 34.5] sampled, ), the best of every sub-0.94 GB Qwen3-1.7B checkpoint; Ternary Bonsai (0.46 GB) leads the non-reasoning columns but scores 46.4 sampled and 42.8 greedy. The official EfficientQAT and BitDistiller score 34.2 and 27.8 at 1.00 GB and 30.0 and 25.3 when their head and embedding are also 2-bit (0.54 GB). It also beats Qwen3-0.6B at 4 bits [60] on MATH-500 (69.2 at 0.40 GB), but not Qwen3-0.6B at 8 bits [61] (76.0 at 0.64 GB) or Q3_K_M [63] (84.3 at 0.94 GB) (Figure 3); the 0.6B models trail it on every non-reasoning column.
Controls behind each stage
Figure 4 summarizes ThinkBit. Each control below removes one stage or design choice under the same frozen hyper-parameters and evaluation (Table 8); table 9 gives the paired reasoning contrasts. Recalibration plus QAT uses 805M tokens and the no-recalibration QAT arms 719M; held-out KL is still falling at the last step.

Figure 4. The framework at a glance. Every weight is an integer codeword times a bf16 scale shared by a group of 128 weights; one checkpoint is kept per model. The rest of this section gives the evidence for each stage.
Reasoning performance differentiates between the configurations. After QAT, removing codeword search, scale recalibration, both, or mixed bit placement keeps the 7-task mean at 53.7–54.8, MMLU at 48.7–49.9 and WikiText-2 at 16.53–16.74 (full framework: 54.6, 49.9 and 16.54), while sampled MATH-500 falls from 43.0 to 28.4–37.0 (Table 8). With a pre-stated ±1.5-point margin, every removal is equivalent on the 7-task mean; MMLU differences of up to 2.1
| Model | Contrast (A vs. B) | A | B | [95% CI] | status | |
| Qwen3-1.7B | codeword search (QAT vs. LW-QAT) | 43.0 | 35.0 | +8.0 [5.6, 10.2] | abl | |
| vs. tuned GPTQ start (all tiers, clipped scales) | 43.0 | 32.6 | +10.3 [8.0, 12.8] | check | ||
| vs. GPTQ start + Hessian-weighted rounding | 43.0 | 37.8 | +5.2 [3.1, 7.5] | check | ||
| recalibration before QAT (QAT vs. QAT-noRec) | 43.0 | 37.0 | +6.0 [3.6, 8.3] | abl | ||
| recalibration on layer-wise-start codewords (LW-QAT vs. QAT-only) | 35.0 | 32.1 | +2.85 [0.85, 4.90] | 0.009 | abl | |
| both (QAT vs. QAT-only) | 43.0 | 32.1 | +10.8 [8.5, 13.0] | abl | ||
| QAT stage (QAT vs. Rec) | 43.0 | 47.5 | −4.5 [−7.0, −2.1] | abl | ||
| same-data QAT baseline (QAT vs. EfficientQAT port, GB) | 43.0 | 28.8 | +14.2 [11.8, 16.8] | check | ||
| codeword search at the recalibrated checkpoint (Rec vs. LW-Rec)1 | 47.2 | 22.8 | +24.4 [21.8, 27.2] | check | ||
| vs. tuned GPTQ start (all tiers, clipped scales) | 47.5 | 18.5 | +28.9 [25.9, 32.1] | check | ||
| vs. GPTQ start + Hessian-weighted rounding | 47.5 | 24.1 | +23.4 [20.4, 26.4] | check | ||
| Checkpoint selection: 313 MATH training problems | ||||||
| Qwen3-1.7B | QAT-hbf16 vs. Rec-hbf16 (0.98 GB) | 22.5 | 28.8 | −6.2 [−9.0, −3.5] | sel | |
| Qwen3-1.7B | QAT-h2 vs. Rec-h2 (0.52 GB) | 21.9 | 27.6 | −5.7 [−7.9, −3.4] | sel | |
| Qwen3-1.7B | Rec-h2 vs. Rec-hbf16 (0.52 vs. 0.98 GB) | 27.6 | 28.8 | −1.2 [−3.7, 1.1] | 0.37 | sel |
| Qwen3-1.7B | codeword search at the recalibrated checkpoint (Rec vs. LW-Rec)1 | 27.8 | 8.6 | +19.2 [15.9, 22.7] | sel | |
| Qwen3-14B | QAT-h2 (3.74 GB) vs. Rec-hbf16 (6.4 GB) | 83.3 | 87.9 | −4.6 [−6.6, −2.7] | sel | |
| QAT-h2 vs. Rec-h2 (3.74 GB) | 83.3 | 87.5 | −4.2 [−6.3, −2.1] | sel |
caption: Codeword search and recalibration each raise sampled MATH-500 significantly, and every selector decision agrees with MATH-500. Paired contrasts on MATH-500 () and on the 313 selection problems. Bold: significant after Holm correction. Status: abl, stage ablation; check, robustness check; sel, selector decision. Separate evaluation run (Rec: 47.2 there, 47.45 in the main run).
:::
points are not equivalent within the margin in 4 of 6 contrasts (section E). QAT itself raises the 7-task mean from 51.5 to 54.6 but lowers sampled MATH-500 from 47.5 to 43.0. HumanEval does not follow the same ordering, so the finding concerns long-form reasoning. A rule with a log-likelihood conjunct would have dropped the search (section E).
QAT baselines. We apply EfficientQAT [5] trained on our data for the same 805M tokens (1.00 GB with a bf16 head) keeps a 7-task mean of 52.0 but scores 28.8 on sampled MATH-500, against 43.0 for the QAT checkpoint of about the same size (QAT-hbf16) and 47.5 for the recalibrated checkpoint (Rec-hbf16) (both ); it closes 56% of its traces (Table 26 and section E). The official EfficientQAT code on the same data scores 34.2 (Table 7). Furthermore at every model size EfficientQAT outperforms BitDistiller [13] on reasoning (MATH-500 and AIME@16).
| Model | Checkpoint | GB | strict | lenient | closed (%) | strict if closed |
| Qwen3-1.7B | QAT+R-h2 (selected) | 0.52 | 73.0 | 74.6 | 87.1 | 83.8 |
| Qwen3-1.7B | Rec-h2 | 0.52 | 46.7 | 47.3 | 86.0 | 54.2 |
| Qwen3-1.7B | QAT-h2 | 0.52 | 42.0 | 44.9 | 70.0 | 60.0 |
| bits per parameter | ||||||
| Qwen3-1.7B | GGUF UD-IQ2_XXS [66] | 0.61 | 1.0 | 2.8 | 46.8 | 2.1 |
| Qwen3-1.7B | GGUF UD-IQ2_M [65] | 0.71 | 24.6 | 27.1 | 46.7 | 52.6 |
| Qwen3-1.7B | GGUF Q2_K [62] | 0.78 | 41.9 | 44.8 | 56.0 | 74.7 |
| >4 bits per parameter | ||||||
| Qwen3-1.7B | GGUF Q3_K_M [63] | 0.94 | 84.3 | 84.7 | 98.4 | 85.7 |
| Qwen3-1.7B | EfficientQAT port, same data | 1.00 | 28.8 | 32.8 | 55.9 | 51.4 |
| Qwen3-1.7B | GGUF Q4_K_M [64] | 1.11 | 90.8 | 90.9 | 99.3 | 91.4 |
| bits per parameter | ||||||
| Qwen3-8B | QAT+R-h2 (selected) | 2.08 | 92.2 | 93.2 | 95.4 | 96.7 |
| Qwen3-8B | Rec-h2 | 2.08 | 86.6 | 87.3 | 94.5 | 91.6 |
| Qwen3-8B | GGUF UD-IQ2_XXS [74] | 2.61 | 35.2 | 49.4 | 39.6 | 89.0 |
| – bits per parameter | ||||||
| Qwen3-8B | GGUF UD-IQ2_M [73] | 3.11 | 92.0 | 92.7 | 96.7 | 95.1 |
| Qwen3-8B | GGUF Q2_K [71] | 3.28 | 91.1 | 92.5 | 95.0 | 95.8 |
| bits per parameter | ||||||
| Qwen3-14B | QAT+R-h2 (selected) | 3.74 | 94.8 | 95.4 | 98.3 | 96.5 |
| Qwen3-14B | Rec-h2 | 3.74 | 92.7 | 93.0 | 96.5 | 96.1 |
| Qwen3-14B | QAT-h2 | 3.74 | 90.5 | 91.2 | 96.8 | 93.6 |
| Qwen3-14B | GGUF UD-IQ2_XXS [69] | 4.47 | 85.2 | 85.8 | 94.6 | 90.1 |
| – bits per parameter | ||||||
| Qwen3-14B | GGUF UD-IQ2_M [68] | 5.43 | 95.0 | 95.2 | 98.9 | 96.0 |
| Qwen3-14B | GGUF Q2_K [67] | 5.75 | 95.0 | 95.0 | 98.8 | 96.1 |
| Smaller model (reference) | ||||||
| Qwen3-4B | GGUF Q4_K_M [70] | 2.50 | 96.2 | 96.3 | 99.6 | 96.6 |
Table 26. QAT+R-h2 closes more traces than Q2_K and is more accurate on them; the earlier 0.52 GB checkpoints win only by closing more traces. Sampled MATH-500 split into traces that close within 32k tokens and strict accuracy on them. Rows grouped by bits per parameter.
Checkpoint selection without the test set. The framework produces two candidates per model, the recalibrated and the QAT checkpoint (Rec, QAT), and the selector decides which one to keep. It samples four completions for each of 313 level-3–5 MATH training problems (disjoint from MATH-500), scores strictly, and keeps the higher mean if the paired sign-flip test gives , else the smaller checkpoint. At 1.7B it keeps the recalibrated checkpoint at both 0.98 and 0.52 GB (Rec-hbf16, Rec-h2), and the smaller of the two; at 14B it keeps the 3.74 GB recalibrated checkpoint (Rec-h2), which beats the QAT checkpoint of equal size (QAT-h2) and ties the 6.4 GB bf16-head version (Rec-hbf16) (section E). MATH-500 agrees with every decision (Table 17 and Table 18). With the reasoning mixture, the selector compares QAT+R-h2 with Rec+R-h2 (§ 5). At 0.52 GB the preference is a termination effect: forced answers tie (section K).
| Model | Contrast (A vs. B) | A | B | Δ [95% CI] | status | |
| Qwen3-1.7B | QAT-h2 vs. Rec-h2 (0.52 GB) | 42.0 | 46.7 | −4.7 [−7.1, −2.2] | post hoc | |
| Rec-h2 vs. GGUF Q2_K [62] (0.78 GB) | 46.7 | 41.9 | +4.8 [2.0, 7.7] | 0.001 | post hoc | |
| Rec-h2 vs. Rec-hbf16 (0.52 vs. 0.98 GB) | 46.7 | 47.5 | −0.8 [−2.7, 1.0] | 0.44 | post hoc | |
| Qwen3-14B | Rec-hbf16 (6.4 GB) vs. QAT-h2 (3.74 GB) | 92.8 | 90.5 | +2.3 [0.9, 3.7] | 0.002 | post hoc |
| QAT-h2 vs. Rec-h2 (3.74 GB) | 90.5 | 92.7 | −2.2 [−3.8, −0.6] | 0.011 | post hoc | |
| codeword search at the recalibrated checkpoint (Rec-hbf16 vs. LW-Rec-hbf16; exploratory) | 92.8 | 89.7 | +3.2 [1.8, 4.7] | post hoc | ||
| same, AIME@16§ | 48.2 | 36.7 | +11.6 [7.3, 16.1] | post hoc | ||
| QAT-only-h2 (3.74 GB) vs. LW-Rec-hbf16 (6.4 GB) | 89.6 | 89.7 | −0.1 [−1.7, 1.5] | 1.0 | post hoc | |
| QAT-only-h2 (3.74 GB) vs. GGUF UD-IQ2_XXS [69] (4.47 GB) | 89.6 | 85.2 | +4.4 [2.6, 6.2] | post hoc | ||
| QAT-h2 vs. QAT-only-h2 (pre-registered) | 90.5 | 89.6 | +0.9 [−0.4, 2.3] | 0.18 | pre-reg. | |
| same, AIME@16 (secondary) | 40.9 | 37.2 | +3.8 [0.7, 7.0] | 0.032 | pre-reg. | |
| QAT (3.74 GB) vs. GGUF UD-IQ2_XXS [69] (4.47 GB) | 90.5 | 85.2 | +5.3 [3.5, 7.1] | post hoc | ||
| Rec-h2 (base-mixture selection, 3.74 GB) vs. GGUF UD-IQ2_XXS [69] (4.47 GB) | 92.7 | 85.2 | +7.5 [5.7, 9.5] | check |
Table 17. At 0.52 GB, Rec-h2 beats QAT-h2 and the larger Q2_K; at 14B every two-bit checkpoint beats the smallest GGUF. Paired MATH-500 contrasts (AIME@16 where noted). Bold: significant after Holm correction together with Table 9. §Separate evaluation run. Status: pre-reg., pre-registered; post hoc, chosen after earlier results; check, robustness check.
| Model | A vs. B | A | B | kept | |
| Qwen3-1.7B | QAT-hbf16 vs. Rec-hbf16 | 22.5 | 28.8 | Rec-hbf16 | |
| Qwen3-1.7B | QAT-h2 vs. Rec-h2 | 21.9 | 27.6 | Rec-h2 | |
| Qwen3-1.7B | Rec-hbf16 vs. Rec-h2 | 28.8 | 27.6 | 0.37 | Rec-h2 |
| Qwen3-14B | QAT-h2 vs. Rec-hbf16 | 83.3 | 87.9 | Rec-hbf16 | |
| Qwen3-14B | QAT-h2 vs. Rec-h2 | 83.3 | 87.5 | Rec-h2 | |
| Qwen3-14B | Rec-hbf16 vs. Rec-h2 | 87.9 | 87.5 | 0.73 | Rec-h2 |
| Qwen3-8B | Rec-hbf16 vs. Rec-h2 | 75.6 | 76.1 | 0.72 | Rec-h2 |
Table 18. Under the base mixture the selector kept the recalibrated checkpoint in every decision, and MATH-500 agrees. A, B: strict accuracy on the 313 selection problems.
Related work
Post-training quantization of LLMs. The first LLM quantizers targeted 8-bit weights and activations, using fine-grained quantization with layer-wise distillation (ZeroQuant [81]), mixed-precision decomposition of outlier features (LLM.int8() [7]) or rescaling that moves outliers from activations to weights (SmoothQuant [78]). For weight-only quantization, GPTQ [19] turns Optimal Brain Compression [18] into one-shot second-order rounding, AWQ [40] protects salient channels by rescaling, and SpQR [9] and SqueezeLLM [31] store sparse outlier weights at higher precision. Learned rounding and clipping optimize layer or block reconstruction (AdaRound [52], BRECQ [38], OmniQuant [57], CBQ [10], Tessaera [37]). Most of these methods are close to lossless at three to four bits but degrade sharply at two. GSQ [6], which we build on, learns each weight’s grid assignment and the group scales jointly through a Gumbel-softmax relaxation of a scalar grid.
Two-bit and sub-two-bit formats. The strongest two-bit results leave the scalar format. Incoherence processing (QuIP [4]), combined with lattice or trellis codebooks (QuIP# [58], QTIP [59]), and additive vector quantization with fine-tuning (AQLM [15], PV-Tuning [48]) reach strong perplexity, and fixed or learned rotations (QuaRot [2], SpinQuant [44]) remove outliers so that weights, activations and the KV cache are easier to quantize. Binary and partially binarized quantization (BiLLM [24], PB-LLM [56]) reaches about one bit per weight at a clear accuracy cost, while binary and ternary models trained from scratch (BitNet [76], BitNet b1.58 [46], Spectra [29]) show that very low-bit weights can be competitive when the pretraining budget allows. We keep a strictly scalar format with one bf16 scale per group of 128 weights, which dequantizes with one multiplication and needs no codebook or rotation at inference; rotations are orthogonal to our stages and could be combined with them.
Quantization-aware training and distillation. QAT trains through the rounding with straight-through estimators [3, 25] or learned step sizes [16]. For LLMs, LLM-QAT [43] distills from the full-precision model on self-generated data, BitDistiller [13] combines asymmetric quantization with self-distillation, EfficientQAT [5] trains all parameters block by block and then only the quantization step sizes end to end, OneBit [79] factors weights into signs and value vectors, and ParetoQ [45] compares 1- to 4-bit QAT within one framework. Parameter-efficient methods train only scales, adapters or normalization layers on frozen low-bit weights (PEQA [30], QLoRA [8], norm tweaking [35]); our scale recalibration is closest to these, but fits scales and gains to the bf16 model’s output distribution with codewords frozen. Our recalibration and QAT distill with the forward KL to the bf16 model [23]; reverse-KL and on-policy distillation [1, 22] are alternatives we do not use. We show that, at a fixed token budget, the post-training starting point and a reasoning-based selector decide how much reasoning survives.
Mixed precision and sensitivity. Since HAWQ, second-order sensitivity has guided bit allocation [11, 12]. KronQ [34] scores each layer with the activation and gradient traces of a Kronecker-factored Hessian approximation (as in K-FAC [49]); we use its score to choose ternary, 2-bit or 3-bit tiers under a bit budget (section 2).
Evaluating quantized models. Systematic evaluations find that perplexity can miss losses on knowledge-intensive tasks [26] and that quantization affects dialogue and long-context tasks differently from basic NLP tasks [36], although perplexity remains a reasonable proxy on many standard benchmarks [28]; perplexity is also unreliable for long-context ability in general [17]. Matched accuracy can hide answer flips [14], and 8-bit and 4-bit weight-only formats are close to lossless across academic and real-world tasks [33]. Scaling studies relate the useful precision to model size and training tokens [32, 53]. For reasoning models, quantization below four bits hurts most on hard problems [42]; whether it lengthens traces is disputed, with no increase reported by Liu et al. [42] and token inflation at 3 and 4 bits by Lian et al. [39]. Qwen3 has been evaluated down to 1 bit with perplexity and zero-shot accuracy [83]; reasoning-specific quantizers use fine-tuning signals [82] or the model’s own reasoning traces as calibration data [77]; and budget forcing [51] controls how long a model thinks. These studies compare a quantized model with its full-precision parent. We use paired tests [50] to show that likelihood also fails to rank competing two-bit configurations, and select checkpoints on held-out reasoning problems instead.
Conclusion
We propose ThinkBit, a strictly scalar two-bit quantization framework designed to preserve reasoning. ThinkBit allocates a ternary, 2-bit or 3-bit tier to each layer by sensitivity, searches the integer codewords block by block, recalibrates the stored scales with the codewords frozen, lets quantization-aware training reassign codewords, and keeps the better checkpoint by a paired reasoning test on training problems. Our main finding is that likelihood-based metrics hide what two-bit quantization does to reasoning: removing codeword search, scale recalibration or tier allocation barely changes the log-likelihood mean but clearly lowers sampled MATH-500, and selecting checkpoints by a reasoning test, without touching the test set, picks the stronger model. Trained on the same data, our checkpoints reason better than EfficientQAT and BitDistiller at every size we compared. With reasoning traces in the final training stage, the selected checkpoints are smaller than every published 2-bit GGUF of the same model; they reason better than all of them at 1.7B, match Q2_K at 8B and 14B at about two thirds of its size, and at 14B come within a few points of bf16 on MATH-500. Files of four or more bits per parameter remain stronger, and some two-bit reasoning traces still do not finish.
There are many potential directions for future work. First, kernels that read 2- and 3-bit codewords directly would turn the smaller files into proportional memory and speed gains at inference time, which the 4-bit containers used here cannot show. Second, training objectives that reward finishing a reasoning trace, such as on-policy distillation from the bf16 teacher or reinforcement learning after quantization, may close much of the remaining gap to bf16. Third, the stage study should be repeated on more model families, mixture-of-experts models and models above 14B, and the tier allocation could be extended below two bits. Finally, selectors for code and tool use would extend checkpoint selection beyond math problems. We will release the code upon acceptance.
Full algorithmic procedure and notation
Algorithm Algorithm 2 presents ThinkBit as procedures, from the bf16 model to the selected checkpoint; Table 10 collects the notation of § 2.
Algorithm 2 ThinkBit, end to end
function TWOBIT()
▷ Stage 1, eq. (5)
▷ algorithm 1
▷ Stage 2
▷ Stage 3
▷ Stage 4
return ▷ Stage 5
function CODEWORDSEARCH()
inputs of block 1 on
for block do
for group of block do ▷ from the group’s current inputs
for layer in do ▷ LayerRefine: eq. (9) via eqs. (11) to (13)
if is ternary then
GPTQ’s scale rule on ;
else
▷ eqs. (9) and (10)
if is a query or key projection then
for every layer of except query and key, which keep
, annealing ▷ eq. (14)
the most likely value of each codeword
▷ inputs of the next block
return
function RECALIBRATE() ▷ frozen
on with clamped, best held-out KL ▷ eq. (15)
return model with ,
function QAT()
▷ eq. (17)
with the STE, scales and gains still training, best held-out KL ▷ eqs. (16) and (18)
return model with and the trained scales and gains
function SELECT()
fraction of 4 sampled answers that are strictly correct, for each
▷ eq. (19)
if then return the candidate with higher mean accuracy
else return the smaller candidate (the recalibrated one if equal in size)
Evaluation settings
Every checkpoint, including the GGUF baselines, is exported to dense bf16 and scored by the same lm-eval and vLLM stack with the same settings (Table 11). Thinking-mode results use the models’ chat template; all other suites use none.
| Suite | Tasks | Shots / format | Decoding |
| 7-task core | ARC-e/c, WinoGrande, HellaSwag, PIQA, LAMBADA, MMLU | lm-eval defaults (0-shot), log-likelihood | – |
| Generative | GSM8K (strict match) | 5-shot, no chat template | greedy |
| HumanEval | 0-shot, pass@1 | greedy | |
| Tool use | BFCL v4: non-live (syntax-tree match), live, multi-turn, irrelevance | official harness, native tool format | greedy, vLLM server |
| Broad suite | MMLU-Pro, TriviaQA, TruthfulQA, SciQ, OpenBookQA, CommonsenseQA, BoolQ, RACE, ARC-C, BBH-CoT, MATH-500, GSM8K-CoT, IFEval, MBPP, HumanEval+ | task defaults; ARC-C 25-shot; BBH 3-shot CoT; GSM8K-CoT 8-shot | greedy |
| Thinking mode | AIME24, AIME25 (30 problems each), MATH-500 | chat template, thinking on, 0-shot | greedy, 1 sample; and sampled (temperature 0.6, top- 0.95, top- 20, 4 samples, strict scoring); 32k new tokens |
Table 11. Evaluation settings. GGUF files are dequantized to bf16; RACE is excluded from retention means because its passages appear in MMLU’s auxiliary data.
Per-task likelihood scores
Each stage raises all seven tasks (Table 12). After QAT the largest remaining gaps to bf16 are on HellaSwag (7.8 points) and MMLU (5.6).
| Qwen3-1.7B | ARC-e | ARC-c | Wino | HSwag | PIQA | LAMB | MMLU | Mean |
| bf16 | 69.7 | 43.0 | 60.9 | 60.4 | 72.2 | 50.5 | 55.5 | 58.9 |
| GSQ-hbf16 | 51.4 | 29.9 | 54.3 | 41.5 | 64.9 | 28.4 | 36.4 | 43.8 |
| Rec-hbf16 | 62.4 | 36.6 | 57.1 | 49.2 | 68.2 | 42.7 | 44.4 | 51.5 |
| QAT-hbf16 | 66.4 | 38.2 | 59.2 | 52.6 | 69.0 | 47.1 | 49.9 | 54.6 |
| QAT-h2 | 65.1 | 37.5 | 61.0 | 51.4 | 69.2 | 45.5 | 48.0 | 54.0 |
| QAT+R-h2 (selected) | 66.3 | 38.6 | 58.3 | 51.2 | 70.4 | 44.7 | 47.0 | 53.8 |
| GGUF Q2_K [62] | 57.7 | 34.6 | 56.5 | 52.2 | 67.4 | 35.9 | 40.0 | 49.2 |
Table 12. The 7-task means of Table 4 task by task (length-normalized accuracy for ARC, HellaSwag and PIQA). Chance-normalized retention after GSQ / recalibration / QAT: 47.6 / 73.1 / 84.2%.
Codeword search across depth and the query–key rule
Figure 5 tracks WikiText-2 perplexity as GSQ quantizes one block after another. On Qwen3-1.7B it stays near bf16 for the first half of the model and rises over the last third.
Figure 5. GSQ perplexity relative to the model’s own bf16 as blocks are quantized (before recalibration), Qwen3-1.7B.
Forcing query and key projections to two bits. ThinkBit keeps query and key projections at bits, a rule chosen on a model without query–key normalization. Without the rule, the allocator makes 15 of 28 query and 3 of 28 key projections ternary and spends the freed bits on 3-bit layers, at the same 2.0 decoder bits per weight. Every subsequent measurement is close to that of the forced run, and neither reasoning difference is significant (table 13); on Qwen3-1.7B the rule is unnecessary.
| Qwen3-1.7B | query/key at bits (ThinkBit) | not forced |
| ternary query / key projections (of 28) | 0 / 0 | 15 / 3 |
| KronQ objective (relative) | 1.000 | 0.997 |
| WikiText-2 after codeword search | 46.91 | 46.35 |
| held-out KL after recalibration | 0.282 | 0.279 |
| WikiText-2 after recalibration | 19.46 | 19.56 |
| sampled MATH-500 after recalibration | 47.5 | 49.3 |
Table 13. Keeping query and key projections at 2 bits or more changes reasoning by less than 2 points (MATH-500 +1.8, ). Otherwise identical settings.
Further ablations
This section details the controls of section 7. Table 14 compares codeword search with stronger GPTQ starts, Table 15 collects the remaining paired contrasts, and Table 16 tests equivalence on the 7-task suite. Unless noted, reasoning is sampled, strictly scored MATH-500 with paired tests.
| Qwen3-1.7B start | WikiText-2 before rec. | WikiText-2 after rec. | held-out KL | MATH-500 after rec. | MATH-500 after QAT | 7-task | GSM8K | HumanEval | BFCL | NL |
| GSQ (framework) | 46.91 | 19.46 | 0.282 | 47.5 | 43.0 | 54.6 | 50.2 | 17.7 | 75.5 | |
| GPTQ, all tiers, clipped scales | 1,379 | 22.02 | 0.402 | 18.5 | 32.6 | 53.8 | 49.4 | 17.7 | 70.0 | |
| GPTQ + Hessian-weighted rounding | 10,352 | 21.06 | 0.367 | 24.1 | 37.8 | 53.8 | 51.9 | 17.7 | 71.9 |
Table 14. Codeword search beats stronger GPTQ starts on reasoning after the same recalibration (rec.) and QAT, while likelihood barely separates them. Bold: best per column.
| Model | Contrast (A vs. B) | sampled MATH-500 A | sampled MATH-500 B | [95% CI] | selector [95% CI] | selector | |
| Qwen3-1.7B | Rec-h2: two recalibration passes vs. one | 46.7 | 44.4 | +2.3 [0.3, 4.2] | 0.027 | +0.6 | 0.71 |
| Qwen3-1.7B | Rec-h2 (1 pass) vs. QAT-h2 | 44.4 | 42.0 | – | – | +5.1 [2.4, 8.0] | |
| Qwen3-1.7B | Rec-hbf16: two passes vs. one pass | 46.6 | 47.5 | –0.9 | 0.38 | –2.3 | 0.052 |
| Qwen3-1.7B | Rec-hbf16: Rec-h2 vs. Rec-hbf16 | 46.7 | 46.6 | +0.1 | 0.96 | – | – |
| Qwen3-1.7B | QAT-h2 vs. QAT-noRec-h2 | 42.0 | 38.2 | +3.8 [1.7, 6.0] | +3.2 [1.0, 5.6] | 0.009 | |
| Qwen3-14B | Rec-h2 vs. Rec-hbf16 | 92.7 | 92.8 | –0.15 [–1.45, 1.20] | 0.88 | –0.4 [–2.2, 1.4] | 0.73 |
Table 15. Quantizing the head and embedding costs no detectable accuracy. Further paired contrasts (pointwise tests, not Holm-corrected).
| QAT vs. | 7-task Δ [90% CI] | equiv. | MMLU Δ [90% CI] | equiv. |
| LW-QAT: layer-wise start instead of GSQ | +0.91 [0.37, 1.44] | yes | +1.22 [0.61, 1.84] | no |
| without the scale recalibration | +0.18 [−0.29, 0.63] | yes | +1.03 [0.47, 1.59] | no |
| QAT-only (neither) | −0.15 [−0.64, 0.34] | yes | −0.02 [−0.69, 0.65] | yes |
| uniform 2-bit, GPTQ, recalibration | +0.20 [−0.35, 0.73] | yes | +0.72 [0.08, 1.39] | yes |
| tuned GPTQ start | +0.79 [0.32, 1.28] | yes | +2.07 [1.47, 2.68] | no |
| GPTQ + Hessian-weighted rounding | +0.80 [0.31, 1.32] | yes | +1.62 [0.98, 2.29] | no |
| Rec (no QAT) | +3.13 [2.63, 3.61] | no | +5.54 [4.91, 6.19] | no |
Table 16. Every stage removal is equivalent to the full framework on the 7-task mean (margin ±1.5); skipping QAT is not. Δ: QAT-hbf16 minus the named checkpoint.
Rounding-only alternatives to QAT. Moving codewords toward their bf16 values without training, then briefly recalibrating the scales, is worse than not moving them (“re-round” rows of Table 8). GSQ’s non-nearest codewords therefore beat nearest rounding; these short controls do not separate QAT’s choice of codewords from its longer optimization.
Scales, gains and reverted reassignments. Recalibrating the scales alone matches recalibrating scales and gains, while the gains alone recover much less (Table 8 and Table 19). Reverting QAT’s MLP reassignments lowers MMLU but raises HumanEval, and reverting its attention reassignments lowers GSM8K (Table 8). These are interventions on a single checkpoint.
Initialization and the reassigned codewords. Latents that start on their codewords () are almost never reassigned, at either learning rate (section I). With the reassignments are single grid steps toward the bf16 weight, concentrated in value projections and the 3-bit tier (section J); their direction largely follows from eq. (17), which starts each latent on the side of its codeword where the bf16 weight lies.
Rules with a likelihood conjunct. Two likelihood rules, written before the reasoning results, would have removed stages that the paired reasoning tests show are needed. Codeword search was to stay only if the layer-wise start lost at least 3 MMLU and 5 thinking-math points; the MMLU gap is 1.2, so the rule drops the search. Recalibration was to stay if skipping it cost more than one MMLU point; it costs 1.03, a boundary case.
Stronger starts. To test whether codeword search only beats a weak start, two starts replace the layer-wise one with the same tier map: GPTQ on every tier with MSE-clipped scales, and the same followed by Hessian-weighted Gumbel rounding. Each then receives the same recalibration and QAT. After recalibration their perplexities are close to ours, but their reasoning is more than 20 points lower; QAT narrows the gap without closing it, while the likelihood and code columns barely separate the three (Table 14; paired tests in table 9).
Remaining paired contrasts. Table 15 lists contrasts on the number of recalibration passes, on recalibration at 0.52 GB and on head precision. A second recalibration pass that quantizes the head and embedding beats a single pass on MATH-500 but not on the selector; with either, the recalibrated checkpoint beats the QAT checkpoint at 0.52 GB. Recalibration before QAT still helps at 0.52 GB, also under budget forcing, but misses the 4-point threshold set in advance, so the 0.98 GB estimate of table 9 is not claimed at this size. Quantizing the head and embedding of the 14B recalibrated checkpoint doesn’t change anything significantly.
Same-data QAT baseline. Our application of EfficientQAT [5] to Qwen3 trains blocks and then the quantization parameters end to end, with 2-bit codewords, per-group zero points and 16-bit scales (groups of 128, 2.14 bits per decoder weight; bf16 head and embedding). Trained on our data for the same 805M tokens as recalibration plus QAT, it is close to our checkpoints on likelihood but trails them on reasoning, mostly because fewer of its traces close (Table 26).
Training data. Removing Wikipedia and MMLU-auxiliary data, and then Evol-Instruct code, from QAT changes the 7-task mean by at most 0.4 (Table 8); these ablations do not cover earlier stages or the self-generated replies.
Equivalence tests on likelihood. The 7-task suite was re-run on eight Qwen3-1.7B checkpoints with per-item logging. For each contrast with the QAT checkpoint, a paired bootstrap gives a 90% interval, and two one-sided tests at level 0.05 declare equivalence when it lies inside a ±1.5-point margin fixed in advance (Table 16). Every removal is equivalent on the 7-task mean. MMLU is not equivalent for four removals, but these differences are far smaller than the MATH-500 differences of the same contrasts. QAT against the recalibrated checkpoint is equivalent on neither measure.
Paired tests by size, head precision and scale
Table 17 extends the paired tests of Table 9. At 0.52 GB the recalibrated checkpoint beats both the QAT checkpoint and the larger Q2_K file [62]; quantizing the head and embedding costs no detectable accuracy; and at 14B every two-bit checkpoint beats the smallest GGUF [69], while the pre-registered QAT-versus-QAT-only contrast is not significant.
How checkpoints are selected
The selector samples four answers per problem on 313 level-3–5 MATH training problems that are disjoint from MATH-500 and from the training prompts, scores them strictly, and keeps the higher mean when the paired sign-flip test gives , otherwise the smaller checkpoint. Table 18 lists every decision; MATH-500 agrees with each of them.
What the recalibration changes
With codewords frozen, training the scales alone lowers held-out KL from 0.641 to 0.282, the same as training scales and RMSNorm gains together, while the gains alone reach 0.418 (Table 19).
Table 20 compares three scales for each group of the frozen codewords : the GSQ scale , the recalibrated scale (eq. (15)), and the weight-space least-squares scale . If recalibration repaired weight error, would follow . It does not: the signs agree in about half of the groups in every tier, the correlation is at most 0.08, and weight error rises in every tier.
Table 21 measures where the output improves. The local error feeds each quantized block the bf16 input of that block; the propagated error runs the whole quantized model. Recalibration and QAT raise both errors, averaged over blocks, while lowering the KL of the next-token distribution.
| Qwen3-1.7B | local block error | propagated error | next-token KL |
| GSQ-hbf16 | 0.079 | 0.214 | 1.300 |
| Rec-hbf16 | 0.094 | 0.282 | 0.571 |
| QAT-hbf16 | 0.122 | 0.373 | 0.371 |
Table 21. Recalibration and QAT make single blocks less faithful but bring the output distribution closer to bf16. Relative block-output error (local: bf16 input; propagated: fully quantized model) and output KL on WikiText-2.
QAT initialization and learning rate
Figure 6 crosses the latent initialization of eq. (17) with the learning rate. Latents that start on their codewords () almost never cross a rounding boundary at either learning rate. With , KL first rises while codewords are reassigned and then falls below its starting value.
Figure 6. QAT latent initialization learning rate on Qwen3-1.7B (short runs from the recalibrated model). (a) Held-out KL relative to the start. (b) Share of codewords changed. With fewer than 0.003% of codewords are reassigned. Validation sequences are the chunks that follow each run’s training slice, so runs with different training lengths are scored on different held-out text; this is why the starting KL (0.251) differs from long QAT’s (0.311). KL values are comparable within a run, not across runs.
Where QAT reassigns codewords
QAT reassigns 6.4% of the codewords. Figure 7 shows that the reassignments concentrate in value projections, the 3-bit tier and the last block, and that the reassigned codewords are mostly those already more than half a grid step from their bf16 value; almost every reassignment is a single step toward that value.
Figure 7. Where long QAT reassigns codewords on Qwen3-1.7B. (a) Share of each projection type’s codewords that QAT changes; value projections (orange) change most, query and key projections (dark blue) least. (b) Share changed in each transformer block, with the share per tier in the inset text. Codewords at the first and last transformer blocks are reassigned more than other blocks(c) Distribution, before QAT, of the distance between each codeword and its bf16 weight , in grid steps with the recalibrated scales, for all codewords (grey) and for those that QAT reassigns (orange). The dotted line at 0.5 is the rounding boundary: most reassigned codewords lie beyond it, and 99.8% of reassignments move one step toward the bf16 weight.
Forcing an answer at a token budget
Budget forcing [51] stops the thinking after a budget of 1k–16k tokens and asks for an answer, which separates finishing a trace from solving the problem. With a forced answer the 0.52 GB checkpoints tie Q2_K [62] at every budget, so their lead without forcing comes from finishing more traces (Table 22).
| Thinking budget (tokens) | 1k | 2k | 4k | 8k | 16k | 32k, unforced |
| bf16 | 74.2 | 77.0 | 82.4 | 86.0 | 90.0 | 90.2 |
| Rec-hbf16 | 40.4 | 43.2 | 44.4 | 44.6 | 45.2 | 40.6 |
| QAT-hbf16 | 42.2 | 43.6 | 44.4 | 43.4 | 39.2 | 39.0 |
| QAT-only-hbf16 | 33.6 | 34.6 | 36.6 | 31.0 | 28.8 | 26.6 |
| 0.52 GB checkpoints (2-bit head and embedding) and Q2_K | ||||||
| Rec-h2, 0.52 GB | 40.6 | 42.0 | 43.4 | 42.6 | 41.0 | 39.0 |
| QAT-h2, 0.52 GB | 40.2 | 43.6 | 45.8 | 42.6 | 37.2 | 38.8 |
| GGUF Q2_K [62], 0.78 GB | 41.4 | 41.8 | 43.2 | 40.4 | 41.4 | 27.8 |
| GGUF UD-IQ2_M [65], 0.71 GB | 28.2 | 26.8 | 27.8 | 25.6 | 24.8 | 14.2 |
| GGUF UD-IQ2_XXS [66], 0.61 GB | 4.0 | 4.0 | 4.6 | 4.0 | 3.8 | 2.8 |
Table 22. With a forced answer the 0.52 GB checkpoints and Q2_K tie, so the strict lead of Rec-h2 comes from finishing more traces. Greedy thinking-mode MATH-500 on Qwen3-1.7B with the answer forced after the given number of thinking tokens.
Trace closing and accuracy on closed traces
Table 23 splits each paired MATH-500 contrast into a change in how often traces close and a change in accuracy on closed traces. Codeword search mainly improves accuracy, whereas the recalibration and QAT mainly change how often traces close.
Packed files and serving
Training checkpoints store every codeword in a 4-bit container. We re-pack checkpoints at their true widths and decode them again; every codeword round-trips bit-exactly and lies on its layer’s grid, and a reference decoder that reads only the packed file rebuilds a checkpoint that loads in stock transformers (Table 24). The measured sizes are within 0.6% of those computed from the grids. For Qwen3-1.7B the bf16 tied embedding is about 64% of the 0.977 GB file, so quantizing the head and embedding matters at small scale.
| Model | Checkpoint | quantized weights | measured | computed | file (GB) |
| Qwen3-1.7B | QAT-hbf16 | 1.41B | 2.011 | 2.000 | 0.977 |
| QAT-h2 | 2.03B | 2.046 | 2.038 | 0.520 | |
| Rec-h2 | 2.03B | 2.046 | 2.038 | 0.520 | |
| Qwen3-14B | QAT-only-h2 | 14.77B | 2.023 | 2.013 | 3.736 |
| Rec-h2 | 14.77B | 2.023 | 2.013 | 3.736 | |
| Qwen3-8B | Rec-h2 | 8.19B | 2.029 | 2.019 | 2.078 |
Table 24. Packed files decode bit-exactly and are within 0.6% of the sizes computed from the grids. Bits per quantized weight include the group scales.
vLLM serves our checkpoints directly from their 4-bit containers (compressed-tensors format, Machete kernels; decoder weights are never expanded, embedding and head stay in bf16) on one H100 (Table 25). These kernels read 4 bits per codeword, so memory and speed reflect the container rather than the 2-bit widths of the packed file; at 1.7B the bf16 embedding and head dominate and decoding is not limited by weight bandwidth.
| Qwen3 | weight memory (GiB) | decode (tokens/s), batch 1 | decode (tokens/s), batch 16 | prefill, 1,024 tokens (ms) | top-1 agreement (%) |
| 1.7B | 3.22 1.85 | 424 408 | 6,232 5,998 | 12.8 15.7 | 98.7 |
| 8B | 15.27 5.68 | 146 212 | 2,225 3,184 | 27.0 28.4 | 99.1 |
| 14B | 27.52 9.31 | 86 150 | 1,341 2,302 | 48.0 45.0 | 98.3 |
Table 25. Serving from packed containers cuts weight memory 1.7–3.0 and speeds up batch-1 decoding at 8B and 14B. vLLM on one H100; memory in GiB.
Retention on a broad benchmark suite
Most knowledge and commonsense tasks keep 80% or more of bf16’s score after QAT, while MATH-500, TriviaQA and HumanEval+ keep about half (Figure 8).
Figure 8. Qwen3-1.7B: retention on 14 benchmarks after recalibration and after QAT (Rec-hbf16, QAT-hbf16) (RACE omitted; its test passages appear in MMLU’s auxiliary training data). Numbers above bars are QAT retention.
Accuracy by model size in bytes
Figure 9 places the Qwen3-1.7B checkpoints among the GGUF files of the same model on four benchmarks, and Table 26 splits sampled MATH-500 into closing rate and accuracy on closed traces for our checkpoints, the GGUF files of Figure 3 and the EfficientQAT port.
Figure 9. Qwen3-1.7B: score against whole-model bytes (generation metrics greedy). Grey: published llama.cpp quantizations (Unsloth [75]; the UD- prefix is omitted in labels). Red star: ours with a 2-bit head and embedding (QAT-h2); orange diamond: ours with the bf16 tied embedding (QAT-hbf16). Dotted: bf16. Over 30 individually scored sub-scores (RACE excluded; greedy MATH counted; differences of at most 0.5 points count as ties) the 0.52 GB QAT checkpoint (QAT-h2) wins 23–26 against each 2-bit GGUF.
The 0.52 GB recalibrated checkpoint selected under the base mixture (Rec-h2) also scores BFCL live 59.9 and multi-turn 2.4 (non-live 73.0) and AIME@16 1.0; like the 2-bit GGUFs of this model, it does not solve competition problems.
Qwen3-14B compared by size
At 3.74 GB our Qwen3-14B checkpoints beat the smallest 14B GGUF (4.47 GB) on every reasoning column, while the larger 2-bit GGUFs and smaller models at four to five bits per parameter score higher (Table 27).
| Checkpoint | Size (GB) ↓ | 7-task ↑ | MMLU ↑ | GSM8K ↑ | HE ↑ | MT ↑ | M500 ↑ | A24/25 ↑ | A@16 ↑ |
| Qwen3-14B bf16 | 29.5 | 74.3 | 77.1 | 88.2 | 57.3 | 40.8 | 97.5 | 76.7/60.0 | 73.6 |
| Smaller models at 4–5 bits per parameter (reference) | |||||||||
| Qwen3-4B GGUF Q4_K_M [70] | 2.50 | 66.7 | 68.3 | 84.8 | 70.1 | 30.0 | 96.3 | 73.3/50.0 | 65.3 |
| Qwen3-8B GGUF Q3_K_M [72] | 4.12 | 69.2 | 71.0 | 86.5 | 61.0 | 31.6 | 95.8 | 56.7/50.0 | 64.1 |
| Qwen3-14B, bits per parameter | |||||||||
| Ours: Rec-h2 | 3.74 | 69.9 | 69.5 | 87.7 | 65.2 | 32.0 | 92.7 | 50.0/33.3 | 46.5 |
| Ours: QAT-h2 | 3.74† | 70.6 | 70.2 | 86.9 | 60.4 | 27.8 | 90.5 | 43.3/26.7 | 40.9 |
| QAT-only-h2 | 3.74 | 70.2 | 69.9 | 85.7 | 62.8 | 27.2 | 89.6 | 30.0/20.0 | 37.2 |
| GGUF UD-IQ2_XXS [69] | 4.47 | 66.7 | 65.2 | 71.2 | 39.0 | 20.8 | 85.2 | 13.3/13.3 | 26.7 |
| Qwen3-14B, 2.6–4 bits per parameter | |||||||||
| Ours: Rec-hbf16 | 6.4* | 70.3§ | 70.1§ | 87.4§ | 56.1§ | 31.0 | 92.8 | 43.3/43.3§ | 48.2§ |
| LW-Rec-hbf16 | 6.4* | 69.1 | 67.9 | 85.2 | 55.5 | 27.8 | 89.7 | 33.3/33.3 | 36.7§ |
| GGUF UD-IQ2_M [68] | 5.43 | 71.0 | 72.0 | 85.2 | 65.2 | 32.2 | 95.0 | 66.7/60.0 | 60.5 |
| GGUF Q2_K [67] | 5.75 | 71.6 | 73.8 | 87.3 | 54.9 | 30.5 | 95.0 | 66.7/60.0 | 63.0 |
Table 27. Table 27: At 3.74 GB our checkpoints beat UD-IQ2_XXS (4.47 GB) on reasoning; Q2_K and larger files remain stronger. Qwen3-14B by whole-model bytes. M500: sampled MATH-500; A24/25: greedy AIME; A@16: mean pass@1 over 16 samples; HE: HumanEval; MT: BFCL multi-turn. Rows grouped by bits per parameter. †Not packed; *computed from the grids; §separate evaluation run.
References
- [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2306.13649
- [2]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2404.00456
- [3]Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.arxiv.org/abs/1308.3432
- [4]Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QuIP: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2307.13304
- [5]Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. EfficientQAT: Efficient quantization-aware training for large language models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2025.arxiv.org/abs/2407.11062
- [6]Alireza Dadgarinia, Soroush Tabesh, Mahdi Nikdan, Michael Helcig, Eldar Kurtic, Maximilian Kleineger, and Dan Alistarh. Gsq: Highly-accurate low-precision scalar quantization for llms via gumbel-softmax sampling. arXiv preprint arXiv:2604.18556, 2026.
- [7]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2208.07339
- [8]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.14314
- [9]Tim Dettmers, Ruslan Svirschveski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SPQR: A sparse-quantized representation for near-lossless LLM weight compression. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2306.03078
- [10]Xin Ding, Xiaoyu Liu, Zhijun Tu, Yun Zhang, Wei Li, Jie Hu, Hanting Chen, Yehui Tang, Zhiwei Xiong, Baoqun Yin, and Yunhe Wang. CBQ: Cross-block quantization for large language models. In International Conference on Learning Representations (ICLR), 2025.
- [11]Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ: Hessian AWare quantization of neural networks with mixed-precision. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.arxiv.org/abs/1905.03696
- [12]Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ-V2: Hessian Aware trace-weighted quantization of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/1911.03852
- [13]Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. BitDistiller: Unleashing the potential of sub-4-bit LLMs via self-distillation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024.arxiv.org/abs/2402.10631
- [14]Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee. Accuracy is not all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2407.09141
- [15]Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2401.06118
- [16]Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinankumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations (ICLR), 2020.
- [17]Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.23771
- [18]Elias Frantar, Sidak Pal Singh, and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2208.11580
- [19]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.17323
- [20]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonnell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602.
- [21]Georgi Gerganov and ggml contributors. llama.cpp: LLM inference in C/C++. https://github.com/ggml-org/llama.cpp, 2023.
- [22]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In International Conference on Learning Representations (ICLR), 2024.DOI
- [23]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
- [24]Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojun Qi. BiLLM: Pushing the limit of post-training quantization for LLMs. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2402.04291
- [25]Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.arxiv.org/abs/1712.05877
- [26]Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing LLMs: The truth is rarely pure and never simple. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.01382
- [27]Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-Softmax. In International Conference on Learning Representations (ICLR), 2017.
- [28]Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. A comprehensive evaluation of quantization strategies for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.arxiv.org/abs/2402.16775
- [29]Ayush Kaushal, Tejas Vaidhya, Arnab Kumar Mondal, Tejas Pandey, Aaryan Bhagat, and Irina Rish. Spectra: Surprising effectiveness of pretraining ternary language models at scale. arXiv preprint arXiv:2407.12327, 2024.arxiv.org/abs/2407.12327
- [30]Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.14152
- [31]Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. SqueezeLLM: Dense-and-sparse quantization. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2306.07629
- [32]Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlivan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2411.04330
- [33]Eldar Kurtić, Alexandre Marques, Shubhrajit Pandit, Mark Kurtz, and Dan Alistarh. “Give me BF16 or give me death”? accuracy-performance trade-offs in LLM quantization. In Annual Meeting of the Association for Computational Linguistics (ACL), 2025.arxiv.org/abs/2411.02355
- [34]Donghyun Lee, Yuhang Li, Ruokai Yin, and Priyadarshini Panda. KronQ: LLM quantization via Kronecker-factored Hessian. In Conference on Language Modeling (COLM), 2026.
- [35]Liang Li, Qingyuan Li, Bo Zhang, and Xiangxiang Chu. Norm tweaking: High-performance low-bit quantization of large language models. In AAAI Conference on Artificial Intelligence (AAAI), 2024.arxiv.org/abs/2309.02784
- [36]Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28480–28524. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/li24bb.html.arxiv.org/abs/2402.18158
- [37]Yuhang Li and Priyadarshini Panda. TesseraQ: Ultra low-bit LLM post-training quantization with block reconstruction. arXiv preprint arXiv:2410.19103, 2024.arxiv.org/abs/2410.19103
- [38]Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.arxiv.org/abs/2102.05426
- [39]Xinyu Lian, Walid Krichene, Beichen Huang, Masahiro Tanaka, Olatunji Ruwase, Li Zhang, and Minjia Zhang. Quantization inflates reasoning: Token inflation as a hidden cost of low-bit reasoning models. arXiv preprint arXiv:2606.25519, 2026.arxiv.org/abs/2606.25519
- [40]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Conference on Machine Learning and Systems (MLSys), 2024.DOI
- [41]Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, and Jianfeng Gao. Bridging discrete and backpropagation: Straight-through and beyond. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2304.08612
- [42]Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezhēng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models. In Conference on Language Modeling (COLM), 2025.arxiv.org/abs/2504.04823
- [43]Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-OAT: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.arxiv.org/abs/2305.17888
- [44]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantization with learned rotations. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2405.16406
- [45]Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, et al. Paretoq: Improving scaling laws in extremely low-bit llm quantization. Advances in Neural Information Processing Systems, 38:91311–91336, 2026.arxiv.org/abs/2502.02631
- [46]Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit LLMs: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 2024.arxiv.org/abs/2402.17764
- [47]Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017.
- [48]Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. PV-Tuning: Beyond straight-through estimation for extreme LLM compression. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2405.14852
- [49]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning (ICML), 2015.
- [50]Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024.arxiv.org/abs/2411.00640
- [51]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannane Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.arxiv.org/abs/2501.19393
- [52]Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020.arxiv.org/abs/2004.10568
- [53]Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisheng Zhang, Haitao Mi, and Dong Yu. Low-bit quantization favors undertrained LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32338–32348, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1555. URL https://aclanthology.org/2025.acl-long.1555/.DOI
- [54]Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models. In International Conference on Machine Learning (ICML), 2025.
- [55]PrismML. Bonsai: 1-bit and ternary language models. Hugging Face checkpoints: https://huggingface.co/prism-ml/Ternary-Bonsai-1.7B-gguf, https://huggingface.co/prism-ml/Bonsai-1.7B-gguf, https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf, https://huggingface.co/prism-ml/Bonsai-8B-gguf, https://huggingface.co/prism-ml/Ternary-Bonsai-1.7B-unpacked, https://huggingface.co/prism-ml/Bonsai-1.7B-unpacked, https://huggingface.co/prism-ml/Ternary-Bonsai-8B-unpacked, https://huggingface.co/prism-ml/Bonsai-8B-unpacked, 2026.
- [56]Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. PB-LLM: Partially binarized large language models. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.00034
- [57]Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Omnidirectionally calibrated quantization for large language models. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2308.13137
- [58]Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QuIP#: Even better LLM quantization with Hadamard incoherence and lattice codebooks. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2402.04396
- [59]Albert Tseng, Qingyao Sun, David Hou, and Christopher De Sa. QTIP: Quantization with trellises and incoherence processing. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2406.11235
- [60]Unsloth AI. Qwen3-0.6B GGUF, Q4_K_M. Hugging Face: https://huggingface.co/unsloth/Qwen3-0.6B-GGUF/blob/50968a4468ef4233ed78cd7c3de230dd1d61a156b/Qwen3-0.6B-Q4_K_M.gguf, 2025. File Qwen3-0.6B-Q4_K_M.gguf, 396,705,472 bytes, revision 50968a4.
- [61]Unsloth AI. Qwen3-0.6B GGUF, Q8_0. Hugging Face: https://huggingface.co/unsloth/Qwen3-0.6B-GGUF/blob/50968a4468ef4233ed78cd7c3de230dd1d61a156b/Qwen3-0.6B-Q8_0.gguf, 2025. File Qwen3-0.6B-Q8_0.gguf, 639,447,744 bytes, revision 50968a4.
- [62]Unsloth AI. Qwen3-1.7B GGUF, Q2_K. Hugging Face: https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/blob/d7f544eead698dbd1f15126ef60b45a1e1933222/Qwen3-1.7B-Q2_K.gguf, 2025. File Qwen3-1.7B-Q2_K.gguf, 777,796,160 bytes, revision d7f544e.
- [63]Unsloth AI. Qwen3-1.7B GGUF, Q3_K_M. Hugging Face: https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/blob/d7f544eead698dbd1f15126ef60b45a1e1933222/Qwen3-1.7B-Q3_K_M.gguf, 2025. File Qwen3-1.7B-Q3_K_M.gguf, 939,539,008 bytes, revision d7f544e.
- [64]Unsloth AI. Qwen3-1.7B GGUF, Q4_K_M. Hugging Face: https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/blob/d7f544eead698dbd1f15126ef60b45a1e1933222/Qwen3-1.7B-Q4_K_M.gguf, 2025. File Qwen3-1.7B-Q4_K_M.gguf, 1,107,409,472 bytes, revision d7f544e.
- [65]Unsloth AI. Qwen3-1.7B GGUF, UD-IQ2_M. Hugging Face: https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/blob/d7f544eead698dbd1f15126ef60b45a1e1933222/Qwen3-1.7B-UD-IQ2_M.gguf, 2025. File Qwen3-1.7B-UD-IQ2_M.gguf, 708,715,072 bytes, revision d7f544e.
- [66]Unsloth AI. Qwen3-1.7B GGUF, UD-IQ2_XXS. Hugging Face: https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/blob/d7f544eead698dbd1f15126ef60b45a1e1933222/Qwen3-1.7B-UD-IQ2_XXS.gguf, 2025. File Qwen3-1.7B-UD-IQ2_XXS.gguf, 605,790,784 bytes, revision d7f544e.
- [67]Unsloth AI. Qwen3-14B GGUF, Q2_K. Hugging Face: https://huggingface.co/unsloth/Qwen3-14B-GGUF/blob/a04a82c4739b3ef5fabda7d10261db2c67dd1985/Qwen3-14B-Q2_K.gguf, 2025. File Qwen3-14B-Q2_K.gguf, 5,753,984,384 bytes, revision a04a82c.
- [68]Unsloth AI. Qwen3-14B GGUF, UD-IQ2_M. Hugging Face: https://huggingface.co/unsloth/Qwen3-14B-GGUF/blob/a04a82c4739b3ef5fabda7d10261db2c67dd1985/Qwen3-14B-UD-IQ2_M.gguf, 2025. File Qwen3-14B-UD-IQ2_M.gguf, 5,425,956,224 bytes, revision a04a82c.
- [69]Unsloth AI. Qwen3-14B GGUF, UD-IQ2_XXS. Hugging Face: https://huggingface.co/unsloth/Qwen3-14B-GGUF/blob/a04a82c4739b3ef5fabda7d10261db2c67dd1985/Qwen3-14B-UD-IQ2_XXS.gguf, 2025. File Qwen3-14B-UD-IQ2_XXS.gguf, 4,470,113,664 bytes, revision a04a82c.
- [70]Unsloth AI. Qwen3-4B GGUF, Q4_K_M. Hugging Face: https://huggingface.co/unsloth/Qwen3-4B-GGUF/blob/22c9fc8a8c770b76a1789366280a6a5a1d1120/Qwen3-4B-Q4_K_M.gguf, 2025. File Qwen3-4B-Q4_K_M.gguf, 2,497,281,312 bytes, revision 22c9fc8.
- [71]Unsloth AI. Qwen3-8B GGUF, Q2_K. Hugging Face: https://huggingface.co/unsloth/Qwen3-8B-GGUF/blob/a6adef130ffb23ddaf1a62fec9dced968c9bc482/Qwen3-8B-Q2_K.gguf, 2025. File Qwen3-8B-Q2_K.gguf, 3,281,733,440 bytes, revision a6adef1.
- [72]Unsloth AI. Qwen3-8B GGUF, Q3_K_M. Hugging Face: https://huggingface.co/unsloth/Qwen3-8B-GGUF/blob/a6adef130ffb23ddaf1a62fec9dced968c9bc482/Qwen3-8B-Q3_K_M.gguf, 2025. File Qwen3-8B-Q3_K_M.gguf, 4,124,161,856 bytes, revision a6adef1.
- [73]Unsloth AI. Qwen3-8B GGUF, UD-IQ2_M. Hugging Face: https://huggingface.co/unsloth/Qwen3-8B-GGUF/blob/a6adef130ffb23ddaf1a62fec9dced968c9bc482/Qwen3-8B-UD-IQ2_M.gguf, 2025. File Qwen3-8B-UD-IQ2_M.gguf, 3,110,897,472 bytes, revision a6adef1.
- [74]Unsloth AI. Qwen3-8B GGUF, UD-IQ2_XXS. Hugging Face: https://huggingface.co/unsloth/Qwen3-8B-GGUF/blob/a6adef130ffb23ddaf1a62fec9dced968c9bc482/Qwen3-8B-UD-IQ2_XXS.gguf, 2025. File Qwen3-8B-UD-IQ2_XXS.gguf, 2,605,221,696 bytes, revision a6adef1.
- [75]Unsloth AI. Qwen3 GGUF quantizations (Unsloth Dynamic 2.0). Hugging Face check-points: https://huggingface.co/unsloth/Qwen3-0.6B-GGUF, https://huggingface.co/unsloth/Qwen3-1.7B-GGUF, https://huggingface.co/unsloth/Qwen3-4B-GGUF, https://huggingface.co/unsloth/Qwen3-8B-GGUF, https://huggingface.co/unsloth/Qwen3-14B-GGUF, 2025.
- [76]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huajie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. BitNet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453, 2023.arxiv.org/abs/2310.11453
- [77]Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, and Anbang Yao. Attend to your own thoughts: Breaking the barrier for post-training quantization of reasoning LLMs through the lens of 1.58-bit quantization. arXiv preprint arXiv:2608.01078, 2026.arxiv.org/abs/2608.01078
- [78]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2211.10438
- [79]Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. OneBit: Towards extremely low-bit large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2402.11295
- [80]An Yang et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [81]Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2206.01861
- [82]Nan Zhang, Eugene Kwek, Yusen Zhang, Muyu Pan, Suhang Wang, Prasenjit Mitra, and Rui Zhang. QuantLRM: Quantization of large reasoning models via fine-tuning signals. arXiv preprint arXiv:2602.02581, 2026.arxiv.org/abs/2602.02581
- [83]Xingyu Zheng, Yuye Li, Haoran Chu, Yue Feng, Xudong Ma, Jie Luo, Jinyang Guo, Haotong Qin, Michele Magno, and Xianglong Liu. An empirical study of Qwen3 quantization. arXiv preprint arXiv:2505.02214, 2025.arxiv.org/abs/2505.02214







