betaFeedback

WorksComputer Science

Quantization of Large Models with Weight Outliers and Sparsity

Abstract

Post-training quantization (PTQ) reduces the memory burden of large model deployment by casting the high precision trained weights to fewer bits. While this promotes memory-efficient inference, it also incurs quantization error that impairs model performance. To compensate for quantization error, the present contribution decomposes the original trained weights into a quantized matrix, a low-rank component, and a sparse matrix that accounts for outlier rows in the weights. Due to analytical intractability and the discrete structure of the decomposition, a simulated annealing search is employed to explore candidate decompositions while avoiding poor local minima. Numerical tests demonstrate the effectiveness of this approach across a broad suite of large language models, including Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B, and Qwen2.5 7B, evaluated on natural language generation benchmarks using 4-bit NF4 and FP4 quantizers. The novel method can markedly outperform direct quantization while incurring minimal memory overhead.

Introduction

Large language models (LLMs) [1, 2] have demonstrated remarkable effectiveness across a wide range of tasks, including natural language understanding [3], code generation [4], and reasoning [5]. Underpinning these capabilities are models with billions of parameters, typically represented in 16-bit floating-point formats (e.g. FP16, BF16). Although training such models requires substantial computational and energy resources, this process is typically performed once. In contrast, the subsequent LLM deployment requires sustained memory and bandwidth throughout the model’s lifetime. As models scale, memory-efficient deployment has become an increasingly important challenge, particularly in resource-constrained and latency-sensitive environments such as edge devices and the cloud.

To address this, several model compression techniques have been advocated. For example; pruning [6] reduces the parameter count by removing weights; knowledge distillation [7] transfers knowledge to a smaller student model; and quantization [8, 9] maps model parameters to lower-precision numerical formats. Among these approaches, low-bit quantization is particularly attractive because it can markedly curtail memory footprint and bandwidth requirements.

Post-training quantization (PTQ) is a popular paradigm because it applies quantization to a model after training, as opposed to alternative paradigms that incorporate quantization during pretraining [10] or fine-tuning [11]. Conventional PTQ typically leverages a calibration dataset to determine the quantization parameters [12, 9], whereas zero-shot PTQ directly quantizes model parameters without additional data. This distinction establishes zero-shot PTQ as remarkably attractive in scenarios where representative calibration data is unavailable. However, a central challenge in PTQ is coping with the associated quantization error, especially when compressing to 4 bits or less. For LLMs, the performance impairment can be exacerbated because errors propagate in the transformer layers and across output tokens in long-sequence inference [8, 13].

A major source of quantization error is the presence of high-magnitude outliers in LLM weight distributions, which can exhibit structured patterns [8, 13]. Directly quantizing such outliers is challenging because they enlarge the quantization range and thereby reduce quantization fidelity for the majority of weights, especially in low-bit regimes. As a result, outliers can contribute disproportionately to quantization error and lead to downstream performance degradation. Prior work has proposed several strategies to mitigate this issue. For instance, Gpt3.int8() [8] and SPQR [13] isolate high-magnitude activation or weight outliers and retain them in higher precision. SmoothQuant [14] transfers activation outliers to weights using per-channel scaling, while AWQ [15] performs saliency-aware weight quantization. QuaRot [16] instead applies rotation matrices to redistribute activation magnitude. Many of these approaches, however, rely on calibration data and can thus be less suitable in fully calibration-free/zero-shot deployment settings.

Rather than directly quantizing pretrained weights, we adopt a calibration-free mixed-precision decomposition framework that factorizes high-precision pretrained weights into three complementary components: a quantized backbone, a high-precision sparse component, and a high-precision low-rank factor. This decomposition is motivated by the observation that, once the high-energy outliers are isolated, the remaining reconstruction error often admits an efficient low-rank approximation capturing the dominant singular directions. Together, the sparse and low-rank components improve reconstruction fidelity while incurring minimal memory overhead and inference latency. Since a global search for an optimal decomposition scales exponentially with weight matrix dimensions, the present contribution leverages a simulated annealing algorithm to explore candidate decompositions while avoiding poor local minima. Experiments on zero-shot natural language generation tasks demonstrate the effectiveness of the proposed approach across a wide range of popular large language models quantized to 4-bit NF4 and FP4 formats.

Preliminaries

Consider a general weight matrix W∈Rm×n\mathbf{W} \in\mathbb{R}^{m \times n} from a pretrained model stored in 16-bit floating-point precision. To reduce the memory footprint during deployment, a quantizer qb:Rm×n→Cbq_b : \mathbb{R}^{m \times n} \to\mathcal{C}_b maps entries of W\mathbf{W} onto a finite bb-bit codebook Cb⊂R\mathcal{C}_b \subset\mathbb{R} and stores the resulting low-precision representation Q:=qb(W)\mathbf{Q} := q_b(\mathbf{W}) in memory. The stored values are converted back to a higher-precision compute format on the fly through a dequantizer db:Cbm×n→Rm×nd_b : \mathcal{C}_b^{m \times n} \to\mathbb{R}^{m \times n}. The overall quantize-then-dequantize operator is denoted by q^b:=db∘qb\hat{q}_b := d_b \circ q_b, which is not an identity mapping owing to quantization error. Accordingly, the quantization error is E:=W−db(Q)\mathbf{E} := \mathbf{W} - d_b(\mathbf{Q}), whose magnitude depends on both the quantizer design and the distribution of the weight entries.

A representative example is normal float (NF) quantization [17] that constructs a codebook with a standard Gaussian prior for the weight entries. Letting Φ−1\Phi^{-1} denote the quantile function of the standard Gaussian distribution N(0,1)\mathcal{N}(0,1), the normalized bb-bit NF codebook is defined by

Squared $\ell_2$ row norms of the quantization error for query, down, and output projection matrices

Figure 1. From left to right: squared ℓ2\ell_2 row norms of the quantization error (NF4, block size 64) for the pretrained query, down, and output projection matrices of Llama 3.1 8B, plotted against layer depth. The quantization error exhibits structured high-energy outlier rows across several layers.

Cb={zkmax⁡1≤j≤2b∣zj∣}k=12b,zk=12Φ−1(k2b+1)+12Φ−1(k+12b+1).\mathcal{C}_b=\left\{\frac{z_k}{\max_{1\le j\le2^b}|z_j|}\right\}_{k=1}^{2^b},\qquad z_k=\frac{1}{2}\Phi^{-1}\left(\frac{k}{2^b+1}\right)+\frac{1}{2}\Phi^{-1}\left(\frac{k+1}{2^b+1}\right).

Given a scaling factor s=max⁡i,j∣wij∣>0s=\max_{i,j}|w_{ij}|>0, the reconstruction is obtained via a nearest-neighbor solution in the normalized domain followed by rescaling

[q^b(W)]ij=s⋅arg⁡min⁡z∈Cb∣s−1wij−z∣.[\hat{\boldsymbol{q}}_b(\mathbf{W})]_{ij}=s\cdot\underset{z\in\mathcal{C}_b}{\arg\min}\left|s^{-1}w_{ij}-z\right|.

That is, the high-precision weights are first normalized to [−1,1][-1,1], quantized to the nearest codebook level, and then mapped back to the original range by scaling with the largest absolute entry of the original weights. In low-bit regimes such as 4-bit quantization, NF4 (i.e. the 4-bit instance of NF quantization) has become a popular choice as it can yield lower quantization error than integer or floating-point alternatives. In practice, the weight matrix can be partitioned into BB non-overlapping blocks of equal size with scaling factor sBs_B assigned block-wise. While this fine-grained approach can account for variation in weight magnitude, quantization remains sensitive to large-magnitude outliers, which can degrade the quantization fidelity around the majority of small weights.

Pretrained LLM weights, specifically, are observed to exhibit both entry-wise and row-wise outliers, i.e., entire rows with significantly larger entries. This outlier pattern propagates to the quantization error as well, even if quantization is performed block-wise. Figure 1 displays the row-wise energy ∥Ei,:∥22\|\mathbf{E}_{i,:}\|_2^2 of the quantization error for the Llama 3.1 8B pretrained weights against layer depth. Direct NF4 weight quantization was applied with a block size of 64 elements. The figure corroborates that outliers concentrate in a small subset of rows, particularly in the multi-layer perceptrons (MLP) of the transformer blocks. Orthogonally, the quantization error can be effectively captured by a low-rank approximation. Although E\mathbf{E} is typically high-rank, it is observed that the leading singular values can be significantly larger than the remaining spectrum [18]. Prior work [19] has also explored low-rank structure in the quantization error, albeit in a LoRA fine-tuning setting. In contrast, this work investigates outlier-aware zero-shot PTQ, where calibration or fine-tuning data may be unavailable. This motivates a co-design of outlier modeling and low-rank approximation for quantization error.

Notation. Calligraphic uppercase letters denote sets. Bold uppercase letters denote matrices, while bold lowercase characters, vectors. For some matrix Z∈Rm×n\mathbf{Z}\in\mathbb{R}^{m\times n}, Zi,:\mathbf{Z}_{i,:} denotes its ii-th row, ∥Z∥F\|\mathbf{Z}\|_{\mathrm{F}} denotes the Frobenius norm, and ∥Z∥2,0\|\mathbf{Z}\|_{2,0} counts the number of nonzero rows. The symbol ∘\circ denotes function composition, and SVDr(Z):=UrΣrVrT\mathrm{SVD}_r(\mathbf{Z}):=\mathbf{U}_r\boldsymbol{\Sigma}_r\mathbf{V}_r^{\mathsf{T}} denotes the rank-rr truncated SVD approximation of Z\mathbf{Z}. The symbol I(⋅)\mathbb{I}(\cdot) is the indicator function, while Hκ:Rm×n→Rm×n\mathbf{H}_\kappa:\mathbb{R}^{m\times n}\to\mathbb{R}^{m\times n} denotes the hard thresholding function Hκ(Z)\mathbf{H}_\kappa(\mathbf{Z}) that preserves the κ∈{0,…,m}\kappa\in\{0,\ldots,m\} highest energy rows of Z\mathbf{Z}, while setting the rest to 0nT\mathbf{0}_n^{\mathsf{T}}.

Outlier-aware zero-shot PTQ via simulated annealing

To cope with the quantization error under row outlier patterns, this paper advocates decomposing the trained weights into a quantized matrix Q∈Cbm×n\mathbf{Q}\in\mathcal{C}_b^{m\times n}, a row-sparse matrix S∈Rm×n\mathbf{S}\in\mathbb{R}^{m\times n}, and a low-rank matrix L∈Rm×n\mathbf{L}\in\mathbb{R}^{m\times n} with rank r≪min⁡(m,n)r\ll\min(m,n). This mixed-precision decomposition of W\mathbf{W} can be captured via the discrete optimization problem

min⁡Q∈Cbm×n min⁡S,L∥W−db(Q)−S−L∥F2\underset{\mathbf{Q}\in\mathcal{C}_b^{m\times n}}{\min}\ \underset{\mathbf{S},\mathbf{L}}{\min}\quad\|\mathbf{W}-d_b(\mathbf{Q})-\mathbf{S}-\mathbf{L}\|_{\mathrm{F}}^2
s.t.rank⁡(L)≤r\text{s.t.}\quad\operatorname{rank}(\mathbf{L})\le r
∥S∥2,0≤κ(1)\|\mathbf{S}\|_{2,0}\le\kappa \tag*{(1)}

where ∥S∥2,0=∑i=1mI{∥Si,:∥2>0}\|\mathbf{S}\|_{2,0}=\sum_{i=1}^{m}\mathbb{I}\{\|\mathbf{S}_{i,:}\|_2>0\}, and κ≪m\kappa\ll m. Although the cost function in (1) is analytically tractable, the caveat is that the search space induced by Q\mathbf{Q} has cardinality 2bmn2^{bmn}. This renders an exhaustive combinatorial search for Q\mathbf{Q} computationally intractable with S\mathbf{S} and L\mathbf{L} given. An alternative is to adopt block coordinate descent solvers cycling between each component. However, such methods are prone to poor local minima in this highly non-convex and discrete search space, and lack a principled mechanism for global exploration.

To address this limitation, this paper adopts a meta-heuristic simulated annealing algorithm, which performs a search over candidate decompositions using temperature-controlled random perturbations.

The temperature follows a pre-defined schedule that decreases over time, thus encouraging exploration of the quantized matrix set Cbm×n\mathcal{C}_{b}^{m \times n} to expose non-local but potentially superior Q\mathbf{Q} candidates.

Specifically, the quantized backbone Q\mathbf{Q} is initialized as the direct quantization Q(0)=qb(W)\mathbf{Q}^{(0)} = q_b(\mathbf{W}), and the row-sparse component is initialized as S(0)=0m×n\mathbf{S}^{(0)} = \mathbf{0}_{m \times n}. Given a decreasing temperature schedule {θk}k=0K−1\{\theta_k\}_{k=0}^{K-1} (e.g. linear, geometric, cosine), the simulated annealing search generates proposals

Πk+1∼f(Πk+1;θk)L~(k+1)=SVD⁡r(W−db(Q(k))+Πk+1),Q~(k+1)=qb(W−L~(k+1)−S(k)),S~(k+1)=Hκ(W−db(Q~(k+1))−L~(k+1))\begin{aligned} \mathbf{\Pi}_{k+1} &\sim f(\mathbf{\Pi}_{k+1};\theta_k) \\ \tilde{\mathbf{L}}^{(k+1)} &= \operatorname{SVD}_{r}\left(\mathbf{W} - d_b\left(\mathbf{Q}^{(k)}\right) + \mathbf{\Pi}_{k+1}\right), \\ \tilde{\mathbf{Q}}^{(k+1)} &= q_b\left(\mathbf{W} - \tilde{\mathbf{L}}^{(k+1)} - \mathbf{S}^{(k)}\right), \\ \tilde{\mathbf{S}}^{(k+1)} &= H_{\kappa}\left(\mathbf{W} - d_b\left(\tilde{\mathbf{Q}}^{(k+1)}\right) - \tilde{\mathbf{L}}^{(k+1)}\right) \end{aligned}

for k=0,…,K−1k = 0, \ldots, K - 1. Πk+1∈Rm×n\mathbf{\Pi}_{k+1} \in\mathbb{R}^{m \times n} is sampled from a pre-specified prior f(Πk+1;θk)f(\mathbf{\Pi}_{k+1};\theta_k), which can be a Gaussian distribution with standard deviation scaled by θk\theta_k, for instance. This perturbation in the low-rank component encourages exploration of Q\mathbf{Q} in regions where the error is approximately low rank while avoiding being trapped around the current iterate Q(k)\mathbf{Q}^{(k)}. The cost of the current proposal is c~k+1=∥W−S~(k+1)−db(Q~(k+1))−L~(k+1)∥F2\tilde{c}_{k+1} = \left\|\mathbf{W} - \tilde{\mathbf{S}}^{(k+1)} - d_b\left(\tilde{\mathbf{Q}}^{(k+1)}\right) - \tilde{\mathbf{L}}^{(k+1)}\right\|_{\mathrm{F}}^2, and the next iterate is taken as

(L(k+1),Q(k+1),S(k+1))={(L~(k+1),Q~(k+1),S~(k+1))w.p. Paccept,(L(k),Q(k),S(k))w.p. 1−Paccept\begin{aligned} \left(\mathbf{L}^{(k+1)},\mathbf{Q}^{(k+1)},\mathbf{S}^{(k+1)}\right) &= \begin{cases} \left(\tilde{\mathbf{L}}^{(k+1)},\tilde{\mathbf{Q}}^{(k+1)},\tilde{\mathbf{S}}^{(k+1)}\right) & \text{w.p. } P_{\mathrm{accept}}, \\ \left(\mathbf{L}^{(k)},\mathbf{Q}^{(k)},\mathbf{S}^{(k)}\right) & \text{w.p. } 1-P_{\mathrm{accept}} \end{cases} \end{aligned}

where the acceptance probability is Paccept=min⁡{e−(c~k+1−ck)/θk,1}P_{\mathrm{accept}} = \min\{e^{-(\tilde{c}_{k+1}-c_k)/\theta_k}, 1\}. Note that a better proposal is always accepted, but a nominally worse proposal can be accepted with probability PacceptP_{\mathrm{accept}}, enabling the trajectory to escape sub-optimal local minima. The algorithm outlining this procedure can be found in section A of the Appendix.

At the final iteration KK, the algorithm outputs a quantized matrix Qf:=Q(K)\mathbf{Q}_f := \mathbf{Q}^{(K)}, together with high-precision row-sparse and low-rank components Sf:=S(K)\mathbf{S}_f := \mathbf{S}^{(K)} and Lf:=L(K)\mathbf{L}_f := \mathbf{L}^{(K)}, corresponding to the final accepted iterate. To reduce the additional memory and compute cost of storing the high-precision components densely, the row sparsity of Sf\mathbf{S}_f is exploited and the low-rank component is factorized with truncated rank-rr SVD. In particular, writing SVD⁡r(Lf)=UrΣrVr⊤\operatorname{SVD}_{r}(\mathbf{L}_f) = \mathbf{U}_r\mathbf{\Sigma}_r\mathbf{V}_r^{\top}, the low-rank component is stored through the factors L1:=UrΣr∈Rm×r\mathbf{L}_1 := \mathbf{U}_r\mathbf{\Sigma}_r \in\mathbb{R}^{m \times r} and L2:=Vr∈Rn×r\mathbf{L}_2 := \mathbf{V}_r \in\mathbb{R}^{n \times r} where Lf=L1L2⊤\mathbf{L}_f = \mathbf{L}_1\mathbf{L}_2^{\top}. The row-sparse component is stored only through its nonzero rows and their indices. Letting K⊆{1,…,m}\mathcal{K} \subseteq\{1,\ldots,m\} denote the index set of kept rows, with ∣K∣=κ|\mathcal{K}| = \kappa, this reduces the storage of Sf\mathbf{S}_f from mnmn to κn\kappa n high-precision entries along with the negligible κ\kappa row indices. Therefore, for an input x∈Rn\mathbf{x} \in\mathbb{R}^{n}, the per-matrix inference becomes xW⊤≈xQf⊤+xSf⊤+(xL2)L1⊤\mathbf{x}\mathbf{W}^{\top} \approx\mathbf{x}\mathbf{Q}_f^{\top} + \mathbf{x}\mathbf{S}_f^{\top} + (\mathbf{x}\mathbf{L}_2)\mathbf{L}_1^{\top}, where the term xSf⊤\mathbf{x}\mathbf{S}_f^{\top} can be evaluated only on the rows indexed by K\mathcal{K}.

Numerical Tests

This section assesses the proposed zero-shot PTQ method on Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B, and Qwen 2.5 7B models, which are evaluated with the LM Evaluation Harness API [20] on GSM8K [21], MBPP [22], and WikiText-2 [23] datasets. Direct quantization, LoFTQ initialization [19], and the proposed method are compared under both NF4 [17] and FP4 quantization. All experiments use a rank r=32r = 32 for L\mathbf{L}, and K=300K = 300 annealing iterations. To control the row sparsity, a percentage-wise row budget ρ\rho is defined a-priori, where κ=⌈ρm⌉\kappa= \lceil\rho m \rceil. Both ρ=0%\rho= 0\% (i.e. S=0m×n\mathbf{S} = \mathbf{0}_{m \times n}) and ρ=1%\rho= 1\% are evaluated. The decomposition is applied to all transformer attention and MLP projections (i.e. q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, and gate_proj). Projections of the same type share a common geometrically decreasing annealing temperature schedule, obtained from a Bayesian optimization pre-processing step (cf. [24]). A single NVIDIA A100 40GB GPU is employed for all tests.

The results are summarized in Table 1, comparing direct zero-shot quantization, the LoFTQ initialization, annealing without row sparsity (ρ=0%\rho= 0\%), and the proposed row-sparse annealing method with ρ=1%\rho= 1\%. Although LoFTQ [19] was originally proposed for LoRA-aware quantization, we include it as a baseline under the same zero-shot PTQ evaluation protocol. Across NF4 and FP4 quantization, the proposed row-sparse plus annealing method achieves the best or near-best performance in most settings, and the gains are especially consistent on WikiText-2, where it obtains the lowest perplexity for every model and quantization format except for the NF4 Llama 3.2 3B case. This suggests that explicitly modeling row-structured outliers can better preserve language-modeling performance under low-bit quantization. In GSM8K, it improves over direct quantization in all cases and typically outperforms both the LoFTQ initialization and annealing without row sparsity, indicating that the sparse correction is beneficial for preserving mathematical reasoning performance under low-bit quantization.

Table 1: Performance comparison on NLG benchmarks using select Llama 3 models and Qwen 2.5 7B. Word perplexity is reported on WikiText-2 (WT-2), while GSM8K and MBPP report exact-match accuracy and pass@1 code-generation accuracy, respectively. Higher is better for GSM8K and MBPP, while lower is better for WT-2.

ModelMethodNF4 GSM8K (↑)NF4 MBPP (↑)NF4 WT-2 (↓)FP4 GSM8K (↑)FP4 MBPP (↑)FP4 WT-2 (↓)
Llama 3.1 8BOriginal (FP16)50.2747.607.5550.2747.607.55
Llama 3.1 8BQLoRA init.44.7347.008.0438.2943.408.51
Llama 3.1 8BLoftQ init.44.1244.807.9838.6744.208.39
Llama 3.1 8BOursρ=0%_{\rho=0\%}44.9646.807.9342.6845.608.32
Llama 3.1 8BOursρ=1%_{\rho=1\%}46.3248.007.8943.9746.408.31
Qwen 2.5 7BOriginal (FP16)83.4063.408.7483.4063.408.74
Qwen 2.5 7BQLoRA init.78.7061.409.0979.7662.409.61
Qwen 2.5 7BLoftQ init.79.9162.609.0679.3861.809.47
Qwen 2.5 7BOursρ=0%_{\rho=0\%}81.5060.009.0177.0357.409.31
Qwen 2.5 7BOursρ=1%_{\rho=1\%}82.4962.809.0180.1462.209.30
Llama 3.2 3BOriginal (FP16)27.4538.609.5127.4538.609.51
Llama 3.2 3BQLoRA init.22.5232.609.9818.7333.4010.42
Llama 3.2 3BLoftQ init.21.5334.209.9319.0335.0010.30
Llama 3.2 3BOursρ=0%_{\rho=0\%}24.7237.809.8319.5636.0010.20
Llama 3.2 3BOursρ=1%_{\rho=1\%}23.5835.009.8622.2131.6010.18
Llama 3.2 1BOriginal (FP16)5.6126.608.075.6126.608.07
Llama 3.2 1BQLoRA init.2.8118.4013.061.3613.4014.28
Llama 3.2 1BLoftQ init.3.7121.6012.922.2710.2013.77
Llama 3.2 1BOursρ=0%_{\rho=0\%}3.9419.4012.732.2015.4013.49
Llama 3.2 1BOursρ=1%_{\rho=1\%}4.7022.2012.692.6515.2013.42

Table 1.

For code generation with MBPP, our methods achieve the best pass@1 score for NF4 quantization across all four models. With FP4, however, plain quantization is slightly better on Qwen 2.5 7B, and annealing without row sparsity performs better on Llama 3.2 3B and Llama 3.2 1B. This could suggest that the benefits of the low-rank and row-sparse corrections can overlap, and the design choice of explicit outlier modeling depends on the model, task, and structure of the quantization error. Overall, the results support that annealing-based decompositions improve over alternative approaches, and adding row-sparse outlier absorption can boost gains with minimal storage overhead (cf. Appendix § 7), particularly for perplexity and GSM8K accuracy.

Conclusion and future work

This paper introduced a calibration-free post-training quantization method that decomposes pretrained LLM weights into quantized, low-rank, and row-sparse components. By explicitly modeling row-structured outliers in the weights, the proposed decomposition improves zero-shot performance over direct quantization and low-rank alternatives across several models and quantization formats. These results suggest that mixed-precision decompositions can be beneficial for low-bit PTQ, particularly when calibration data or fine-tuning data is unavailable. Future work will investigate using the low-rank factors as an initialization for LoRA fine-tuning, exploring whether zero-shot performance correlates with downstream adaptation capacity.

Algorithm pseudocode

This Appendix section provides the pseudocode of the present contribution in Algorithm 1. The algorithm inputs are the pretrained high-precision weights W\mathbf{W}, the number of annealing steps KK, the rank rr of the low-rank component L\mathbf{L}, the sparsity-controlling parameter κ\kappa for the row-sparse S\mathbf{S}, and the temperature schedule θk:=θ0(θK−1/θ0)k/(K−1)\theta_k := \theta_0(\theta_{K-1}/\theta_0)^{k/(K-1)}. While the temperatures are geometrically decreasing, alternative schedules such as cosine or linear can be readily employed. Another design choice concerns the perturbation distribution, which in this work is defined as a zero-mean Gaussian with standard deviation proportional to the temperature. We leave alternative schedule and perturbation design choices for future research.

Algorithm 1 Simulated annealing solver for quantized, low-rank, and row-sparse decomposition

  1. Input: W\mathbf{W}, KK, θ0\theta_0, θK−1\theta_{K-1}, rr, κ\kappa

  2. Initialize: Q(0)=qb(W)\mathbf{Q}^{(0)} = q_b(\mathbf{W}), S(0)=0m×n\mathbf{S}^{(0)} = \mathbf{0}_{m \times n}, c0=+∞c_0 = +\infty

  3. for k=0,…,K−1k = 0, \ldots, K - 1 do

  4. θk=θ0(θK−1/θ0)k/(K−1)\theta_k = \theta_0(\theta_{K-1}/\theta_0)^{k/(K-1)}

  5. sample [Πk+1]ij∼N(0,θk2/4)[\mathbf{\Pi}_{k+1}]_{ij} \sim\mathcal{N}(0,\theta_k^2/4), ∀i,j\forall i,j

  6. L~(k+1)=SVD⁡r(W−db(Q(k))+Πk+1)\tilde{\mathbf{L}}^{(k+1)} = \operatorname{SVD}_r(\mathbf{W} - d_b(\mathbf{Q}^{(k)}) + \mathbf{\Pi}_{k+1})

  7. Q~(k+1)=qb(W−L~(k+1)−S(k))\tilde{\mathbf{Q}}^{(k+1)} = q_b(\mathbf{W} - \tilde{\mathbf{L}}^{(k+1)} - \mathbf{S}^{(k)})

  8. S~(k+1)=Hκ(W−db(Q~(k+1))−L~(k+1))\tilde{\mathbf{S}}^{(k+1)} = H_\kappa(\mathbf{W} - d_b(\tilde{\mathbf{Q}}^{(k+1)}) - \tilde{\mathbf{L}}^{(k+1)})

  9. c~k+1=∥W−S~(k+1)−db(Q~(k+1))−L~(k+1)∥F2\tilde{c}_{k+1} = \lVert\mathbf{W} - \tilde{\mathbf{S}}^{(k+1)} - d_b(\tilde{\mathbf{Q}}^{(k+1)}) - \tilde{\mathbf{L}}^{(k+1)}\rVert_F^2

  10. Paccept=min⁡{e−(c~k+1−ck)/θk,1}P_{\mathrm{accept}} = \min\{e^{-(\tilde{c}_{k+1}-c_k)/\theta_k}, 1\}

  11. sample u∼Uniform⁡[0,1]u \sim\operatorname{Uniform}[0,1]

  12. if Paccept>uP_{\mathrm{accept}} > u then

  13. (L(k+1),Q(k+1),S(k+1))=(L~(k+1),Q~(k+1),S~(k+1))(\mathbf{L}^{(k+1)}, \mathbf{Q}^{(k+1)}, \mathbf{S}^{(k+1)}) = (\tilde{\mathbf{L}}^{(k+1)}, \tilde{\mathbf{Q}}^{(k+1)}, \tilde{\mathbf{S}}^{(k+1)})

  14. ck+1=c~k+1c_{k+1} = \tilde{c}_{k+1}

  15. else

  16. (L(k+1),Q(k+1),S(k+1))=(L(k),Q(k),S(k))(\mathbf{L}^{(k+1)}, \mathbf{Q}^{(k+1)}, \mathbf{S}^{(k+1)}) = (\mathbf{L}^{(k)}, \mathbf{Q}^{(k)}, \mathbf{S}^{(k)})

  17. ck+1=ckc_{k+1} = c_k

  18. Output: Q(K)\mathbf{Q}^{(K)}, S(K)\mathbf{S}^{(K)}, L(K)\mathbf{L}^{(K)}

Memory overhead analysis

The memory overhead is measured in bits per parameter, and is calculated for all seven attention and MLP projection types in every layer. The low-rank components L1L_1 and L2L_2 are stored in FP32 and employ a rank r=32r = 32. The non-zero rows of each row-sparse matrix SS and their indices are stored in FP16 and FP32 respectively. For both the NF4 and FP4 quantization schemes, one FP32 scaling factor is used per block of 64 weights without double quantization. The remaining parameters are stored in FP16. Table 2 provides a summary of the results.

ModelOriginal (FP16)QLoRA init.LoftQ init.Oursρ=0%_{\rho=0\%}Oursρ=1%_{\rho=1\%}
Llama 3.1 8B16.0006.0056.3396.3396.477
Qwen 2.5 7B16.0006.1466.4866.4866.621
Llama 3.2 3B16.0005.9116.3956.3956.533
Llama 3.2 1B16.0006.9457.5297.5297.653

Table 2. Memory storage comparison across quantization schemes in bits per parameter. Values apply to both NF4 and FP4 quantization schemes.

At a rank of r=32r = 32, our method with ρ=1%\rho= 1\% incurs an additional 0.12–0.14 bits per parameter relative to the ρ=0%\rho= 0\% or the alternating optimization cases. These estimates include quantization scales and row indices.

References

  1. [1]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al., “Mistral 7B,” arXiv:2310.06825, 2023.arxiv.org/abs/2310.06825
  2. [2]Aaron Graffiatiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al., “The Llama 3 herd of models,” arXiv:2407.21783, 2024.
  3. [3]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, “SuperGLUE: A stickier benchmark for general-purpose language understanding systems,” in Proc. Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1905.00537
  4. [4]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al., “Evaluating Large Language Models Trained on Code,” arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  5. [5]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi, “PIQA: Reasoning about Physical Commonsense in Natural Language,” Proc. AAAI Conf. Artif. Intel., 2020.arxiv.org/abs/1911.11641
  6. [6]Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman, “SliceGPT: Compress large language models by deleting rows and columns,” in Proc. Int. Conf. on Learning Representations (ICLR), 2024.arxiv.org/abs/2401.15024
  7. [7]Luyang Fang, Xiaowei Yu, Jiazhuang Cai, Yongkai Chen, Shushan Wu, Zhengliang Liu, Zhenyuan Yang, Haoran Lu, Xilin Gong, Yufang Liu, et al., “Knowledge distillation and dataset distillation of large language models: Emerging trends, challenges, and future directions,” Artificial Intelligence Review, 2025.
  8. [8]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer, “Gpt3.int8(): 8-bit matrix multiplication for transformers at scale,” in Proc. Neural Information Processing Systems (NeurIPS), 2022.
  9. [9]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, “GPTQ: Accurate Quantization for Generative Pre-trained Transformers,” in Proc. Int. Conf. on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.17323
  10. [10]Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra, “LLM-QAT: Data-free quantization aware training for large language models,” in Proc. Conf. Assoc. Comput. Linguist. Meet. (ACL), 2024.
  11. [11]Quan Wei, Chung-Yiu Yau, Hoi To Wai, Yang Zhao, Dongyeop Kang, Youngsuk Park, and Mingyi Hong, “RoSTE: An efficient quantization-aware supervised fine-tuning approach for large language models,” in Proc. Int. Conf. on Machine Learning (ICML), 2025.arxiv.org/abs/2502.09003
  12. [12]Elias Frantar and Dan Alistarh, “Optimal brain compression: A framework for accurate post-training quantization and pruning,” in Proc. Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2208.11580
  13. [13]Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh, “SpQR: A sparse-quantized representation for near-lossless LLM weight compression,” in Proc. Int. Conf. on Learning Representations (ICLR), 2024.arxiv.org/abs/2306.03078
  14. [14]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” in Proc. Int. Conf. on Machine Learning (ICML), 2023.arxiv.org/abs/2211.10438
  15. [15]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han, “AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration,” in Proc. Conf. on Machine Learn. and Syst., 2024.DOI
  16. [16]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman, “QuaRot: Outlier-free 4-bit inference in rotated llms,” in Proc. Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2404.00456
  17. [17]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer, “QLoRA: Efficient finetuning of quantized llms,” in Proc. Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.14314
  18. [18]Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci, “CALDERA: Compressing large language models using low rank and low precision decomposi- tion,” in Proc. Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2405.18886
  19. [19]Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao, “LoftQ: LoRA-fine-tuning-aware quantization for large language models,” in Proc. Int. Conf. on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.08659
  20. [20]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPoFi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, “The language model evaluation harness,” 2024.
  21. [21]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman, “Training Verifiers to Solve Math Word Problems,” arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  22. [22]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton, “Program Synthesis with Large Language Models,” arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
  23. [23]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher, “Pointer sentinel mixture models,” in ICLR, 2017.arxiv.org/abs/1609.07843
  24. [24]Roman Garnett, Bayesian Optimization, Cambridge University Press, 2023.DOI

Paper details

vlnc:W_b68b4rsm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents