betaFeedback

WorksComputer Science

Complete-SVD: A Functional Norm Approach to Low-Rank LLM Compression

Abstract

Post-training low-rank compression of neural networks must not only approximate the original weights but also preserve the model's input-output behavior. Recent output-aware methods measure the compression error beyond the compressed layer, at the model output, weighting logit changes by the observed next token or by the model's predicted probabilities. We propose Complete-SVD, a method for large language models that minimizes the L2L_2 distance between the logits of the original and compressed models, weighting all logit changes equally. Linearizing the network around the original weights, together with a Kronecker-factored approximation, reduces this objective to a weighted SVD with a closed-form solution. Its output-side factor is intractable to compute; we estimate it without bias from random probes, at a cost of KK backward passes per sequence, independent of vocabulary size and sequence length. At matched settings with uniform ranks and WikiText-2 calibration, Complete-SVD achieves the lowest C4 and PTB perplexity in every setting, and at ratio 0.7 under LEMS rank allocation. Replacing the L2L_2 distance by the Kullback-Leibler divergence (KL-SVD), with the same estimation framework, lowers calibration perplexity in most settings and raises downstream accuracy, but raises cross-corpus perplexity. Complete-SVD also improves IFEval instruction-following accuracy over the strongest prior output-aware methods throughout.

Introduction and Related Work

Large language models have driven rapid progress across reasoning and generation tasks ((Kaplan et al., 2020)), but their growing parameter counts make inference prohibitively expensive. Model compression reduces this cost through quantization ([6] (Lin et al. 2024; Xiao et al. 2023)), pruning ([5] (Ma et al. 2023; Sun et al. 2024)), knowledge distillation ([8]), or low-rank factorization, which replaces each weight matrix by the product of two smaller matrices. Low-rank factorization retains dense operations that run efficiently on standard hardware. Post-training low-rank methods differ in their per-layer objective, i.e., the criterion minimized by the low-rank approximation of each layer; rank allocation, storage schemes ((Wang et al., 2025b)), and cross-layer basis sharing ((Wang et al., 2025a)) are complementary to it. This work concerns the objective, that is, which change in the model the approximation of a layer should penalize.

Error at the compressed layer. Let WW be the weight matrix of a linear layer and W^\widehat{W} its approximation of rank at most rr. Truncated singular value decomposition (SVD) returns the W^\widehat{W} closest to WW in Frobenius norm, which treats all weight directions equally, and replacing the weights of an LLM by these approximations severely degrades its performance. Activation-aware methods instead minimize the error on the layer outputs, ∥X(W−W^)∥F\lVert X(W-\widehat{W})\rVert_F, where the rows of XX are input activations collected on a small calibration set of text sequences. ASVD ((Yuan et al., 2023)) rescales WW by per-channel activation magnitudes, and SVD-LLM ((Wang et al., 2025c)) solves this problem in closed form by whitening with (XTX)12(X^\mathsf{T}X)^{\frac{1}{2}} before LoRA fine-tuning. For tensor decompositions of convolutional networks, Kalle et al. (Kalle et al. [12]) cast this principle as a functional norm: the L2L_2 distance, over the data distribution, between the original and compressed networks up to the compressed layer. This norm involves only the upstream network, i.e., the preceding layers, and reduces to the activation-aware objective for a linear layer. The error at the layer output is not, however, the error of the model: it propagates through the downstream network, i.e., the following layers, which amplifies it along some directions and attenuates it along others, and, in a causal transformer, carries it to the predictions at all subsequent positions.

Error at the model output. Output-aware methods measure the error at the model output. Each compares the outputs of the original and compressed models through a discrepancy, whose local quadratic approximation weights a change of the layer output by its effect on the logits and by the importance that the discrepancy assigns to each logit change (Figure 1, panel b). A first family preserves the training loss through the empirical Fisher matrix, built from loss gradients at the observed next token. FWSVD [9] uses a diagonal approximation, GFWSVD [1] a Kronecker factorization fitted from a limited number of sequence-level gradients, and KFAC-SVD (Thoma et al., 2026) Kronecker factors estimated from token-level statistics, which alleviate the rank deficiency of GFWSVD. For each token, the empirical Fisher only penalizes the change in the log-probability of the observed next token, a single direction among all possible logit changes (Kunstner et al., 2019). A second family preserves the predictive distribution through the Kullback–Leibler (KL) divergence, whose local quadratic approximation is governed by the true Fisher information (Amari, 1998). IO-SVD (Abbasi et al., 2026) compresses LLMs with this objective. IO-SVD approximates the true Fisher matrix through two reductions: it retains only the effect of a perturbation at a given position on the prediction at that same position, and it restricts the predictive distribution to its most probable tokens, renormalized. Both reductions are deterministic, so the resulting bias does not vanish as the calibration set grows (Appendix § 10.5.1).

Figure 1. Output geometry changes the compression tradeoff. (a) The output-aware methods can be written in this Kronecker-factored form and differ mainly in the output kernel K\mathcal{K}: Cinput=ΣinputC_{\mathrm{input}} = \Sigma_{\mathrm{input}}, and Cgrad=ΣgradC_{\mathrm{grad}} = \Sigma_{\mathrm{grad}} for Complete-SVD. (b) Illustrative next-token example. Fisher-based objectives compare p=softmax⁡(o)p = \operatorname{softmax}(o), whereas Complete-SVD compares the logits oo; local output costs hold to leading order in δo\delta o. Link width is the weight of each token in this cost, and black ticks mark original values. Bars share a logarithmic scale (o=log⁡p+const⁡o = \log p + \operatorname{const}); compressed values are schematic. (c) Perplexity reduction 100(PPLKL−PPLComplete)/PPLKL100(\mathrm{PPL}_{\mathrm{KL}} - \mathrm{PPL}_{\mathrm{Complete}})/\mathrm{PPL}_{\mathrm{KL}} of Complete-SVD relative to KL-SVD at compression ratios (CR) 0.9 and 0.7, with uniform ranks. WikiText-2 is the calibration domain.

Complete-SVD. Our objective is the L2L_2 functional distance between the original and compressed networks, i.e., the mean squared Euclidean distance between their logits over the data distribution. Every logit change is thus penalized with equal weight, whereas the Fisher objectives weight each logit change by the observed next token or by the probabilities that the model predicts on the calibration contexts (Figure 1, panel b). We find that the uniform weighting better preserves perplexity on corpora other than the calibration one (Figure 1, panel c). A Kronecker-factored approximation (K-FAC) ([4] (Martens and Grosse 2015)) of the objective reduces the objective to a weighted SVD with a closed-form solution. Its two weights are the input covariance Σinput\Sigma_{\mathrm{input}} of activation-aware methods and the Gram matrix Σgrad\Sigma_{\mathrm{grad}} of the logit Jacobians, which measures how strongly each direction of the layer output affects the logits.

Tractable estimation. The choice of objective also determines the cost of the output-side weight. The empirical Fisher is inexpensive because it involves only the loss, a single scalar whose gradient takes one backward pass per sequence. The KL and L2L_2 objectives involve every logit at every position, and computing Σgrad\Sigma_{\mathrm{grad}} exactly takes one backward pass per vocabulary entry and per position, tens of millions per sequence for Llama 2-7B. The linearity of the LM head absorbs the vocabulary exactly, without any estimation, into the D×DD \times D Gram matrix of the LM-head weights, with DD the hidden dimension: the number of backward passes per output position drops from VV to DD, a factor of about 31 for Llama 3-8B (V≈128kV \approx128\mathrm{k}, D=4096D = 4096), and the resulting expression of Σgrad\Sigma_{\mathrm{grad}} is still exact. Random probes with random signs on the output positions then yield an unbiased estimator of Σgrad\Sigma_{\mathrm{grad}} costing KK backward passes per sequence, independently of the vocabulary size and the sequence length. With probes drawn from the predictive distribution, the same construction gives an unbiased estimator of the true Fisher, which yields KL-SVD, the KL objective of IO-SVD without its reductions (see Appendix § 10.5.1 for details). Comparing KL-SVD with Complete-SVD then separates the choice of objective from the approximation used to compute it.

Our contributions are the following.

  • Complete functional norm. We extend the functional-norm objective of Kalle et al. [12] from the upstream network to the full model, and reduce it to a closed-form weighted SVD that penalizes all logit changes equally (§ 2.2).

  • An unbiased estimator of Σgrad\Sigma_{\mathrm{grad}} whose cost is KK backward passes per sequence, independently of the vocabulary size and the sequence length, validated against the exact matrix at reduced scale (Section 2.3). Drawing the probes from the predictive distribution, the same construction yields an unbiased estimator of the true Fisher matrix FsF_s at the same cost. Substituting it for Σgrad\Sigma_{\mathrm{grad}} gives KL-SVD, the KL objective of IO-SVD without its reductions, which we show to be deterministic deviations from FsF_s that persist for any calibration budget (Appendix § 10.§ 5). KL-SVD shares every other component with Complete-SVD and therefore isolates the effect of the objective from that of its estimation.

  • Empirical evaluation. On Mistral-7B, Llama 3-8B and Qwen3-8B calibrated on WikiText-2 with uniform ranks, Complete-SVD achieves the lowest C4 and PTB perplexity among all compared methods at both tested compression ratios (0.9 and 0.7). On Llama 3-8B at ratio 0.9, it reduces PTB perplexity from 30.9 with IO-SVD, the strongest prior baseline, to 23.0. Without fine-tuning and combined with GPTQ, Complete-SVD compresses Llama 2-7B from 13.5 GB to 1.9 GB at a WikiText-2 perplexity of 16.7 (§ 3).

Method

Problem Statement

We suppose that we are in a supervised learning setting, where we have a data distribution D\mathcal{D} on X×Y\mathcal{X} \times\mathcal{Y} and a per-sample loss function ℓ:Y×Y→R+\ell: \mathcal{Y} \times\mathcal{Y} \to\mathbb{R}_{+} (for classification this is typically the cross-entropy loss and Y=RV\mathcal{Y} = \mathbb{R}^{V} with VV the number of classes). The goal of supervised learning is to find a function f:X→Yf : \mathcal{X} \to\mathcal{Y} such that the expected loss L(fw)=Ex,y∼D(ℓ(fw(x),y))\mathcal{L}(f_w) = \mathbb{E}_{x,y \sim\mathcal{D}}(\ell(f_w(x), y)) is small. We suppose that we already have such a function fWf_W that has low expected loss. We can decompose this function as a pre-computation aa, a fully-connected layer with weights WW (LWL_W) and a post-computation qq such that fW=q∘LW∘af_W = q \circ L_W \circ a:

fW:x↦a(x)↦LWs(x)↦o(1)f_W : x \mapsto a(x) \xmapsto{L_W} s(x) \mapsto o \tag*{(1)}

with x∈Xx \in\mathcal{X}, a(x)∈Rnaa(x) \in\mathbb{R}^{n_a}, s(x)∈Rnss(x) \in\mathbb{R}^{n_s}, o∈RVo \in\mathbb{R}^{V}. We aim to replace the fully-connected layer with two fully-connected layers LW1L_{W_1} and LW2L_{W_2} such that fW≈q∘LW2∘LW1∘af_W \approx q \circ L_{W_2} \circ L_{W_1} \circ a with the constraint that the intermediate dimension is smaller than a specific value kk and the expected loss of this new function is close to the expected loss of fWf_W. We note W^=W1×W2\widehat{W} = W_1 \times W_2 the effective weight of the new function. In practice, we aim to solve:

argmin⁡W^ L(fW^) such that rank⁡(W^)≤k.(2)\underset{\widehat{W}}{\operatorname{argmin}}\ \mathcal{L}(f_{\widehat{W}})\ \text{such that}\ \operatorname{rank}(\widehat{W}) \le k. \tag*{(2)}

Norm Optimization

We minimize the L2L_2 functional distance ∥fW−fW^∥2\lVert f_W - f_{\widehat{W}}\rVert^2; alternative objectives (KL, empirical Fisher) are derived in Appendix B. Since optimizing this distance directly is intractable, we linearize fW^f_{\widehat{W}} around WW.

Theorem 1. With the notation introduced above, let Σinput≔Ex∼D[a(x)(a(x))T]\Sigma_{\mathrm{input}} \coloneqq\mathbb{E}_{x\sim\mathcal{D}}[a(x)(a(x))^\mathsf{T}] denote the input covariance and Σgrad≔Ex∼D[(∂fW(x)∂s)T∂fW(x)∂s]\Sigma_{\mathrm{grad}} \coloneqq\mathbb{E}_{x\sim\mathcal{D}}[\left(\frac{\partial f_W(x)}{\partial s}\right)^\mathsf{T}\frac{\partial f_W(x)}{\partial s}] the Gram matrix of the output Jacobians at the compressed layer. Then:

(i) Exact first-order expansion. Let GW≔Ex∼D[(∂fW(x)∂W)T∂fW(x)∂W]∈Rnsna×nsnaG_W \coloneqq\mathbb{E}_{x\sim\mathcal{D}}[\left(\frac{\partial f_W(x)}{\partial W}\right)^\mathsf{T}\frac{\partial f_W(x)}{\partial W}] \in\mathbb{R}^{n_s n_a \times n_s n_a}. Then

∥fW−fW^∥=∥GW12dW→∥F+o(∥dW∥F).(3)\lVert f_W-f_{\widehat{W}}\rVert=\left\lVert G_W^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F+o(\lVert\mathrm{d}W\rVert_F). \tag*{(3)}

(ii) Kronecker-factored approximation. Under the K-FAC structural approximation GW≈Σinput⊗ΣgradG_W \approx\Sigma_{\mathrm{input}} \otimes\Sigma_{\mathrm{grad}}, the right-hand side of Equation (3) admits the closed-form approximation

∥GW12dW→∥F≈∥Σinput12dWΣgrad12∥F.(4)\left\lVert G_W^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F \approx\left\lVert\Sigma_{\mathrm{input}}^{\frac{1}{2}}\mathrm{d}W\Sigma_{\mathrm{grad}}^{\frac{1}{2}}\right\rVert_F. \tag*{(4)}

Proof Sketch. Linearizing fWf_W around WW gives ∥fW−fW^∥≈∥GW12dW→∥F\lVert f_W-f_{\widehat{W}}\rVert\approx\left\lVert G_W^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F with GW=∂fW∂WT∂fW∂WG_W = \frac{\partial f_W}{\partial W}^{\mathsf{T}}\frac{\partial f_W}{\partial W}. Approximating the Gram matrix GW≈Σinput⊗ΣgradG_W \approx\Sigma_{\mathrm{input}} \otimes\Sigma_{\mathrm{grad}} via K-FAC (Dangel et al., [3]) and leveraging the Kronecker property (X⊗Y)vec⁡(Z)=vec⁡(YXZT)(X \otimes Y)\operatorname{vec}(Z)=\operatorname{vec}(YX Z^{\mathsf{T}}) yields the weighted Frobenius norm. See Appendix A for the full derivation. This optimization problem is solvable efficiently in closed-form via SVD. See Appendix § 12 for the computation of Σinput12\Sigma_{\mathrm{input}}^{\frac{1}{2}} and Σgrad12\Sigma_{\mathrm{grad}}^{\frac{1}{2}}.

Tractable Computation of Σgrad\Sigma_{\mathrm{grad}}

Theorem 1 reduces compression to the closed-form weighted Frobenius norm, but the computation of output-Jacobian Gram matrix Σgrad\Sigma_{\mathrm{grad}} requires one backward pass per output dimension, i.e., VV backward passes per token which is intractable for V≈32kV \approx32\mathrm{k}–128k128\mathrm{k} in modern LLMs.

LM-head factorization. Let f=WUhf=W_Uh be the predicted logits with WU∈RV×DW_U \in\mathbb{R}^{V\times D} the LM-head weight and hh the last hidden state. Using the chain rule, Σgrad\Sigma_{\mathrm{grad}} is a sum of (∂hn,t′/∂sn,t)TWUTWU(∂hn,t′/∂sn,t)(\partial h_{n,t'}/\partial s_{n,t})^\mathsf{T}W_U^\mathsf{T}W_U(\partial h_{n,t'}/\partial s_{n,t}) terms, where ∂hn,t′/∂sn,t\partial h_{n,t'}/\partial s_{n,t} is the sensitivity of the hidden state at output position t′t' to position tt. The vocabulary dimension is absorbed once into the precomputed Gram matrix MG≔WUTWUM_G \coloneqq W_U^\mathsf{T}W_U, reducing the number of cotangent directions from VV to DD, a factor ≈8\approx8 for Llama 2-7B and ≈31\approx31 for Llama 3-8B.

Broadcast backward. The remaining factor DD is the hidden size, and still dominates the calibration budget. Drawing a Rademacher matrix Ψ∈{±1}V×K\Psi\in\{\pm1\}^{V\times K} and setting Z≔WUTΨZ \coloneqq W_U^\mathsf{T}\Psi gives K≤DK \le D cotangent directions zkz_k with EΨ[zkzkT]=MG\mathbb{E}_{\Psi}[z_k z_k^\mathsf{T}] = M_G. Each direction is broadcast to every position of the LM-head input, weighted by an independent random sign εn,t′(k)\varepsilon_{n,t'}^{(k)}, and propagated in a single backward pass: the broadcast accumulates the contribution of all positions in one traversal of the computation graph, while the signs annihilate the cross-position terms in expectation.

Together these give the broadcast estimator Σ~grad\widetilde{\Sigma}_{\mathrm{grad}}, costing one forward pass and KK backward passes per sequence, independent of both VV and TnT_n, and unbiased for Σgrad\Sigma_{\mathrm{grad}} conditionally on the calibration set (Proposition 4). Against the exact Σgrad\Sigma_{\mathrm{grad}}, computable at reduced scale, Σ~grad\widetilde{\Sigma}_{\mathrm{grad}} recovers most of the compression quality (Table 13).

KL-SVD. With probes drawn from the predictive distribution instead of Rademacher columns, the same construction yields an unbiased estimator, at the same cost, of the true Fisher matrix Fs=E[JT(diag⁡(p)−ppT)J]F_s=\mathbb{E}[J^\mathsf{T}(\operatorname{diag}(p)-pp^\mathsf{T})J], where J≔∂fW∂sJ\coloneqq\frac{\partial f_W}{\partial s}, of the KL divergence between the predictive distributions (Appendix § 10.§ 5). Substituting it for Σgrad\Sigma_{\mathrm{grad}} yields KL-SVD, all other components are shared with Complete-SVD (Figure 1).

Experiments

We compare our method, Complete-SVD, against state-of-the-art SVD-based compression methods, including standard SVD, ASVD Yuan et al., 2023, FWSVD Hsu et al., [9], GFWSVD Chekalina et al., [1], SVD-LLM Wang et al., 2025c, Dobi-SVD Wang et al., 2025b, KFAC-SVD Thoma et al., 2026, and IO-SVD Abbasi et al., 2026 as well as structured pruning methods such as LLM-Pruner Ma et al., 2023 and Wanda-SP Sun et al., 2024. We further demonstrate that Complete-SVD is compatible with state-of-the-art quantization methods by combining it with GPTQ Frantar et al., [6]. To validate our approach, we conduct experiments on Llama 2-7B Touvron et al., 2023 Base and Chat, Llama 3-8B Grattafiori et al., [7], Mistral-7B Jiang et al., [11] and Qwen3-8B Yang et al., 2025, evaluating on the language modeling datasets WikiText-2 Merity et al., 2017, C4 Raffel et al., 2020, and PTB Marcus et al., 1993, the downstream benchmarks OpenbookQA Mihaylov et al., 2018, WinoGrande Sakaguchi et al., 2021, HellaSwag Zellers et al., 2019, PIQA Bisk et al., 2020, ARC-c, and ARC-e Clark et al., [2], and the generation dataset IFEval Zhou et al., 2023. The covariance matrices Σgrad\Sigma_{\mathrm{grad}} and Σinput\Sigma_{\mathrm{input}} are computed from 256 randomly sampled sequences of length 2048 from WikiText-2 dataset (See Appendix § 10.§ 3 for the calibration cost). Each experiment runs on a single GPU (NVIDIA H100 or H200), except the inference-efficiency measurements (Appendix § 13). The compression ratio (CR) is the parameter count of the compressed layers divided by that of the original layers.

LEMS Rank allocation. To test whether Complete-SVD composes with a published rank allocator, we use LEMS Thoma et al., 2026 with the authors’ procedure: one KL probe per layer, with that layer reduced to 10% of its parameters, on 32 calibration windows of WikiText-2; the harmonic cross-layer term; ranks in multiples of 8; and an exact integer-program solve (HiGHS) Huangfu and Hall, [10] at the target budget. Ranks are allocated under each method’s own whitening, so every method receives the allocation LEMS finds for it, and all use KFAC-SVD’s damping. We fix the depth-prior weight to γ∈{0,5}\gamma\in\{0, 5\} for every method and report both; the LEMS authors instead tune hyperparameters per cell, which gives stronger KFAC-SVD results at ratio 0.7 than our fixed-γ\gamma runs. The budget is 90% / 70% of the parameters of all attention and MLP projections; no layer is left dense.

Comparison to State-of-the-art Low-Rank SVD Methods

We evaluate Complete-SVD against state-of-the-art SVD-based baselines on Mistral-7B, Llama 3-8B and Qwen3-8B at compression ratios of 0.9 and 0.7 (Table 1). For a fair comparison, KFAC-SVD, IO-SVD, KL-SVD and Complete-SVD share the same uniform rank allocation and covariance regularization procedure (Appendix § 12). On the calibration corpus (WikiText-2), Complete-SVD remains within 1.3 perplexity points of the stronger of KFAC-SVD and IO-SVD in every configuration. The consistent advantage appears off the calibration distribution: Complete-SVD achieves the lowest C4 and PTB perplexity in all six configurations. On Llama 3-8B and Qwen3-8B, it reduces C4 perplexity relative to the stronger of these two baselines by 1–3% at ratio 0.9 and 5–15% at ratio 0.7, and PTB perplexity by 16–44%. Average zero-shot accuracy remains comparable, ranging from 1.4 percentage points below to 0.9 point above the stronger baseline. Under generative evaluation (Table 2), Complete-SVD outperforms both KFAC-SVD and IO-SVD on all three models at ratios 0.9 and 0.8. To verify that the perplexity gains are not specific to WikiText-2 calibration, Table 4 repeats the comparison on Llama 3-8B with C4 as the calibration set. Complete-SVD slightly improves on the stronger baseline on C4 at ratio 0.9 and reduces its perplexity by 20% at ratio 0.7, while reducing perplexity by 13–26% on WikiText-2 and 27–43% on PTB.

CRMethodMistral-7B Wiki↓Mistral-7B C4↓Mistral-7B PTB↓Mistral-7B Acc↑Llama 3-8B Wiki↓Llama 3-8B C4↓Llama 3-8B PTB↓Llama 3-8B Acc↑Qwen3-8B Wiki↓Qwen3-8B C4↓Qwen3-8B PTB↓Qwen3-8B Acc↑
Baseline5.38.127.764.06.19.59.963.39.715.515.462.0
FWSVD9.513.858.456.942.160.069.848.917.024.831.553.8
ASVD9.113.449.157.365.177.6207.747.920.430.036.451.2
SVD-LLM6.515.166.156.810.139.649.952.312.527.436.756.0
DOBI-SVD7.115.263.655.211.234.842.453.013.528.935.855.5
0.9GFWSVD31.765.1305.740.725703828849233.4171.5350.5765.646.7
KFAC-SVD6.213.144.056.98.825.732.654.511.523.828.057.1
IO-SVD6.112.840.056.98.423.230.956.211.023.728.957.9
KL-SVD6.112.840.157.38.322.827.156.911.224.629.157.7
Complete-SVD6.412.739.956.98.722.423.055.811.723.423.456.7
FWSVD34.850.3215.144.2716.4837.5194033.441.468.890.742.7
ASVD28.242.9163.646.11098994003603333.070.797.0131.543.1
SVD-LLM11.042.6169.544.434.6376.0600.937.717.163.1136.245.8
DOBI-SVD12.439.7170.842.931.5500.1223.738.518.563.894.346.2
0.7GFWSVD25503404535632.341799536717222231.821685276275081534.1
KFAC-SVD9.334.6110.745.819.4151.3253.540.714.750.094.046.8
IO-SVD8.833.197.845.616.8111.8219.243.314.458.7120.249.2
KL-SVD10.334.798.846.616.0103.0172.745.614.249.982.848.6
Complete-SVD10.132.990.745.416.895.3123.744.215.047.567.747.8

Table 1. Compression results on Mistral-7B, Llama 3-8B and Qwen3-8B with uniform rank allocation. Results for IO-SVD are reproduced by us; all other baselines are reported from Thoma et al. (2026). Accuracy (Acc) reported as average zero-shot accuracy value on 6 downstream datasets.

CRMethodMistral 7BQwen3 8BLlama 2 7B
Baseline16.824.430.3
0.9KFAC-SVD12.417.220.2
IO-SVD13.912.222.4
KL-SVD12.417.727.7
Complete-SVD17.421.326.6
0.8KFAC-SVD11.814.622.2
IO-SVD7.410.515.3
KL-SVD11.313.323.5
Complete-SVD12.918.125.3
CR Methodγ\gammaMistral-7B Wiki↓\downarrowC4↓\downarrowPTB↓\downarrowAcc↑\uparrowLlama 3-8B Wiki↓\downarrowC4↓\downarrowPTB↓\downarrowAcc↑\uparrowQwen3-8B Wiki↓\downarrowC4↓\downarrowPTB↓\downarrowAcc↑\uparrow
Baseline-5.38.127.764.06.19.59.963.39.715.515.462.0
KFAC-SVD5.48.626.862.36.711.611.560.59.916.215.961.9
KL-SVD05.48.526.562.46.611.411.261.09.916.216.061.8
Complete-SVD5.48.627.062.26.611.411.161.09.916.216.061.8
KFAC-SVD5.58.829.362.37.112.612.962.010.116.716.661.1
KL-SVD55.48.727.262.67.012.312.062.110.116.716.861.3
Complete-SVD5.48.727.462.37.012.211.661.310.016.616.560.9
KFAC-SVD8.530.3100.947.615.682.8162.545.812.233.748.052.1
KL-SVD08.228.883.748.313.165.0117.547.312.033.250.353.5
Complete-SVD8.727.974.246.813.460.977.346.011.931.136.352.8
KFAC-SVD7.620.165.252.011.838.188.250.013.034.866.652.8
KL-SVD57.318.456.553.111.034.273.151.513.435.769.052.8
Complete-SVD7.518.051.651.611.132.550.850.613.233.247.751.8

Table 2. IFEval (0-shot, prompt-level strict accuracy, without applying a chat template) under low-rank compression at compression ratios (CR) 0.9 and 0.8. Llama 2-7B is the Chat variant.

CR MethodLlama 3-8B Wiki ↓\downarrowC4 ↓\downarrowPTB ↓\downarrowAcc ↑\uparrow
Baseline6.19.59.963.3
KFAC-SVD17.715.923.655.9
IO-SVD14.716.125.856.8
KL-SVD15.215.518.357.6
Complete-SVD12.815.717.356.8
KFAC-SVD89.047.3158.643.6
IO-SVD69.545.5169.044.9
KL-SVD60.136.7108.546.4
Complete-SVD51.236.289.945.7

Table 4. Compression results of output-aware methods on Llama 3-8B (WikiText-2, C4, and PTB perplexity, zero-shot accuracy) when C4 dataset is used for calibration with 256 samples and 2048 token sequence.

Comparison with KL-SVD. Comparing Complete-SVD with KL-SVD further isolates the effect of the compression objective, as both methods also share the probe-based estimation framework. Relative to KL-SVD, Complete-SVD has 4–6% higher WikiText-2 perplexity in five of the six configurations and 0.4–1.4 percentage points lower average zero-shot accuracy in all six. Nevertheless, Complete-SVD achieves lower C4 and PTB perplexity in every configuration, with reductions of up to 7% and 28%, respectively. These results suggest that the Complete-SVD objective better preserves performance beyond the calibration corpus, at the cost of slightly higher calibration perplexity and lower zero-shot accuracy. With C4 calibration (Table 4) on Llama 3-8B, KL-SVD attains slightly lower C4 perplexity at ratio 0.9 (15.5 against 15.7), but Complete-SVD reduces WikiText-2 perplexity by 16% and PTB perplexity by 5%. At ratio 0.7, Complete-SVD is lower on all three corpora, by 15% on WikiText-2, 1% on C4 and 17% on PTB. As with WikiText-2 calibration, KL-SVD attains higher average zero-shot accuracy at both ratios.

Sensitivity-aware rank allocation. Table 3 reports the output-aware methods under LEMS. The allocator decides how many directions each layer keeps, the objective which ones. At ratio 0.9, allocation alone brings every method close to the uncompressed model (Llama 3-8B C4: 11.4 against 9.5, versus 22.4 with uniform ranks), leaving little for the objective to change: Complete-SVD and KL-SVD differ by at most 0.5 perplexity points. At ratio 0.7, the objective matters again: Complete-SVD achieves the lowest C4 and PTB perplexity on all three models and for both values of γ\gamma, and its PTB reduction over the strongest competitor (9–34%) is at least as large as with uniform ranks (7–28%). The ordering holds for both γ\gamma although γ\gamma changes perplexity by up to a factor of two, so it does not hinge on the allocator’s tuning. As under uniform ranks, KL-SVD keeps the higher zero-shot accuracy.

Comparison with pruning methods. Following the setup of Dobi-SVD (Wang et al., 2025b), we also compare Complete-SVD with post-training pruning methods (Table 5). At 10.1 GB, LLM-Pruner and Wanda-SP reach WikiText-2 perplexities of 10.5 and 12.1, against 7.5 for

Memory (GB)10.17.76.8
Wiki↓\downarrow
LLM-Pruner10.546.3253.1
Wanda-SP12.138.6249.2
Complete-SVD7.513.319.8
Dobi-SVD*5.97.99.5
Complete-SVD*5.87.08.4

Table 5. Perplexity results (Wiki-2) of Complete-SVD versus state-of-the-art pruning methods and Dobi-SVD on Llama 2-7B (Base). Methods marked with * use remapping storage.

caption: Compression results on Mistral-7B, Llama 3-8B and Qwen3-8B with LEMS rank allocation at matched budget and fixed hyperparameters.

:::

Complete-SVD; at 6.8 GB, both exceed 249, whereas Complete-SVD attains 19.8. Under the remapping storage of Dobi-SVD (marked , Appendix § 11), Complete-SVD outperforms Dobi-SVD* at every memory budget, although it uses uniform ranks whereas Dobi-SVD* uses its loss-aware rank allocation.

Ablation and Sensitivity Studies

Ablation Study on Covariance Components. We analyze the individual contributions of Σgrad\Sigma_{\mathrm{grad}} and Σinput\Sigma_{\mathrm{input}} to the SVD compression objective on Llama 2-7B (Base). As shown in Table 6, while each matrix individually improves over standard SVD, their combination consistently yields the best performance across all compression ratios. At

At 50% compression ratio, Complete-SVD achieves substantially lower perplexity than using Σinput\Sigma_{\mathrm{input}} alone or Σgrad\Sigma_{\mathrm{grad}} alone, demonstrating that each component contributes complementary information to the functional norm.

Robustness to Calibration Data. We assess the sensitivity of Complete-SVD to its random draws and calibration budget (Figure 2). At the reference budget (N=256N = 256 sequences, K=64K = 64 probes), varying the calibration seed (resampling the sequences) give WikiText-2 perplexities in {7.53,7.54,7.57}\{7.53, 7.54, 7.57\} (std scal=0.02s_{\mathrm{cal}} = 0.02), and the estimator seed (redrawing the probes Ψ\Psi and signs ϵ\epsilon of Σ^grad\hat{\Sigma}_{\mathrm{grad}}) give {7.49,7.53,7.55}\{7.49, 7.53, 7.55\} (std sest=0.03s_{\mathrm{est}} = 0.03). Reducing NN from 256 to 32 slightly raises perplexity monotonically from 7.53 to 7.61, and varying KK over {64,128,256}\{64, 128, 256\} changes it by at most 0.03 (within noise). We use N=256N = 256 and K=64K = 64 as a compromise between cost and precision. Table 7 compares covariance matrices estimated on WikiText-2, our default, against C4:

WikiText-2 perplexity versus sample size and probe budget

Figure 2. WikiText-2 perplexity of Complete-SVD on Llama 2-7B (Base) at 80% compression ratio versus the sample size NN (left, K=64K = 64) and the probe budget KK (right, N=256N = 256). Error bars show ±σ\pm\sigma, with σ=σcal2+σest2≈0.04\sigma= \sqrt{\sigma_{\mathrm{cal}}^{2} + \sigma_{\mathrm{est}}^{2}} \approx0.04 the single-run standard deviation estimated from seed repeats at the reference budget.

Cal. SetWiki ↓\downarrowC4 ↓\downarrowPTB ↓\downarrowAcc. ↑\uparrow
Wiki-27.515.256.950.0
C49.911.656.051.3

Table 7. Effect of the calibration set (WikiText-2 vs. C4) on Complete-SVD applied to Llama 2-7B (Base) at 80% compression ratio. Average accuracy is computed over 6 downstream datasets.

CRMethodCinputC_{\mathrm{input}}CgradC_{\mathrm{grad}}Wiki ↓\downarrow
BaselineN/AN/A5.47
0.8SVDIdId19 438.3
0.8Gradient onlyIdΣgrad\Sigma_{\mathrm{grad}}21.8
0.8Forward onlyΣinput\Sigma_{\mathrm{input}}Id8.5
0.8Complete-SVDΣinput\Sigma_{\mathrm{input}}Σgrad\Sigma_{\mathrm{grad}}7.5
0.7SVDIdId31 062.3
0.7Gradient onlyIdΣgrad\Sigma_{\mathrm{grad}}81.9
0.7Forward onlyΣinput\Sigma_{\mathrm{input}}Id10.9
0.7Complete-SVDΣinput\Sigma_{\mathrm{input}}Σgrad\Sigma_{\mathrm{grad}}9.3
0.6SVDIdId44 494.7
0.6Gradient onlyIdΣgrad\Sigma_{\mathrm{grad}}290.0
0.6Forward onlyΣinput\Sigma_{\mathrm{input}}Id17.3
0.6Complete-SVDΣinput\Sigma_{\mathrm{input}}Σgrad\Sigma_{\mathrm{grad}}13.3
0.5SVDIdId53 670.8
0.5Gradient onlyIdΣgrad\Sigma_{\mathrm{grad}}2211.0
0.5Forward onlyΣinput\Sigma_{\mathrm{input}}Id37.9
0.5Complete-SVDΣinput\Sigma_{\mathrm{input}}Σgrad\Sigma_{\mathrm{grad}}22.4

caption: WikiText-2 perplexity of Llama 2-7B (Base) when optimizing ∥Cinput12dWCgrad12∥F\left\|C_{\mathrm{input}}^{\frac{1}{2}}dW C_{\mathrm{grad}}^{\frac{1}{2}}\right\|_{F}.

:::

each set favors its own distribution in perplexity, with comparable downstream accuracy.

Compatibility with quantization. We verify this by applying 3-bit weight-only GPTQ quantization (W3A16) to Llama 2-7B compressed by Complete-SVD at several ratios (Table 8). At ratio 0.8, quantization reduces the memory footprint

CRMethodWiki ↓\downarrowMem ↓\downarrow
0.8Complete-SVD7.510.1 GB
0.8Complete-SVD + GPTQ W3A169.22.5 GB
0.6Complete-SVD13.37.7 GB
0.6Complete-SVD + GPTQ W3A1616.71.9 GB

Table 8. Perplexity (Wiki) and model size for Complete-SVD combined with GPTQ quantization on Llama 2-7B (Base). Memory reports the model parameter storage after compression.

from 10.1 GB to 2.5 GB at a WikiText-2 perplexity of 9.2; at ratio 0.6, the model occupies 1.9 GB, 14% of the original FP16 size. The low-rank factors produced by Complete-SVD therefore remain compatible with quantization, and the two techniques can be combined for deployment on memory-constrained devices.

Conclusion

We introduced Complete-SVD, a post-training low-rank compression method that minimizes the L2L_2 distance between the logits of the original and compressed networks over the data distribution. We make the Gram matrix of logit Jacobians, Σgrad\Sigma_{\mathrm{grad}}, tractable by exactly reducing its vocabulary dimension to the hidden dimension and estimating it without bias using KK backward passes per sequence, independent of vocabulary size and sequence length. As a controlled counterpart, we proposed KL-SVD which replaces L2L_2 distance with KL divergence between predictive distributions, and uses an unbiased estimator of the true Fisher matrix.

Across Mistral-7B, Llama 3-8B and Qwen3-8B, Complete-SVD achieves the lowest C4 and PTB perplexity among all compared methods, including KL-SVD, under WikiText-2 calibration, with covariance regularization held fixed across the output-aware methods. This cross-corpus advantage holds under both uniform and sensitivity-aware (LEMS) rank allocation, the latter at ratio 0.7, and persists with C4 calibration. Complete-SVD also improves IFEval instruction-following accuracy over KFAC-SVD and IO-SVD in every tested setting, and over KL-SVD in all but one setting. KL-SVD, in turn, attains equal or higher average zero-shot accuracy in every configuration. Complete-SVD requires no fine-tuning and is compatible with post-training quantization. These results suggest that, for post-training low-rank compression, matching the logits of the original model is a simple and effective alternative to preserving either the loss on the observed next token or the predicted distribution.

Limitations

Firstly, our evaluation is limited to models in the 7–8B parameter range. Evaluating larger models would provide stronger evidence that the observed behavior of the different output-aware methods persists at scale. Secondly, as in most post-training SVD-based methods, each layer is compressed using statistics collected from the original model, so the errors introduced by previously compressed layers are not taken into account. AA-SVD (Sinha and Fleuret, 2026) mitigates this by combining an activation-aware factorization with a gradient-based block-wise refinement on the activations of the partially compressed network, at the cost of no longer being training-free. Reducing calibration cost and estimator variance without sacrificing the full functional objective remains an important direction for future work. Finally, the complete functional norm is not specific to low-rank factorization. GPTQ (Frantar et al., [6]) and SparseGPT (Frantar and Alistarh, [5]) minimize

∥Σinput1/2dW∥F2\left\lVert\Sigma_{\mathrm{input}}^{1/2}\mathrm{d}W \right\rVert_F^2

, that is, the objective of Theorem 1 with Σgrad=Id\Sigma_{\mathrm{grad}} = \mathrm{I}_{d}. Substituting Σgrad\Sigma_{\mathrm{grad}} yields an output-aware objective for quantization and pruning, although the closed-form SVD solution no longer applies and a dedicated solver is required, as in YAQA (Tseng et al., 2026) for the KL objective.

References

A Proof on Complete Functional Norm

Theorem 1. With the notation introduced in § 2.1, let Σinput≔Ex∼D[a(x)a(x)T]\Sigma_{\mathrm{input}} \coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left[a(x)a(x)^{\mathsf{T}}\right] denote the input covariance and Σgrad≔Ex∼D[(∂fW(x)∂s)T∂fW(x)∂s]\Sigma_{\mathrm{grad}} \coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left[\left(\frac{\partial f_W(x)}{\partial s}\right)^{\mathsf{T}}\frac{\partial f_W(x)}{\partial s}\right] the Gram matrix of the output Jacobians at the compressed layer. Then:

(i) Exact first-order expansion. Let GW≔Ex∼D[(∂fW(x)∂W)T∂fW(x)∂W]∈Rnsna×nsnaG_W \coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left[\left(\frac{\partial f_W(x)}{\partial W}\right)^{\mathsf{T}}\frac{\partial f_W(x)}{\partial W}\right]\in\mathbb{R}^{n_s n_a\times n_s n_a}. Then

∥fW−fW^∥=∥GW12dW→∥F+o(∥dW∥F).(5)\lVert f_W-f_{\widehat{W}}\rVert=\left\lVert G_W^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F+o(\lVert\mathrm{d}W\rVert_F). \tag*{(5)}

(ii) Kronecker-factored approximation. Under the K-FAC structural approximation GW≈Σinput⊗ΣgradG_W\approx\Sigma_{\mathrm{input}}\otimes\Sigma_{\mathrm{grad}}, the right-hand side of Equation (5) admits the closed-form approximation

∥GW12dW→∥F≈∥Σinput12dWΣgrad12∥F.(6)\left\lVert G_W^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F\approx\left\lVert\Sigma_{\mathrm{input}}^{\frac{1}{2}}\mathrm{d}W\Sigma_{\mathrm{grad}}^{\frac{1}{2}}\right\rVert_F. \tag*{(6)}

Proof. Step 1 (exact first-order expansion). Writing the first-order Taylor development of fWf_W around WW,

fW^−fW=∂fW∂WdW→+o(∥dW∥F).(7)f_{\widehat{W}}-f_W=\frac{\partial f_W}{\partial W}\overrightarrow{\mathrm{d}W}+o(\lVert\mathrm{d}W\rVert_F). \tag*{(7)}

Taking norms and expanding the inner product (full derivation in Appendix B.§ 2),

∥fW−fW^∥2=∥∂fW∂WdW→∥F2+o(∥dW∥F2)(8)\begin{aligned} \lVert f_W-f_{\widehat{W}}\rVert^2 \\ &=\left\lVert\frac{\partial f_W}{\partial W}\overrightarrow{\mathrm{d}W}\right\rVert_F^2+o(\lVert\mathrm{d}W\rVert_F^2) \tag*{(8)} \end{aligned}
=(dW→)TGWdW→+o(∥dW∥F2)(9)=(\overrightarrow{\mathrm{d}W})^{\mathsf{T}}G_W\overrightarrow{\mathrm{d}W}+o(\lVert\mathrm{d}W\rVert_F^2) \tag*{(9)}

where GW≔(∂fW∂W)T∂fW∂WG_W\coloneqq\left(\frac{\partial f_W}{\partial W}\right)^{\mathsf{T}}\frac{\partial f_W}{\partial W}. Taking square roots yields Equation (5).

Step 2 (K-FAC approximation). The matrix GWG_W has size nsna×nsnan_s n_a\times n_s n_a and is intractable to store or factorize at LLM scale. We invoke the K-FAC structural approximation (Dangel et al., [3]),

GW≈Σinput⊗Σgrad,(10)G_W\approx\Sigma_{\mathrm{input}}\otimes\Sigma_{\mathrm{grad}}, \tag*{(10)}

where Σinput≔Ex∼D[a(x)a(x)T]\Sigma_{\mathrm{input}}\coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left[a(x)a(x)^{\mathsf{T}}\right] and Σgrad≔Ex∼D[(∂fW(x)∂s)T∂fW(x)∂s]\Sigma_{\mathrm{grad}}\coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left[\left(\frac{\partial f_W(x)}{\partial s}\right)^{\mathsf{T}}\frac{\partial f_W(x)}{\partial s}\right]. This is a structural approximation: see Remark 2.

Combining the Kronecker identities (X⊗Y)12=X12⊗Y12(X\otimes Y)^{\frac{1}{2}}=X^{\frac{1}{2}}\otimes Y^{\frac{1}{2}} and (X⊗Y)vec⁡(Z)=vec⁡(XZYT)(X\otimes Y)\operatorname{vec}(Z)=\operatorname{vec}(XZY^{\mathsf{T}}) with Equation (10) gives

∥GW12dW→∥F≈∥(Σinput⊗Σgrad)12dW→∥F=∥Σinput12dWΣgrad12∥F,(11)\begin{aligned} \left\lVert G_W^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F\approx\left\lVert(\Sigma_{\mathrm{input}}\otimes\Sigma_{\mathrm{grad}})^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F \\ &=\left\lVert\Sigma_{\mathrm{input}}^{\frac{1}{2}}\mathrm{d}W\Sigma_{\mathrm{grad}}^{\frac{1}{2}}\right\rVert_F, \tag*{(11)} \end{aligned}

which is Equation (6). □

Remark 2. The two approximation symbols in Theorem 1 are not interchangeable. The remainder in Equation (5) is o(∥dW∥F)o(\lVert\mathrm{d}W\rVert_F) and vanishes as dW→0\mathrm{d}W\to0. The K-FAC step in Equation (6), by contrast, replaces the full Gram matrix GWG_W by its Kronecker-factored approximation; the resulting structural error εKFAC≔∥GW−Σinput⊗Σgrad∥F\varepsilon_{\mathrm{KFAC}}\coloneqq\lVert G_W-\Sigma_{\mathrm{input}}\otimes\Sigma_{\mathrm{grad}}\rVert_F is independent of dW\mathrm{d}W and does not vanish in any limit. We adopt the K-FAC approximation because it admits a closed-form SVD solution and has been empirically validated as a tight approximation of the layer-wise curvature for fully-connected layers (Dangel et al., [3]; Eschenhagen et al., [4]); we do not claim it is asymptotically tight in dW\mathrm{d}W.

B Reconstruction Error

The first idea to find a good W^\widehat{W} from WW is to minimize ∥W^−W∥F\lVert\widehat{W}-W\rVert_F which can be done by computing the SVD of WW and keeping the top kk singular values. However, this does not take into account the effect of the change in weight on the loss. Hence, we can aim instead to minimize the change in loss. Writing dW=W^−W\mathrm{d}W=\widehat{W}-W and dW→\overrightarrow{\mathrm{d}W} the vectorization of dW\mathrm{d}W we have the following first-order Taylor expansion of the loss around WW:

L(fW^)−L(fW)=⟨∇WL(fW),dW→⟩2+o(∥dW∥F)(12)\begin{aligned} \mathcal{L}(f_{\widehat{W}})-\mathcal{L}(f_W) \\ &=\left\langle\nabla_W\mathcal{L}(f_W),\overrightarrow{\mathrm{d}W}\right\rangle_2+o(\lVert\mathrm{d}W\rVert_F) \tag*{(12)} \end{aligned}

This suggests choosing W^\widehat{W} in order to minimize ⟨∇WL(fW),dW→⟩\left\langle\nabla_W\mathcal{L}(f_W),\overrightarrow{\mathrm{d}W}\right\rangle. However this would lead to only approximate fWf_W on the correct label and does not give a rich feedback. As shown for Knowledge distillation (Hinton et al., [8]), it is better to copy the full distribution and to have a richer feedback. In addition, we suppose that fWf_W is a local minimizer (as it is an already trained function), we can expect that ∇WL(fW)≈0\nabla_W\mathcal{L}(f_W)\approx0 and that the gradient information is limited.

Hence we can either directly minimize the functional distance between the output of fW^f_{\widehat{W}} and fWf_W:

∥fW−fW^∥2=Ex∼D(∥fW(x)−fW^(x)∥F2)(13)\begin{aligned} \lVert f_W-f_{\widehat{W}}\rVert^2 \\ &= \mathbb{E}_{x\sim\mathcal{D}}\left(\left\|f_W(x)-f_{\widehat{W}}(x)\right\|_F^2\right) \tag*{(13)} \end{aligned}

or minimize the expected KL divergence between the output of fW^f_{\widehat{W}} and fWf_W:

KL⁡(fW,fW^)(14)\operatorname{KL}(f_W,f_{\widehat{W}}) \tag*{(14)}
≔Ex∼D(KL⁡(Softmax⁡(fW(x)),Softmax⁡(fW^(x))))\coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left(\operatorname{KL}\left(\operatorname{Softmax}(f_W(x)),\operatorname{Softmax}(f_{\widehat{W}}(x))\right)\right)

two objectives that capture the full output distribution and not only the value at the correct label. We show now how we can optimize those objectives efficiently.

Norm Optimization

What we can show is that minimizing Equation (12), Equation (14) or Equation (13) can be done with similar techniques. Each of those objectives can be approximated by a quadratic form of dW→\overrightarrow{\mathrm{d}W} with a curvature matrix that can be approximated using the Kronecker-factored approximation ([3]) as a Kronecker product of two matrices CinputC_{\mathrm{input}} and CgradC_{\mathrm{grad}}.

Hilbertian Norm Minimization

Writing the first order development of fWf_W around WW we have:

fW^−fW=∂fW∂WdW→+o(∥dW∥F)(15)f_{\widehat{W}}-f_W=\frac{\partial f_W}{\partial W}\overrightarrow{\mathrm{d}W}+o(\|\mathrm{d}W\|_F) \tag*{(15)}

Hence:

∥fW−fW^∥2=∥∂fW∂WdW→∥2+⟨∂fW∂WdW→,o(∥dW∥F)⟩+o(∥dW∥F)2(16)\begin{aligned} \|f_W-f_{\widehat{W}}\|^2=\left\|\frac{\partial f_W}{\partial W}\overrightarrow{\mathrm{d}W}\right\|^2 \\ &+\left\langle\frac{\partial f_W}{\partial W}\overrightarrow{\mathrm{d}W},o(\|\mathrm{d}W\|_F)\right\rangle+o(\|\mathrm{d}W\|_F)^2 \tag*{(16)} \end{aligned}
=∥∂fW∂WdW→∥2+⟨∂fW∂W,o(∥dW∥F)dW→T⟩+o(∥dW∥F2)(17)\begin{aligned} =\left\|\frac{\partial f_W}{\partial W}\overrightarrow{\mathrm{d}W}\right\|^2 \\ &+\left\langle\frac{\partial f_W}{\partial W},o(\|\mathrm{d}W\|_F)\overrightarrow{\mathrm{d}W}^{\mathsf{T}}\right\rangle+o(\|\mathrm{d}W\|_F^2) \tag*{(17)} \end{aligned}
=∥((∂fW∂W)T∂fW∂W)12dW→∥F2+o(∥dW∥F2)(18)\begin{aligned} =\left\|\left(\left(\frac{\partial f_W}{\partial W}\right)^{\mathsf{T}}\frac{\partial f_W}{\partial W}\right)^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\|_F^2 \\ &+o(\|\mathrm{d}W\|_F^2) \tag*{(18)} \end{aligned}

Then taking the square-root we have:

∥fW−fW^∥=∥((∂fW∂W)T∂fW∂W)12dW→∥F+o(∥dW∥F)(19)\begin{aligned} \|f_W-f_{\widehat{W}}\|=\left\|\left(\left(\frac{\partial f_W}{\partial W}\right)^{\mathsf{T}}\frac{\partial f_W}{\partial W}\right)^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\|_F \\ &+o(\|\mathrm{d}W\|_F) \tag*{(19)} \end{aligned}

This is still expensive to compute, as for fully-connected layers, GW≔(∂fW∂W)T∂fW∂WG_W\coloneqq\left(\frac{\partial f_W}{\partial W}\right)^{\mathsf{T}}\frac{\partial f_W}{\partial W} is a matrix of size nsna×nsnan_s n_a\times n_s n_a. To further reduce the computational cost, we can use the Kronecker-factored approximation ([3]) to approximate GWG_W by the Kronecker product of two matrices Σinput≔Ex∼D(a(x)a(x)T)\Sigma_{\mathrm{input}}\coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left(a(x)a(x)^{\mathsf{T}}\right) and Σgrad≔Ex∼D((∂fW∂s(x))T∂fW∂s(x))\Sigma_{\mathrm{grad}}\coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left(\left(\frac{\partial f_W}{\partial s}(x)\right)^{\mathsf{T}}\frac{\partial f_W}{\partial s}(x)\right): GW≈Σinput⊗ΣgradG_W\approx\Sigma_{\mathrm{input}}\otimes\Sigma_{\mathrm{grad}}.

Direct Loss Minimization

We can minimize the absolute value as we can suppose that we won’t get a lower loss after compression. Rewriting the objective:

∣⟨∇WL(fW),dW→⟩∣2=∥∇WL(fW)TdW→∥F=∥(∇WL(fW)(∇WL(fW))T)12dW→∥F(20)\begin{aligned} \left|\left\langle\nabla_W\mathcal{L}(f_W),\overrightarrow{\mathrm{d}W}\right\rangle\right|_2 \\ &=\left\|\nabla_W\mathcal{L}(f_W)^{\mathsf{T}}\overrightarrow{\mathrm{d}W}\right\|_F \\ &=\left\|\left(\nabla_W\mathcal{L}(f_W)(\nabla_W\mathcal{L}(f_W))^{\mathsf{T}}\right)^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\|_F \tag*{(20)} \end{aligned}

KL Minimization

A classical result in information theory is that the change in KL divergence can be approximated by a quadratic form of the Fisher information matrix of the distribution fW(x)f_W(x) with respect to the parameters WW noted FW(x)F_W(x):

KL⁡(Softmax⁡(fW(x)),Softmax⁡(fW^(x)))=12(dW→)TFW(x)dW→+o(∥dW∥F2)(21)\operatorname{KL}(\operatorname{Softmax}(f_W(x)),\operatorname{Softmax}(f_{\widehat{W}}(x))) =\frac{1}{2}\left(\overrightarrow{\mathrm{d}W}\right)^{\mathsf{T}}F_W(x)\overrightarrow{\mathrm{d}W}+o(\|\mathrm{d}W\|_F^2) \tag*{(21)}

where the Fisher information matrix is defined for pW(x)≔Softmax⁡(fW(x))p_W(x)\coloneqq\operatorname{Softmax}(f_W(x)) the label distribution predicted by the model parameterized by WW as:

FW(x)=Ey∼pW(x)[∇Wlog⁡pW(y∣x)×(∇Wlog⁡pW(y∣x))T].(22)F_W(x)=\mathbb{E}_{y\sim p_W(x)}\left[\nabla_W\log p_W(y\mid x)\times\left(\nabla_W\log p_W(y\mid x)\right)^{\mathsf{T}}\right]. \tag*{(22)}

Hence taking the expectation over the data distribution, we have with FW≔Ex∼D(FW(x))F_W\coloneqq\mathbb{E}_{x\sim\mathcal{D}}(F_W(x))

KL⁡(fW,fW^)=12(dW→)TFWdW→+o(∥dW∥F2)≈12∥FW12dW→∥F2(23)\begin{aligned} \operatorname{KL}(f_W,f_{\widehat{W}}) \\ &= \frac{1}{2}\left(\overrightarrow{\mathrm{d}W}\right)^{\mathsf{T}}F_W\overrightarrow{\mathrm{d}W}+o\left(\lVert\mathrm{d}W\rVert_F^2\right) \\ &\approx\frac{1}{2}\left\lVert F_W^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F^2 \tag*{(23)} \end{aligned}

and as shown by Dangel et al. [3], FWF_W can be approximated by the Kronecker product of two matrices Σinput\Sigma_{\mathrm{input}} and FsF_s: FW≈Σinput⊗FsF_W \approx\Sigma_{\mathrm{input}} \otimes F_s where FsF_s is the expected Fisher information matrix of the distribution fW(x)f_W(x) with respect to values s(x)s(x). If we note pW(x)≔Softmax⁡(fW(x))p_W(x) \coloneqq\operatorname{Softmax}(f_W(x)) we have:

Fs=Ex∼D[Fs(x)](24)F_s=\mathbb{E}_{x\sim\mathcal{D}}\left[F_s(x)\right] \tag*{(24)}
=Ex∼D[Ey∼pW(x)[(∇s(x)log⁡pW(y∣x))×(∇s(x)log⁡pW(y∣x))T]](25)=\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim p_W(x)}\left[\left(\nabla_{s(x)}\log p_W(y\mid x)\right)\times\left(\nabla_{s(x)}\log p_W(y\mid x)\right)^{\mathsf{T}}\right]\right] \tag*{(25)}

Optimization of the Quadratic Forms

All of above objectives can be formulated as minimizing ∥C12dW→∥F\left\lVert C^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F where CC is a conditioning matrix that can be approximated by the Kronecker product of two matrices CinputC_{\mathrm{input}} and CgradC_{\mathrm{grad}}. Using the properties of the Kronecker product, (X⊗Y)12=X12⊗Y12(X\otimes Y)^{\frac{1}{2}}=X^{\frac{1}{2}}\otimes Y^{\frac{1}{2}} and (X⊗Y)vec⁡(Z)=vec⁡(XZYT)(X\otimes Y)\operatorname{vec}(Z)=\operatorname{vec}(XZY^{\mathsf{T}}) (where vec⁡\operatorname{vec} is the row-wise vectorization), we have:

d(fW,f^W)≈∥(Cinput⊗Cgrad)12dW→∥F=∥Cinput12dWCgrad12∥F(26)\begin{aligned} d(f_W,\widehat{f}_W)\approx\left\lVert\left(C_{\mathrm{input}}\otimes C_{\mathrm{grad}}\right)^{\frac{1}{2}}\overrightarrow{\mathrm{d}W}\right\rVert_F \\ &=\left\lVert C_{\mathrm{input}}^{\frac{1}{2}}\mathrm{d}W C_{\mathrm{grad}}^{\frac{1}{2}}\right\rVert_F \tag*{(26)} \end{aligned}

This final optimization problem can be solved efficiently in closed-form using the SVD.

MethodCinputC_{\mathrm{input}}CgradC_{\mathrm{grad}}
Empirical Fisher curvature ([1])Σinput\Sigma_{\mathrm{input}}EsE_s
KL curvatureΣinput\Sigma_{\mathrm{input}}FsF_s
L2 curvatureΣinput\Sigma_{\mathrm{input}}Σgrad\Sigma_{\mathrm{grad}}

Table 9. Kronecker-factored curvatures of the output-aware objectives. Each objective weights the approximation error of WW by Cinput⊗CgradC_{\mathrm{input}}\otimes C_{\mathrm{grad}}. The input factor Cinput=ΣinputC_{\mathrm{input}}=\Sigma_{\mathrm{input}} is common to all methods, which differ only in the output factor CgradC_{\mathrm{grad}}.

Fisher and Empirical Fisher

The empirical Fisher matrix is called “empirical” because it evaluates the gradient at the observed label instead of taking the expectation over the predictive distribution of the model, as the true Fisher matrix does. The full Fisher matrix is defined as:

FW(x)=Ey∼pW(x)[∇Wlog⁡pW(y∣x)×(∇Wlog⁡pW(y∣x))T](27)F_W(x)=\mathbb{E}_{y\sim p_W(x)}\left[\nabla_W\log p_W(y\mid x)\times\left(\nabla_W\log p_W(y\mid x)\right)^{\mathsf{T}}\right] \tag*{(27)}

a common approximation is to replace the expectation by the value at the correct label y(x)y(x). This is a logical choice as the correct label is the most important one for the loss.

If we approximate pW(x)p_W(x) by a Dirac distribution on the correct label δ(y(x))\delta(y(x)), we have:

FW(x)≈Ey∼δ(y(x))[∇Wlog⁡pW(y∣x)×(∇Wlog⁡pW(y∣x))T](28)F_W(x)\approx\mathbb{E}_{y\sim\delta(y(x))}\left[\nabla_W\log p_W(y\mid x)\times\left(\nabla_W\log p_W(y\mid x)\right)^{\mathsf{T}}\right] \tag*{(28)}
=∇Wlog⁡pW(y(x)∣x)×(∇Wlog⁡pW(y(x)∣x))T(29)=\nabla_W\log p_W(y(x)\mid x)\times\left(\nabla_W\log p_W(y(x)\mid x)\right)^{\mathsf{T}} \tag*{(29)}

As the cross-entropy loss is ℓ(x,y)=−log⁡(pW(y(x)∣x))+constant\ell(x,y)=-\log(p_W(y(x)\mid x))+\text{constant}, where y(x)y(x) denote the label of xx, we have:

=∇Wℓ(fW(x,y(x)))(∇Wℓ(fW(x,y(x))))T=EW(x)(30)\begin{aligned} =\nabla_W\ell(f_W(x,y(x)))\left(\nabla_W\ell(f_W(x,y(x)))\right)^{\mathsf{T}} \\ &=E_W(x) \tag*{(30)} \end{aligned}

EE is called the empirical Fisher. Similar to the True Fisher, it can be approximated by a Kronecker product of two matrices:

EW≈Σinput⊗Es(31)E_W\approx\Sigma_{\mathrm{input}}\otimes E_s \tag*{(31)}

All the CgradC_{\mathrm{grad}} can be defined with a similar structure:

Σgrad=Ex∼D((∂fW∂s(x))T∂fW∂s(x))(32)\Sigma_{\mathrm{grad}}=\mathbb{E}_{x\sim\mathcal{D}}\left(\left(\frac{\partial f_W}{\partial s}(x)\right)^{\mathsf{T}}\frac{\partial f_W}{\partial s}(x)\right) \tag*{(32)}

For the Fisher using the chain rule (see Dangel et al. [3] for more details) we have:

Fs=Ex∼D[Ey∼pW(x)[(∇s(x)log⁡pW(y∣x))××(∇s(x)log⁡pW(y∣x))T(33)\begin{aligned} F_s=\mathbb{E}_{x\sim\mathcal{D}}[\mathbb{E}_{y\sim p_W(x)}[(\nabla_{s(x)}\log p_W(y\mid x))\times\\ &\times\left(\nabla_s(x)\log p_W(y\mid x)\right)^{\mathsf T} \tag*{(33)} \end{aligned}
=Ex∼D((∂fW∂s(x))T[diag⁡(pW(x))−pW(x)pW(x)T]∂fW∂s(x))(34)= \mathbb{E}_{x\sim\mathcal{D}}\left(\left(\frac{\partial f_W}{\partial s}(x)\right)^{\mathsf T}\left[\operatorname{diag}(p_W(x))-p_W(x)p_W(x)^{\mathsf T}\right]\frac{\partial f_W}{\partial s}(x)\right) \tag*{(34)}

The same trick can be applied to the empirical Fisher:

Es=Ex∼D[∇sL(f)(x)∇sL(f)(x)T](35)E_s=\mathbb{E}_{x\sim\mathcal{D}}\left[\nabla_s\mathcal{L}(f)(x)\nabla_s\mathcal{L}(f)(x)^{\mathsf T}\right] \tag*{(35)}
=Ex∼D[(∂fW∂s(x))T(pW(x)−ey)(pW(x)−ey)T(∂fW∂s(x))](36)=\mathbb{E}_{x\sim\mathcal{D}}\left[\left(\frac{\partial f_W}{\partial s}(x)\right)^{\mathsf T}(p_W(x)-e_y)(p_W(x)-e_y)^{\mathsf T}\left(\frac{\partial f_W}{\partial s}(x)\right)\right] \tag*{(36)}

So all methods can be formulated as:

Cgrad=Ex∼D((∂fW∂s(x))TK(x,y)∂fW∂s(x))(37)C_{\mathrm{grad}}=\mathbb{E}_{x\sim\mathcal{D}}\left(\left(\frac{\partial f_W}{\partial s}(x)\right)^{\mathsf T}\mathcal{K}(x,y)\frac{\partial f_W}{\partial s}(x)\right) \tag*{(37)}

with:

MethodK(x,y)\mathcal{K}(x,y)
Σgrad\Sigma_{\mathrm{grad}} (L2L_2 norm)II
EsE_s (Empirical Fisher)(pW(x)−ey)(pW(x)−ey)T(p_W(x)-e_y)(p_W(x)-e_y)^{\mathsf T}
FsF_s (True Fisher)diag⁡(pW(x))−pW(x)pW(x)T\operatorname{diag}(p_W(x))-p_W(x)p_W(x)^{\mathsf T}

Table 10. Output-space kernel K(x,y)\mathcal{K}(x,y) inducing CgradC_{\mathrm{grad}}.

Computing the Curvature Matrix

The calibration set X={xn}n=1N\mathcal{X}=\{x_n\}_{n=1}^{N} consists of NN token sequences, with sequence xnx_n of length TT. The total number of token positions is Ntok≔NTN_{\mathrm{tok}}\coloneqq NT. For position tt in sequence nn, the forward map decomposes as

xn⟼an,t∈Rna↦LWsn,t∈Rns↦qfn,t∈RV(38)x_n \longmapsto a_{n,t}\in\mathbb{R}^{n_a}\xmapsto{\mathrm{L}_W}s_{n,t}\in\mathbb{R}^{n_s}\xmapsto{q}f_{n,t}\in\mathbb{R}^{V} \tag*{(38)}

where an,ta_{n,t} is the input of the layer LW:a↦WTa\mathrm{L}_W:a\mapsto W^{\mathsf T}a at position tt, sn,ts_{n,t} is its output, and qq denotes the rest of the network up to the logits fn,tf_{n,t}. We denote by hn,t∈RDh_{n,t}\in\mathbb{R}^{D} the input of the LM head, i.e. the last hidden state produced by qq before the linear projection to the vocabulary. The LM head is itself linear,

fn,t=WUhn,t,WU∈RV×D,(39)f_{n,t}=W_Uh_{n,t},\quad W_U\in\mathbb{R}^{V\times D}, \tag*{(39)}

where WUW_U denotes the LM head weight.

The matrix Σgrad∈Rns×ns\Sigma_{\mathrm{grad}}\in\mathbb{R}^{n_s\times n_s} is defined as

Σgrad≔Ex∼D((∂fW∂s(x))T∂fW∂s(x)).(40)\Sigma_{\mathrm{grad}}\coloneqq\mathbb{E}_{x\sim\mathcal{D}}\left(\left(\frac{\partial f_W}{\partial s}(x)\right)^{\mathsf T}\frac{\partial f_W}{\partial s}(x)\right). \tag*{(40)}

For sequence inputs, the expectation in Equation (40) expands as a sum over all (t,t′)(t,t') pairs with t′≥tt'\ge t, the causal constraint ∂fn,t′/∂sn,t=0\partial f_{n,t'}/\partial s_{n,t}=0 for t′<tt'<t removing the remainder:

Σgrad=1Ntok∑n=1N∑t=1T∑t′=tT(Jn,t,t′)TJn,t,t′(41)\Sigma_{\mathrm{grad}}=\frac{1}{N_{\mathrm{tok}}}\sum_{n=1}^{N}\sum_{t=1}^{T}\sum_{t'=t}^{T}(J_{n,t,t'})^{\mathsf T}J_{n,t,t'} \tag*{(41)}

with Jn,t,t′≔∂fn,t′∂sn,t∈RV×nsJ_{n,t,t'}\coloneqq\frac{\partial f_{n,t'}}{\partial s_{n,t}}\in\mathbb{R}^{V\times n_s}.

Its direct evaluation is intractable: reverse-mode differentiation delivers one row of Jn,t,t′J_{n,t,t'} per backward pass and isolates one output position t′t' per cotangent assignment, so Equation (41) costs T×VT\times V backward passes per sequence, which is 6.6×1076.6\times10^{7} for Llama 2-7B at T=2048T=2048.

This cost is the product of two independent axes: the vocabulary index, of size VV, and the output-position index, of size TT. We remove the first exactly, down to the hidden size DD, by absorbing the LM head into a Gram matrix (Appendix § 10.§ 2); we then reduce both remaining axes to a single user-chosen budget KK by two applications of one randomization principle, stated next. The resulting estimator costs KK backward passes per sequence, independently of VV and TT, and is unbiased for Equation (41).

A Rademacher Annihilation Lemma

Both axes are removed by the same mechanism: a random signed superposition, whose second moment retains the diagonal terms of a sum of Gram matrices and annihilates the cross terms in expectation.

Lemma 3. Let A1,…,Am∈Rp×qA_1,\ldots,A_m\in\mathbb{R}^{p\times q} and let ϵ1,…,ϵm\epsilon_1,\ldots,\epsilon_m be i.i.d. Rademacher variables, P[ϵi=±1]=1/2\mathbb{P}[\epsilon_i=\pm1]=1/2. Then

Eϵ[(∑i=1mϵiAi)T(∑i=1mϵiAi)]=∑i=1m(Ai)TAi.(42)\mathbb{E}_{\epsilon}\left[\left(\sum_{i=1}^{m}\epsilon_iA_i\right)^{\mathsf T}\left(\sum_{i=1}^{m}\epsilon_iA_i\right)\right]=\sum_{i=1}^{m}(A_i)^{\mathsf T}A_i. \tag*{(42)}

Proof. Expanding the product gives ∑i,i′ϵiϵi′(Ai)TAi′\sum_{i,i'}\epsilon_i\epsilon_{i'}(A_i)^{\mathsf T}A_{i'}. Since the signs are independent and centred, Eϵ[ϵiϵi′]=δii′\mathbb{E}_{\epsilon}[\epsilon_i\epsilon_{i'}]=\delta_{ii'}, which anni-

lates every i≠i′i \ne i' term and preserves the mm diagonal terms. □

The left-hand side of Equation (42) is computed from the single superposed matrix ∑iεiAi\sum_i \varepsilon_i A_i, whereas the right-hand side names mm separate matrices. Whenever the index ii is one that reverse-mode differentiation charges a backward pass for, Lemma 3 converts a cost of mm into a cost of one, at the price of a variance that we quantify in Appendix § 10.§ 3. We apply it twice: with ii ranging over the vocabulary and with ii ranging over output positions (Appendix § 10.§ 3).

Eliminating the Vocabulary Dimension

The chain rule applied to the LM head fn,t′=WUhn,t′f_{n,t'} = W_U h_{n,t'} gives

Jn,t,t′=WUHn,t,t′,(43)J_{n,t,t'} = W_U H_{n,t,t'}, \tag*{(43)}

with Hn,t,t′≔∂hn,t′∂sn,t∈RD×nsH_{n,t,t'} \coloneqq\frac{\partial h_{n,t'}}{\partial s_{n,t}} \in\mathbb{R}^{D \times n_s}, so that WUW_U factors out of every summand of Equation (41):

Σgrad=1Ntok∑n,t∑t′=tT(Hn,t,t′)TMGHn,t,t′,(44)\Sigma_{\mathrm{grad}} = \frac{1}{N_{\mathrm{tok}}} \sum_{n,t} \sum_{t'=t}^{T} (H_{n,t,t'})^{\mathsf{T}} M_G H_{n,t,t'}, \tag*{(44)}

where MG≔WUTWU∈RD×DM_G \coloneqq W_U^{\mathsf{T}}W_U \in\mathbb{R}^{D \times D}. The vocabulary dimension is absorbed once and for all into the head Gram matrix MGM_G, which is precomputed at negligible cost.

The matrix MGM_G is symmetric positive semi-definite, so it admits the spectral decomposition

MG=∑j=1Duj(uj)T,uj≔λjvj,(45)M_G = \sum_{j=1}^{D} u_j (u_j)^{\mathsf{T}}, \qquad u_j \coloneqq\sqrt{\lambda_j}v_j, \tag*{(45)}

where (λj,vj)j=1D(\lambda_j,v_j)_{j=1}^{D} are the eigenpairs of MGM_G. Substituting Equation (45) into Equation (44) gives

Σgrad=1Ntok∑n,t,t′∑j=1D(Hn,t,t′)Tuj(uj)THn,t,t′=1Ntok∑n=1N∑j=1D∑t=1T∑t′=tT(gn,t,t′,j)Tgn,t,t′,j,(46)\begin{aligned} \Sigma_{\mathrm{grad}} &= \frac{1}{N_{\mathrm{tok}}} \sum_{n,t,t'} \sum_{j=1}^{D} (H_{n,t,t'})^{\mathsf{T}} u_j (u_j)^{\mathsf{T}} H_{n,t,t'} \\ &= \frac{1}{N_{\mathrm{tok}}} \sum_{n=1}^{N} \sum_{j=1}^{D} \sum_{t=1}^{T} \sum_{t'=t}^{T} (g_{n,t,t',j})^{\mathsf{T}}g_{n,t,t',j}, \tag*{(46)} \end{aligned}

where

gn,t,t′,j≔(uj)THn,t,t′=∂((uj)Thn,t′)∂sn,t∈R1×ns(47)\begin{aligned} g_{n,t,t',j} \coloneqq(u_j)^{\mathsf{T}}H_{n,t,t'} \\ &= \frac{\partial\left((u_j)^{\mathsf{T}}h_{n,t'}\right)}{\partial s_{n,t}} \in\mathbb{R}^{1 \times n_s} \tag*{(47)} \end{aligned}

is the derivative at sn,ts_{n,t} of the scalar (uj)Thn,t′(u_j)^{\mathsf{T}}h_{n,t'}, obtained by placing the cotangent uju_j at the LM-head input of the single output position t′t'.

The reduction is exact and lowers the cost per output position from VV to DD backward passes, a factor V/D≈8V/D \approx8 for Llama 2-7B and ≈31\approx31 for Llama 3-8B. The eigenbasis (uj)j=1D(u_j)_{j=1}^{D} makes the DD required cotangent directions explicit, one backward pass each, and Equation (46) is the form from which the estimator of Appendix § 10.§ 3 is built.

The broadcast estimator

Reducing the Backward Passes. The factor DD is nonetheless the hidden size of the model, and is the dominant term in the calibration budget. We reduce it to a user-chosen K≤DK \le D by applying Lemma 3 along the vocabulary index. To this end we use, in place of the spectral decomposition Equation (45), the row decomposition of the same matrix MG=∑v=1Vwv(wv)TM_G = \sum_{v=1}^{V} w_v(w_v)^{\mathsf{T}} with wvw_v the vv-th row of WUW_U. Draw a Rademacher matrix Ψ∈{±1}V×K\Psi\in\{\pm1\}^{V \times K} and set Z≔WUTΨ∈RD×KZ \coloneqq W_U^{\mathsf{T}}\Psi\in\mathbb{R}^{D \times K}, such that

zk≔WUTψk=∑v=1VΨvkwv,(48)z_k \coloneqq W_U^{\mathsf{T}}\psi_k = \sum_{v=1}^{V}\Psi_{vk}w_v, \tag*{(48)}

with ψk\psi_k the kk-th column of Ψ\Psi. Lemma 3 applied with Av=(wv)TA_v = (w_v)^{\mathsf{T}} yields, for each column independently,

EΨ[zk(zk)T]=∑v=1Vwv(wv)T=MG,(49)\mathbb{E}_{\Psi}\left[z_k(z_k)^{\mathsf{T}}\right] = \sum_{v=1}^{V}w_v(w_v)^{\mathsf{T}} = M_G, \tag*{(49)}

hence

1KEΨ[ZZT]=MG.(50)\frac{1}{K}\mathbb{E}_{\Psi}\left[ZZ^{\mathsf{T}}\right] = M_G. \tag*{(50)}

The KK columns of ZZ replace the DD eigendirections uju_j of Equation (45) as cotangent directions.

Eliminating the Output-Position Dimension. Writing gn,t,t′,k≔(zk)THn,t,t′g_{n,t,t',k} \coloneqq(z_k)^{\mathsf{T}}H_{n,t,t'} for the projected gradient of Equation (47) along the direction zkz_k, a single backward pass differentiates a scalar that has already been summed over output positions: seeding the cotangent zkz_k at every LM-head input hn,t′h_{n,t'} returns ∑t′≥tgn,t,t′,k\sum_{t' \ge t}g_{n,t,t',k} for all input positions tt at once. Squaring this quantity aggregates the displacements of distinct output positions before taking the norm, which is not the summand of Equation (46). Isolating one output position per backward pass restores it, at a cost of TT passes per direction.

We instead weight each output position by an independent random sign. Let εn,t′(k)∈{±1}\varepsilon_{n,t'}^{(k)} \in\{\pm1\} be i.i.d. Rademacher variables over (n,t′,k)(n,t',k), drawn independently of Ψ\Psi, and consider the estimator

Σ~grad≔1NtokK∑n=1N∑k=1K∑t=1T(g~n,t,k)Tg~n,t,k(51)\widetilde{\Sigma}_{\mathrm{grad}} \coloneqq\frac{1}{N_{\mathrm{tok}}K}\sum_{n=1}^{N}\sum_{k=1}^{K}\sum_{t=1}^{T}\left(\widetilde{g}_{n,t,k}\right)^{\mathsf{T}}\widetilde{g}_{n,t,k} \tag*{(51)}

with g~n,t,k≔∑t′=tTεn,t′(k)gn,t,t′,k∈R1×ns.\widetilde{g}_{n,t,k} \coloneqq\sum_{t'=t}^{T}\varepsilon_{n,t'}^{(k)}g_{n,t,t',k} \in\mathbb{R}^{1\times n_s}.

The vector g~n,t,k\widetilde{g}_{n,t,k} is the derivative at sn,ts_{n,t} of the signed projected scalar ∑t′εn,t′(k)(zk)Thn,t′\sum_{t'}\varepsilon_{n,t'}^{(k)}(z_k)^{\mathsf{T}}h_{n,t'}, so it is still obtained for all tt in one backward pass: the signs change the cotangent placed at each position, not the number of traversals of the computation graph. Per sequence the cost is one forward pass and KK backward passes, independent of both VV and TT.

Applying Lemma 3 along the output-position index annihilates the cross-position terms t′≠t′′t'\ne t'' in expectation and preserves the diagonal, so each summand of Equation (51) is unbiased for the corresponding summand of Equation (46) along the direction zkz_k.

Proposition 4. Conditionally on the calibration set, EΨ,ε[Σ~grad]=Σgrad\mathbb{E}_{\Psi,\varepsilon}[\widetilde{\Sigma}_{\mathrm{grad}}]=\Sigma_{\mathrm{grad}} entrywise, with Σgrad\Sigma_{\mathrm{grad}} as in Equation (41). Moreover Σ~grad⪰0\widetilde{\Sigma}_{\mathrm{grad}}\succeq0 in every realization of (Ψ,ε)(\Psi,\varepsilon).

Proof. Fix (n,t)(n,t) and kk. Conditioning on Ψ\Psi and taking the expectation over ε\varepsilon first, Lemma 3 with At′=gn,t,t′,kA_{t'}=g_{n,t,t',k} gives

Eε[(g~n,t,k)Tg~n,t,k]=∑t′≥t(Hn,t,t′)TzkzkTHn,t,t′.\mathbb{E}_{\varepsilon}\left[(\widetilde{g}_{n,t,k})^{\mathsf{T}}\widetilde{g}_{n,t,k}\right]=\sum_{t'\ge t}(H_{n,t,t'})^{\mathsf{T}}z_kz_k^{\mathsf{T}}H_{n,t,t'}.

Taking the expectation over Ψ\Psi and applying Equation (50) replaces zkzkTz_kz_k^{\mathsf{T}} by MGM_G, which is Equation (44) term by term; the two randomizations commute because Ψ\Psi and ε\varepsilon are independent. Positive semidefiniteness is immediate from Equation (51), every summand being an outer product of a real row vector. □

The uniform seed does not admit the same statement. Writing Σ^grad\widehat{\Sigma}_{\mathrm{grad}} for Equation (51) with εn,t′(k)≡1\varepsilon_{n,t'}^{(k)}\equiv1, and

Sn,t′,t′′(k)≔∑t=1T((gn,t,t′,k)Tgn,t,t′′,k+(gn,t,t′′,k)Tgn,t,t′,k)∈Rns×ns(52)S_{n,t',t''}^{(k)}\coloneqq\sum_{t=1}^{T}\left((g_{n,t,t',k})^{\mathsf{T}}g_{n,t,t'',k}+(g_{n,t,t'',k})^{\mathsf{T}}g_{n,t,t',k}\right)\in\mathbb{R}^{n_s\times n_s} \tag*{(52)}

for the symmetrized contribution of the output-position pair t′<t′′t'<t'', the two estimators differ by a term that no choice of budget removes.

Proposition 5. For every realization of Ψ\Psi,

Σ^grad=Eε[Σ~grad]+1NtokK∑n=1N∑k=1K∑t′<t′′Sn,t′,t′′(k)(53)\widehat{\Sigma}_{\mathrm{grad}}=\mathbb{E}_{\varepsilon}[\widetilde{\Sigma}_{\mathrm{grad}}]+\frac{1}{N_{\mathrm{tok}}K}\sum_{n=1}^{N}\sum_{k=1}^{K}\sum_{t'<t''}S_{n,t',t''}^{(k)} \tag*{(53)}

and the second term is deterministic given the calibration set and the probes.

Proof. Setting all signs to one in Equation (51) gives (g~n,t,k)Tg~n,t,k=∑t′,t′′(gn,t,t′,k)Tgn,t,t′′,k(\widetilde{g}_{n,t,k})^{\mathsf{T}}\widetilde{g}_{n,t,k}=\sum_{t',t''}(g_{n,t,t',k})^{\mathsf{T}}g_{n,t,t'',k}. Splitting the double sum into its diagonal t′=t′′t'=t'' and its off-diagonal part, and pairing (t′,t′′)(t',t'') with (t′′,t′)(t'',t') in the latter, gives Equation (53) with the diagonal part equal to Eε[Σ~grad]\mathbb{E}_{\varepsilon}[\widetilde{\Sigma}_{\mathrm{grad}}] by Lemma 3. No sign variable appears in the remainder. □

The second term of Equation (53) is an average over the NN calibration sequences of the cross-position blocks Equation (52). It does not decrease with the number of sequences or with the number of probes, and vanishes only if those blocks cancel on average.

As shown in Table 11, the proposed estimator Σ~grad\widetilde{\Sigma}_{\mathrm{grad}} reduces the per-sequence calibration cost for Llama 2-7B from TV≈6.6×107TV\approx6.6\times10^{7} backward passes for direct evaluation to K=64K=64, a reduction of about six orders of magnitude. On a single H200 GPU, the full Complete-SVD pipeline takes about 80 minutes for both Llama 3-8B and Mistral-7B (Table 12), most of which is spent estimating Σgrad\Sigma_{\mathrm{grad}}; since Σinput\Sigma_{\mathrm{input}} and Σgrad\Sigma_{\mathrm{grad}} do not depend on the target rank, this cost is incurred once per model

ConstructionVocabulary axisBackward passes
Direct evaluation of Equation (41)VV exactTV=6.6×107TV=6.6\times10^{7}
Head-Gram spectral, Equation (46)DD exactTD=8.4×106TD=8.4\times10^{6}
Σ~grad\widetilde{\Sigma}_{\mathrm{grad}}, Equation (51)KK probesK=64K=64

Table 11. Calibration cost per sequence, in backward passes, for the constructions of Appendix § 10. Numerical column instantiated for Llama 2-7B (V=32000V=32000, D=4096D=4096, T=2048T=2048) with K=64K=64.

ModelΣinput\Sigma_{\mathrm{input}}Σgrad\Sigma_{\mathrm{grad}}SVDTotal
Llama 3-8B12 min64 min5 min∼80\sim80 min
Mistral-7B12 min61 min5 min∼80\sim80 min
CRΣgrad\Sigma_{\mathrm{grad}} (exact)Σ^grad\widehat{\Sigma}_{\mathrm{grad}}Σ~grad\widetilde{\Sigma}_{\mathrm{grad}}
0.9023.423.823.6
0.8525.626.225.9
0.8029.029.829.6

Table 12. Per-stage wall-clock cost of the Complete-SVD compression pipeline on Llama 3-8B and Mistral-7B on a single H200 GPU: the input covariance estimation (Σinput\Sigma_{\mathrm{input}}), the Gram matrix of the output Jacobians estimation (Σgrad\Sigma_{\mathrm{grad}}) and SVD computation. Reported with 256 calibration samples of length 2048.

and calibration set and is shared across all compression ratios.

Validation Against the Exact Reference

At reduced scale Equation (46) is directly computable, so the estimator can be validated end to end. Table 13 compares against this reference both Σ~grad\widetilde{\Sigma}_{\mathrm{grad}} and the broadcast construction Σ^grad\widehat{\Sigma}_{\mathrm{grad}}, obtained from Equation (51) by replacing the random signs with εn,t′(k)≡1\varepsilon_{n,t'}^{(k)} \equiv1.

As shown in Table 13, the exact reference is the best perplexity attainable under this weighting and both estimators sit above it. Relative to Σ^grad\widehat{\Sigma}_{\mathrm{grad}}, the proposed estimator Σ~grad\widetilde{\Sigma}_{\mathrm{grad}} closes the gap to the reference at each compression ratio.

Extension to a Tractable Fisher Matrix Estimator

The construction of Appendix § 10 extends to the true Fisher matrix of the model, obtained by replacing the identity weighting of Equation (41) with the curvature of the predictive distribution. Write pn,t′≔Softmax⁡(fn,t′)∈RVp_{n,t'} \coloneqq\operatorname{Softmax}(f_{n,t'}) \in\mathbb{R}^{V} for the predictive distribution at output position t′t' of sequence nn, and

Cn,t′≔diag⁡(pn,t′)−pn,t′(pn,t′)T∈RV×V(54)C_{n,t'} \coloneqq\operatorname{diag}(p_{n,t'}) - p_{n,t'}(p_{n,t'})^{\mathsf{T}} \in\mathbb{R}^{V \times V} \tag*{(54)}

for the Hessian of the negative log-likelihood with respect to the logits, evaluated at fn,t′f_{n,t'}. Taking the contexts from the calibration set and the labels from the model itself, the score of the sequence log-likelihood decomposes over output positions, and the resulting Fisher matrix at the layer pre-activation is

Fs=1Ntok∑n=1N∑t=1T∑t′=tT(Jn,t,t′)TCn,t′Jn,t,t′,(55)F_s = \frac{1}{N_{\mathrm{tok}}}\sum_{n=1}^{N}\sum_{t=1}^{T}\sum_{t'=t}^{T}(J_{n,t,t'})^{\mathsf{T}}C_{n,t'}J_{n,t,t'}, \tag*{(55)}

with Jn,t,t′J_{n,t,t'} as in Equation (41). No cross-output-position terms appear: conditionally on the context, the sampled labels at distinct positions are independent and each per-position score is centred, so the corresponding expectations vanish. Equation (55) is therefore exact, and plays for the Fisher matrix the role that Equation (41) plays for Σgrad\Sigma_{\mathrm{grad}}.

The vocabulary-elimination step of Appendix § 10.§ 2 does not transfer. Substituting Equation (43) into Equation (55) gives (Hn,t,t′)TMn,t′Hn,t,t′(H_{n,t,t'})^{\mathsf{T}}M_{n,t'}H_{n,t,t'}, with Mn,t′≔WUTCn,t′WU∈RD×DM_{n,t'} \coloneqq W_U^{\mathsf{T}}C_{n,t'}W_U \in\mathbb{R}^{D \times D}, but Mn,t′M_{n,t'} now depends on the output position, so it cannot be factored out of the sum as the constant MGM_G was. Recovering the eigenbasis construction of Equation (46) would require one spectral decomposition and DD cotangent directions per output position, hence TDTD backward passes per sequence, which is the cost Appendix § 10.§ 3 was designed to remove.

Probes matched to the predictive distribution. We instead randomize along the vocabulary index as in Equation (48), but with a probe distribution matched to Cn,t′C_{n,t'} rather than to MGM_G. Let ev∈RVe_v \in\mathbb{R}^{V} denote the vv-th canonical basis vector. The matrix Equation (54) admits the exact expansion

Cn,t′=∑v=1Vpn,t′,v(ev−pn,t′)(ev−pn,t′)T=Ey∼pn,t′[(ey−pn,t′)(ey−pn,t′)T],(56)\begin{aligned} C_{n,t'} = \sum_{v=1}^{V}p_{n,t',v}(e_v-p_{n,t'})(e_v-p_{n,t'})^{\mathsf{T}} \\ &= \mathbb{E}_{y\sim p_{n,t'}}\left[(e_y-p_{n,t'})(e_y-p_{n,t'})^{\mathsf{T}}\right], \tag*{(56)} \end{aligned}

where pn,t′,vp_{n,t',v} is the vv-th entry of pn,t′p_{n,t'} and the expectation is over a single categorical draw yy from pn,t′p_{n,t'}. The weighted sum over the vocabulary is thus a plain expectation over a term visited at its own weight, and the budget KK now indexes independent draws rather than Rademacher columns. Let yn,t′(k)∼pn,t′y_{n,t'}^{(k)} \sim p_{n,t'} be independent over (n,t′,k)(n,t',k), and define the cotangent directions

zn,t′(k)≔WUT(eyn,t′(k)−pn,t′)(57)z_{n,t'}^{(k)} \coloneqq W_U^{\mathsf{T}}\left(e_{y_{n,t'}^{(k)}}-p_{n,t'}\right) \tag*{(57)}
=wyn,t′(k)−WUTpn,t′∈RD,(58)= w_{y_{n,t'}^{(k)}}-W_U^{\mathsf{T}}p_{n,t'} \in\mathbb{R}^{D}, \tag*{(58)}

with wvw_v the vv-th row of WUW_U as in Appendix § 10.§ 3. Each direction is one row of the LM head corrected by the single vector WUTpn,t′W_U^{\mathsf{T}}p_{n,t'}, so beyond the logits themselves no object of size V×VV \times V or V×DV \times D is formed at any position, and Equation (56) gives

Ey[zn,t′(k)(zn,t′(k))T]=Mn,t′.\mathbb{E}_{y}\left[z_{n,t'}^{(k)}(z_{n,t'}^{(k)})^{\mathsf{T}}\right] = M_{n,t'}.

The estimator. Unlike the probes Equation (48), the directions Equation (57) vary with the output position, which is what allows the position-dependent curvature to be represented. Writing γn,t,t′,k≔(zn,t′(k))THn,t,t′\gamma_{n,t,t',k} \coloneqq\left(z_{n,t'}^{(k)}\right)^{\mathsf{T}} H_{n,t,t'} for the projected gradient of Equation (47) along zn,t′(k)z_{n,t'}^{(k)}, the estimator is

F~s≔1NtokK∑n=1N∑k=1K∑t=1T(γ~n,t,k)Tγ~n,t,k(59)\widetilde{F}_s \coloneqq\frac{1}{N_{\mathrm{tok}}K}\sum_{n=1}^{N}\sum_{k=1}^{K}\sum_{t=1}^{T}\left(\widetilde{\gamma}_{n,t,k}\right)^{\mathsf{T}}\widetilde{\gamma}_{n,t,k} \tag*{(59)}

with γ~n,t,k≔∑t′=tTγn,t,t′,k∈R1×ns\widetilde{\gamma}_{n,t,k} \coloneqq\sum_{t'=t}^{T}\gamma_{n,t,t',k} \in\mathbb{R}^{1 \times n_s}.

γ~n,t,k≔∑t′=tTγn,t,t′,k∈R1×ns.(60)\widetilde{\gamma}_{n,t,k} \coloneqq\sum_{t'=t}^{T}\gamma_{n,t,t',k} \in\mathbb{R}^{1 \times n_s}. \tag*{(60)}

As in Equation (51), γ~n,t,k\widetilde{\gamma}_{n,t,k} is obtained for all input positions tt in a single backward pass, by seeding the cotangent zn,t′(k)z_{n,t'}^{(k)} at the LM-head input of every output position t′t'. The probes are held fixed in this differentiation: although zn,t′(k)z_{n,t'}^{(k)} is a function of pn,t′p_{n,t'} and hence of hn,t′(k)h_{n,t'}^{(k)}, the scalar differentiated is ∑t′=1T(zn,t′(k))Thn,t′(k)\sum_{t'=1}^{T}\left(z_{n,t'}^{(k)}\right)^{\mathsf{T}}h_{n,t'}^{(k)} with each zn,t′(k)z_{n,t'}^{(k)} treated as a constant, so that Equation (57) enters through the cotangent alone and not through the computation graph.

No sign variables appear in Equation (60). The role played by Lemma 3 along the output-position index in Equation (51) is here played by the probe construction itself: the directions Equation (57) are drawn independently at each output position and satisfy Ey[zn,t′(k)]=0\mathbb{E}_{y}\left[z_{n,t'}^{(k)}\right]=0, so the cross-position terms are annihilated in expectation without an auxiliary randomization. The estimator Equation (51) requires the signs for a different reason. Its probe zkz_k is centred as well, but the same realization is seeded at every output position, so the relevant moment for t′≠t′′t'\ne t'' is EΨ[zkzkT]=MG≠0\mathbb{E}_{\Psi}\left[z_k z_k^{\mathsf{T}}\right]=M_G\ne0 and the cross terms survive; independent signs restore the cancellation that independent draws provide here for free.

Proposition 6. Conditionally on the calibration set, Ey[F~s]=Fs\mathbb{E}_{y}\left[\widetilde{F}_s\right]=F_s entrywise, with FsF_s as in Equation (55). Moreover F~s⪰0\widetilde{F}_s\succeq0 in every realization of the draws.

Proof. Fix (n,t)(n,t) and kk. Expanding the outer product in Equation (60) over two output-position indices gives

∑t′≥t∑t′′≥t(Hn,t,t′)T(zn,t′(k))Tzn,t′′(k)Hn,t,t′′.\sum_{t'\ge t}\sum_{t''\ge t}\left(H_{n,t,t'}\right)^{\mathsf{T}}\left(z_{n,t'}^{(k)}\right)^{\mathsf{T}}z_{n,t''}^{(k)}H_{n,t,t''}.

The draws being independent across output positions and centred, Ey[(zn,t′(k))Tzn,t′′(k)]\mathbb{E}_{y}\left[\left(z_{n,t'}^{(k)}\right)^{\mathsf{T}}z_{n,t''}^{(k)}\right] equals Mn,t′M_{n,t'} when t′=t′′t'=t'' and vanishes otherwise, leaving ∑t′≥t(Hn,t,t′)TMn,t′Hn,t,t′\sum_{t'\ge t}\left(H_{n,t,t'}\right)^{\mathsf{T}}M_{n,t'}H_{n,t,t'}, which is Equation (55) term by term by Equation (43). Positive semidefiniteness is immediate from Equation (60), every summand being an outer product of a real row vector. □

The cost per sequence is one forward pass and KK backward passes, independent of VV and TT, matching Table 11. The forward pass supplies both the predictive distributions entering Equation (57) and the graph traversed by the backward passes, and no spectral decomposition is required at any stage. The budget KK controls the variance alone: Proposition 6 holds for every K≥1K\ge1, so the centre is fixed and the variance decreases as O(1/K)O(1/K), in contrast to constructions that restrict Equation (56) to a fixed subset of the vocabulary and renormalize pn,t′p_{n,t'} on it, whose error is deterministic and does not vanish with the calibration budget.

Comparison with the Top-mm Construction of IO-SVD

IO-SVD targets the same output-side curvature as Equation (55) and, like the present work, pairs it with an input activation covariance under a Kronecker factorization. It differs in how the two intractable axes are removed, and both of its reductions introduce a deterministic deviation from Equation (55) rather than a variance that decreases with the calibration budget.

The output-position axis. IO-SVD defines its output-side curvature through the same-position Jacobian ∂zt/∂hℓ,t\partial z_t/\partial h_{\ell,t}, so its accumulated object is the single term t′=tt'=t of Equation (55):

1Ntok∑n=1N∑t=1T(Jn,t,t)TCn,tJn,t,t.(61)\frac{1}{N_{\mathrm{tok}}}\sum_{n=1}^{N}\sum_{t=1}^{T}\left(J_{n,t,t}\right)^{\mathsf{T}}C_{n,t}J_{n,t,t}. \tag*{(61)}

In a causal model the pre-activation sn,ts_{n,t} influences the logits at every position t′≥tt'\ge t, so a perturbation at input position tt contributes to T−t+1T-t+1 terms of Equation (55) and Equation (61) retains one of them. The omission is structural. It does not depend on the vocabulary budget mm, it is not reduced by enlarging the calibration set, and its extent grows with sequence length.

The vocabulary axis. IO-SVD restricts Equation (54) to the mm largest entries of the predictive distribution and renormalizes the probabilities on that support. Fix a sequence and an output position and suppress the indices (n,t′)(n,t'). Let SS collect the mm largest entries of pp, let pS∈Rmp_S\in\mathbb{R}^{m} be the restriction of pp to SS, let π≔1TpS\pi\coloneqq\mathbf{1}^{\mathsf{T}}p_S be its mass, and let CSC_S be the principal submatrix of CC indexed by SS. The renormalized curvature is CIO≔diag⁡(pS/π)−pSpST/π2C^{\mathrm{IO}}\coloneqq\operatorname{diag}(p_S/\pi)-p_Sp_S^{\mathsf{T}}/\pi^2, and a direct computation gives

CIO=1π(CS−1−ππpSpST).(62)C^{\mathrm{IO}} = \frac{1}{\pi}\left(C_S - \frac{1-\pi}{\pi}p_Sp_S^{\mathsf{T}}\right). \tag*{(62)}

The truncation therefore does more than set the entries outside SS to zero: it inflates the retained block by π−1\pi^{-1} and over-subtracts the rank-one term by π−2\pi^{-2}. Since π\pi varies from one output position to the next, Equation (62) reweights positions against one another rather than applying a common scale, and the traces differ, Tr⁡(CIO)=1−∥pS/π∥2\operatorname{Tr}(C^{\mathrm{IO}})=1-\lVert p_S/\pi\rVert^2 against Tr⁡(C)=1−∥p∥2\operatorname{Tr}(C)=1-\lVert p\rVert^2. Restricting Equation (56) to SS without renormalizing would instead give the partial sum ∑v∈Spv(ev−p)(ev−p)T\sum_{v\in S}p_v(e_v-p)(e_v-p)^{\mathsf{T}}, which is the head term of a stratified form of our estimator: positive semi-definite, free of the distortion Equation (62), and monotone in mm.

Both deviations are deterministic given the calibration set. They persist at any mm short of VV and at any number of calibration sequences, so the corresponding error is a bias rather than a variance. By contrast, Proposition 6 places the centre of F~s\widetilde{F}_s at Equation (55) for every K≥1K \ge1, leaving only an O(1/K)O(1/K) variance, and the two constructions coincide neither in the output positions they retain nor in the curvature they assign to those retained.

SVD Compression with KL Divergence Objective

To further validate the design of Complete-SVD, we compare it against an alternative objective based on the Kullback-Leibler (KL) divergence between the original and compressed model output distributions discussed in Appendix B.§ 4. The KL-based objective is connected to the true Fisher information matrix, which arises as the second-order expansion of the KL divergence and constitutes the principled curvature metric for probabilistic models. Furthermore, in our experiment, we use the unbiased estimator of True Fisher information matrix described in Appendix § 10.§ 5.

The configurations in Table 14 share the rank allocation and covariance regularization procedure (Appendix § 12). Compared with IO-SVD, KL-SVD yields gains that depend on the model and compression ratio. On Llama 3-8B, it lowers perplexity on all three corpora at both ratios, with larger improvements under stronger compression: at ratio 0.7, C4 perplexity decreases from 111.8 to 103.0 and PTB perplexity from 219.2 to 172.7, corresponding to reductions of 7.9% and 21.2%. On Qwen3-8B, IO-SVD is marginally better on all three corpora at ratio 0.9, whereas KL-SVD improves all three perplexities at ratio 0.7, reducing C4 and PTB perplexity by 15.0% and 31.1%, respectively. On Mistral-7B, the two methods are nearly identical in perplexity at ratio 0.9, whereas KL-SVD yields higher perplexity on all three corpora at ratio 0.7, particularly on WikiText-2 (10.3 versus 8.8). Average zero-shot accuracy is higher with KL-SVD on Mistral-7B and Llama 3-8B at both ratios, but slightly lower on Qwen3-8B. Furthermore, on IFEval (Table 2), KL-SVD improves prompt-level strict accuracy over IO-SVD by 2.8–8.2 percentage points in all but one setting; the exception is Mistral-7B at ratio 0.9, where accuracy decreases from 13.9 to 12.4.

CR MethodMistral-7BLlama 3-8BQwen3-8B
Wiki ↓C4 ↓PTB ↓Acc ↑Wiki ↓C4 ↓PTB ↓Acc ↑Wiki ↓C4 ↓PTB ↓Acc ↑
Baseline5.38.127.764.06.19.59.963.39.715.515.462.0
KFAC-SVD6.213.144.056.98.825.732.654.511.523.828.057.1
IO-SVD6.112.840.056.98.423.230.956.211.023.728.957.9
KL-SVD6.112.840.157.38.322.827.156.911.224.629.157.7
Complete-SVD6.412.739.956.98.722.423.055.811.723.423.456.7
KFAC-SVD9.334.6110.745.819.4151.3253.540.714.750.094.046.8
IO-SVD8.833.197.845.616.8111.8219.243.314.458.7120.249.2
KL-SVD10.334.798.846.616.0103.0172.745.614.249.982.848.6
Complete-SVD10.132.990.745.416.895.3123.744.215.047.567.747.8

Table 14. Output-aware methods from Table 1 (WikiText-2 calibration, uniform rank allocation), repeated here for the comparison of KL-SVD with IO-SVD. Accuracy (Acc) is the average zero-shot accuracy over 6 downstream datasets.

Remapping storage

Standard SVD compression stores the truncated factors U~k∈Rns×k\widetilde{U}_k \in\mathbb{R}^{n_s\times k} and V~k∈Rna×k\widetilde{V}_k \in\mathbb{R}^{n_a\times k} separately, with footprint k(ns+na)k(n_s+n_a) and parameter ratio α=k(ns+na)/nsna\alpha= k(n_s + n_a)/n_s n_a. This ratio saturates before kk reaches min⁡(ns,na)\min(n_s,n_a) (for ns=nan_s = n_a, α=1\alpha= 1 already at k=na/2k = n_a/2), so α\alpha is not in one-to-one correspondence with kk over (0,1](0,1]. Following Dobi-SVD Wang et al. (2025b), assuming ns≥nan_s \ge n_a, we adopt a remapping scheme that halves the precision of the overlapping na×kn_a \times k submatrices of U~k\widetilde{U}_k and V~k\widetilde{V}_k and merges them into a single ns×kn_s \times k matrix, yielding a parameter ratio α~=k/min⁡(ns,na)\widetilde{\alpha} = k/\min(n_s,n_a) in one-to-one correspondence with kk. Under remapping, we thus choose the ranks uniformly as k=⌊α~⋅min⁡(ns,na)⌋k = \lfloor\widetilde{\alpha} \cdot\min(n_s,n_a) \rfloor.

Implementation Details: Covariance Square Roots

Regularization following KFAC-SVD. We follow the covariance regularization and Cholesky factorization used in KFAC-SVD for both the input covariance and the output Jacobian-Gram covariance. For a covariance matrix M∈Rd×dM \in\mathbb{R}^{d \times d}, we first apply M←(1−α)M+α(tr⁡(M)d)IM \leftarrow(1-\alpha)M + \alpha\left(\frac{\operatorname{tr}(M)}{d}\right)I, using α=0.1\alpha= 0.1 for the input covariance and α=0.3\alpha= 0.3 for Σgrad\Sigma_{\mathrm{grad}}. We then compute its Cholesky factorization M=LL⊤M = LL^{\top}. If the Cholesky decomposition fails because of finite-precision negative eigenvalues, we compute the smallest eigenvalue smin⁡s_{\min} and shift the diagonal as M←M+(−smin⁡+10−6)IM \leftarrow M + (-s_{\min} + 10^{-6})I before retrying the factorization.

Inference Efficiency

Inference throughput. We measure the generation throughput, in tokens per second, of Llama 2-7B (Base) compressed by Complete-SVD at ratios of 40%, 60%, and 80% (Figure 3). On an NVIDIA A100 (80 GB), compression increases throughput at every ratio, with the largest relative gains at large batch sizes and short sequence lengths, consistent with the reduced parameter footprint of the linear layers. On a memory-limited NVIDIA Quadro P5000 (16 GB), which can still hold the original model, the model at 40% ratio is about 4×4 \times faster than the original at batch size 128. At batch size 256, the original model runs out of memory, whereas the models at 40% and 60% ratios still run and reach their highest throughput.

Figure 3. Inference throughput (tokens/sec) of Llama 2-7B (100%) and its Complete-SVD compressed variants at 40%, 60%, and 80% ratios. (a) Varying batch size with sequence length 32 on an A100 (80GB). (b) Varying sequence length with batch size 64 on an A100. (c) Varying batch size with sequence length 32 on a Quadro P5000 (16GB). “OOM” denotes an out-of-memory failure of the baseline.

Experiments on VLMs

ScienceQA-IMG is the image-based subset of the ScienceQA test split. It contains 2,017 multiple-choice science questions from elementary and high-school curricula, covering natural, social and language science; each question comes with an image and two to five answer options, and performance is reported as top-1 accuracy (%) of the selected option.

SEED-Bench is a human-annotated multiple-choice benchmark for multimodal comprehension. We use its image subset (SEED-Bench-IMG), whose four-option questions span nine dimensions: scene understanding, instance identity, instance attributes, instance location, instance counting, spatial relation, instance interaction, visual reasoning and text recognition; performance is reported as top-1 accuracy (%).

Complete-SVD^q Complete-SVD allocates ranks uniformly, so that every compressed projection retains the same fraction of its parameters. **Complete-SVD**^q keeps the Complete-SVD factorization unchanged and replaces this uniform allocation with the sensitivity-aware rank allocator of QSVD, which separates the contribution of the factorization from that of the rank allocation. For a projection W∈Rm×nW \in\mathbb{R}^{m \times n}, let Σgrad12WΣinput12=∑iσiuivi⊤\Sigma_{\mathrm{grad}}^{\frac{1}{2}} W \Sigma_{\mathrm{input}}^{\frac{1}{2}} = \sum_i \sigma_i u_i v_i^{\top} denote the singular value decomposition of the whitened weight, so that W=∑iσiΣgrad−12uivi⊤Σinput−12W = \sum_i \sigma_i \Sigma_{\mathrm{grad}}^{-\frac{1}{2}}u_i v_i^{\top}\Sigma_{\mathrm{input}}^{-\frac{1}{2}}. Each component is scored by Ii=σi(1N∑n=1Ngn,i2)12I_i = \sigma_i\left(\frac{1}{N}\sum_{n=1}^{N}g_{n,i}^{2}\right)^{\frac{1}{2}}, where

gn,i=∂Ln∂σi=⟨∇WLn,Σgrad−12uiviTΣinput−12⟩(63)g_{n,i} = \frac{\partial\mathcal{L}_n}{\partial\sigma_i} = \left\langle\nabla_W \mathcal{L}_n, \Sigma_{\mathrm{grad}}^{-\frac{1}{2}} u_i v_i^{T} \Sigma_{\mathrm{input}}^{-\frac{1}{2}} \right\rangle \tag*{(63)}

is the derivative of the cross-entropy of the nn-th calibration sample with respect to σi\sigma_i, obtained with the gradient pass of QSVD on the same N=256N = 256 calibration samples used to estimate Σinput\Sigma_{\mathrm{input}} and Σgrad\Sigma_{\mathrm{grad}}. As in QSVD, the components of all query, key and value projections are ranked jointly and the kk highest-scoring ones are retained, regardless of their position in the spectrum. Following the parameter-matched budget of WSVD, k=⌊ρ∑ℓmℓnℓmℓ+nℓ⌋k = \left\lfloor\rho\sum_{\ell} \frac{m_{\ell} n_{\ell}}{m_{\ell} + n_{\ell}} \right\rfloor, so that the factorized projections hold a fraction ρ\rho of the original parameters. Let I⊂{1,…,min⁡(m,n)}\mathcal{I} \subset\{1, \ldots, \min(m,n)\} denote the indices of the components retained for a projection, which need not be the leading ones, and let UIU_{\mathcal{I}}, VIV_{\mathcal{I}} and σI\boldsymbol{\sigma}_{\mathcal{I}} collect the corresponding singular vectors and values. The factors are then A=diag⁡(σI)12VITΣinput−12A = \operatorname{diag}(\boldsymbol{\sigma}_{\mathcal{I}})^{\frac{1}{2}} V_{\mathcal{I}}^{T} \Sigma_{\mathrm{input}}^{-\frac{1}{2}} and B=Σgrad−12UIdiag⁡(σI)12B = \Sigma_{\mathrm{grad}}^{-\frac{1}{2}} U_{\mathcal{I}} \operatorname{diag}(\boldsymbol{\sigma}_{\mathcal{I}})^{\frac{1}{2}}, with rank ∣I∣|\mathcal{I}|; uniform Complete-SVD corresponds to I={1,…,r}\mathcal{I} = \{1, \ldots, r\}.

ModelMethodScienceQA-IMG ↑ 90%80%70%60%50%SEED-Bench ↑ 90%80%70%60%50%Avg. ↑
LLaVA1.5 7BASVD49.9350.1247.1036.6919.1954.2753.5348.3537.1724.1742.05
LLaVA1.5 7BSVD-LLM65.4463.7161.9257.4155.5357.8957.5055.3354.6455.3158.47
LLaVA1.5 7BQSVD67.7268.2267.0865.0562.3759.8459.0759.7859.0058.2362.64
LLaVA1.5 7BWSVD68.1767.7267.2865.8965.4960.1060.1759.8960.1860.4663.54
LLaVA1.5 7BIO-SVD67.9268.0768.1766.7365.9960.2960.3260.2260.1560.6463.85
LLaVA1.5 7BComplete-SVD66.4866.5866.8365.7966.3958.9858.5158.0956.1956.7262.06
LLaVA1.5 7BComplete-SVDQ^{Q}67.2866.6866.4466.1465.3460.0559.8759.8459.4259.2163.03
LLaVA1.5 7BFP16Accuracy: 68.01Accuracy: 60.1864.10
LLaVA1.5 13BASVD71.3971.5970.0070.2569.5161.9261.9161.5461.5160.7166.03
LLaVA1.5 13BSVD-LLM71.0570.8570.3070.3570.3062.2862.3462.2562.0863.0166.48
LLaVA1.5 13BQSVD71.8971.9971.4971.5471.3962.6162.6462.8262.6362.5267.15
LLaVA1.5 13BWSVD71.9971.8472.5371.5971.4462.5262.6862.3862.3762.3767.17
LLaVA1.5 13BIO-SVD71.7972.1472.2972.1472.6862.6462.4962.4662.4562.2967.34
LLaVA1.5 13BComplete-SVD71.1571.9971.8972.0971.5962.5862.6562.4462.5462.3067.12
LLaVA1.5 13BComplete-SVDQ^{Q}71.4971.5971.7971.9471.4462.4162.5362.4462.4462.5467.06
LLaVA1.5 13BFP16Accuracy: 71.83Accuracy: 62.5367.18
SmolVLM 2BASVD29.303.970.2017.851.500.958.96
SmolVLM 2BSVD-LLM40.0617.203.8232.4915.894.6019.01
SmolVLM 2BQSVD77.0062.7742.5964.8050.4636.2455.64
SmolVLM 2BWSVD76.3071.7460.9365.7863.2954.4565.42
SmolVLM 2BIO-SVD83.3482.6582.3068.3668.6568.1575.58
SmolVLM 2BComplete-SVD79.3376.5573.7766.6765.6964.6371.11
SmolVLM 2BComplete-SVDQ^{Q}83.1483.5982.5568.6468.3968.0275.72
SmolVLM 2BFP16Accuracy: 84.53Accuracy: 68.4776.53

Table 15. Accuracy evaluation of different methods under FP16.

References

  1. [1]Viktoriia A. Chekalina, Daniil Moskovsky, Tatyana Matveeva, Andrey Kuznetsov, and Evgeny Frolov. 2026. Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. In Proceedings of the 43rd International Conference on Machine Learning, pages 13381–13399.
  2. [2]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv: 1803.05457 [cs].arxiv.org/abs/1803.05457
  3. [3]Felix Dangel, Bálint Mucsányi, Tobias Weber, and Runa Eschenhagen. 2025. Kronecker-Factored Approximate Curvature (KFAC) From Scratch. arXiv: 2507.05127 [cs].arxiv.org/abs/2507.05127
  4. [4]Runa Eschenhagen, Alexander Immer, Richard E. Turner, Frank Schneider, and Philipp Hennig. 2023. Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures. In Advances in Neural Information Processing Systems.
  5. [5]Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. In Proceedings of the 40th International Conference on Machine Learning, pages 10323–10337.arxiv.org/abs/2301.00774
  6. [6]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate Quantization for Generative Pre-Trained Transformers. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2210.17323
  7. [7]Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. arXiv: 2407.21783 [cs].arxiv.org/abs/2407.21783
  8. [8]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv: 1503.02531 [stat].arxiv.org/abs/1503.02531
  9. [9]Yen-Chang Hsu, Ting Hua, Sung-En Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022. Language Model Compression with Weighted Low-Rank Factorization. In The Tenth International Conference on Learning Representations.arxiv.org/abs/2207.00112
  10. [10]Q. Huangfu and J. A. J. Hall. 2018. Parallelizing the Dual Revised Simplex Method. Mathematical Programming 10(1):119–142.arxiv.org/abs/1503.01889
  11. [11]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv: 2310.06825 [cs.CL].arxiv.org/abs/2310.06825
  12. [12]Alper Kalle, Théo Rudkiewicz, Mohamed Ouerfelli, and Mohamed Tamaazousti. 2025. Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks. In Advances in Neural Information Processing Systems.arxiv.org/abs/2511.04494

Paper details

vlnc:W_2p4yw6wy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents