WorksComputer Science
Complete-SVD: A Functional Norm Approach to Low-Rank LLM Compression
Abstract
Post-training low-rank compression of neural networks must not only approximate the original weights but also preserve the model's input-output behavior. Recent output-aware methods measure the compression error beyond the compressed layer, at the model output, weighting logit changes by the observed next token or by the model's predicted probabilities. We propose Complete-SVD, a method for large language models that minimizes the distance between the logits of the original and compressed models, weighting all logit changes equally. Linearizing the network around the original weights, together with a Kronecker-factored approximation, reduces this objective to a weighted SVD with a closed-form solution. Its output-side factor is intractable to compute; we estimate it without bias from random probes, at a cost of backward passes per sequence, independent of vocabulary size and sequence length. At matched settings with uniform ranks and WikiText-2 calibration, Complete-SVD achieves the lowest C4 and PTB perplexity in every setting, and at ratio 0.7 under LEMS rank allocation. Replacing the distance by the Kullback-Leibler divergence (KL-SVD), with the same estimation framework, lowers calibration perplexity in most settings and raises downstream accuracy, but raises cross-corpus perplexity. Complete-SVD also improves IFEval instruction-following accuracy over the strongest prior output-aware methods throughout.
Introduction and Related Work
Large language models have driven rapid progress across reasoning and generation tasks ((Kaplan et al., 2020)), but their growing parameter counts make inference prohibitively expensive. Model compression reduces this cost through quantization ([6] (Lin et al. 2024; Xiao et al. 2023)), pruning ([5] (Ma et al. 2023; Sun et al. 2024)), knowledge distillation ([8]), or low-rank factorization, which replaces each weight matrix by the product of two smaller matrices. Low-rank factorization retains dense operations that run efficiently on standard hardware. Post-training low-rank methods differ in their per-layer objective, i.e., the criterion minimized by the low-rank approximation of each layer; rank allocation, storage schemes ((Wang et al., 2025b)), and cross-layer basis sharing ((Wang et al., 2025a)) are complementary to it. This work concerns the objective, that is, which change in the model the approximation of a layer should penalize.
Error at the compressed layer. Let be the weight matrix of a linear layer and its approximation of rank at most . Truncated singular value decomposition (SVD) returns the closest to in Frobenius norm, which treats all weight directions equally, and replacing the weights of an LLM by these approximations severely degrades its performance. Activation-aware methods instead minimize the error on the layer outputs, , where the rows of are input activations collected on a small calibration set of text sequences. ASVD ((Yuan et al., 2023)) rescales by per-channel activation magnitudes, and SVD-LLM ((Wang et al., 2025c)) solves this problem in closed form by whitening with before LoRA fine-tuning. For tensor decompositions of convolutional networks, Kalle et al. (Kalle et al. [12]) cast this principle as a functional norm: the distance, over the data distribution, between the original and compressed networks up to the compressed layer. This norm involves only the upstream network, i.e., the preceding layers, and reduces to the activation-aware objective for a linear layer. The error at the layer output is not, however, the error of the model: it propagates through the downstream network, i.e., the following layers, which amplifies it along some directions and attenuates it along others, and, in a causal transformer, carries it to the predictions at all subsequent positions.
Error at the model output. Output-aware methods measure the error at the model output. Each compares the outputs of the original and compressed models through a discrepancy, whose local quadratic approximation weights a change of the layer output by its effect on the logits and by the importance that the discrepancy assigns to each logit change (Figure 1, panel b). A first family preserves the training loss through the empirical Fisher matrix, built from loss gradients at the observed next token. FWSVD [9] uses a diagonal approximation, GFWSVD [1] a Kronecker factorization fitted from a limited number of sequence-level gradients, and KFAC-SVD (Thoma et al., 2026) Kronecker factors estimated from token-level statistics, which alleviate the rank deficiency of GFWSVD. For each token, the empirical Fisher only penalizes the change in the log-probability of the observed next token, a single direction among all possible logit changes (Kunstner et al., 2019). A second family preserves the predictive distribution through the Kullback–Leibler (KL) divergence, whose local quadratic approximation is governed by the true Fisher information (Amari, 1998). IO-SVD (Abbasi et al., 2026) compresses LLMs with this objective. IO-SVD approximates the true Fisher matrix through two reductions: it retains only the effect of a perturbation at a given position on the prediction at that same position, and it restricts the predictive distribution to its most probable tokens, renormalized. Both reductions are deterministic, so the resulting bias does not vanish as the calibration set grows (Appendix § 10.5.1).
Figure 1. Output geometry changes the compression tradeoff. (a) The output-aware methods can be written in this Kronecker-factored form and differ mainly in the output kernel : , and for Complete-SVD. (b) Illustrative next-token example. Fisher-based objectives compare , whereas Complete-SVD compares the logits ; local output costs hold to leading order in . Link width is the weight of each token in this cost, and black ticks mark original values. Bars share a logarithmic scale (); compressed values are schematic. (c) Perplexity reduction of Complete-SVD relative to KL-SVD at compression ratios (CR) 0.9 and 0.7, with uniform ranks. WikiText-2 is the calibration domain.
Complete-SVD. Our objective is the functional distance between the original and compressed networks, i.e., the mean squared Euclidean distance between their logits over the data distribution. Every logit change is thus penalized with equal weight, whereas the Fisher objectives weight each logit change by the observed next token or by the probabilities that the model predicts on the calibration contexts (Figure 1, panel b). We find that the uniform weighting better preserves perplexity on corpora other than the calibration one (Figure 1, panel c). A Kronecker-factored approximation (K-FAC) ([4] (Martens and Grosse 2015)) of the objective reduces the objective to a weighted SVD with a closed-form solution. Its two weights are the input covariance of activation-aware methods and the Gram matrix of the logit Jacobians, which measures how strongly each direction of the layer output affects the logits.
Tractable estimation. The choice of objective also determines the cost of the output-side weight. The empirical Fisher is inexpensive because it involves only the loss, a single scalar whose gradient takes one backward pass per sequence. The KL and objectives involve every logit at every position, and computing exactly takes one backward pass per vocabulary entry and per position, tens of millions per sequence for Llama 2-7B. The linearity of the LM head absorbs the vocabulary exactly, without any estimation, into the Gram matrix of the LM-head weights, with the hidden dimension: the number of backward passes per output position drops from to , a factor of about 31 for Llama 3-8B (, ), and the resulting expression of is still exact. Random probes with random signs on the output positions then yield an unbiased estimator of costing backward passes per sequence, independently of the vocabulary size and the sequence length. With probes drawn from the predictive distribution, the same construction gives an unbiased estimator of the true Fisher, which yields KL-SVD, the KL objective of IO-SVD without its reductions (see Appendix § 10.5.1 for details). Comparing KL-SVD with Complete-SVD then separates the choice of objective from the approximation used to compute it.
Our contributions are the following.
Complete functional norm. We extend the functional-norm objective of Kalle et al. [12] from the upstream network to the full model, and reduce it to a closed-form weighted SVD that penalizes all logit changes equally (§ 2.2).
An unbiased estimator of whose cost is backward passes per sequence, independently of the vocabulary size and the sequence length, validated against the exact matrix at reduced scale (Section 2.3). Drawing the probes from the predictive distribution, the same construction yields an unbiased estimator of the true Fisher matrix at the same cost. Substituting it for gives KL-SVD, the KL objective of IO-SVD without its reductions, which we show to be deterministic deviations from that persist for any calibration budget (Appendix § 10.§ 5). KL-SVD shares every other component with Complete-SVD and therefore isolates the effect of the objective from that of its estimation.
Empirical evaluation. On Mistral-7B, Llama 3-8B and Qwen3-8B calibrated on WikiText-2 with uniform ranks, Complete-SVD achieves the lowest C4 and PTB perplexity among all compared methods at both tested compression ratios (0.9 and 0.7). On Llama 3-8B at ratio 0.9, it reduces PTB perplexity from 30.9 with IO-SVD, the strongest prior baseline, to 23.0. Without fine-tuning and combined with GPTQ, Complete-SVD compresses Llama 2-7B from 13.5 GB to 1.9 GB at a WikiText-2 perplexity of 16.7 (§ 3).
Method
Problem Statement
We suppose that we are in a supervised learning setting, where we have a data distribution on and a per-sample loss function (for classification this is typically the cross-entropy loss and with the number of classes). The goal of supervised learning is to find a function such that the expected loss is small. We suppose that we already have such a function that has low expected loss. We can decompose this function as a pre-computation , a fully-connected layer with weights () and a post-computation such that :
with , , , . We aim to replace the fully-connected layer with two fully-connected layers and such that with the constraint that the intermediate dimension is smaller than a specific value and the expected loss of this new function is close to the expected loss of . We note the effective weight of the new function. In practice, we aim to solve:
Norm Optimization
We minimize the functional distance ; alternative objectives (KL, empirical Fisher) are derived in Appendix B. Since optimizing this distance directly is intractable, we linearize around .
Theorem 1. With the notation introduced above, let denote the input covariance and the Gram matrix of the output Jacobians at the compressed layer. Then:
(i) Exact first-order expansion. Let . Then
(ii) Kronecker-factored approximation. Under the K-FAC structural approximation , the right-hand side of Equation (3) admits the closed-form approximation
Proof Sketch. Linearizing around gives with . Approximating the Gram matrix via K-FAC (Dangel et al., [3]) and leveraging the Kronecker property yields the weighted Frobenius norm. See Appendix A for the full derivation. This optimization problem is solvable efficiently in closed-form via SVD. See Appendix § 12 for the computation of and .
Tractable Computation of
Theorem 1 reduces compression to the closed-form weighted Frobenius norm, but the computation of output-Jacobian Gram matrix requires one backward pass per output dimension, i.e., backward passes per token which is intractable for – in modern LLMs.
LM-head factorization. Let be the predicted logits with the LM-head weight and the last hidden state. Using the chain rule, is a sum of terms, where is the sensitivity of the hidden state at output position to position . The vocabulary dimension is absorbed once into the precomputed Gram matrix , reducing the number of cotangent directions from to , a factor for Llama 2-7B and for Llama 3-8B.
Broadcast backward. The remaining factor is the hidden size, and still dominates the calibration budget. Drawing a Rademacher matrix and setting gives cotangent directions with . Each direction is broadcast to every position of the LM-head input, weighted by an independent random sign , and propagated in a single backward pass: the broadcast accumulates the contribution of all positions in one traversal of the computation graph, while the signs annihilate the cross-position terms in expectation.
Together these give the broadcast estimator , costing one forward pass and backward passes per sequence, independent of both and , and unbiased for conditionally on the calibration set (Proposition 4). Against the exact , computable at reduced scale, recovers most of the compression quality (Table 13).
KL-SVD. With probes drawn from the predictive distribution instead of Rademacher columns, the same construction yields an unbiased estimator, at the same cost, of the true Fisher matrix , where , of the KL divergence between the predictive distributions (Appendix § 10.§ 5). Substituting it for yields KL-SVD, all other components are shared with Complete-SVD (Figure 1).
Experiments
We compare our method, Complete-SVD, against state-of-the-art SVD-based compression methods, including standard SVD, ASVD Yuan et al., 2023, FWSVD Hsu et al., [9], GFWSVD Chekalina et al., [1], SVD-LLM Wang et al., 2025c, Dobi-SVD Wang et al., 2025b, KFAC-SVD Thoma et al., 2026, and IO-SVD Abbasi et al., 2026 as well as structured pruning methods such as LLM-Pruner Ma et al., 2023 and Wanda-SP Sun et al., 2024. We further demonstrate that Complete-SVD is compatible with state-of-the-art quantization methods by combining it with GPTQ Frantar et al., [6]. To validate our approach, we conduct experiments on Llama 2-7B Touvron et al., 2023 Base and Chat, Llama 3-8B Grattafiori et al., [7], Mistral-7B Jiang et al., [11] and Qwen3-8B Yang et al., 2025, evaluating on the language modeling datasets WikiText-2 Merity et al., 2017, C4 Raffel et al., 2020, and PTB Marcus et al., 1993, the downstream benchmarks OpenbookQA Mihaylov et al., 2018, WinoGrande Sakaguchi et al., 2021, HellaSwag Zellers et al., 2019, PIQA Bisk et al., 2020, ARC-c, and ARC-e Clark et al., [2], and the generation dataset IFEval Zhou et al., 2023. The covariance matrices and are computed from 256 randomly sampled sequences of length 2048 from WikiText-2 dataset (See Appendix § 10.§ 3 for the calibration cost). Each experiment runs on a single GPU (NVIDIA H100 or H200), except the inference-efficiency measurements (Appendix § 13). The compression ratio (CR) is the parameter count of the compressed layers divided by that of the original layers.
LEMS Rank allocation. To test whether Complete-SVD composes with a published rank allocator, we use LEMS Thoma et al., 2026 with the authors’ procedure: one KL probe per layer, with that layer reduced to 10% of its parameters, on 32 calibration windows of WikiText-2; the harmonic cross-layer term; ranks in multiples of 8; and an exact integer-program solve (HiGHS) Huangfu and Hall, [10] at the target budget. Ranks are allocated under each method’s own whitening, so every method receives the allocation LEMS finds for it, and all use KFAC-SVD’s damping. We fix the depth-prior weight to for every method and report both; the LEMS authors instead tune hyperparameters per cell, which gives stronger KFAC-SVD results at ratio 0.7 than our fixed- runs. The budget is 90% / 70% of the parameters of all attention and MLP projections; no layer is left dense.
Comparison to State-of-the-art Low-Rank SVD Methods
We evaluate Complete-SVD against state-of-the-art SVD-based baselines on Mistral-7B, Llama 3-8B and Qwen3-8B at compression ratios of 0.9 and 0.7 (Table 1). For a fair comparison, KFAC-SVD, IO-SVD, KL-SVD and Complete-SVD share the same uniform rank allocation and covariance regularization procedure (Appendix § 12). On the calibration corpus (WikiText-2), Complete-SVD remains within 1.3 perplexity points of the stronger of KFAC-SVD and IO-SVD in every configuration. The consistent advantage appears off the calibration distribution: Complete-SVD achieves the lowest C4 and PTB perplexity in all six configurations. On Llama 3-8B and Qwen3-8B, it reduces C4 perplexity relative to the stronger of these two baselines by 1–3% at ratio 0.9 and 5–15% at ratio 0.7, and PTB perplexity by 16–44%. Average zero-shot accuracy remains comparable, ranging from 1.4 percentage points below to 0.9 point above the stronger baseline. Under generative evaluation (Table 2), Complete-SVD outperforms both KFAC-SVD and IO-SVD on all three models at ratios 0.9 and 0.8. To verify that the perplexity gains are not specific to WikiText-2 calibration, Table 4 repeats the comparison on Llama 3-8B with C4 as the calibration set. Complete-SVD slightly improves on the stronger baseline on C4 at ratio 0.9 and reduces its perplexity by 20% at ratio 0.7, while reducing perplexity by 13–26% on WikiText-2 and 27–43% on PTB.
| CR | Method | Mistral-7B Wiki↓ | Mistral-7B C4↓ | Mistral-7B PTB↓ | Mistral-7B Acc↑ | Llama 3-8B Wiki↓ | Llama 3-8B C4↓ | Llama 3-8B PTB↓ | Llama 3-8B Acc↑ | Qwen3-8B Wiki↓ | Qwen3-8B C4↓ | Qwen3-8B PTB↓ | Qwen3-8B Acc↑ |
| Baseline | 5.3 | 8.1 | 27.7 | 64.0 | 6.1 | 9.5 | 9.9 | 63.3 | 9.7 | 15.5 | 15.4 | 62.0 | |
| FWSVD | 9.5 | 13.8 | 58.4 | 56.9 | 42.1 | 60.0 | 69.8 | 48.9 | 17.0 | 24.8 | 31.5 | 53.8 | |
| ASVD | 9.1 | 13.4 | 49.1 | 57.3 | 65.1 | 77.6 | 207.7 | 47.9 | 20.4 | 30.0 | 36.4 | 51.2 | |
| SVD-LLM | 6.5 | 15.1 | 66.1 | 56.8 | 10.1 | 39.6 | 49.9 | 52.3 | 12.5 | 27.4 | 36.7 | 56.0 | |
| DOBI-SVD | 7.1 | 15.2 | 63.6 | 55.2 | 11.2 | 34.8 | 42.4 | 53.0 | 13.5 | 28.9 | 35.8 | 55.5 | |
| 0.9 | GFWSVD | 31.7 | 65.1 | 305.7 | 40.7 | 2570 | 3828 | 8492 | 33.4 | 171.5 | 350.5 | 765.6 | 46.7 |
| KFAC-SVD | 6.2 | 13.1 | 44.0 | 56.9 | 8.8 | 25.7 | 32.6 | 54.5 | 11.5 | 23.8 | 28.0 | 57.1 | |
| IO-SVD | 6.1 | 12.8 | 40.0 | 56.9 | 8.4 | 23.2 | 30.9 | 56.2 | 11.0 | 23.7 | 28.9 | 57.9 | |
| KL-SVD | 6.1 | 12.8 | 40.1 | 57.3 | 8.3 | 22.8 | 27.1 | 56.9 | 11.2 | 24.6 | 29.1 | 57.7 | |
| Complete-SVD | 6.4 | 12.7 | 39.9 | 56.9 | 8.7 | 22.4 | 23.0 | 55.8 | 11.7 | 23.4 | 23.4 | 56.7 | |
| FWSVD | 34.8 | 50.3 | 215.1 | 44.2 | 716.4 | 837.5 | 1940 | 33.4 | 41.4 | 68.8 | 90.7 | 42.7 | |
| ASVD | 28.2 | 42.9 | 163.6 | 46.1 | 10989 | 9400 | 36033 | 33.0 | 70.7 | 97.0 | 131.5 | 43.1 | |
| SVD-LLM | 11.0 | 42.6 | 169.5 | 44.4 | 34.6 | 376.0 | 600.9 | 37.7 | 17.1 | 63.1 | 136.2 | 45.8 | |
| DOBI-SVD | 12.4 | 39.7 | 170.8 | 42.9 | 31.5 | 500.1 | 223.7 | 38.5 | 18.5 | 63.8 | 94.3 | 46.2 | |
| 0.7 | GFWSVD | 2550 | 3404 | 5356 | 32.3 | 41799 | 53671 | 72222 | 31.8 | 21685 | 27627 | 50815 | 34.1 |
| KFAC-SVD | 9.3 | 34.6 | 110.7 | 45.8 | 19.4 | 151.3 | 253.5 | 40.7 | 14.7 | 50.0 | 94.0 | 46.8 | |
| IO-SVD | 8.8 | 33.1 | 97.8 | 45.6 | 16.8 | 111.8 | 219.2 | 43.3 | 14.4 | 58.7 | 120.2 | 49.2 | |
| KL-SVD | 10.3 | 34.7 | 98.8 | 46.6 | 16.0 | 103.0 | 172.7 | 45.6 | 14.2 | 49.9 | 82.8 | 48.6 | |
| Complete-SVD | 10.1 | 32.9 | 90.7 | 45.4 | 16.8 | 95.3 | 123.7 | 44.2 | 15.0 | 47.5 | 67.7 | 47.8 |
Table 1. Compression results on Mistral-7B, Llama 3-8B and Qwen3-8B with uniform rank allocation. Results for IO-SVD are reproduced by us; all other baselines are reported from Thoma et al. (2026). Accuracy (Acc) reported as average zero-shot accuracy value on 6 downstream datasets.
| CR | Method | Mistral 7B | Qwen3 8B | Llama 2 7B | |||||||||
| Baseline | 16.8 | 24.4 | 30.3 | ||||||||||
| 0.9 | KFAC-SVD | 12.4 | 17.2 | 20.2 | |||||||||
| IO-SVD | 13.9 | 12.2 | 22.4 | ||||||||||
| KL-SVD | 12.4 | 17.7 | 27.7 | ||||||||||
| Complete-SVD | 17.4 | 21.3 | 26.6 | ||||||||||
| 0.8 | KFAC-SVD | 11.8 | 14.6 | 22.2 | |||||||||
| IO-SVD | 7.4 | 10.5 | 15.3 | ||||||||||
| KL-SVD | 11.3 | 13.3 | 23.5 | ||||||||||
| Complete-SVD | 12.9 | 18.1 | 25.3 | ||||||||||
| CR Method | Mistral-7B Wiki | C4 | PTB | Acc | Llama 3-8B Wiki | C4 | PTB | Acc | Qwen3-8B Wiki | C4 | PTB | Acc | |
| Baseline | - | 5.3 | 8.1 | 27.7 | 64.0 | 6.1 | 9.5 | 9.9 | 63.3 | 9.7 | 15.5 | 15.4 | 62.0 |
| KFAC-SVD | 5.4 | 8.6 | 26.8 | 62.3 | 6.7 | 11.6 | 11.5 | 60.5 | 9.9 | 16.2 | 15.9 | 61.9 | |
| KL-SVD | 0 | 5.4 | 8.5 | 26.5 | 62.4 | 6.6 | 11.4 | 11.2 | 61.0 | 9.9 | 16.2 | 16.0 | 61.8 |
| Complete-SVD | 5.4 | 8.6 | 27.0 | 62.2 | 6.6 | 11.4 | 11.1 | 61.0 | 9.9 | 16.2 | 16.0 | 61.8 | |
| KFAC-SVD | 5.5 | 8.8 | 29.3 | 62.3 | 7.1 | 12.6 | 12.9 | 62.0 | 10.1 | 16.7 | 16.6 | 61.1 | |
| KL-SVD | 5 | 5.4 | 8.7 | 27.2 | 62.6 | 7.0 | 12.3 | 12.0 | 62.1 | 10.1 | 16.7 | 16.8 | 61.3 |
| Complete-SVD | 5.4 | 8.7 | 27.4 | 62.3 | 7.0 | 12.2 | 11.6 | 61.3 | 10.0 | 16.6 | 16.5 | 60.9 | |
| KFAC-SVD | 8.5 | 30.3 | 100.9 | 47.6 | 15.6 | 82.8 | 162.5 | 45.8 | 12.2 | 33.7 | 48.0 | 52.1 | |
| KL-SVD | 0 | 8.2 | 28.8 | 83.7 | 48.3 | 13.1 | 65.0 | 117.5 | 47.3 | 12.0 | 33.2 | 50.3 | 53.5 |
| Complete-SVD | 8.7 | 27.9 | 74.2 | 46.8 | 13.4 | 60.9 | 77.3 | 46.0 | 11.9 | 31.1 | 36.3 | 52.8 | |
| KFAC-SVD | 7.6 | 20.1 | 65.2 | 52.0 | 11.8 | 38.1 | 88.2 | 50.0 | 13.0 | 34.8 | 66.6 | 52.8 | |
| KL-SVD | 5 | 7.3 | 18.4 | 56.5 | 53.1 | 11.0 | 34.2 | 73.1 | 51.5 | 13.4 | 35.7 | 69.0 | 52.8 |
| Complete-SVD | 7.5 | 18.0 | 51.6 | 51.6 | 11.1 | 32.5 | 50.8 | 50.6 | 13.2 | 33.2 | 47.7 | 51.8 |
Table 2. IFEval (0-shot, prompt-level strict accuracy, without applying a chat template) under low-rank compression at compression ratios (CR) 0.9 and 0.8. Llama 2-7B is the Chat variant.
| CR Method | Llama 3-8B Wiki | C4 | PTB | Acc |
| Baseline | 6.1 | 9.5 | 9.9 | 63.3 |
| KFAC-SVD | 17.7 | 15.9 | 23.6 | 55.9 |
| IO-SVD | 14.7 | 16.1 | 25.8 | 56.8 |
| KL-SVD | 15.2 | 15.5 | 18.3 | 57.6 |
| Complete-SVD | 12.8 | 15.7 | 17.3 | 56.8 |
| KFAC-SVD | 89.0 | 47.3 | 158.6 | 43.6 |
| IO-SVD | 69.5 | 45.5 | 169.0 | 44.9 |
| KL-SVD | 60.1 | 36.7 | 108.5 | 46.4 |
| Complete-SVD | 51.2 | 36.2 | 89.9 | 45.7 |
Table 4. Compression results of output-aware methods on Llama 3-8B (WikiText-2, C4, and PTB perplexity, zero-shot accuracy) when C4 dataset is used for calibration with 256 samples and 2048 token sequence.
Comparison with KL-SVD. Comparing Complete-SVD with KL-SVD further isolates the effect of the compression objective, as both methods also share the probe-based estimation framework. Relative to KL-SVD, Complete-SVD has 4–6% higher WikiText-2 perplexity in five of the six configurations and 0.4–1.4 percentage points lower average zero-shot accuracy in all six. Nevertheless, Complete-SVD achieves lower C4 and PTB perplexity in every configuration, with reductions of up to 7% and 28%, respectively. These results suggest that the Complete-SVD objective better preserves performance beyond the calibration corpus, at the cost of slightly higher calibration perplexity and lower zero-shot accuracy. With C4 calibration (Table 4) on Llama 3-8B, KL-SVD attains slightly lower C4 perplexity at ratio 0.9 (15.5 against 15.7), but Complete-SVD reduces WikiText-2 perplexity by 16% and PTB perplexity by 5%. At ratio 0.7, Complete-SVD is lower on all three corpora, by 15% on WikiText-2, 1% on C4 and 17% on PTB. As with WikiText-2 calibration, KL-SVD attains higher average zero-shot accuracy at both ratios.
Sensitivity-aware rank allocation. Table 3 reports the output-aware methods under LEMS. The allocator decides how many directions each layer keeps, the objective which ones. At ratio 0.9, allocation alone brings every method close to the uncompressed model (Llama 3-8B C4: 11.4 against 9.5, versus 22.4 with uniform ranks), leaving little for the objective to change: Complete-SVD and KL-SVD differ by at most 0.5 perplexity points. At ratio 0.7, the objective matters again: Complete-SVD achieves the lowest C4 and PTB perplexity on all three models and for both values of , and its PTB reduction over the strongest competitor (9–34%) is at least as large as with uniform ranks (7–28%). The ordering holds for both although changes perplexity by up to a factor of two, so it does not hinge on the allocator’s tuning. As under uniform ranks, KL-SVD keeps the higher zero-shot accuracy.
Comparison with pruning methods. Following the setup of Dobi-SVD (Wang et al., 2025b), we also compare Complete-SVD with post-training pruning methods (Table 5). At 10.1 GB, LLM-Pruner and Wanda-SP reach WikiText-2 perplexities of 10.5 and 12.1, against 7.5 for
| Memory (GB) | 10.1 | 7.7 | 6.8 |
| Wiki | |||
| LLM-Pruner | 10.5 | 46.3 | 253.1 |
| Wanda-SP | 12.1 | 38.6 | 249.2 |
| Complete-SVD | 7.5 | 13.3 | 19.8 |
| Dobi-SVD* | 5.9 | 7.9 | 9.5 |
| Complete-SVD* | 5.8 | 7.0 | 8.4 |
Table 5. Perplexity results (Wiki-2) of Complete-SVD versus state-of-the-art pruning methods and Dobi-SVD on Llama 2-7B (Base). Methods marked with * use remapping storage.
caption: Compression results on Mistral-7B, Llama 3-8B and Qwen3-8B with LEMS rank allocation at matched budget and fixed hyperparameters.
:::
Complete-SVD; at 6.8 GB, both exceed 249, whereas Complete-SVD attains 19.8. Under the remapping storage of Dobi-SVD (marked , Appendix § 11), Complete-SVD outperforms Dobi-SVD* at every memory budget, although it uses uniform ranks whereas Dobi-SVD* uses its loss-aware rank allocation.
Ablation and Sensitivity Studies
Ablation Study on Covariance Components. We analyze the individual contributions of and to the SVD compression objective on Llama 2-7B (Base). As shown in Table 6, while each matrix individually improves over standard SVD, their combination consistently yields the best performance across all compression ratios. At
At 50% compression ratio, Complete-SVD achieves substantially lower perplexity than using alone or alone, demonstrating that each component contributes complementary information to the functional norm.
Robustness to Calibration Data. We assess the sensitivity of Complete-SVD to its random draws and calibration budget (Figure 2). At the reference budget ( sequences, probes), varying the calibration seed (resampling the sequences) give WikiText-2 perplexities in (std ), and the estimator seed (redrawing the probes and signs of ) give (std ). Reducing from 256 to 32 slightly raises perplexity monotonically from 7.53 to 7.61, and varying over changes it by at most 0.03 (within noise). We use and as a compromise between cost and precision. Table 7 compares covariance matrices estimated on WikiText-2, our default, against C4:

Figure 2. WikiText-2 perplexity of Complete-SVD on Llama 2-7B (Base) at 80% compression ratio versus the sample size (left, ) and the probe budget (right, ). Error bars show , with the single-run standard deviation estimated from seed repeats at the reference budget.
| Cal. Set | Wiki | C4 | PTB | Acc. |
| Wiki-2 | 7.5 | 15.2 | 56.9 | 50.0 |
| C4 | 9.9 | 11.6 | 56.0 | 51.3 |
Table 7. Effect of the calibration set (WikiText-2 vs. C4) on Complete-SVD applied to Llama 2-7B (Base) at 80% compression ratio. Average accuracy is computed over 6 downstream datasets.
| CR | Method | Wiki | ||
| Baseline | N/A | N/A | 5.47 | |
| 0.8 | SVD | Id | Id | 19 438.3 |
| 0.8 | Gradient only | Id | 21.8 | |
| 0.8 | Forward only | Id | 8.5 | |
| 0.8 | Complete-SVD | 7.5 | ||
| 0.7 | SVD | Id | Id | 31 062.3 |
| 0.7 | Gradient only | Id | 81.9 | |
| 0.7 | Forward only | Id | 10.9 | |
| 0.7 | Complete-SVD | 9.3 | ||
| 0.6 | SVD | Id | Id | 44 494.7 |
| 0.6 | Gradient only | Id | 290.0 | |
| 0.6 | Forward only | Id | 17.3 | |
| 0.6 | Complete-SVD | 13.3 | ||
| 0.5 | SVD | Id | Id | 53 670.8 |
| 0.5 | Gradient only | Id | 2211.0 | |
| 0.5 | Forward only | Id | 37.9 | |
| 0.5 | Complete-SVD | 22.4 |
caption: WikiText-2 perplexity of Llama 2-7B (Base) when optimizing .
:::
each set favors its own distribution in perplexity, with comparable downstream accuracy.
Compatibility with quantization. We verify this by applying 3-bit weight-only GPTQ quantization (W3A16) to Llama 2-7B compressed by Complete-SVD at several ratios (Table 8). At ratio 0.8, quantization reduces the memory footprint
| CR | Method | Wiki | Mem |
| 0.8 | Complete-SVD | 7.5 | 10.1 GB |
| 0.8 | Complete-SVD + GPTQ W3A16 | 9.2 | 2.5 GB |
| 0.6 | Complete-SVD | 13.3 | 7.7 GB |
| 0.6 | Complete-SVD + GPTQ W3A16 | 16.7 | 1.9 GB |
Table 8. Perplexity (Wiki) and model size for Complete-SVD combined with GPTQ quantization on Llama 2-7B (Base). Memory reports the model parameter storage after compression.
from 10.1 GB to 2.5 GB at a WikiText-2 perplexity of 9.2; at ratio 0.6, the model occupies 1.9 GB, 14% of the original FP16 size. The low-rank factors produced by Complete-SVD therefore remain compatible with quantization, and the two techniques can be combined for deployment on memory-constrained devices.
Conclusion
We introduced Complete-SVD, a post-training low-rank compression method that minimizes the distance between the logits of the original and compressed networks over the data distribution. We make the Gram matrix of logit Jacobians, , tractable by exactly reducing its vocabulary dimension to the hidden dimension and estimating it without bias using backward passes per sequence, independent of vocabulary size and sequence length. As a controlled counterpart, we proposed KL-SVD which replaces distance with KL divergence between predictive distributions, and uses an unbiased estimator of the true Fisher matrix.
Across Mistral-7B, Llama 3-8B and Qwen3-8B, Complete-SVD achieves the lowest C4 and PTB perplexity among all compared methods, including KL-SVD, under WikiText-2 calibration, with covariance regularization held fixed across the output-aware methods. This cross-corpus advantage holds under both uniform and sensitivity-aware (LEMS) rank allocation, the latter at ratio 0.7, and persists with C4 calibration. Complete-SVD also improves IFEval instruction-following accuracy over KFAC-SVD and IO-SVD in every tested setting, and over KL-SVD in all but one setting. KL-SVD, in turn, attains equal or higher average zero-shot accuracy in every configuration. Complete-SVD requires no fine-tuning and is compatible with post-training quantization. These results suggest that, for post-training low-rank compression, matching the logits of the original model is a simple and effective alternative to preserving either the loss on the observed next token or the predicted distribution.
Limitations
Firstly, our evaluation is limited to models in the 7–8B parameter range. Evaluating larger models would provide stronger evidence that the observed behavior of the different output-aware methods persists at scale. Secondly, as in most post-training SVD-based methods, each layer is compressed using statistics collected from the original model, so the errors introduced by previously compressed layers are not taken into account. AA-SVD (Sinha and Fleuret, 2026) mitigates this by combining an activation-aware factorization with a gradient-based block-wise refinement on the activations of the partially compressed network, at the cost of no longer being training-free. Reducing calibration cost and estimator variance without sacrificing the full functional objective remains an important direction for future work. Finally, the complete functional norm is not specific to low-rank factorization. GPTQ (Frantar et al., [6]) and SparseGPT (Frantar and Alistarh, [5]) minimize
, that is, the objective of Theorem 1 with . Substituting yields an output-aware objective for quantization and pruning, although the closed-form SVD solution no longer applies and a dedicated solver is required, as in YAQA (Tseng et al., 2026) for the KL objective.
References
A Proof on Complete Functional Norm
Theorem 1. With the notation introduced in § 2.1, let denote the input covariance and the Gram matrix of the output Jacobians at the compressed layer. Then:
(i) Exact first-order expansion. Let . Then
(ii) Kronecker-factored approximation. Under the K-FAC structural approximation , the right-hand side of Equation (5) admits the closed-form approximation
Proof. Step 1 (exact first-order expansion). Writing the first-order Taylor development of around ,
Taking norms and expanding the inner product (full derivation in Appendix B.§ 2),
where . Taking square roots yields Equation (5).
Step 2 (K-FAC approximation). The matrix has size and is intractable to store or factorize at LLM scale. We invoke the K-FAC structural approximation (Dangel et al., [3]),
where and . This is a structural approximation: see Remark 2.
Combining the Kronecker identities and with Equation (10) gives
which is Equation (6). □
Remark 2. The two approximation symbols in Theorem 1 are not interchangeable. The remainder in Equation (5) is and vanishes as . The K-FAC step in Equation (6), by contrast, replaces the full Gram matrix by its Kronecker-factored approximation; the resulting structural error is independent of and does not vanish in any limit. We adopt the K-FAC approximation because it admits a closed-form SVD solution and has been empirically validated as a tight approximation of the layer-wise curvature for fully-connected layers (Dangel et al., [3]; Eschenhagen et al., [4]); we do not claim it is asymptotically tight in .
B Reconstruction Error
The first idea to find a good from is to minimize which can be done by computing the SVD of and keeping the top singular values. However, this does not take into account the effect of the change in weight on the loss. Hence, we can aim instead to minimize the change in loss. Writing and the vectorization of we have the following first-order Taylor expansion of the loss around :
This suggests choosing in order to minimize . However this would lead to only approximate on the correct label and does not give a rich feedback. As shown for Knowledge distillation (Hinton et al., [8]), it is better to copy the full distribution and to have a richer feedback. In addition, we suppose that is a local minimizer (as it is an already trained function), we can expect that and that the gradient information is limited.
Hence we can either directly minimize the functional distance between the output of and :
or minimize the expected KL divergence between the output of and :
two objectives that capture the full output distribution and not only the value at the correct label. We show now how we can optimize those objectives efficiently.
Norm Optimization
What we can show is that minimizing Equation (12), Equation (14) or Equation (13) can be done with similar techniques. Each of those objectives can be approximated by a quadratic form of with a curvature matrix that can be approximated using the Kronecker-factored approximation ([3]) as a Kronecker product of two matrices and .
Hilbertian Norm Minimization
Writing the first order development of around we have:
Hence:
Then taking the square-root we have:
This is still expensive to compute, as for fully-connected layers, is a matrix of size . To further reduce the computational cost, we can use the Kronecker-factored approximation ([3]) to approximate by the Kronecker product of two matrices and : .
Direct Loss Minimization
We can minimize the absolute value as we can suppose that we won’t get a lower loss after compression. Rewriting the objective:
KL Minimization
A classical result in information theory is that the change in KL divergence can be approximated by a quadratic form of the Fisher information matrix of the distribution with respect to the parameters noted :
where the Fisher information matrix is defined for the label distribution predicted by the model parameterized by as:
Hence taking the expectation over the data distribution, we have with
and as shown by Dangel et al. [3], can be approximated by the Kronecker product of two matrices and : where is the expected Fisher information matrix of the distribution with respect to values . If we note we have:
Optimization of the Quadratic Forms
All of above objectives can be formulated as minimizing where is a conditioning matrix that can be approximated by the Kronecker product of two matrices and . Using the properties of the Kronecker product, and (where is the row-wise vectorization), we have:
This final optimization problem can be solved efficiently in closed-form using the SVD.
| Method | ||
| Empirical Fisher curvature ([1]) | ||
| KL curvature | ||
| L2 curvature |
Table 9. Kronecker-factored curvatures of the output-aware objectives. Each objective weights the approximation error of by . The input factor is common to all methods, which differ only in the output factor .
C Links between the different curvature matrices
Fisher and Empirical Fisher
The empirical Fisher matrix is called “empirical” because it evaluates the gradient at the observed label instead of taking the expectation over the predictive distribution of the model, as the true Fisher matrix does. The full Fisher matrix is defined as:
a common approximation is to replace the expectation by the value at the correct label . This is a logical choice as the correct label is the most important one for the loss.
If we approximate by a Dirac distribution on the correct label , we have:
As the cross-entropy loss is , where denote the label of , we have:
is called the empirical Fisher. Similar to the True Fisher, it can be approximated by a Kronecker product of two matrices:
Links in the Computations
All the can be defined with a similar structure:
For the Fisher using the chain rule (see Dangel et al. [3] for more details) we have:
The same trick can be applied to the empirical Fisher:
So all methods can be formulated as:
with:
| Method | |
| ( norm) | |
| (Empirical Fisher) | |
| (True Fisher) |
Table 10. Output-space kernel inducing .
Computing the Curvature Matrix
The calibration set consists of token sequences, with sequence of length . The total number of token positions is . For position in sequence , the forward map decomposes as
where is the input of the layer at position , is its output, and denotes the rest of the network up to the logits . We denote by the input of the LM head, i.e. the last hidden state produced by before the linear projection to the vocabulary. The LM head is itself linear,
where denotes the LM head weight.
The matrix is defined as
For sequence inputs, the expectation in Equation (40) expands as a sum over all pairs with , the causal constraint for removing the remainder:
with .
Its direct evaluation is intractable: reverse-mode differentiation delivers one row of per backward pass and isolates one output position per cotangent assignment, so Equation (41) costs backward passes per sequence, which is for Llama 2-7B at .
This cost is the product of two independent axes: the vocabulary index, of size , and the output-position index, of size . We remove the first exactly, down to the hidden size , by absorbing the LM head into a Gram matrix (Appendix § 10.§ 2); we then reduce both remaining axes to a single user-chosen budget by two applications of one randomization principle, stated next. The resulting estimator costs backward passes per sequence, independently of and , and is unbiased for Equation (41).
A Rademacher Annihilation Lemma
Both axes are removed by the same mechanism: a random signed superposition, whose second moment retains the diagonal terms of a sum of Gram matrices and annihilates the cross terms in expectation.
Lemma 3. Let and let be i.i.d. Rademacher variables, . Then
Proof. Expanding the product gives . Since the signs are independent and centred, , which anni-
lates every term and preserves the diagonal terms. □
The left-hand side of Equation (42) is computed from the single superposed matrix , whereas the right-hand side names separate matrices. Whenever the index is one that reverse-mode differentiation charges a backward pass for, Lemma 3 converts a cost of into a cost of one, at the price of a variance that we quantify in Appendix § 10.§ 3. We apply it twice: with ranging over the vocabulary and with ranging over output positions (Appendix § 10.§ 3).
Eliminating the Vocabulary Dimension
The chain rule applied to the LM head gives
with , so that factors out of every summand of Equation (41):
where . The vocabulary dimension is absorbed once and for all into the head Gram matrix , which is precomputed at negligible cost.
The matrix is symmetric positive semi-definite, so it admits the spectral decomposition
where are the eigenpairs of . Substituting Equation (45) into Equation (44) gives
where
is the derivative at of the scalar , obtained by placing the cotangent at the LM-head input of the single output position .
The reduction is exact and lowers the cost per output position from to backward passes, a factor for Llama 2-7B and for Llama 3-8B. The eigenbasis makes the required cotangent directions explicit, one backward pass each, and Equation (46) is the form from which the estimator of Appendix § 10.§ 3 is built.
The broadcast estimator
Reducing the Backward Passes. The factor is nonetheless the hidden size of the model, and is the dominant term in the calibration budget. We reduce it to a user-chosen by applying Lemma 3 along the vocabulary index. To this end we use, in place of the spectral decomposition Equation (45), the row decomposition of the same matrix with the -th row of . Draw a Rademacher matrix and set , such that
with the -th column of . Lemma 3 applied with yields, for each column independently,
hence
The columns of replace the eigendirections of Equation (45) as cotangent directions.
Eliminating the Output-Position Dimension. Writing for the projected gradient of Equation (47) along the direction , a single backward pass differentiates a scalar that has already been summed over output positions: seeding the cotangent at every LM-head input returns for all input positions at once. Squaring this quantity aggregates the displacements of distinct output positions before taking the norm, which is not the summand of Equation (46). Isolating one output position per backward pass restores it, at a cost of passes per direction.
We instead weight each output position by an independent random sign. Let be i.i.d. Rademacher variables over , drawn independently of , and consider the estimator
with
The vector is the derivative at of the signed projected scalar , so it is still obtained for all in one backward pass: the signs change the cotangent placed at each position, not the number of traversals of the computation graph. Per sequence the cost is one forward pass and backward passes, independent of both and .
Applying Lemma 3 along the output-position index annihilates the cross-position terms in expectation and preserves the diagonal, so each summand of Equation (51) is unbiased for the corresponding summand of Equation (46) along the direction .
Proposition 4. Conditionally on the calibration set, entrywise, with as in Equation (41). Moreover in every realization of .
Proof. Fix and . Conditioning on and taking the expectation over first, Lemma 3 with gives
Taking the expectation over and applying Equation (50) replaces by , which is Equation (44) term by term; the two randomizations commute because and are independent. Positive semidefiniteness is immediate from Equation (51), every summand being an outer product of a real row vector. □
The uniform seed does not admit the same statement. Writing for Equation (51) with , and
for the symmetrized contribution of the output-position pair , the two estimators differ by a term that no choice of budget removes.
Proposition 5. For every realization of ,
and the second term is deterministic given the calibration set and the probes.
Proof. Setting all signs to one in Equation (51) gives . Splitting the double sum into its diagonal and its off-diagonal part, and pairing with in the latter, gives Equation (53) with the diagonal part equal to by Lemma 3. No sign variable appears in the remainder. □
The second term of Equation (53) is an average over the calibration sequences of the cross-position blocks Equation (52). It does not decrease with the number of sequences or with the number of probes, and vanishes only if those blocks cancel on average.
As shown in Table 11, the proposed estimator reduces the per-sequence calibration cost for Llama 2-7B from backward passes for direct evaluation to , a reduction of about six orders of magnitude. On a single H200 GPU, the full Complete-SVD pipeline takes about 80 minutes for both Llama 3-8B and Mistral-7B (Table 12), most of which is spent estimating ; since and do not depend on the target rank, this cost is incurred once per model
Table 11. Calibration cost per sequence, in backward passes, for the constructions of Appendix § 10. Numerical column instantiated for Llama 2-7B (, , ) with .
| Model | SVD | Total | ||
| Llama 3-8B | 12 min | 64 min | 5 min | min |
| Mistral-7B | 12 min | 61 min | 5 min | min |
| CR | (exact) | |||
| 0.90 | 23.4 | 23.8 | 23.6 | |
| 0.85 | 25.6 | 26.2 | 25.9 | |
| 0.80 | 29.0 | 29.8 | 29.6 |
Table 12. Per-stage wall-clock cost of the Complete-SVD compression pipeline on Llama 3-8B and Mistral-7B on a single H200 GPU: the input covariance estimation (), the Gram matrix of the output Jacobians estimation () and SVD computation. Reported with 256 calibration samples of length 2048.
and calibration set and is shared across all compression ratios.
Validation Against the Exact Reference
At reduced scale Equation (46) is directly computable, so the estimator can be validated end to end. Table 13 compares against this reference both and the broadcast construction , obtained from Equation (51) by replacing the random signs with .
As shown in Table 13, the exact reference is the best perplexity attainable under this weighting and both estimators sit above it. Relative to , the proposed estimator closes the gap to the reference at each compression ratio.
Extension to a Tractable Fisher Matrix Estimator
The construction of Appendix § 10 extends to the true Fisher matrix of the model, obtained by replacing the identity weighting of Equation (41) with the curvature of the predictive distribution. Write for the predictive distribution at output position of sequence , and
for the Hessian of the negative log-likelihood with respect to the logits, evaluated at . Taking the contexts from the calibration set and the labels from the model itself, the score of the sequence log-likelihood decomposes over output positions, and the resulting Fisher matrix at the layer pre-activation is
with as in Equation (41). No cross-output-position terms appear: conditionally on the context, the sampled labels at distinct positions are independent and each per-position score is centred, so the corresponding expectations vanish. Equation (55) is therefore exact, and plays for the Fisher matrix the role that Equation (41) plays for .
The vocabulary-elimination step of Appendix § 10.§ 2 does not transfer. Substituting Equation (43) into Equation (55) gives , with , but now depends on the output position, so it cannot be factored out of the sum as the constant was. Recovering the eigenbasis construction of Equation (46) would require one spectral decomposition and cotangent directions per output position, hence backward passes per sequence, which is the cost Appendix § 10.§ 3 was designed to remove.
Probes matched to the predictive distribution. We instead randomize along the vocabulary index as in Equation (48), but with a probe distribution matched to rather than to . Let denote the -th canonical basis vector. The matrix Equation (54) admits the exact expansion
where is the -th entry of and the expectation is over a single categorical draw from . The weighted sum over the vocabulary is thus a plain expectation over a term visited at its own weight, and the budget now indexes independent draws rather than Rademacher columns. Let be independent over , and define the cotangent directions
with the -th row of as in Appendix § 10.§ 3. Each direction is one row of the LM head corrected by the single vector , so beyond the logits themselves no object of size or is formed at any position, and Equation (56) gives
The estimator. Unlike the probes Equation (48), the directions Equation (57) vary with the output position, which is what allows the position-dependent curvature to be represented. Writing for the projected gradient of Equation (47) along , the estimator is
with .
As in Equation (51), is obtained for all input positions in a single backward pass, by seeding the cotangent at the LM-head input of every output position . The probes are held fixed in this differentiation: although is a function of and hence of , the scalar differentiated is with each treated as a constant, so that Equation (57) enters through the cotangent alone and not through the computation graph.
No sign variables appear in Equation (60). The role played by Lemma 3 along the output-position index in Equation (51) is here played by the probe construction itself: the directions Equation (57) are drawn independently at each output position and satisfy , so the cross-position terms are annihilated in expectation without an auxiliary randomization. The estimator Equation (51) requires the signs for a different reason. Its probe is centred as well, but the same realization is seeded at every output position, so the relevant moment for is and the cross terms survive; independent signs restore the cancellation that independent draws provide here for free.
Proposition 6. Conditionally on the calibration set, entrywise, with as in Equation (55). Moreover in every realization of the draws.
Proof. Fix and . Expanding the outer product in Equation (60) over two output-position indices gives
The draws being independent across output positions and centred, equals when and vanishes otherwise, leaving , which is Equation (55) term by term by Equation (43). Positive semidefiniteness is immediate from Equation (60), every summand being an outer product of a real row vector. □
The cost per sequence is one forward pass and backward passes, independent of and , matching Table 11. The forward pass supplies both the predictive distributions entering Equation (57) and the graph traversed by the backward passes, and no spectral decomposition is required at any stage. The budget controls the variance alone: Proposition 6 holds for every , so the centre is fixed and the variance decreases as , in contrast to constructions that restrict Equation (56) to a fixed subset of the vocabulary and renormalize on it, whose error is deterministic and does not vanish with the calibration budget.
Comparison with the Top- Construction of IO-SVD
IO-SVD targets the same output-side curvature as Equation (55) and, like the present work, pairs it with an input activation covariance under a Kronecker factorization. It differs in how the two intractable axes are removed, and both of its reductions introduce a deterministic deviation from Equation (55) rather than a variance that decreases with the calibration budget.
The output-position axis. IO-SVD defines its output-side curvature through the same-position Jacobian , so its accumulated object is the single term of Equation (55):
In a causal model the pre-activation influences the logits at every position , so a perturbation at input position contributes to terms of Equation (55) and Equation (61) retains one of them. The omission is structural. It does not depend on the vocabulary budget , it is not reduced by enlarging the calibration set, and its extent grows with sequence length.
The vocabulary axis. IO-SVD restricts Equation (54) to the largest entries of the predictive distribution and renormalizes the probabilities on that support. Fix a sequence and an output position and suppress the indices . Let collect the largest entries of , let be the restriction of to , let be its mass, and let be the principal submatrix of indexed by . The renormalized curvature is , and a direct computation gives
The truncation therefore does more than set the entries outside to zero: it inflates the retained block by and over-subtracts the rank-one term by . Since varies from one output position to the next, Equation (62) reweights positions against one another rather than applying a common scale, and the traces differ, against . Restricting Equation (56) to without renormalizing would instead give the partial sum , which is the head term of a stratified form of our estimator: positive semi-definite, free of the distortion Equation (62), and monotone in .
Both deviations are deterministic given the calibration set. They persist at any short of and at any number of calibration sequences, so the corresponding error is a bias rather than a variance. By contrast, Proposition 6 places the centre of at Equation (55) for every , leaving only an variance, and the two constructions coincide neither in the output positions they retain nor in the curvature they assign to those retained.
SVD Compression with KL Divergence Objective
To further validate the design of Complete-SVD, we compare it against an alternative objective based on the Kullback-Leibler (KL) divergence between the original and compressed model output distributions discussed in Appendix B.§ 4. The KL-based objective is connected to the true Fisher information matrix, which arises as the second-order expansion of the KL divergence and constitutes the principled curvature metric for probabilistic models. Furthermore, in our experiment, we use the unbiased estimator of True Fisher information matrix described in Appendix § 10.§ 5.
The configurations in Table 14 share the rank allocation and covariance regularization procedure (Appendix § 12). Compared with IO-SVD, KL-SVD yields gains that depend on the model and compression ratio. On Llama 3-8B, it lowers perplexity on all three corpora at both ratios, with larger improvements under stronger compression: at ratio 0.7, C4 perplexity decreases from 111.8 to 103.0 and PTB perplexity from 219.2 to 172.7, corresponding to reductions of 7.9% and 21.2%. On Qwen3-8B, IO-SVD is marginally better on all three corpora at ratio 0.9, whereas KL-SVD improves all three perplexities at ratio 0.7, reducing C4 and PTB perplexity by 15.0% and 31.1%, respectively. On Mistral-7B, the two methods are nearly identical in perplexity at ratio 0.9, whereas KL-SVD yields higher perplexity on all three corpora at ratio 0.7, particularly on WikiText-2 (10.3 versus 8.8). Average zero-shot accuracy is higher with KL-SVD on Mistral-7B and Llama 3-8B at both ratios, but slightly lower on Qwen3-8B. Furthermore, on IFEval (Table 2), KL-SVD improves prompt-level strict accuracy over IO-SVD by 2.8–8.2 percentage points in all but one setting; the exception is Mistral-7B at ratio 0.9, where accuracy decreases from 13.9 to 12.4.
| CR Method | Mistral-7B | Llama 3-8B | Qwen3-8B | |||||||||
| Wiki ↓ | C4 ↓ | PTB ↓ | Acc ↑ | Wiki ↓ | C4 ↓ | PTB ↓ | Acc ↑ | Wiki ↓ | C4 ↓ | PTB ↓ | Acc ↑ | |
| Baseline | 5.3 | 8.1 | 27.7 | 64.0 | 6.1 | 9.5 | 9.9 | 63.3 | 9.7 | 15.5 | 15.4 | 62.0 |
| KFAC-SVD | 6.2 | 13.1 | 44.0 | 56.9 | 8.8 | 25.7 | 32.6 | 54.5 | 11.5 | 23.8 | 28.0 | 57.1 |
| IO-SVD | 6.1 | 12.8 | 40.0 | 56.9 | 8.4 | 23.2 | 30.9 | 56.2 | 11.0 | 23.7 | 28.9 | 57.9 |
| KL-SVD | 6.1 | 12.8 | 40.1 | 57.3 | 8.3 | 22.8 | 27.1 | 56.9 | 11.2 | 24.6 | 29.1 | 57.7 |
| Complete-SVD | 6.4 | 12.7 | 39.9 | 56.9 | 8.7 | 22.4 | 23.0 | 55.8 | 11.7 | 23.4 | 23.4 | 56.7 |
| KFAC-SVD | 9.3 | 34.6 | 110.7 | 45.8 | 19.4 | 151.3 | 253.5 | 40.7 | 14.7 | 50.0 | 94.0 | 46.8 |
| IO-SVD | 8.8 | 33.1 | 97.8 | 45.6 | 16.8 | 111.8 | 219.2 | 43.3 | 14.4 | 58.7 | 120.2 | 49.2 |
| KL-SVD | 10.3 | 34.7 | 98.8 | 46.6 | 16.0 | 103.0 | 172.7 | 45.6 | 14.2 | 49.9 | 82.8 | 48.6 |
| Complete-SVD | 10.1 | 32.9 | 90.7 | 45.4 | 16.8 | 95.3 | 123.7 | 44.2 | 15.0 | 47.5 | 67.7 | 47.8 |
Table 14. Output-aware methods from Table 1 (WikiText-2 calibration, uniform rank allocation), repeated here for the comparison of KL-SVD with IO-SVD. Accuracy (Acc) is the average zero-shot accuracy over 6 downstream datasets.
Remapping storage
Standard SVD compression stores the truncated factors and separately, with footprint and parameter ratio . This ratio saturates before reaches (for , already at ), so is not in one-to-one correspondence with over . Following Dobi-SVD Wang et al. (2025b), assuming , we adopt a remapping scheme that halves the precision of the overlapping submatrices of and and merges them into a single matrix, yielding a parameter ratio in one-to-one correspondence with . Under remapping, we thus choose the ranks uniformly as .
Implementation Details: Covariance Square Roots
Regularization following KFAC-SVD. We follow the covariance regularization and Cholesky factorization used in KFAC-SVD for both the input covariance and the output Jacobian-Gram covariance. For a covariance matrix , we first apply , using for the input covariance and for . We then compute its Cholesky factorization . If the Cholesky decomposition fails because of finite-precision negative eigenvalues, we compute the smallest eigenvalue and shift the diagonal as before retrying the factorization.
Inference Efficiency
Inference throughput. We measure the generation throughput, in tokens per second, of Llama 2-7B (Base) compressed by Complete-SVD at ratios of 40%, 60%, and 80% (Figure 3). On an NVIDIA A100 (80 GB), compression increases throughput at every ratio, with the largest relative gains at large batch sizes and short sequence lengths, consistent with the reduced parameter footprint of the linear layers. On a memory-limited NVIDIA Quadro P5000 (16 GB), which can still hold the original model, the model at 40% ratio is about faster than the original at batch size 128. At batch size 256, the original model runs out of memory, whereas the models at 40% and 60% ratios still run and reach their highest throughput.
Figure 3. Inference throughput (tokens/sec) of Llama 2-7B (100%) and its Complete-SVD compressed variants at 40%, 60%, and 80% ratios. (a) Varying batch size with sequence length 32 on an A100 (80GB). (b) Varying sequence length with batch size 64 on an A100. (c) Varying batch size with sequence length 32 on a Quadro P5000 (16GB). “OOM” denotes an out-of-memory failure of the baseline.
Experiments on VLMs
ScienceQA-IMG is the image-based subset of the ScienceQA test split. It contains 2,017 multiple-choice science questions from elementary and high-school curricula, covering natural, social and language science; each question comes with an image and two to five answer options, and performance is reported as top-1 accuracy (%) of the selected option.
SEED-Bench is a human-annotated multiple-choice benchmark for multimodal comprehension. We use its image subset (SEED-Bench-IMG), whose four-option questions span nine dimensions: scene understanding, instance identity, instance attributes, instance location, instance counting, spatial relation, instance interaction, visual reasoning and text recognition; performance is reported as top-1 accuracy (%).
Complete-SVD^q Complete-SVD allocates ranks uniformly, so that every compressed projection retains the same fraction of its parameters. **Complete-SVD**^q keeps the Complete-SVD factorization unchanged and replaces this uniform allocation with the sensitivity-aware rank allocator of QSVD, which separates the contribution of the factorization from that of the rank allocation. For a projection , let denote the singular value decomposition of the whitened weight, so that . Each component is scored by , where
is the derivative of the cross-entropy of the -th calibration sample with respect to , obtained with the gradient pass of QSVD on the same calibration samples used to estimate and . As in QSVD, the components of all query, key and value projections are ranked jointly and the highest-scoring ones are retained, regardless of their position in the spectrum. Following the parameter-matched budget of WSVD, , so that the factorized projections hold a fraction of the original parameters. Let denote the indices of the components retained for a projection, which need not be the leading ones, and let , and collect the corresponding singular vectors and values. The factors are then and , with rank ; uniform Complete-SVD corresponds to .
| Model | Method | ScienceQA-IMG ↑ 90% | 80% | 70% | 60% | 50% | SEED-Bench ↑ 90% | 80% | 70% | 60% | 50% | Avg. ↑ |
| LLaVA1.5 7B | ASVD | 49.93 | 50.12 | 47.10 | 36.69 | 19.19 | 54.27 | 53.53 | 48.35 | 37.17 | 24.17 | 42.05 |
| LLaVA1.5 7B | SVD-LLM | 65.44 | 63.71 | 61.92 | 57.41 | 55.53 | 57.89 | 57.50 | 55.33 | 54.64 | 55.31 | 58.47 |
| LLaVA1.5 7B | QSVD | 67.72 | 68.22 | 67.08 | 65.05 | 62.37 | 59.84 | 59.07 | 59.78 | 59.00 | 58.23 | 62.64 |
| LLaVA1.5 7B | WSVD | 68.17 | 67.72 | 67.28 | 65.89 | 65.49 | 60.10 | 60.17 | 59.89 | 60.18 | 60.46 | 63.54 |
| LLaVA1.5 7B | IO-SVD | 67.92 | 68.07 | 68.17 | 66.73 | 65.99 | 60.29 | 60.32 | 60.22 | 60.15 | 60.64 | 63.85 |
| LLaVA1.5 7B | Complete-SVD | 66.48 | 66.58 | 66.83 | 65.79 | 66.39 | 58.98 | 58.51 | 58.09 | 56.19 | 56.72 | 62.06 |
| LLaVA1.5 7B | Complete-SVD | 67.28 | 66.68 | 66.44 | 66.14 | 65.34 | 60.05 | 59.87 | 59.84 | 59.42 | 59.21 | 63.03 |
| LLaVA1.5 7B | FP16 | Accuracy: 68.01 | Accuracy: 60.18 | 64.10 | ||||||||
| LLaVA1.5 13B | ASVD | 71.39 | 71.59 | 70.00 | 70.25 | 69.51 | 61.92 | 61.91 | 61.54 | 61.51 | 60.71 | 66.03 |
| LLaVA1.5 13B | SVD-LLM | 71.05 | 70.85 | 70.30 | 70.35 | 70.30 | 62.28 | 62.34 | 62.25 | 62.08 | 63.01 | 66.48 |
| LLaVA1.5 13B | QSVD | 71.89 | 71.99 | 71.49 | 71.54 | 71.39 | 62.61 | 62.64 | 62.82 | 62.63 | 62.52 | 67.15 |
| LLaVA1.5 13B | WSVD | 71.99 | 71.84 | 72.53 | 71.59 | 71.44 | 62.52 | 62.68 | 62.38 | 62.37 | 62.37 | 67.17 |
| LLaVA1.5 13B | IO-SVD | 71.79 | 72.14 | 72.29 | 72.14 | 72.68 | 62.64 | 62.49 | 62.46 | 62.45 | 62.29 | 67.34 |
| LLaVA1.5 13B | Complete-SVD | 71.15 | 71.99 | 71.89 | 72.09 | 71.59 | 62.58 | 62.65 | 62.44 | 62.54 | 62.30 | 67.12 |
| LLaVA1.5 13B | Complete-SVD | 71.49 | 71.59 | 71.79 | 71.94 | 71.44 | 62.41 | 62.53 | 62.44 | 62.44 | 62.54 | 67.06 |
| LLaVA1.5 13B | FP16 | Accuracy: 71.83 | Accuracy: 62.53 | 67.18 | ||||||||
| SmolVLM 2B | ASVD | 29.30 | 3.97 | 0.20 | 17.85 | 1.50 | 0.95 | 8.96 | ||||
| SmolVLM 2B | SVD-LLM | 40.06 | 17.20 | 3.82 | 32.49 | 15.89 | 4.60 | 19.01 | ||||
| SmolVLM 2B | QSVD | 77.00 | 62.77 | 42.59 | 64.80 | 50.46 | 36.24 | 55.64 | ||||
| SmolVLM 2B | WSVD | 76.30 | 71.74 | 60.93 | 65.78 | 63.29 | 54.45 | 65.42 | ||||
| SmolVLM 2B | IO-SVD | 83.34 | 82.65 | 82.30 | 68.36 | 68.65 | 68.15 | 75.58 | ||||
| SmolVLM 2B | Complete-SVD | 79.33 | 76.55 | 73.77 | 66.67 | 65.69 | 64.63 | 71.11 | ||||
| SmolVLM 2B | Complete-SVD | 83.14 | 83.59 | 82.55 | 68.64 | 68.39 | 68.02 | 75.72 | ||||
| SmolVLM 2B | FP16 | Accuracy: 84.53 | Accuracy: 68.47 | 76.53 |
Table 15. Accuracy evaluation of different methods under FP16.
References
- [1]Viktoriia A. Chekalina, Daniil Moskovsky, Tatyana Matveeva, Andrey Kuznetsov, and Evgeny Frolov. 2026. Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. In Proceedings of the 43rd International Conference on Machine Learning, pages 13381–13399.
- [2]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv: 1803.05457 [cs].arxiv.org/abs/1803.05457
- [3]Felix Dangel, Bálint Mucsányi, Tobias Weber, and Runa Eschenhagen. 2025. Kronecker-Factored Approximate Curvature (KFAC) From Scratch. arXiv: 2507.05127 [cs].arxiv.org/abs/2507.05127
- [4]Runa Eschenhagen, Alexander Immer, Richard E. Turner, Frank Schneider, and Philipp Hennig. 2023. Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures. In Advances in Neural Information Processing Systems.
- [5]Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. In Proceedings of the 40th International Conference on Machine Learning, pages 10323–10337.arxiv.org/abs/2301.00774
- [6]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate Quantization for Generative Pre-Trained Transformers. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2210.17323
- [7]Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. arXiv: 2407.21783 [cs].arxiv.org/abs/2407.21783
- [8]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv: 1503.02531 [stat].arxiv.org/abs/1503.02531
- [9]Yen-Chang Hsu, Ting Hua, Sung-En Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022. Language Model Compression with Weighted Low-Rank Factorization. In The Tenth International Conference on Learning Representations.arxiv.org/abs/2207.00112
- [10]Q. Huangfu and J. A. J. Hall. 2018. Parallelizing the Dual Revised Simplex Method. Mathematical Programming 10(1):119–142.arxiv.org/abs/1503.01889
- [11]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv: 2310.06825 [cs.CL].arxiv.org/abs/2310.06825
- [12]Alper Kalle, Théo Rudkiewicz, Mohamed Ouerfelli, and Mohamed Tamaazousti. 2025. Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks. In Advances in Neural Information Processing Systems.arxiv.org/abs/2511.04494

