betaFeedback

WorksComputer Science

SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers

Abstract

Pretrained transformers spend a large share of their inference compute inside self-attention, yet the per-head query-key interaction is structurally low-rank by construction. We ask a diagnostic question: across layers and architectures, which heads actually use the full rank budget on real data, and which do not? We propose SVD-Attention, a post-hoc method that decomposes the per-head interaction matrix MhM_h and scores every singular mode by both its spectral magnitude and its activation variance across tokens on a small calibration set. The score keeps the modes the head can amplify and that real data actually activates, and discards the rest; the resulting factorization is exact for the retained modes and needs no finetuning. Across seven pretrained models and five task types, the diagnostic finds a task-dependent pattern: redundancy sits in the early to middle layers for sentiment and image classification and for generation, and in the late layers for dense prediction and extractive question answering. It is also predictive: truncating the layers it flags costs at most 1.751.75 F1, while truncating the same number of layers elsewhere in the same model costs up to 5050 F1. SVD-Attention is an analysis tool rather than a compression method: acting on it reduces inference FLOPs by 0.80.8-1.5%1.5\% on its own and by 88-11%11\% when composed with weight pruning, with 00-4%4\% change in task quality.

Introduction

Multi-head self-attention is the central building block of modern transformers, and a long line of work has sought to reduce its cost by changing the attention pattern [1, 15], approximating the softmax kernel [5, 16, 10] (Wang et al. 2020a), or removing redundancy from pretrained projection weights after training (Yuan et al., 2023; Wang et al., 2025; Sanyal et al., 2026). We follow this third direction.

Existing post-hoc methods decompose Wq,hW_{q,h} and Wk,hW_{k,h} independently, which leaves their product, the object the softmax actually consumes, unconstrained. We instead decompose the per-head interaction matrix Mh=Wq,hWk,hTM_h = W_{q,h}W_{k,h}^{\mathsf{T}} directly. MhM_h has rank at most dhd_h, so the head has algebraic room for many directions but uses far fewer in practice: a singular direction only affects the head’s output if real input tokens vary along it, and otherwise contributes a constant offset that softmax absorbs. We outline our method in Figure 1.

Figure 1. Conceptual representation of our proposed SVD-Attention is shown in (a). The SVD-Attention method is outlined in (b).

We therefore score each mode of MhM_h by the product of its singular value and the geometric mean of its query- and key-side activation variances on a small calibration set. Keeping the smallest set of modes that captures a (1−ϵ)(1-\epsilon) fraction of that importance gives an exact rank-rhr_h factorization with no finetuning. We use this diagnostic to ask three questions. First, does adding data to the score change which modes look important compared to ranking by singular value alone (RQ1)? Second, where in a pretrained transformer does the diagnostic find the most redundancy, and does the pattern depend on the task (RQ2)? Third, is the diagnostic predictive: when we remove the modes it flags, does performance hold up where the score is low and drop where it is high (RQ3)?

We run the diagnostic on seven pretrained transformers spanning language understanding (Distil-BERT), extractive question answering (RoBERTa-base, MiniLM), generation (Llama-2-7B), vision classification (ViT, Swin), and dense prediction (DPT-Large). Redundancy sits in the early to middle layers for sentiment and image classification and for generation, and in the late layers for dense prediction and extractive question answering. Acting on the diagnostic reduces inference FLOPs by 0.8-1.5% on its own; composing it with weight pruning (Sun et al., 2024) raises the reduction to 8-11%, with task quality within 0-4% of baseline and no gradient updates. In the flagged layers more than half of the average head’s rank budget can be discarded for 0.16 F1 (RoBERTa-base on SQuAD 2.0), and the same diagnostic predicts where truncation instead fails.

Related Work

Efficient attention. The quadratic cost of softmax attention has motivated linear and sparse approximations [5, 15] (Wang et al. 2020a), and recent work continues to close the gap with softmax [16, 10]. These methods change the attention computation itself. Our work is orthogonal: we keep softmax attention intact and study redundancy in the pretrained projection weights.

Token-level compression. A complementary direction reduces the number of tokens rather than the per-token cost: Token Merging groups similar tokens at each layer and increases throughput with negligible accuracy drop [3]. Token merging shrinks the sequence length while our method shrinks the per-head channel dimension dhd_h, so the two axes are complementary.

Post-training SVD of weights. Closest to our setting, ASVD (Yuan et al., 2023) compresses LLM projections by reweighting columns according to activation magnitudes before SVD, and SVD-LLM (Wang et al., 2025) adds a truncation-aware whitening step that links singular value loss to output loss. Both compress each weight matrix in isolation and are not specific to attention. We differ in three ways. First, our unit of decomposition is the per-head interaction matrix MhM_h, the object the softmax consumes; compressing Wq,hW_{q,h} and Wk,hW_{k,h} separately leaves their product unconstrained. Second, our score uses activation variance in the singular bases, which captures discriminability rather than magnitude: a mode contributing a near-constant offset to every token is washed out by softmax even when its activations are large. Third, we use the construction as a diagnostic for where redundancy lives across layers and tasks, not only as a compression algorithm.

Magnitude-based pruning. WANDA (Sun et al., 2024) prunes individual weights by the product of weight magnitude and input activation norm, with no retraining. WANDA and our method are both training-free and calibration-based but target different units: WANDA removes individual entries within each weight matrix, while we remove whole singular modes of MhM_h. We compose with WANDA as a stronger baseline in § 5.

Attention rank and head pruning. Elhage et al. [9] analyze the combined QK matrix as a circuit object, the same object we decompose, and Bhojanapalli et al. [2] study the per-head rank budget as an expressivity limit; our finding is the complement, that real data often does not use a budget that is algebraically sufficient. Michel et al. [18] and Voita et al. (2019) prune whole heads, a coarser unit than ours: a head that is indispensable as a whole can still spend most of its rank budget on modes that no token varies along. DRONE [4] and FWSVD [12] are data-aware low-rank compressors, but like ASVD and SVD-LLM their objective is reconstruction error, which ranks a spectrally large but data-inactive mode highly.

SVD-Attention Method

Figure 1 outlines the method; § 3.8 gives the complete algorithm.

Setup

Let X∈RN×dX \in\mathbb{R}^{N \times d} be the input to a self-attention layer with HH heads, head dimension dh=d/Hd_h = d/H, and per-head projections Wq,h,Wk,h∈Rd×dhW_{q,h}, W_{k,h} \in\mathbb{R}^{d \times d_h}. The attention logits inside head hh depend on XX only through the interaction matrix

Mh=Wq,hWk,hT∈Rd×d,(1)M_h = W_{q,h}W_{k,h}^{\mathsf{T}} \in\mathbb{R}^{d \times d}, \tag*{(1)}

since QhKhT=XMhXTQ_hK_h^{\mathsf{T}} = XM_hX^{\mathsf{T}}. By construction rank⁡(Mh)<dh\operatorname{rank}(M_h) < d_h, so MhM_h lives in a subspace of dimension at most dhd_h inside a d×dd \times d matrix. We ask whether real input data uses this rank budget fully, or whether some modes are unused in practice and can be removed.

Decomposition

We decompose MhM_h directly with the singular value decomposition:

Mh=UhΣhVhT,(2)M_h = U_h \Sigma_h V_h^{\mathsf{T}}, \tag*{(2)}

where Uh,Vh∈Rd×dU_h, V_h \in\mathbb{R}^{d \times d} are orthogonal and Σh=diag⁡(σ1,…,σd)\Sigma_h = \operatorname{diag}(\sigma_1,\ldots,\sigma_d) with σ1≥⋯≥σd≥0\sigma_1 \ge\cdots\ge\sigma_d \ge0. Because rank⁡(Mh)≤dh\operatorname{rank}(M_h) \le d_h, at most dhd_h singular values are nonzero. The columns of UhU_h form a basis for queries and those of VhV_h a basis for keys; the two bases are in general distinct.

Data-Aware Importance

The singular value σi\sigma_i measures how much mode ii can amplify a query-key interaction, but not whether real tokens differ along that mode. A direction along which all tokens project to the same value adds a near-constant offset to every attention logit and is washed out by the softmax, so it contributes nothing to the head’s output regardless of its singular value. We capture this with a data-aware variance term.

Given a small calibration set C={X(c)}c=1∣C∣C = \{X^{(c)}\}_{c=1}^{|C|} of layer inputs, we project each example into the two bases and measure how much tokens vary along each singular direction:

vi(U)=1∣C∣∑cVar⁡t[(X(c)Uh)t,i],(3)v_i^{(U)} = \frac{1}{|C|} \sum_c \operatorname{Var}_t\left[(X^{(c)}U_h)_{t,i}\right], \tag*{(3)}

and analogously vi(V)v_i^{(V)} using VhV_h, with the variance taken across tokens t∈{1,…,N}t \in\{1,\ldots,N\} and averaged over calibration examples. We combine the weight-side capacity and the data-side activation into a single importance score:

Ii=σi⋅vi(U)vi(V).(4)\mathcal{I}_i = \sigma_i \cdot\sqrt{v_i^{(U)}v_i^{(V)}}. \tag*{(4)}

The geometric mean is the right combination because it goes to zero whenever either side is zero: if queries vary along mode ii but all keys project to the same value, the dot product qs,ikt,iq_{s,i}k_{t,i} is constant across key tokens and the mode produces no differentiation (Yuan et al., 2023). The arithmetic mean would give such a mode misleadingly moderate importance.

Adaptive Rank Selection and Replacement

We sort modes by Ii\mathcal{I}_i in descending order and pick the smallest rhr_h such that the top modes capture at least a (1−ϵ)(1-\epsilon) fraction of the linear importance:

rh=min⁡{r:∑i=1rI(i)∑i=1dI(i)≥1−ϵ},(5)r_h = \min\left\{r : \frac{\sum_{i=1}^{r}\mathcal{I}_{(i)}}{\sum_{i=1}^{d}\mathcal{I}_{(i)}} \ge1-\epsilon\right\}, \tag*{(5)}

where I(i)\mathcal{I}_{(i)} denotes the ii-th largest score. Each head chooses its own rank, so heads that use less of the rank budget are compressed more aggressively. Let Uh(r),Vh(r)∈Rd×rhU_h^{(r)}, V_h^{(r)} \in\mathbb{R}^{d \times r_h} collect the columns of UhU_h and VhV_h at the top-rhr_h modes and let σ(r)=(σπ(1),…,σπ(rh))\sigma^{(r)} = (\sigma_{\pi(1)},\ldots,\sigma_{\pi(r_h)}) collect their singular values. We replace the per-head projections by

Wq,h∗=Uh(r)diag⁡(σ(r)),Wk,h∗=Vh(r)diag⁡(σ(r)).(6)W_{q,h}^{*} = U_h^{(r)}\operatorname{diag}\left(\sqrt{\sigma^{(r)}}\right), \qquad W_{k,h}^{*} = V_h^{(r)}\operatorname{diag}\left(\sqrt{\sigma^{(r)}}\right). \tag*{(6)}

This factorization is exact for the retained modes:

Mh∗=Wq,h∗(Wk,h∗)T=Uh(r)diag⁡(σ(r))(Vh(r))T,M_h^{*} = W_{q,h}^{*}(W_{k,h}^{*})^{\mathsf{T}} = U_h^{(r)}\operatorname{diag}\left(\sigma^{(r)}\right)(V_h^{(r)})^{\mathsf{T}},

the truncated SVD of MhM_h at rank rhr_h. The only approximation error comes from the discarded modes, and is bounded (§ 3.6).

Inference Cost

The replacement weights have shape d×rhd \times r_h instead of d×dhd \times d_h, so each compressed head reduces both the projection FLOPs and the attention-logit FLOPs by a factor of rh/dhr_h/d_h. The decomposition is performed once offline, and the reported reduction assumes the discarded query and key dimensions are not computed. For models using rotary position embeddings (RoPE), such as Llama-2-7B, we apply the same post-hoc truncation and evaluate it empirically (§ 3.7). We report analytic FLOPs throughput, as the post-training compression literature we build on does; realizing them as wall-clock speedup requires an attention kernel operating at the reduced dimension rhr_h, since padding the discarded dimensions back erases the saving.

Truncation Error Bound

We bound the spectral-norm error introduced by truncating MhM_h to rank rhr_h using our data-aware ordering.

Setup. Let Mh=UhΣhVhTM_h = U_h\Sigma_hV_h^{\mathsf{T}} be the SVD of the per-head interaction matrix, with singular values σ1≥⋯≥σd≥0\sigma_1 \ge\cdots\ge\sigma_d \ge0. Let π\pi be the permutation that sorts modes by Ii=σivi(U)vi(V)\mathcal{I}_i = \sigma_i\sqrt{v_i^{(U)}v_i^{(V)}} in descending order, and let S={π(1),…,π(rh)}S = \{\pi(1),\ldots,\pi(r_h)\} be the indices of the retained modes. Define

M~h=∑i∈SσiuiviT.(7)\widetilde{M}_h = \sum_{i \in S} \sigma_i u_i v_i^{\mathsf{T}}. \tag*{(7)}

This is exactly the rank-rhr_h matrix produced by Algorithm 1.

Proposition. The spectral-norm truncation error satisfies

∥Mh−M~h∥2=max⁡i∉Sσi.(8)\lVert M_h-\widetilde{M}_h\rVert_2=\max_{i\notin S}\sigma_i. \tag*{(8)}

Proof. By construction Mh−M~h=∑i∉SσiuiviTM_h-\widetilde{M}_h=\sum_{i\notin S}\sigma_i u_i v_i^{\mathsf{T}}. The vectors {ui}i∉S\{u_i\}_{i\notin S} are mutually orthonormal because they are columns of UhU_h; the same holds for {vi}i∉S\{v_i\}_{i\notin S}. Hence Mh−M~hM_h-\widetilde{M}_h has SVD ∑i∉SσiuiviT\sum_{i\notin S}\sigma_i u_i v_i^{\mathsf{T}} already in canonical form, and its spectral norm equals its largest singular value, max⁡i∉Sσi\max_{i\notin S}\sigma_i. □\square

Comparison with Eckart–Young. The Eckart–Young theorem Eckart and Young, [8] gives

min⁡rank⁡(B)≤rh∥Mh−B∥2=σrh+1.(9)\min_{\operatorname{rank}(B)\le r_h}\lVert M_h-B\rVert_2=\sigma_{r_h+1}. \tag*{(9)}

The minimum is attained by retaining the top-rhr_h singular values in spectral order. When the data-aware ordering agrees with the spectral ordering, our truncation is optimal in the spectral norm. When it disagrees, the gap is the difference between our retained set and the top-rhr_h spectral set:

∥Mh−M~h∥2−σrh+1=(max⁡i∉Sσi)−σrh+1.(10)\lVert M_h-\widetilde{M}_h\rVert_2-\sigma_{r_h+1}=\left(\max_{i\notin S}\sigma_i\right)-\sigma_{r_h+1}. \tag*{(10)}

The right-hand side is non-negative and is zero when no high-spectral mode is discarded in favor of a lower-spectral one.

Why we accept this gap. We trade spectral-norm optimality for a property the spectral ranking lacks: modes that are spectrally large but data-inactive contribute a near-constant offset to the attention logits. Concretely, if all tokens project to nearly the same value cˉi\bar{c}_i along mode ii, then mode ii’s contribution to the logit qsTktq_s^{\mathsf{T}}k_t is approximately σicˉi2\sigma_i\bar{c}_i^2 for every (s,t)(s,t) pair. A constant added to every entry of a softmax row leaves the output distribution unchanged, so retaining such a mode wastes rank budget without changing attention. The variance term in Ii\mathcal{I}_i is the data-side analogue of σi\sigma_i and quantifies precisely how much each mode contributes beyond a constant offset. The gap in Eq. (10) therefore upper-bounds a quantity we explicitly do not want to minimize.

Frobenius bound. The same argument applies to the Frobenius norm:

∥Mh−M~h∥F2=∑i∉Sσi2.(11)\lVert M_h-\widetilde{M}_h\rVert_F^2=\sum_{i\notin S}\sigma_i^2. \tag*{(11)}

The Eckart–Young Frobenius lower bound is ∑i>rhσi2\sum_{i>r_h}\sigma_i^2, and the data-aware Frobenius error exceeds this by exactly the spectral mass we sacrifice to keep data-active modes.

SVD-Attention with RoPE

Llama-2-7B applies rotary position embeddings (RoPE) Su et al., 2024 to the queries and keys after the linear projections. For positions ss and tt, the RoPE-augmented queries and keys are

q~s=Rsqs,k~t=Rtkt,(12)\widetilde{q}_s=R_s q_s,\qquad\widetilde{k}_t=R_t k_t, \tag*{(12)}

where qs=(XWq,h)sq_s=(XW_{q,h})_s, kt=(XWk,h)tk_t=(XW_{k,h})_t, and Rs,Rt∈Rdh×dhR_s,R_t\in\mathbb{R}^{d_h\times d_h} are orthogonal, position-dependent rotations. The resulting attention score is

q~sTk~t=qsTRsTRtkt=(XWq,h)sTRsTRt(XWk,h)t(13)\widetilde{q}_s^{\mathsf{T}}\widetilde{k}_t=q_s^{\mathsf{T}}R_s^{\mathsf{T}}R_tk_t=(XW_{q,h})_s^{\mathsf{T}}R_s^{\mathsf{T}}R_t(XW_{k,h})_t \tag*{(13)}

Unlike standard attention, the effective query-key interaction is therefore not fully described by a single position-independent matrix MhM_h: the bilinear form depends on relative position through RsTRtR_s^{\mathsf{T}}R_t.

We nonetheless compute the SVD of MhM_h and use it as a diagnostic proxy for the query-key interaction in RoPE models. After selecting the retained modes, we replace the query and key projections with the truncated factors

Wq,h∗=Uh(r)diag⁡(σ(r)),Wk,h∗=Vh(r)diag⁡(σ(r)),(14)W_{q,h}^{*}=U_h^{(r)}\operatorname{diag}\left(\sqrt{\sigma^{(r)}}\right),\qquad W_{k,h}^{*}=V_h^{(r)}\operatorname{diag}\left(\sqrt{\sigma^{(r)}}\right), \tag*{(14)}

or, when an implementation expects the original head dimension dhd_h, their zero-padded versions. The standard RoPE operation is then applied to the resulting queries and keys without changing the RoPE module itself, inducing the score

(q~s∗)Tk~t∗=(XWq,h∗)sTRsTRt(XWk,h∗)t,(15)(\widetilde{q}_s^{*})^{\mathsf{T}}\widetilde{k}_t^{*}=(XW_{q,h}^{*})_s^{\mathsf{T}}R_s^{\mathsf{T}}R_t(XW_{k,h}^{*})_t, \tag*{(15)}

which has the same RoPE form as the original model. This is a practical post-hoc approximation, not an exact low-rank factorization of the full position-dependent interaction; the exact factorization statement in § 3.4 applies to the position-independent matrix MhM_h. The three arguments below say why the approximation is nonetheless well founded.

Mode selection is position-independent. Writing RsTRt=RDR_s^{\mathsf{T}}R_t=R_D with D=t−sD=t-s, the RoPE logit is xsTM(D)xtx_s^{\mathsf{T}}M^{(D)}x_t with M(D)=Wq,hRDWk,hTM^{(D)}=W_{q,h}R_DW_{k,h}^{\mathsf{T}}, so the effective interaction is a relative-position family and MhM_h is its D=0D=0 member. But RDR_D is orthogonal and acts inside the head space, after the projection. Hence for every DD the column space of M(D)M^{(D)} lies in that of Wq,hW_{q,h}, its row space in that of Wk,hW_{k,h}, and rank⁡(M(D))≤dh\operatorname{rank}(M^{(D)}) \le d_h. RoPE re-weights directions the head already reads; it adds no rank and cannot make the head read an input direction that Wq,hW_{q,h} or Wk,hW_{k,h} does not span. Which input directions a head reads is what our diagnostic measures, and the activation variances behind it are computed on the layer input XX, which RoPE never touches, so the selection criterion is position-independent by construction.

The truncation error is bounded uniformly in position. Let qs∗q_s^* and kt∗k_t^* be the queries and keys produced by the truncated projections. The logit error splits into two cross terms, each a product of a truncation error on one side and a rotated but not rescaled vector on the other. Because RDR_D preserves length, Cauchy–Schwarz gives

∣q~sTk~t−(qs∗)Tkt∗∣≤∥qs−qs∗∥∥kt∥+∥qs∗∥∥kt−kt∗∥,(16)\left|\tilde{q}_s^{\mathsf{T}}\tilde{k}_t-\left(q_s^*\right)^{\mathsf{T}}k_t^*\right| \le\lVert q_s-q_s^*\rVert\lVert k_t\rVert+\lVert q_s^*\rVert\lVert k_t-k_t^*\rVert, \tag*{(16)}

which contains no DD. A rotation changes the orientation of the error but not its magnitude, so the failure mode a nontrivial interaction would produce, an error that is small at short range and grows with distance, is precisely what the isometry precludes. This relies on the same RDR_D acting on the exact and on the truncated queries and keys, which holds because RoPE is applied in the original head coordinates.

Changes for a discarded mode. What RoPE does change is the fate of a discarded mode. If every token projects to nearly the same value along mode ii, its contribution to the logit is approximately σici(U)ci(V)(uiTRDvi)\sigma_i c_i^{(U)}c_i^{(V)}(u_i^{\mathsf{T}}R_Dv_i). This varies with DD but is identical for every token pair at that offset. Without RoPE it is a constant that softmax absorbs exactly; with RoPE it becomes a fixed positional bias that varies with distance but carries no information about which tokens sit at those positions. Discarding such a mode therefore removes at most a positional bias, never content-dependent signal. This is weaker than exact absorption, and we state it as such.

Empirical check. In Tab. 3, truncating the flagged Llama-2-7B layers {8,…,11}\{8,\ldots,11\} keeps WikiText-2 perplexity within 0.11 of baseline (5.58 against 5.47) and MMLU within 0.65 points (44.65 against 45.30), while truncating the non-flagged layers {4,…,7}\{4,\ldots,7\} costs 2.64 perplexity (8.11) and 4.45 MMLU (40.85). A proxy breaking down under rotation would degrade both; instead the diagnostic separates them cleanly.

Grouped-query attention. Llama-2-7B uses standard multi-head attention. For grouped-query attention (GQA), where multiple query heads share a key/value head, a RoPE-aware extension would also need to account for the shared key projection across the group. A fully RoPE-aware variant would further need to account for the rotary coordinate structure explicitly, for example by selecting complete rotary coordinate pairs or by decomposing the relative-position-dependent family rather than the single matrix MhM_h. We leave both to future work.

Algorithm

Algorithm 1 summarizes the full SVD-Attention procedure for one attention layer and one head. The procedure is applied independently across heads and across the layers selected by the diagnostic. The calibration pass that produces {X(c)}c=1∣C∣\{X^{(c)}\}_{c=1}^{|C|} is a single forward pass per example with hooks attached at the input of each attention layer, run once before any decomposition.

Experimental Setup

We evaluate on seven pretrained transformers across five task types: DistilBERT (Sanh et al., 2019) on SST-5 (Socher et al., 2013) for sentiment classification (Accuracy); RoBERTa-base (Liu et al., [14]) and MiniLM (Wang et al., 2020b) on SQuAD 2.0 (Rajpurkar et al., [19]) and NewsQA (Trischler et al., 2017) for extractive question answering (F1); Llama-2-7B (Touvron et al., 2023) on TruthfulQA (Lin et al., [13]) (ROUGE-L), MMLU (Hendrycks et al., [11]) (5-shot Accuracy) and WikiText-2 (Merity et al., [17]) (perplexity) for generation; Swin-B (Liu et al., [15]) and ViT-B/16 (Dosovitskiy et al., [7]) on ImageNet-1K (Deng et al., [6]) for image classification at 224×224224 \times224 (Top-1); and DPT-Large (Ranftl et al., [20]) on ADE20K (Zhou et al., 2017) for semantic segmentation (mIoU). All models are public checkpoints; we apply no finetuning or architectural changes.

Calibration. We draw 20000, 2000, 200, and 50 examples from the training split/benchmarks of ImageNet-1K, SST-5, TruthfulQA, and ADE20K Algorithm 1 SVD-ATTENTION for one head

Require: Pretrained projections Wq,h,Wk,h∈Rd×dhW_{q,h}, W_{k,h} \in\mathbb{R}^{d \times d_h}; calibration inputs {X(c)}c=1∣C∣\{X^{(c)}\}_{c=1}^{|C|} at this layer; threshold ϵ\epsilon

Ensure: Replacement projections Wq,h∗,Wk,h∗∈Rd×rhW^{*}_{q,h}, W^{*}_{k,h} \in\mathbb{R}^{d \times r_h}

  1. Mh←Wq,hWk,hTM_h \leftarrow W_{q,h}W_{k,h}^{\mathsf{T}} ▷ interaction matrix

  2. Uh,Σh,VhT←SVD⁡(Mh)U_h, \Sigma_h, V_h^{\mathsf{T}} \leftarrow\operatorname{SVD}(M_h) ▷ Σh=diag⁡(σ1,…,σd)\Sigma_h = \operatorname{diag}(\sigma_1,\ldots,\sigma_d)

  3. for c=1,…,∣C∣c = 1,\ldots,|C| do

  4. X^U(c)←X(c)Uh\hat{X}^{(c)}_U \leftarrow X^{(c)}U_h

  5. X^V(c)←X(c)Vh\hat{X}^{(c)}_V \leftarrow X^{(c)}V_h

  6. end for

  7. for i=1,…,di = 1,\ldots,d do

  8. vi(U)←1∣C∣∑cVar⁡t[X^U,:,i(c)]v_i^{(U)} \leftarrow\frac{1}{|C|}\sum_c \operatorname{Var}_t[\hat{X}^{(c)}_{U,:,i}]

  9. vi(V)←1∣C∣∑cVar⁡t[X^V,:,i(c)]v_i^{(V)} \leftarrow\frac{1}{|C|}\sum_c \operatorname{Var}_t[\hat{X}^{(c)}_{V,:,i}]

  10. Ii←σi⋅vi(U)vi(V)\mathcal{I}_i \leftarrow\sigma_i \cdot\sqrt{v_i^{(U)}v_i^{(V)}}

  11. end for

  12. Sort modes by Ii\mathcal{I}_i in descending order, giving permutation π\pi

  13. rh←min⁡{r:∑i≤rIπ(i)∑iIπ(i)≥1−ϵ}r_h \leftarrow\min\left\{r : \frac{\sum_{i \leq r}\mathcal{I}_{\pi(i)}}{\sum_i \mathcal{I}_{\pi(i)}} \geq1-\epsilon\right\}

  14. Uh(r)←[Uh,:,π(1)⋯Uh,:,π(rh)]U_h^{(r)} \leftarrow[U_{h,:, \pi(1)} \cdots U_{h,:, \pi(r_h)}]

  15. Vh(r)←[Vh,:,π(1)⋯Vh,:,π(rh)]V_h^{(r)} \leftarrow[V_{h,:, \pi(1)} \cdots V_{h,:, \pi(r_h)}]

  16. σ(r)←(σπ(1),…,σπ(rh))\sigma^{(r)} \leftarrow(\sigma_{\pi(1)},\ldots,\sigma_{\pi(r_h)})

  17. Wq,h∗←Uh(r)diag⁡(σ(r))W^{*}_{q,h} \leftarrow U_h^{(r)}\operatorname{diag}(\sqrt{\sigma^{(r)}})

  18. Wk,h∗←Vh(r)diag⁡(σ(r))W^{*}_{k,h} \leftarrow V_h^{(r)}\operatorname{diag}(\sqrt{\sigma^{(r)}})

  19. return Wq,h∗,Wk,h∗W^{*}_{q,h}, W^{*}_{k,h}

respectively, and compute the per-mode variances in one forward pass per example with hooks at each attention layer. For the extractive QA models we calibrate on 1,303 SQuAD 2.0 examples (1% of train) unless stated otherwise; § 5.3 shows the layer-level answer is unchanged from 100 to 19,548 examples.

Knobs and baselines. The diagnostic exposes an energy threshold ϵ\epsilon and a set of target layers; we sweep ϵ∈{0.1,…,0.6}\epsilon\in\{0.1,\ldots,0.6\} and report representative configurations (layer-wise sweep in § 5.2). We compare against the unmodified model, against Sun et al. (2024) at sparsity 0.1, and against our method composed with WANDA on top, using the same calibration set throughout.

Metrics. We report task quality and the change in inference FLOPs relative to the unmodified model, computed from the per-head retained rank rhr_h and post-WANDA sparsity.

Methodϵ=0.1\epsilon= 0.1ϵ=0.2\epsilon= 0.2ϵ=0.3\epsilon= 0.3
Singular value only77.8477.1376.12
SVD-Attention (data-aware)77.9477.6276.86
Gap+0.10+0.49+0.74
DatasetLayersrhr_h (sv)F1 (sv)rhr_h (ours)F1 (ours)
SQuAD 2.0{8,…,11}\{8,\ldots,11\}38.663.0829.082.85
NewsQA{8,…,11}\{8,\ldots,11\}38.659.3327.069.06
NewsQA{4,…,7}\{4,\ldots,7\}38.520.7328.859.77

Table 1. Ablation of the ranking rule, holding the unit of decomposition and the layer selection fixed. (a) On ViT-B the data-aware score wins at every ϵ\epsilon and the gap widens as more modes are discarded. (b) On RoBERTa-base the singular-value-only rule keeps roughly ten more modes per head and still loses up to 39 F1.

Results

Answering the Research Questions

RQ1: Does data add information beyond singular value? To isolate the variance term, we compare our ranking against ranking by σi\sigma_i alone at three values of ϵ\epsilon (Tab. 1a). The data-aware score wins at every ϵ\epsilon, and the gap widens monotonically: +0.10 Top-1 at ϵ=0.1\epsilon= 0.1, +0.49 at ϵ=0.2\epsilon= 0.2, and +0.74 at ϵ=0.3\epsilon= 0.3. This is the expected signature of a useful data-aware term: at small ϵ\epsilon few modes are discarded and the two rankings nearly agree, but as more are removed the singular-value-only ranking starts discarding modes that real data uses. The effect is far larger on RoBERTa-base at ϵ=0.3\epsilon= 0.3 (Tab. 1b), where that rule keeps roughly ten more modes per head and still loses up to 39 F1: on NewsQA layers {4,…,7}\{4,\ldots,7\} it scores 20.73 against our 59.77. A mode can therefore be spectrally large and contribute only a near-constant offset, which is precisely the mode a reconstruction-based objective would retain and ours discards.

RQ2: Where does redundancy live, and does it depend on the task? The answer depends on the task type. For the three classification models (DistilBERT, ViT-B, Swin-B) and for generation on Llama-2-7B, the diagnostic flags the early to middle layers as the most compressible, though not the very earliest: DistilBERT at {0,1,2}\{0,1,2\}, Swin-B at {4,…,7}\{4,\ldots,7\}, ViT-B at {6,7,8}\{6,7,8\}, and Llama-2-7B at {8,…,11}\{8,\ldots,11\} (Tab. 2). For dense prediction (DPT-Large on ADE20K) the pattern shifts to the late layers {16,…,19}\{16,\ldots,19\}, where truncation preserves mIoU exactly. Extractive QA joins dense prediction rather than classification: the diagnostic

ModelLayersϵ\epsilonMetricΔ\Delta FLOPs
DistilBERT on SST-5 (Accuracy %)
Baseline——49.95—
WANDA-only——49.82-9.65%
Ours{0,1,2}\{0,1,2\}0.150.95-0.83%
Ours + WANDA{0,1,2}\{0,1,2\}0.151.09-10.39%
RoBERTa-base on NewsQA (F1)
Baseline——70.81—
Ours{8,…,11}\{8,\ldots,11\}0.170.74-1.47%
Ours{8,…,11}\{8,\ldots,11\}0.369.06-3.63%
Ours + WANDA{8,…,11}\{8,\ldots,11\}0.170.61-10.33%
Ours + WANDA{8,…,11}\{8,\ldots,11\}0.369.04-12.32%
Llama-2-7B on TruthfulQA (ROUGE-L)
Baseline——0.5362—
WANDA-only——0.5359-9.55%
Ours{8,…,11}\{8,\ldots,11\}0.10.4952-1.13%
Ours + WANDA{8,…,11}\{8,\ldots,11\}0.10.4948-10.57%
ViT-B on ImageNet-1K (Top-1 %)
Baseline——81.08—
WANDA-only——81.02-9.45%
Ours{6,7,8}\{6,7,8\}0.177.94-0.76%
Ours + WANDA{6,7,8}\{6,7,8\}0.177.94-10.13%
Swin-B on ImageNet-1K (Top-1 %)
Baseline——84.71—
WANDA-only——84.67-9.49%
Ours{4,…,7}\{4,\ldots,7\}0.184.68-1.29%
Ours + WANDA{4,…,7}\{4,\ldots,7\}0.184.67-10.65%
DPT-Large on ADE20K (mIoU)
Baseline——45.99—
WANDA-only——46.02-6.98%
Ours{16,…,19}\{16,\ldots,19\}0.146.02-1.54%
Ours + WANDA{16,…,19}\{16,\ldots,19\}0.146.02-8.40%

caption: Main results across seven pretrained transformers and five task types; settings whose metrics do not fit this format are reported in Tab. 3. Ours applies SVD-Attention (data-aware SVD truncation); Ours + WANDA applies WANDA pruning at sparsity 0.1 on top. Metric measures task quality in units written in bold. Δ\Delta FLOPs is the % reduction in inference FLOPs relative to baseline.

:::

flags RoBERTa-base at {8,…,11}\{8,\ldots,11\} of 12 layers and MiniLM at {9,10,11}\{9,10,11\}, replicating across two architectures and two datasets (Tab. 3a). A single reading covers both reversals. Dense prediction and span extraction need token-level detail preserved through the early and middle blocks, so heads there use more of their rank budget and the redundancy shifts late; in sentiment and image classification and in generation the early and middle layers do generic token-level processing and the heavy task-specific work is concentrated in the late blocks.

RQ3: Is the diagnostic predictive? Truncating the diagnostic-selected layers preserves task quality across all seven models (Tab. 2, Tab. 3). The clearest evidence comes from the QA models, where the metric has the most headroom: at ϵ=0.3\epsilon= 0.3, truncating the flagged layers costs at most 1.75 F1 in every setting, while truncating the same number of layers elsewhere in the same model costs up to 50 F1

Model / datasetBaselineFlaggedNon-flagged
RoBERTa / SQuAD 2.083.0182.85 {8,…,11}\{8,\ldots,11\}52.23 {4,…,7}\{4,\ldots,7\}
RoBERTa / NewsQA70.8169.06 {8,…,11}\{8,\ldots,11\}58.85 {0,…,3}\{0,\ldots,3\}
MiniLM / SQuAD 2.077.7777.12 {6,…,8}\{6,\ldots,8\}69.98 {6,…,8}\{6,\ldots,8\}
MiniLM / NewsQA56.0355.58 {9,…,11}\{9,\ldots,11\}5.94 {3,…,5}\{3,\ldots,5\}

Table a. Extractive QA at ϵ=0.3\epsilon= 0.3 (F1)

MetricBaselineFlagged {8,…,11}\{8,\ldots,11\}Non-flagged {4,…,7}\{4,\ldots,7\}
MMLU 5-shot (%)45.3044.6540.86
WikiText-2 PPL (↓\downarrow)5.475.588.11

Table b. Llama-2-7B at ϵ=0.1\epsilon= 0.1

Table 3: Flagged against non-flagged layer groups at matched ϵ\epsilon, for the settings whose metrics do not fit the format of Tab. 2. Flagged is the layer group the diagnostic selects; Non-flagged truncates the same number of layers elsewhere in the same model. Layer sets in braces.

(MiniLM on NewsQA, 55.58 against 5.94). Llama-2-7B separates the same way on probes denser than ROUGE-L (Tab. 3b): at the flagged layers MMLU falls 0.65 point and perplexity rises 0.11, while the non-flagged layers cost 4.45 MMLU and 2.64 perplexity. Effects are smaller on the remaining models: Swin-B drops 0.03 Top-1 at −1.29%-1.29\% FLOPs, ViT-B drops 3.14 Top-1 at −0.76%-0.76\%, and DPT-Large is unchanged in mIoU at −1.54%-1.54\%. DistilBERT scores +1.00 accuracy above baseline at −0.83%-0.83\% FLOPs, but we do not read this as an improvement: differences of this size are small for a five-class metric and Table 5 in § 5.2 shows several layer choices above baseline, so we claim only that quality is preserved. The metric does detect damage when there is damage: sweeping ϵ\epsilon from 0.1 to 1.0 on DistilBERT layers {0,…,5}\{0,\ldots,5\}, accuracy falls monotonically from 50.5 to 38.2. Composing with WANDA at 10% sparsity adds roughly nine percentage points of FLOP reduction while changing quality by at most 0.13 F1 on RoBERTa-base/NewsQA. Most of the composition figure is WANDA’s; the point of the composition is that the two methods remove different weights, so their savings add rather than compete. The diagnostic also predicts where compression fails (§ 5.2), its layer-level answer is stable under distribution shift, calibration draw, and calibration size (§ 5.3), and within the flagged layers the quality-versus-ϵ\epsilon frontier is smooth rather than a cliff (§ 5.4).

LayersAcc. (%)SVD Δ\Delta FLOPs+ WANDA
{0,1}\{0,1\}50.63−1.79%-1.79\%−11.26%-11.26\%
{2,3}\{2,3\}49.64−1.53%-1.53\%−11.03%-11.03\%
{0,1,2}\{0,1,2\}50.95−0.83%-0.83\%−10.39%-10.39\%
{3,4,5}\{3,4,5\}50.62−1.57%-1.57\%−11.06%-11.06\%
{0,…,5}\{0,\ldots,5\}50.50−4.15%-4.15\%−13.38%-13.38\%

Table 5. DistilBERT layer-wise sweep on SST-5 at ϵ=0.1\epsilon= 0.1. Several layer choices land slightly above the baseline (49.95), including the flagged group {0,1,2}\{0,1,2\}; we read this as quality being preserved rather than improved.

Layer-Wise Sweep

Tabs. 4, 5, 6, 7 report the layer-wise sweep at ϵ=0.1\epsilon= 0.1 for four of the seven models. These results back the layer-selection claims in § 5.1: the configuration the diagnostic flags as most redundant (bold row) gives the best accuracy-FLOPs tradeoff, and truncating in the wrong layers degrades performance sharply.

LayersTop-1 (%)SVD Δ\Delta FLOPs+ WANDA
{0,1,2}\{0,1,2\}77.22−2.47%-2.47\%−11.67%-11.67\%
{3,4,5}\{3,4,5\}77.92−1.09%-1.09\%−10.43%-10.43\%
{6,7,8}\{6,7,8\}77.94−0.76%-0.76\%−10.13%-10.13\%
{9,10,11}\{9,10,11\}71.63−0.62%-0.62\%−10.00%-10.00\%
{0,…,5}\{0,\ldots,5\}71.07−3.56%-3.56\%−12.66%-12.66\%
{6,…,11}\{6,\ldots,11\}67.60−1.38%-1.38\%−10.69%-10.69\%
{0,…,11}\{0,\ldots,11\}47.89−4.94%-4.94\%−13.90%-13.90\%

Table 4. ViT-B layer-wise sweep on ImageNet-1K at ϵ=0.1\epsilon= 0.1. Truncating middle layers {6,7,8}\{6,7,8\} preserves Top-1 within 3.14 points of the baseline (81.08). Pushing truncation to layers {9,10,11}\{9,10,11\} or wider drops accuracy sharply.

LayersTop-1 (%)SVD Δ\Delta FLOPs+ WANDA
{0,…,3}\{0,\ldots,3\}84.68−1.70%-1.70\%−11.06%-11.06\%
{4,…,7}\{4,\ldots,7\}84.68−1.29%-1.29\%−10.65%-10.65\%
{8,…,11}\{8,\ldots,11\}84.64−0.62%-0.62\%−10.05%-10.05\%
{12,…,15}\{12,\ldots,15\}84.24−0.58%-0.58\%−10.01%-10.01\%
{16,…,19}\{16,\ldots,19\}83.41−0.50%-0.50\%−9.94%-9.94\%
{20,…,23}\{20,\ldots,23\}80.88−0.54%-0.54\%−9.98%-9.98\%
all 24 layers76.33−5.22%-5.22\%−14.26%-14.26\%

Table 6. Swin-B layer-wise sweep on ImageNet-1K at ϵ=0.1\epsilon= 0.1. Accuracy is well preserved in the first three blocks (layers 0-11) and degrades monotonically as truncation moves to later blocks.

Robustness of the Diagnostic

Our pipeline is deterministic. We use public pretrained checkpoints, we do no training or finetuning, and we run only forward passes for calibration and evaluation, so no random seed enters the decomposition, the mode selection, or the metric, and repeated runs are bit-identical. The one real source of variation is which examples land in the calibration set. We separate the three ways a calibration set can vary and test each: the distribution it is drawn from, which examples happen to be drawn, and how many.

LayersmIoUSVD Δ\Delta FLOPs+ WANDA
{0,…,3}\{0,\ldots,3\}44.29−2.00%-2.00\%−8.82%-8.82\%
{4,…,7}\{4,\ldots,7\}45.38−1.55%-1.55\%−8.40%-8.40\%
{8,…,11}\{8,\ldots,11\}45.73−1.52%-1.52\%−8.37%-8.37\%
{12,…,15}\{12,\ldots,15\}45.95−1.70%-1.70\%−8.54%-8.54\%
{16,…,19}\{16,\ldots,19\}46.02−1.54%-1.54\%−8.40%-8.40\%
{20,…,23}\{20,\ldots,23\}45.99−1.01%-1.01\%−7.91%-7.91\%
all 24 layers43.32−9.31%-9.31\%−15.56%-15.56\%

Table 7. DPT-Large layer-wise sweep on ADE20K at ϵ=0.1\epsilon= 0.1. mIoU rises monotonically as the truncation window moves later, the opposite of ViT-B and Swin-B; late layers {16,…,19}\{16,\ldots,19\} preserve baseline mIoU.

Different distribution. We take RoBERTa-base finetuned on SQuAD 2.0, calibrate the diagnostic only on SQuAD 2.0 data, and evaluate on NewsQA (baseline F1 57.95 in this shifted setting). We repeat the experiment on MiniLM at ϵ=0.6\epsilon= 0.6, where truncation is most destructive and instability would be easiest to see (baseline F1 48.39). Table 8 shows the layers flagged redundant stay redundant under shift and the layers flagged critical stay critical.

RoBERTa-base, ϵ=0.3\epsilon= 0.3 (F1)
Layers truncatedcalib 1%calib 15%
{8,…,11}\{8,\ldots,11\} (flagged redundant)56.2556.03
{0,…,3}\{0,\ldots,3\}51.5553.34
{4,…,7}\{4,\ldots,7\} (flagged critical)25.8725.49
MiniLM, ϵ=0.6\epsilon= 0.6 (F1)
{9,10,11}\{9,10,11\} (flagged redundant)37.45
{3,4,5}\{3,4,5\}15.88
{6,7,8}\{6,7,8\} (flagged critical)5.73

Table 8. Robustness to distribution shift. The diagnostic is calibrated on SQuAD 2.0 and evaluated on NewsQA. On MiniLM the flagged layers hold up even when 60% of the importance mass is discarded, while truncating the critical layers {6,7,8}\{6,7,8\} costs over 42 F1.

Different examples, same size. We rebuilt the calibration set five times with different random draws of the same size on DistilBERT/SST-5 and reran the whole diagnostic each time. Tab. 9 reports ϵ=0.9\epsilon= 0.9, the most aggressive truncation we ran, because that is where instability would be easiest to see. The spread across draws is under 0.2 accuracy and the ranking of layer groups is identical in all five draws. The same holds at every ϵ\epsilon from 0.5 to 0.9, where the largest standard deviation anywhere in the sweep is 0.39.

Layers truncatedAccuracy (mean ±\pm std)Min-Max
{0,1}\{0,1\} (flagged)45.73 ±\pm 0.0645.70–45.84
{2,3}\{2,3\}44.49 ±\pm 0.1644.21–44.57
{4,5}\{4,5\}44.25 ±\pm 0.1244.07–44.39

Table 9. Robustness to the calibration draw. DistilBERT on SST-5 at ϵ=0.9\epsilon= 0.9, mean over five independent draws of the same size.

Calibration size. We swept the calibration budget on RoBERTa-base/SQuAD 2.0 across more than two orders of magnitude at ϵ=0.3\epsilon= 0.3 (Table 10). The flagged layers vary by only 0.25 F1 as the calibration set grows from 100 to 19,548 examples, and the ordering of the layer groups is identical at every size. The same holds on MiniLM under distribution shift, where at every ϵ\epsilon from 0.2 to 0.6 all four layer groups return the same F1 whether we calibrate with 1, 16, 64, or 256 examples. A very small calibration budget therefore recovers the same mode selection, which also means the diagnostic is cheap to run.

Calib. size (share){0,…,3}\{0,\ldots,3\}{4,…,7}\{4,\ldots,7\}{8,…,11}\{8,\ldots,11\}
100 (0.08%)76.8756.9583.10
1,303 (1%)77.0652.2382.85
6,516 (5%)75.2351.7682.86
13,032 (10%)75.4153.5182.97
19,548 (15%)75.5853.5782.95

Table 10. Robustness to calibration size. RoBERTa-base on SQuAD 2.0 at ϵ=0.3\epsilon= 0.3 (F1, baseline 83.01). Bold column is the layer group the diagnostic flags as redundant.

Rank-Energy Frontier in the Flagged Layers

The energy threshold ϵ\epsilon controls how much of the importance mass is discarded, and therefore how many modes each head keeps. Table 11 sweeps it on the layers the diagnostic flags in RoBERTa-base/SQuAD 2.0. The mean retained rank falls from 49 to 38 to 29 of 64 as ϵ\epsilon goes from 0.1 to 0.3, while F1 moves from 83.24 to 82.85 against a baseline of 83.01; discarding more than half of the average head’s rank budget therefore costs 0.16 F1.

ϵ=0.1\epsilon= 0.1ϵ=0.2\epsilon= 0.2ϵ=0.3\epsilon= 0.3
F183.2483.0982.85
Mean retained rank rhr_h493829

Table 11. RoBERTa-base on SQuAD 2.0, flagged layers {8,…,11}\{8,\ldots,11\} (baseline F1 83.01, dh=64d_h = 64). The quality-versus-ϵ\epsilon frontier is smooth: more than half of the average head’s rank budget is discarded at ϵ=0.3\epsilon= 0.3 for a loss of 0.16 F1.

Two things follow. First, the redundancy the diagnostic reports is large in absolute terms rather than a marginal trim at the tail of the spectrum, which is what the small whole-model FLOP percentages in Tab. 2 might otherwise suggest. Second, ϵ\epsilon behaves as a usable dial rather than a threshold that has to be tuned precisely: within the flagged layers quality degrades gradually, so it is the layer selection rather than the exact energy level that determines whether truncation is safe. The contrast with Tab. 3, where the same ϵ\epsilon applied to non-flagged layers costs up to 50 F1, is the point.

Conclusion

We presented SVD-Attention, a data-aware spectral diagnostic that scores each singular mode of the per-head interaction matrix MhM_h by its singular value and the geometric mean of query- and key-side activation variances, and discards the rest without finetuning. Two findings emerge. First, the layer at which redundancy concentrates depends on task type: early to middle for sentiment and image classification and for generation, late for dense prediction and extractive question answering. Second, acting on the diagnostic yields measurable FLOP reductions across all seven models we tested and composes with weight-level pruning such as WANDA. The diagnostic also predicts where compression fails, positioning SVD-Attention as a measurement tool first and a compression tool second.

Limitations

We acknowledge several limitations. First, our compression gains are modest in absolute terms compared with token-level methods, primarily because we compress only multi-head self-attention. Our contribution is the diagnostic and its no-finetuning property, not a state-of-the-art compression ratio. Second, we report analytic FLOPs. Turning them into wall-clock speedup requires an attention kernel at the reduced per-head dimension, since padding the discarded dimensions back erases the saving. Third, under RoPE the query-key interaction is a relative-position family rather than a single matrix. We argue in § 3.7 that mode selection is nonetheless position-independent and that the truncation error is bounded uniformly in position, but we do not claim an exact factorization of the full family and we do not treat grouped-query attention. Finally, we do not evaluate interactions with quantization or with attention-pattern modifications such as linear attention; we expect the gains to compose, but this has not been verified empirically.

Acknowledgments

This work was conducted in part during an internship at Dolby Laboratories. This work was also supported in part by CoCoSys, one of seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA.

References

  1. [1]Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.arxiv.org/abs/2004.05150
  2. [2]Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. 2020. Low-rank bottleneck in multi-head attention models. In International Conference on Machine Learning (ICML).arxiv.org/abs/2002.07028
  3. [3]Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2022. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461.arxiv.org/abs/2210.09461
  4. [4]Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. 2021. Drone: Data-aware low-rank compression for large nlp models. In Advances in Neural Information Processing Systems (NeurIPS).
  5. [5]Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andrea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, and 1 others. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794.arxiv.org/abs/2009.14794
  6. [6]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  7. [7]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR).
  8. [8]Carl Eckart and Gale Young. 1936. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211–218.DOI
  9. [9]Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, and 1 others. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12.
  10. [10]Qihang Fan, Huaibo Huang, and Ran He. 2025. Breaking the low-rank dilemma of linear attention. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).arxiv.org/abs/2411.07635
  11. [11]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR).
  12. [12]Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2207.00112
  13. [13]Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252.arxiv.org/abs/2109.07958
  14. [14]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.arxiv.org/abs/1907.11692
  15. [15]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE International Conference on Computer Vision (ICCV).arxiv.org/abs/2103.14030
  16. [16]Weikang Meng, Yadan Luo, Xin Li, Dongmei Jiang, and Zheng Zhang. 2025. Polaformer: Polarity-aware linear attention for vision transformers. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2501.15061
  17. [17]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR).arxiv.org/abs/1609.07843
  18. [18]Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/1905.10650
  19. [19]Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Annual Meeting of the Association for Computational Linguistics (ACL).arxiv.org/abs/1806.03822
  20. [20]René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021. Vision transformers for dense prediction. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12159–12168. IEEE.arxiv.org/abs/2103.13413

Paper details

vlnc:W_jrkkme7q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents