Introduction
Fine-tuning large pretrained models on downstream tasks is effective but expensive: updating all parameters of a billion-scale model requires substantial memory and compute. Parameter-efficient fine-tuning (PEFT) solves this problem by adapting a large pretrained model while updating only a small subset of parameters. Some of the representative approaches include adapter modules Houlsby et al. [4 ] , prefix tuning Li and Liang (2021), prompt tuning, and low-rank adaptation (LoRA, Hu et al. 2022). Among these, LoRA stands out as an attractive option because it keeps the pretrained backbone frozen and modifies selected linear maps by adding trainable low-rank updates. More formally, for a weight matrix W 0 ∈ R m × n W_0 \in\mathbb{R}^{m \times n} W 0 ∈ R m × n , LoRA parameterizes the adapted weight as
W = W 0 + T , T = U V T , U ∈ R m × r , V ∈ R n × r , W = W_0 + T,\qquad T = UV^{\mathsf{T}},\qquad U \in\mathbb{R}^{m \times r}, V \in\mathbb{R}^{n \times r}, W = W 0 + T , T = U V T , U ∈ R m × r , V ∈ R n × r ,
The rank r r r is typically chosen by the practitioner as a hyperparameter rather than something derived from the fine-tuning objective itself (Hu et al. 2022; Schulman and Lab [18 ] ). A fundamental theoretical question is therefore: When does the regularized fine-tuning objective itself admit a low-rank global minimizer, and what controls the rank?
Most existing theory addresses nearby questions through linearization or simplified training dynamics. In the NTK regime, Jang et al. [8 ] shows that full fine-tuning admits a low-rank solution and that sufficiently overparameterized LoRA eliminates spurious local minima. Afzal et al. [1 ] analyze fine-tuning through linearization under an explicit ℓ 2 \ell_2 ℓ 2 bias toward the pretrained model. Beyond classical linearization, Kim et al. [9 ] study LoRA dynamics and prove a dichotomy between convergence to a low-rank global minimizer and convergence to a qualitatively different high-rank solution. These works are complementary to ours, but they do not answer a more structural question: what property of the objective function makes low-rank optima available in the first place?
Contributions. Our answer starts with a simple observation: once the backbone is frozen, the loss does not observe an adapted matrix T T T directly, but only through a finite-rank linear map determined by the frozen activations. Concretely, if H = [ h 1 , … , h N ] H = [h_1,\ldots,h_N] H = [ h 1 , … , h N ] is the matrix of frozen input features across the N N N training samples, the empirical loss depends on T T T only through the product T H TH T H . We call rank ( Φ ) \operatorname{rank}(\Phi) rank ( Φ ) , where Φ ( T ) = T H \Phi(T)=TH Φ ( T ) = T H , the bottleneck rank of the adapted block. The same principle applies blockwise to multiple adapted stages and attention heads. Trace-norm regularization then selects a low-rank representative among all matrices that induce the same observed output. This yields exact low-rank existence results without linearizing the network around the pretrained weights. For multi-head attention (Vaswani et al. [19 ] ), the same principle becomes sharper because attention introduces additional invariances: row-wise softmax is unchanged by additive row constants in the score matrix, and value updates are observable only after attention weighting and the frozen output projection.
Table 1. Comparison with the closest related theoretical analyses of LoRA.
Our contribution is summarized as follows.
Low-rank representer theorem for trace-norm fine-tuning. For a frozen network in which an adapted block T T T is observed only through a finite-rank linear map Φ ( T ) \Phi(T) Φ ( T ) of bottleneck rank d Φ d_{\Phi} d Φ , we prove that the trace-norm-regularized problem has a global minimizer whose rank is controlled by d Φ d_{\Phi} d Φ ; see Theorem 4.1 . The multi-stage extension is Theorem 4.3 .
Rank bounds from attention invariances. For multi-head attention, query and key updates are observable only through centered score subspaces because row-wise softmax is invariant to row-wise additive constants. Value updates are observable only after attention weighting and frozen output projection. Combining these invariances with projector compression gives the optimizer-dependent subspace rank bounds in Theorem 4.4 .
Generic first-order rank deficiency for factorized LoRA. For the weight-decayed LoRA factorization, we prove that after a generic positive semidefinite perturbation, full-column-rank first-order stationary points are excluded whenever the Pataki–Barvinok rank threshold exceeds the dimension of the associated family of symmetric KKT matrices; see Theorem 4.7 . The multistage and attention consequences are Theorem C.9 and Corollary 4.8. This is a first-order exclusion theorem, not a no-spurious-local-minima theorem; Proposition 4.9 gives an explicit limitation.
Linearized and kernel views of fine-tuning. The closest prior work to ours is Jang et al. [8 ] , which studies LoRA in the NTK regime. Using K o u t K_{\mathrm{out}} K out for the number of output coordinates in the linearized training predictions, their low-rank existence theorem gives a solution satisfying r ( r + 1 ) 2 ≤ K o u t N \frac{r(r+1)}{2} \le K_{\mathrm{out}}N 2 r ( r + 1 ) ≤ K out N . Thus the rank scale in that analysis is r = ⌊ 8 K o u t N + 1 − 1 2 ⌋ r = \left\lfloor\frac{\sqrt{8K_{\mathrm{out}}N+1}-1}{2} \right\rfloor r = ⌊ 2 8 K out N + 1 − 1 ⌋ , or r = O ( K o u t N ) r = O(\sqrt{K_{\mathrm{out}}N}) r = O ( K out N ) in the scalar-output convention. Our results improve this comparison in two ways. First, our bottleneck reduction is exact for frozen upstream computation and does not linearize the network around the pretrained weights. Second, the optimizer’s rank scale is r ≤ min { K o u t , N } r \le\min\{K_{\mathrm{out}},N\} r ≤ min { K out , N } in Attention, and this is better than Jang et al. [8 ] ’s bound. Afzal et al. [1 ] also analyzed fine-tuning through linearization under an explicit ℓ 2 \ell_2 ℓ 2 bias toward the pretrained model and quantified when nonlinear fine-tuning stays close to the linearized problem. In contrast, our reduction is exact once upstream computation is frozen; no approximation-by-linearization argument is needed.
Optimization theory beyond linearization. Recent works also study LoRA outside the lazy regime. Kim et al. [9 ] analyzes the LoRA loss landscape and proves that training converges either to a low-rank global minimizer or to a qualitatively different high-rank solution. Xu et al. (2025) study gradient flow for LoRA in matrix factorization and show how initialization and singular-space alignment shape the final error. Liu et al. [13 ] compares the landscapes of LoRA, GaLore, and full-rank adaptation through Polyak–Łojasiewicz-type regions and spurious local minima. Zhang and Pilanci [25 ] analyzes a Riemannian preconditioner for LoRA in the infinite-width setting. These works ask how LoRA training behaves. Our question comes earlier: before one analyzes dynamics, when does the frozen regularized objective itself admit low-rank optima, and when must first-order stationary points of the factored problem be rank-deficient?
Attention and Transformer expressivity. Since transformers are built around attention (Vaswani et al. [19 ] ), we instantiate our framework for Q / K / V Q/K/V Q / K / V projections in attention modules. Orthogonally, a rich literature studies the expressive power and limitations of self-attention and transformers [23 , 12 , 3 , 26 ] . Complementary work studies what functions low-rank adaptation can represent; Zeng and Lee [24 ] characterized the expressive power of low-rank adaptation in several settings. For Transformer architectures, Hu et al. (2025) studied computational limitations of LoRA fine-tuning.
Problem setup
We study fine-tuning of a pretrained network in which all weights are frozen except for a specified collection of linear maps. The full notation tables are collected in Appendix § 8 ; here we introduce only what is needed to state the main results.
Single adapted block. For a frozen weight matrix W 0 ∈ R m × n W_0 \in\mathbb{R}^{m \times n} W 0 ∈ R m × n , we write the adapted weight as W = W 0 + T W = W_0 + T W = W 0 + T with update T ∈ R m × n T \in\mathbb{R}^{m \times n} T ∈ R m × n . Given training data S d a t a = { ( x i , y i ) } i = 1 N S_{\mathrm{data}} = \{(x_i,y_i)\}_{i=1}^{N} S data = {( x i , y i ) } i = 1 N , let h i ∈ R n h_i \in\mathbb{R}^{n} h i ∈ R n denote the frozen input activation at this layer for sample x i x_i x i , and define the frozen feature matrix
H : = [ h 1 , … , h N ] ∈ R n × N . H := [h_1,\ldots,h_N] \in\mathbb{R}^{n \times N}. H := [ h 1 , … , h N ] ∈ R n × N .
When the input h i h_i h i is independent of the update T T T and all other parameters are fixed, the network output on sample x i x_i x i depends on T T T only through T h i Th_i T h i (Lemma B.1 ).
Stacking across samples, the empirical loss depends on T T T only through Φ ( T ) : = T H ∈ R m × N \Phi(T) := TH \in\mathbb{R}^{m \times N} Φ ( T ) := T H ∈ R m × N . The map Φ \Phi Φ is linear, and its range has dimension exactly m rank ( H ) ≤ m min { n , N } m \operatorname{rank}(H) \le m\min\{n,N\} m rank ( H ) ≤ m min { n , N } .
Definition 3.1 (Bottleneck rank). For a linear map Φ : R m × n → E \Phi: \mathbb{R}^{m \times n} \to\mathcal{E} Φ : R m × n → E through which the loss observes an adapted block T T T , the bottleneck rank is
d Φ : = rank ( Φ ) = dim R ( Φ ) . d_{\Phi} := \operatorname{rank}(\Phi) = \dim\mathcal{R}(\Phi). d Φ := rank ( Φ ) = dim R ( Φ ) .
When Φ ( T ) = T H \Phi(T) = TH Φ ( T ) = T H as above, we have d Φ = m rank ( H ) d_{\Phi} = m\operatorname{rank}(H) d Φ = m rank ( H ) . The bottleneck rank quantifies how much information the frozen network extracts from the update T T T : any two updates T , T ′ T,T' T , T ′ with T H = T ′ H TH = T'H T H = T ′ H produce identical training predictions.
Trace-norm regularization and the LoRA factorization. We consider the nuclear-norm-regularized (trace-norm-regularized) objective
min T ∈ R m × n L v a n i l l a ( T ) + λ ∥ T ∥ ⋆ , λ > 0 , (1) \min_{T\in\mathbb{R}^{m\times n}} \mathcal{L}^{\mathrm{vanilla}}(T) + \lambda\|T\|_{\star}, \qquad\lambda> 0, \tag*{(1)} T ∈ R m × n min L vanilla ( T ) + λ ∥ T ∥ ⋆ , λ > 0 , (1)
where L v a n i l l a ( T ) \mathcal{L}^{\mathrm{vanilla}}(T) L vanilla ( T ) is the empirical loss with update T T T and ∥ T ∥ ⋆ \|T\|_{\star} ∥ T ∥ ⋆ denotes the nuclear norm. The factorized LoRA objective with weight decay at rank r r r is
L λ l o r a ( U , V ) : = L v a n i l l a ( U V T ) + λ 2 ( ∥ U ∥ F 2 + ∥ V ∥ F 2 ) , U ∈ R m × r , V ∈ R n × r . \mathcal{L}_{\lambda}^{\mathrm{lora}}(U,V) := \mathcal{L}^{\mathrm{vanilla}}(UV^{\mathsf{T}}) + \frac{\lambda}{2}\left(\|U\|_{\mathrm{F}}^{2}+\|V\|_{\mathrm{F}}^{2}\right), \qquad U\in\mathbb{R}^{m\times r},\ V\in\mathbb{R}^{n\times r}. L λ lora ( U , V ) := L vanilla ( U V T ) + 2 λ ( ∥ U ∥ F 2 + ∥ V ∥ F 2 ) , U ∈ R m × r , V ∈ R n × r .
By the standard equivalence between nuclear-norm regularization and balanced weight decay [17 ] , the factorized problem has the same global minimum as (1) restricted to rank ≤ r \le r ≤ r :
min U , V L λ l o r a ( U , V ) = min T ∈ R m × n rank ( T ) ≤ r { L v a n i l l a ( T ) + λ ∥ T ∥ ⋆ } . \min_{U,V}\mathcal{L}_{\lambda}^{\mathrm{lora}}(U,V)=\min_{\substack{T\in\mathbb{R}^{m\times n}\\ \operatorname{rank}(T)\le r}}\left\{\mathcal{L}^{\mathrm{vanilla}}(T)+\lambda\|T\|_{\star}\right\}. U , V min L λ lora ( U , V ) = T ∈ R m × n rank ( T ) ≤ r min { L vanilla ( T ) + λ ∥ T ∥ ⋆ } .
Thus, whenever the unrestricted trace-norm problem (1) admits a global minimizer of rank ≤ r \le r ≤ r , the rank-r r r LoRA factorization incurs no loss of optimality.
Attention mechanism. For multi-head attention with head set A \mathcal{A} A , head a ∈ A a\in\mathcal{A} a ∈ A produces adapted queries, keys, and values on sample i i i :
Q ~ i ( a ) = ( W Q , 0 ( a ) + T Q ( a ) ) H Q , i , K ~ i ( a ) = ( W K , 0 ( a ) + T K ( a ) ) H K , i , V ~ i ( a ) = ( W V , 0 ( a ) + T V ( a ) ) H V , i . \widetilde{Q}_{i}^{(a)}=(W_{Q,0}^{(a)}+T_{Q}^{(a)})H_{Q,i},\qquad\widetilde{K}_{i}^{(a)}=(W_{K,0}^{(a)}+T_{K}^{(a)})H_{K,i},\qquad\widetilde{V}_{i}^{(a)}=(W_{V,0}^{(a)}+T_{V}^{(a)})H_{V,i}. Q i ( a ) = ( W Q , 0 ( a ) + T Q ( a ) ) H Q , i , K i ( a ) = ( W K , 0 ( a ) + T K ( a ) ) H K , i , V i ( a ) = ( W V , 0 ( a ) + T V ( a ) ) H V , i .
where H S , i H_{S,i} H S , i is the frozen token matrix entering projection S ∈ { Q , K , V } S\in\{Q,K,V\} S ∈ { Q , K , V } for sample i i i . Queries and keys share output dimension d k d_k d k ; values have output dimension d v d_v d v . The attention matrix and block output are
A i ( a ) = softmax r o w ( ( Q ~ i ( a ) ) T K ~ i ( a ) d k + C i M ) , O i = ∑ a ∈ A W O ( a ) V ~ i ( a ) ( A i ( a ) ) T . A_{i}^{(a)}=\operatorname{softmax}_{\mathrm{row}}\left(\frac{(\widetilde{Q}_{i}^{(a)})^{\mathsf{T}}\widetilde{K}_{i}^{(a)}}{\sqrt{d_k}}+C_{i}^{M}\right),\qquad O_i=\sum_{a\in\mathcal{A}}W_{O}^{(a)}\widetilde{V}_{i}^{(a)}(A_{i}^{(a)})^{\mathsf{T}}. A i ( a ) = softmax row ( d k ( Q i ( a ) ) T K i ( a ) + C i M ) , O i = a ∈ A ∑ W O ( a ) V i ( a ) ( A i ( a ) ) T .
where C i M C_i^M C i M is an attention mask and W O ( a ) W_O^{(a)} W O ( a ) is the frozen output-projection block for head a a a . We write H S : = [ H S , 1 , … , H S , N ] H_S:=[H_{S,1},\ldots,H_{S,N}] H S := [ H S , 1 , … , H S , N ] for the sample-stacked feature matrices. For sample i i i , let ϱ Q , i \varrho_{Q,i} ϱ Q , i and ϱ K , i \varrho_{K,i} ϱ K , i denote the numbers of query and key token positions, respectively, so that A i ( a ) ∈ R ϱ Q , i × ϱ K , i A_i^{(a)}\in\mathbb{R}^{\varrho_{Q,i}\times\varrho_{K,i}} A i ( a ) ∈ R ϱ Q , i × ϱ K , i . In the stage-indexed notation used below, at stage t t t , these quantities are written as ϱ t , Q , i \varrho_{t,Q,i} ϱ t , Q , i and ϱ t , K , i \varrho_{t,K,i} ϱ t , K , i .
Multiple adapted layers. When L L L consecutive attention modules are adapted (indexed so that larger t t t denotes a more upstream stage), the frozen features at stage t t t may depend on the updates at stages τ > t \tau>t τ > t . This creates a stagewise bottleneck structure, formalized as follows.
Definition 3.2 (L-stage block-linear dependence). Let L ∈ N L\in\mathbb{N} L ∈ N and let S t \mathcal{S}_t S t be finite index sets for t ∈ [ L ] t\in[L] t ∈ [ L ] . Let Z L + 1 \mathcal{Z}_{L+1} Z L + 1 be a singleton and let Z t \mathcal{Z}_t Z t be finite-dimensional inner-product spaces for t ∈ [ L ] t\in[L] t ∈ [ L ] . We say that a loss map L ( { T t , S } t ∈ [ L ] , S ∈ S t ) \mathcal{L}(\{T_{t,S}\}_{t\in[L],S\in\mathcal{S}_t}) L ({ T t , S } t ∈ [ L ] , S ∈ S t ) has L-stage block-linear dependence if there exist maps
Φ t , S : R m t , S × n t , S × Z t + 1 → E t , S , g t : ( ∏ S ∈ S t E t , S ) × Z t + 1 → Z t ( t ∈ [ L ] ) , \Phi_{t,S}:\mathbb{R}^{m_{t,S}\times n_{t,S}}\times\mathcal{Z}_{t+1}\to\mathcal{E}_{t,S},\qquad g_t:\left(\prod_{S\in\mathcal{S}_t}\mathcal{E}_{t,S}\right)\times\mathcal{Z}_{t+1}\to\mathcal{Z}_t\quad(t\in[L]), Φ t , S : R m t , S × n t , S × Z t + 1 → E t , S , g t : ( S ∈ S t ∏ E t , S ) × Z t + 1 → Z t ( t ∈ [ L ]) , and g 0 : Z 1 → R g_0:\mathcal{Z}_1\to\mathbb{R} g 0 : Z 1 → R such that:
for every fixed z t + 1 ∈ Z t + 1 z_{t+1}\in\mathcal{Z}_{t+1} z t + 1 ∈ Z t + 1 , the map T ↦ Φ t , S ( T , z t + 1 ) T\mapsto\Phi_{t,S}(T,z_{t+1}) T ↦ Φ t , S ( T , z t + 1 ) is linear;
if z L + 1 z_{L+1} z L + 1 denotes the unique point of Z L + 1 \mathcal{Z}_{L+1} Z L + 1 and
z t : = g t ( { Φ t , S ( T t , S , z t + 1 ) } S ∈ S t , z t + 1 ) , t = L , L − 1 , … , 1 , z_t:=g_t\left(\{\Phi_{t,S}(T_{t,S},z_{t+1})\}_{S\in\mathcal{S}_t},z_{t+1}\right),\qquad t=L,L-1,\ldots,1, z t := g t ( { Φ t , S ( T t , S , z t + 1 ) } S ∈ S t , z t + 1 ) , t = L , L − 1 , … , 1 , then L ( { T t , S } ) = g 0 ( z 1 ) \mathcal{L}(\{T_{t,S}\})=g_0(z_1) L ({ T t , S }) = g 0 ( z 1 ) .
The key feature is that each block T t , S T_{t,S} T t , S enters stage t t t through a linear map Φ t , S \Phi_{t,S} Φ t , S , but the stage transition maps g t g_t g t may couple the blocks in an arbitrary (possibly nonconvex) way. Stagewise affine shifts (e.g., the frozen term W t , S , 0 H t , S ( z t + 1 ) W_{t,S,0}H_{t,S}(z_{t+1}) W t , S , 0 H t , S ( z t + 1 ) ) are absorbed into the stage maps g t g_t g t . For attention, the concrete map is Φ t , ( a , S ) ( T , z ) = T H t , S ( z ) \Phi_{t,(a,S)}(T,z)=TH_{t,S}(z) Φ t , ( a , S ) ( T , z ) = T H t , S ( z ) , where H t , S ( z ) H_{t,S}(z) H t , S ( z ) is the frozen token matrix at stage t t t . An output projection adapted alongside Q / K / V Q/K/V Q / K / V receives inputs that depend on those same-stage updates.
Stagewise bottleneck ranks. For z t + 1 ∈ Z t + 1 r e a c h z_{t+1} \in\mathcal{Z}_{t+1}^{\mathrm{reach}} z t + 1 ∈ Z t + 1 reach (the set of reachable upstream states; see Definition A.2 in the appendix), define
d t , S ( z t + 1 ) : = dim R ( Φ t , S ( ⋅ , z t + 1 ) ) , d t , S max : = sup z t + 1 ∈ Z t + 1 r e a c h d t , S ( z t + 1 ) . d_{t,S}(z_{t+1}) := \dim\mathcal{R}\bigl(\Phi_{t,S}(\cdot,z_{t+1})\bigr), \qquad d_{t,S}^{\max} := \sup_{z_{t+1}\in\mathcal{Z}_{t+1}^{\mathrm{reach}}} d_{t,S}(z_{t+1}). d t , S ( z t + 1 ) := dim R ( Φ t , S ( ⋅ , z t + 1 ) ) , d t , S m a x := z t + 1 ∈ Z t + 1 reach sup d t , S ( z t + 1 ) .
The recursive upstream dimension budget is
β L + 1 : = 0 , β t : = β t + 1 + ∑ S ∈ S t d t , S max , t = L , L − 1 , … , 1. \beta_{L+1} := 0, \qquad\beta_t := \beta_{t+1} + \sum_{S\in\mathcal{S}_t} d_{t,S}^{\max}, \qquad t=L,L-1,\ldots,1. β L + 1 := 0 , β t := β t + 1 + S ∈ S t ∑ d t , S m a x , t = L , L − 1 , … , 1.
Intuitively, β t \beta_t β t upper-bounds the intrinsic dimension of the stage-t t t reachable state family.
Standing assumptions. We use two mild assumptions throughout, which are exactly the standard evaluation-mode condition.
Assumption 1 (Well-posed objective). The loss function g g g (or g 0 g_0 g 0 in L L L -stage formulation, through which the empirical loss factors) is bounded below and lower semicontinuous, and all maps in the stagewise decomposition are continuous.
Assumption 2 (Deterministic forward pass). The forward pass is deterministic per sample (no Dropout, no BatchNorm).
Main results
All of our results follow from the same principle: freezing the backbone makes each adapted block observable only through a finite-rank linear map, and trace-norm regularization selects a low-rank representative among all updates that produce the same observations.
Low-rank minimizers for a single frozen block
Theorem 4.1 (Low-rank global minimizers from bottleneck rank). Let Assumptions 1 and 2 hold. Suppose the empirical loss depends on T T T only through a linear map Φ ( T ) \Phi(T) Φ ( T ) of bottleneck rank d Φ d_{\Phi} d Φ . Then the trace-norm-regularized problem (3.1) admits a global minimizer. Moreover, for every global minimizer T ~ \widetilde{T} T , there exists a global minimizer T ⋆ T^\star T ⋆ with
Φ ( T ⋆ ) = Φ ( T ~ ) and rank ( T ⋆ ) ( rank ( T ⋆ ) + 1 ) 2 ≤ d Φ . \Phi(T^\star)=\Phi(\widetilde{T}) \qquad\text{and} \qquad\frac{\operatorname{rank}(T^\star)\bigl(\operatorname{rank}(T^\star)+1\bigr)}{2}\le d_{\Phi}. Φ ( T ⋆ ) = Φ ( T ) and 2 rank ( T ⋆ ) ( rank ( T ⋆ ) + 1 ) ≤ d Φ . Consequently,
rank ( T ⋆ ) ≤ min { m , n , ⌊ 8 d Φ + 1 − 1 2 ⌋ } . \operatorname{rank}(T^\star)\le\min\left\{m,n,\left\lfloor\frac{\sqrt{8d_{\Phi}+1}-1}{2}\right\rfloor\right\}. rank ( T ⋆ ) ≤ min { m , n , ⌊ 2 8 d Φ + 1 − 1 ⌋ } .
Proof idea. Among all PSD lifts Z ⪰ 0 Z\succeq0 Z ⪰ 0 with Φ ( Z ‾ ) = Φ ( T ~ ) \Phi(\overline{Z})=\Phi(\widetilde{T}) Φ ( Z ) = Φ ( T ) , choose one that minimizes tr ( Z ) \operatorname{tr}(Z) tr ( Z ) . Any extreme point of this spectrahedron satisfies the Barvinok–Pataki rank bound ([16 , 2 ] ): its rank r r r obeys r ( r + 1 ) / 2 ≤ d Φ r(r+1)/2\le d_{\Phi} r ( r + 1 ) /2 ≤ d Φ . The off-diagonal block Z ‾ \overline{Z} Z recovers T ⋆ T^\star T ⋆ with the same or smaller nuclear norm. The full proof is in Appendix C.1.
Remark 4.2 (Single-block rank scales). For a single frozen block with Φ ( T ) = T H \Phi(T)=TH Φ ( T ) = T H , we have d Φ ≤ m rank ( H ) ≤ m N d_{\Phi}\le m\operatorname{rank}(H)\le mN d Φ ≤ m rank ( H ) ≤ m N . When the observed output dimension is smaller than min { K o u t , rank ( H ) } ≤ min { K o u t , N } \min\{K_{\mathrm{out}},\operatorname{rank}(H)\}\le\min\{K_{\mathrm{out}},N\} min { K out , rank ( H )} ≤ min { K out , N } , this has the same worst-case counting scale as O ( K o u t N ) O(\sqrt{K_{\mathrm{out}}N}) O ( K out N ) in Jang et al. [8 ] .
Extension to L L L consecutive adapted layers. The next theorem concerns multiple adapted stages, each potentially containing several heads and projections. The same argument extends by stagewise induction: fix upstream stages, apply Theorem 4.1 to the conditional objective at each stage, and propagate downward.
Theorem 4.3 (Low-rank global minimizers for L L L stages). Under Assumptions 1 and 2, suppose the loss has L L L -stage block-linear dependence (Definition 3.2) and let all λ t , S > 0 \lambda_{t,S}>0 λ t , S > 0 . Then the trace-norm-regularized objective admits a global minimizer { T t , S ⋆ } \{T^\star_{t,S}\} { T t , S ⋆ } satisfying, for every stage t ∈ [ L ] t\in[L] t ∈ [ L ] and block S ∈ S t S\in\mathcal{S}_t S ∈ S t ,
r t , S ( r t , S + 1 ) 2 ≤ d t , S ( z t + 1 ⋆ ) ≤ d t , S max , r t , S : = rank ( T t , S ⋆ ) . \frac{r_{t,S}(r_{t,S}+1)}{2}\le d_{t,S}(z^\star_{t+1})\le d_{t,S}^{\max}, \qquad r_{t,S}:=\operatorname{rank}(T^\star_{t,S}). 2 r t , S ( r t , S + 1 ) ≤ d t , S ( z t + 1 ⋆ ) ≤ d t , S m a x , r t , S := rank ( T t , S ⋆ ) .
The case L = 1 L = 1 L = 1 recovers Theorem 4.1. For generic frozen linear layers, these bounds can still be loose because d t , S max d_{t,S}^{\max} d t , S m a x may be large. The next subsection shows that multi-head attention substantially tightens the effective bottleneck.
Rank bounds for multi-head attention
For attention modules, the bottleneck rank from Theorem 4.3 can be refined using two invariances below that are specific to the attention mechanism.
Invariance 1: score centering. Row-wise softmax satisfies softmax r o w ( M + C ) = softmax r o w ( M P 1 ⊥ + C ) \operatorname{softmax}_{\mathrm{row}}(M + C) = \operatorname{softmax}_{\mathrm{row}}(MP_1^\perp+ C) softmax row ( M + C ) = softmax row ( M P 1 ⊥ + C ) , where M , C ∈ R Q × k M, C \in\mathbb{R}^{Q \times k} M , C ∈ R Q × k (allowing − ∞ -\infty − ∞ entries in the mask C C C ) and P 1 ⊥ = I k − 1 k 1 k 1 k T ∈ R k × k P_1^\perp= I_k - \frac{1}{k}\mathbf{1}_k\mathbf{1}_k^{\mathsf{T}} \in\mathbb{R}^{k \times k} P 1 ⊥ = I k − k 1 1 k 1 k T ∈ R k × k (Lemma C.3). Thus, query/key scores are observed only after centering across key columns, reducing the key-token dimension from ϱ K , i \varrho_{K,i} ϱ K , i to ϱ K , i − 1 \varrho_{K,i} - 1 ϱ K , i − 1 . For queries and keys, the loss observes the update T Q ( a ) T_Q^{(a)} T Q ( a ) only through the centered key subspace , and similarly for T K ( a ) T_K^{(a)} T K ( a ) through the query subspace.
Invariance 2: value-projection compression. Value updates are observable to the loss only after two compressions: (i) attention-weighted pooling by A t , i ( a ) A_{t,i}^{(a)} A t , i ( a ) , which projects onto a subspace of dimension rank ( A t , i ( a ) ) \operatorname{rank}(A_{t,i}^{(a)}) rank ( A t , i ( a ) ) ; and (ii) the frozen output projection W O ( a ) W_{O}^{(a)} W O ( a ) .
Projector compression principle. Both invariances are instances of the following general lemma: if Φ ( T ) = Φ ( P L T P R ) \Phi(T) = \Phi(P_LTP_R) Φ ( T ) = Φ ( P L T P R ) for orthogonal projectors P L P_L P L , P R P_R P R , then T ⋆ = P L T P ~ R T^\star= P_LT\widetilde{P}_R T ⋆ = P L T P R is an equally optimal global minimizer with rank ( T ⋆ ) ≤ min { rank ( P L ) , rank ( P R ) } \operatorname{rank}(T^\star) \le\min\{\operatorname{rank}(P_L),\operatorname{rank}(P_R)\} rank ( T ⋆ ) ≤ min { rank ( P L ) , rank ( P R )} (Lemma C.1).
Combining these ingredients gives per-head, per-projection rank certificates:
Theorem 4.4 (Rank bounds for multi-head attention from invariances). Let Assumptions 1 and 2 hold. Consider L L L consecutive multi-head attention modules with adapted blocks S t ⊆ A t × { Q , K , V } S_t \subseteq A_t \times\{Q, K, V\} S t ⊆ A t × { Q , K , V } , fixed output-projection blocks W t , O ( a ) W_{t,O}^{(a)} W t , O ( a ) , and regularization weights λ t , S ( a ) > 0 \lambda_{t,S}^{(a)} > 0 λ t , S ( a ) > 0 . Every attention row has at least one unmasked key. Then the nuclear-norm-regularized objective admits a global minimizer. Moreover, there exists a global minimizer { T t , S ( a ) ⋆ } \{T_{t,S}^{(a)\star}\} { T t , S ( a ) ⋆ } that minimizes total rank ∑ t , a , S rank ( T t , S ( a ) ) \sum_{t,a,S} \operatorname{rank}(T_{t,S}^{(a)}) ∑ t , a , S rank ( T t , S ( a ) ) over the global minimizer set, and this minimizer satisfies:
Query bound:
rank ( T t , Q ( a ) ⋆ ) ≤ min { d k , t , rank ( H t , Q ⋆ ) , rank ( H t , K ⋆ ) , ∑ i = 1 N ( ϱ t , K , i − 1 ) } ∀ ( a , Q ) ∈ S t . \operatorname{rank}(T_{t,Q}^{(a)\star}) \le\min\left\{d_{k,t},\operatorname{rank}(H_{t,Q}^\star),\operatorname{rank}(H_{t,K}^\star),\sum_{i=1}^{N}(\varrho_{t,K,i}-1)\right\}\quad\forall\,(a,Q)\in S_t. rank ( T t , Q ( a ) ⋆ ) ≤ min { d k , t , rank ( H t , Q ⋆ ) , rank ( H t , K ⋆ ) , i = 1 ∑ N ( ϱ t , K , i − 1 ) } ∀ ( a , Q ) ∈ S t .
Key bound:
rank ( T t , K ( a ) ⋆ ) ≤ min { d k , t , rank ( H t , Q ⋆ ) , rank ( H t , K ⋆ ) , ∑ i = 1 N ( ϱ t , K , i − 1 ) } ∀ ( a , K ) ∈ S t . \operatorname{rank}(T_{t,K}^{(a)\star}) \le\min\left\{d_{k,t},\operatorname{rank}(H_{t,Q}^\star),\operatorname{rank}(H_{t,K}^\star),\sum_{i=1}^{N}(\varrho_{t,K,i}-1)\right\}\quad\forall\,(a,K)\in S_t. rank ( T t , K ( a ) ⋆ ) ≤ min { d k , t , rank ( H t , Q ⋆ ) , rank ( H t , K ⋆ ) , i = 1 ∑ N ( ϱ t , K , i − 1 ) } ∀ ( a , K ) ∈ S t .
Value bound:
rank ( T t , V ( a ) ⋆ ) ≤ min { d v , t , rank ( H t , V ⋆ ) , ∑ i = 1 N rank ( A t , i ( a ) ⋆ ) } ∀ ( a , V ) ∈ S t . \operatorname{rank}(T_{t,V}^{(a)\star}) \le\min\left\{d_{v,t},\operatorname{rank}(H_{t,V}^\star),\sum_{i=1}^{N}\operatorname{rank}(A_{t,i}^{(a)\star})\right\}\quad\forall\,(a,V)\in S_t. rank ( T t , V ( a ) ⋆ ) ≤ min { d v , t , rank ( H t , V ⋆ ) , i = 1 ∑ N rank ( A t , i ( a ) ⋆ ) } ∀ ( a , V ) ∈ S t .
Here, H t , S ⋆ H_{t,S}^\star H t , S ⋆ and A t , i ( a ) ⋆ A_{t,i}^{(a)\star} A t , i ( a ) ⋆ are the frozen features and attention matrices evaluated at the same global minimizer whose rank is being bounded.
Interpretation. Our bounds have the following implications. First, query and key ranks are controlled by the centered key/query subspace. The term ∑ i = 1 N ( ϱ t , K , i − 1 ) \sum_{i=1}^{N}(\varrho_{t,K,i}-1) ∑ i = 1 N ( ϱ t , K , i − 1 ) arises from the score-centering invariance. Second, value rank is controlled by attention sparsity. The term ∑ i = 1 N rank ( A t , i ( a ) ⋆ ) \sum_{i=1}^{N}\operatorname{rank}(A_{t,i}^{(a)\star}) ∑ i = 1 N rank ( A t , i ( a ) ⋆ ) reflects the effective number of distinct attention patterns. When attention is sparse or concentrated on a few dominant keys per query, rank ( A t , i ( a ) ⋆ ) \operatorname{rank}(A_{t,i}^{(a)\star}) rank ( A t , i ( a ) ⋆ ) is small, yielding a tight value-rank bound. Third, the frozen output projection provides an unconditional value-rank bound. Since rank ( T t , V ( a ) ⋆ ) ≤ rank ( W O ( a ) T ) ≤ d v , t \operatorname{rank}(T_{t,V}^{(a)\star}) \le\operatorname{rank}(W_{O}^{(a)\mathsf{T}}) \le d_{v,t} rank ( T t , V ( a ) ⋆ ) ≤ rank ( W O ( a ) T ) ≤ d v , t , the value update can never exceed the per-head value dimension, regardless of the data.
Adapted output projections. The results above are extended for output projection in the next theorem. The preceding rank certificates assumed fixed output-projection weights. They extend structurally when one also adapts the full concatenated-head output projection, provided its input is treated as fixed only conditionally on the other adapted blocks.
Theorem 4.5 (Attention rank bounds with adapted output projections). In the setting of Theorem 4.4, suppose that at each selected stage t t t the full output projection W t , O , 0 ∈ R d o u t , t × ∣ A t ∣ d v , t W_{t,O,0} \in\mathbb{R}^{d_{\mathrm{out},t} \times|\mathcal{A}_t|d_{v,t}} W t , O , 0 ∈ R d out , t × ∣ A t ∣ d v , t also receives an update T t , O T_{t,O} T t , O of the same shape with nuclear-norm weight λ t , O > 0 \lambda_{t,O} > 0 λ t , O > 0 . Then the nuclear-norm-regularized objective admits a global minimizer that minimizes the total rank of all adapted Q / K / V / O Q/K/V/O Q / K / V / O blocks over the global minimizer set. At every such minimizer, the query and key bounds of Theorem 4.4 hold unchanged, and
rank ( T t , V ( a ) ⋆ ) ≤ min { d v , t , rank ( H t , V ⋆ ) , ∑ i = 1 N rank ( A t , i ( a ) ⋆ ) } , \operatorname{rank}(T_{t,V}^{(a)\star}) \le\min\left\{d_{v,t},\operatorname{rank}(H_{t,V}^{\star}),\sum_{i=1}^{N}\operatorname{rank}(A_{t,i}^{(a)\star})\right\}, rank ( T t , V ( a ) ⋆ ) ≤ min { d v , t , rank ( H t , V ⋆ ) , i = 1 ∑ N rank ( A t , i ( a ) ⋆ ) } , rank ( T t , O ⋆ ) ≤ min { d o u t , t , ∣ A t ∣ d v , t , ∑ i = 1 N ϱ t , Q , i } , \operatorname{rank}(T_{t,O}^{\star}) \le\min\left\{d_{\mathrm{out},t},|\mathcal{A}_t|d_{v,t},\sum_{i=1}^{N}\varrho_{t,Q,i}\right\}, rank ( T t , O ⋆ ) ≤ min { d out , t , ∣ A t ∣ d v , t , i = 1 ∑ N ϱ t , Q , i } , where all features and attention matrices are evaluated at this minimizer.
A value update T t , V ( a ) T_{t,V}^{(a)} T t , V ( a ) enters the stage output only after left multiplication by the head-a a a column block W ~ t , O ( a ) ∈ R d o u t , t × d v , t \widetilde{W}_{t,O}^{(a)} \in\mathbb{R}^{d_{\mathrm{out},t} \times d_{v,t}} W t , O ( a ) ∈ R d out , t × d v , t of the adapted output projection W t , O , 0 + T t , O W_{t,O,0}+T_{t,O} W t , O , 0 + T t , O . Adapting the output projection changes which directions of the d v , t d_{v,t} d v , t -dimensional head output are retained, but it cannot enlarge that output. The update T t , O T_{t,O} T t , O differs from the Q / K / V Q/K/V Q / K / V updates because its input, the attention-weighted values, depends on the same-stage Q / K / V Q/K/V Q / K / V updates. The proof therefore compresses each block conditionally on the other blocks instead of using a joint linear map, and for the same reason Corollary 4.8 does not apply to T t , O T_{t,O} T t , O .
Generic rank deficiency of first-order stationary points
The results in § 4.1 –§ 4.2 concern global minimizers of the convex trace-norm objective. In practice, LoRA solves the nonconvex factorized problem L l o r a ( U , V ) \mathcal{L}^{\mathrm{lora}}(U,V) L lora ( U , V ) via gradient descent, which may converge to stationary points that are not global minimizers. We now show that the bottleneck structure also constrains the rank at stationary points of the factorized problem.
Assumption 3 (Differentiability for stationarity). The induced empirical loss L \mathcal{L} L is differentiable with respect to every adapted block.
Assumption 4 (Stagewise C 1 C^1 C 1 regularity for stagewise linear maps). In Definition 3.2, for every stage t ∈ [ L ] t \in[L] t ∈ [ L ] , every block S ∈ S t S \in\mathcal{S}_t S ∈ S t , every basis vector e j e_j e j of E t , S \mathcal{E}_{t,S} E t , S , and every basis matrix B k B_k B k of R m t , S × n t , S \mathbb{R}^{m_{t,S} \times n_{t,S}} R m t , S × n t , S , the maps Z t + 1 ↦ Φ t , S ( Z t + 1 ) ( e j ) \mathcal{Z}_{t+1} \mapsto\Phi_{t,S}(\mathcal{Z}_{t+1})(e_j) Z t + 1 ↦ Φ t , S ( Z t + 1 ) ( e j ) , Z t + 1 ↦ Φ t , S ( Z t + 1 ) ( B k Z t + 1 ) \mathcal{Z}_{t+1} \mapsto\Phi_{t,S}(\mathcal{Z}_{t+1})(B_k\mathcal{Z}_{t+1}) Z t + 1 ↦ Φ t , S ( Z t + 1 ) ( B k Z t + 1 ) are C 1 C^1 C 1 on an open neighborhood of Z t + 1 r e a c h \mathcal{Z}_{t+1}^{\mathrm{reach}} Z t + 1 reach . Here, Φ t , S \Phi_{t,S} Φ t , S is the stage-shifted map of Φ t , S \Phi_{t,S} Φ t , S . Moreover, each transition map g t g_t g t is C 1 C^1 C 1 on an open neighborhood of the reachable set of its arguments.
Assumption 4 is a regularity assumption on how the moving linear observation maps depend on upstream states, and common evaluation-mode attention blocks satisfy this condition on bounded reachable sets.
Why a perturbation is needed. The KKT conditions at stationarity gives ( B m , n ( G ) + λ I m + n ) R = 0 (\mathcal{B}_{m,n}(G)+\lambda I_{m+n})R=0 ( B m , n ( G ) + λ I m + n ) R = 0 with G = ∇ T L G=\nabla_T\mathcal{L} G = ∇ T L and B m , n ( G ) = [ 0 G G T 0 ] \mathcal{B}_{m,n}(G)=\begin{bmatrix}0&G\\G^{\mathsf{T}}&0\end{bmatrix} B m , n ( G ) = [ 0 G T G 0 ] . Full column rank of R R R forces an r r r -dimensional nullspace, whereas bottleneck structure restricts G G G to the low-dimensional range R ( Φ ⋆ ) \mathcal{R}(\Phi^{\star}) R ( Φ ⋆ ) (Lemma C.7 ). A generic PSD perturbation makes this low-dimensional symmetric matrix family avoid such a high-dimensional nullspace.
Definition 4.6 (Perturbed LoRA objective). For perturbation P ∈ S + m + n P \in\mathbb{S}_{+}^{m+n} P ∈ S + m + n , define
L λ , P l o r a ( U , V ) : = L v a n i l l a ( U V T ) + λ 2 ( ∥ U ∥ F 2 + ∥ V ∥ F 2 ) + 1 2 ⟨ P , R R T ⟩ . \mathcal{L}_{\lambda,P}^{\mathrm{lora}}(U,V) := \mathcal{L}^{\mathrm{vanilla}}(UV^{\mathsf{T}}) + \frac{\lambda}{2}\left(\lVert U\rVert_F^2+\lVert V\rVert_F^2\right) + \frac{1}{2}\langle P,RR^{\mathsf{T}}\rangle. L λ , P lora ( U , V ) := L vanilla ( U V T ) + 2 λ ( ∥ U ∥ F 2 + ∥ V ∥ F 2 ) + 2 1 ⟨ P , R R T ⟩ .
Theorem 4.7 (Generic exclusion of full-rank stationary points). Fix d ≥ 1 d \ge1 d ≥ 1 , r ≥ 1 r \ge1 r ≥ 1 , λ > 0 \lambda> 0 λ > 0 , and let G ⊆ S d \mathcal{G} \subseteq\mathbb{S}^{d} G ⊆ S d be contained in a finite union of C 1 C^{1} C 1 images of dimension at most p p p . If r ( r + 1 ) 2 > p \frac{r(r+1)}{2} > p 2 r ( r + 1 ) > p , then for Lebesgue-a.e. P ∈ S + d P \in\mathbb{S}_{+}^{d} P ∈ S + d , every matrix R ∈ R d × r R \in\mathbb{R}^{d \times r} R ∈ R d × r satisfying ( G + λ I + P ) R = 0 (G + \lambda I + P)R = 0 ( G + λ I + P ) R = 0 for some G ∈ G G \in\mathcal{G} G ∈ G has rank ( R ) < r \operatorname{rank}(R) < r rank ( R ) < r .
Proof idea. For each s ≥ r s \ge r s ≥ r , the set of symmetric matrices with nullspace dimension exactly s s s is a smooth embedded submanifold of S d \mathbb{S}^{d} S d of codimension s ( s + 1 ) / 2 s(s+1)/2 s ( s + 1 ) /2 . Therefore, the set of symmetric matrices with nullspace dimension at least r r r is a finite union of such manifolds, not generally a single submanifold. The family G + λ I \mathcal{G} + \lambda I G + λ I has dimension at most p p p . When r ( r + 1 ) 2 > p \frac{r(r+1)}{2} > p 2 r ( r + 1 ) > p , the translated family G + λ I + P \mathcal{G} + \lambda I + P G + λ I + P generically misses this submanifold by a dimension-counting argument (Lemma C.6 ).
Application to attention. Using the stagewise gradient-range characterization (Lemma C.8 ), the family G t , S \mathcal{G}_{t,S} G t , S for each attention block is C 1 C^{1} C 1 -parameterizable with dimension at most β t + 1 + d t , S max \beta_{t+1} + d^{\max}_{t,S} β t + 1 + d t , S m a x . This yields:
Corollary 4.8 (Generic rank deficiency for L L L attention modules). Under Assumptions 1, 2, and mild differentiability conditions (Assumptions 3 and 4), consider L L L consecutive attention modules. For each adapted block α = ( a , S ) ∈ S t \alpha= (a, S) \in\mathcal{S}_{t} α = ( a , S ) ∈ S t , let r H , t , α : = sup z t + 1 rank ( H t , S ( z t + 1 ) ) r_{H,t,\alpha} := \sup_{z_{t+1}} \operatorname{rank}(H_{t,S}(z_{t+1})) r H , t , α := sup z t + 1 rank ( H t , S ( z t + 1 )) . If
r t , α ( r t , α + 1 ) 2 > β t + 1 + m t , α r H , t , α max ∀ t ∈ [ L ] , α ∈ S t , \frac{r_{t,\alpha}(r_{t,\alpha}+1)}{2} > \beta_{t+1} + m_{t,\alpha}r^{\max}_{H,t,\alpha} \qquad\forall t \in[L], \alpha\in\mathcal{S}_{t}, 2 r t , α ( r t , α + 1 ) > β t + 1 + m t , α r H , t , α m a x ∀ t ∈ [ L ] , α ∈ S t , then for Lebesgue-a.e. PSD perturbations { P t , α } \{P_{t,\alpha}\} { P t , α } , every first-order stationary point of the perturbed LoRA objective satisfies rank ( R t , α ) < r t , α \operatorname{rank}(R_{t,\alpha}) < r_{t,\alpha} rank ( R t , α ) < r t , α .
Limitation: spurious local minima persist. This is a first-order exclusion theorem, not a no-spurious-local-minima result. Indeed, spurious local minima can occur even for arbitrarily overparameterized LoRA:
Proposition 4.9 (Spurious local minima for overparameterized LoRA). For any rank r ≥ 2 r \ge2 r ≥ 2 , any λ > 0 \lambda> 0 λ > 0 , and any ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) , there exists a smooth nonconvex loss g g g such that for every P ∈ S + 2 P \in\mathbb{S}_{+}^{2} P ∈ S + 2 with ∥ P ∥ F ≤ ϵ \lVert P\rVert_{F} \le\epsilon ∥ P ∥ F ≤ ϵ , the perturbed rank-r r r LoRA objective has a strict local minimum at R = 0 R = 0 R = 0 that is not global.
This clarifies the scope of the generic rank-deficiency result: it guarantees that stationary points are rank-deficient (the LoRA factorization does not waste its rank budget at stationary points), but it does not guarantee that all such points are globally optimal.
Perturbation transfer and generalization
Near-optimality of the perturbed factorization. The PSD perturbation in Theorem 4.7 can be taken arbitrarily small. The following result shows that global minimizers of the perturbed factorized objective remain near-optimal for the original trace-norm problem:
Theorem 4.10 (Perturbation-to-unperturbed transfer). Let I \mathcal{I} I be any finite index set of adapted blocks. For each j ∈ I j \in\mathcal{I} j ∈ I , fix a factorization rank r j r_j r j and a regularization parameter λ j > 0 \lambda_j > 0 λ j > 0 . Assume that the trace-norm-regularized problem admits a global minimizer { T j ∗ } \{T_j^*\} { T j ∗ } satisfying rank ( T j ∗ ) ≤ r j ∀ j ∈ I \operatorname{rank}(T_j^*) \le r_j \ \forall j \in\mathcal{I} rank ( T j ∗ ) ≤ r j ∀ j ∈ I , and let R j ∗ : = [ U j ∗ V j ∗ ] R_j^* := \begin{bmatrix} U_j^* \\ V_j^* \end{bmatrix} R j ∗ := [ U j ∗ V j ∗ ] for a balanced factorization of T j ∗ T_j^* T j ∗ . Let { ( U ^ j , V ^ j ) } \{(\widehat{U}_j,\widehat{V}_j)\} {( U j , V j )} be a global minimizer of the perturbed LoRA objective with parameters λ j , P j \lambda_j, P_j λ j , P j (Definition 4.6 ). Then
L ( { T ^ j } ) + ∑ j λ j ∥ T ^ j ∥ ∗ ≤ min { T j } { L ( { T j } ) + ∑ j λ j ∥ T j ∥ ∗ } + 1 2 ∑ j ⟨ P j , R j ∗ R j ∗ ⊤ ⟩ . \mathcal{L}(\{\widehat{T}_j\}) + \sum_j \lambda_j \lVert\widehat{T}_j\rVert_* \le\min_{\{T_j\}} \left\{\mathcal{L}(\{T_j\}) + \sum_j \lambda_j \lVert T_j\rVert_*\right\} + \frac{1}{2}\sum_j \langle P_j, R_j^*R_j^{*\top}\rangle. L ({ T j }) + j ∑ λ j ∥ T j ∥ ∗ ≤ { T j } min { L ({ T j }) + j ∑ λ j ∥ T j ∥ ∗ } + 2 1 j ∑ ⟨ P j , R j ∗ R j ∗ ⊤ ⟩ . In particular, if ∥ P j ∥ 2 ≤ ε j \lVert P_j\rVert_2 \le\varepsilon_j ∥ P j ∥ 2 ≤ ε j , the gap is at most ∑ j ε j ∥ T j ∗ ∥ ∗ \sum_j \varepsilon_j\lVert T_j^*\rVert_* ∑ j ε j ∥ T j ∗ ∥ ∗ .
Our assumptions satisfy that the trace-norm-regularized problem admits a global minimizer, and this theorem can be applied to a single block index set S \mathcal{S} S , a stagewise index set S t \mathcal{S}_t S t , and the flattened attention index set S ~ : = { ( t , a , S ) : t ∈ [ L ] , ( a , S ) ∈ S t } \widetilde{\mathcal{S}} := \{(t,a,S) : t \in[L], (a,S) \in\mathcal{S}_t\} S := {( t , a , S ) : t ∈ [ L ] , ( a , S ) ∈ S t } .
Generalization. Uniform generalization bounds for trace-norm-bounded updates in frozen networks are governed by E [ ∥ E H ⊤ ∥ 2 ] \mathbb{E}[\lVert E H^{\top}\rVert_2] E [∥ E H ⊤ ∥ 2 ] , where E E E is a Rademacher sign matrix. The key identity is the trace/spectral duality:
sup ∥ T ∥ ∗ ≤ D ⟨ T , E H ⊤ ⟩ F = D ∥ E H ⊤ ∥ 2 , \sup_{\lVert T\rVert_{\ast}\le D} \langle T, E H^{\top}\rangle_F = D\lVert E H^{\top}\rVert_2, ∥ T ∥ ∗ ≤ D sup ⟨ T , E H ⊤ ⟩ F = D ∥ E H ⊤ ∥ 2 ,
which reduces the Rademacher complexity of the loss class to the spectral norm of random-sign-weighted frozen features. The full single-block bound (Theorem B.5 ) and the L L L -stage attention extension (Theorem B.8 ) are stated and proved in Appendix B.
Experiment
We test the predicted rank threshold on SST-2, MNLI, and RTE [20 ] using Qwen2.5-1.5B [22 ] . Only one query-projection block in one head of layers 7 and 8 is adapted; after a classification-head warm-up, both the backbone and classifier are frozen. The threshold is the smallest rank at which the upstream and downstream rank proxies both saturate.
Two rank-bound regimes. For the selected query blocks, Theorem 4.4 gives the upper bound
rank ( T t , Q ( a ) ⋆ ) ≤ min ( d k , t , rank ( H t , Q ⋆ ) , rank ( H t , K ⋆ ) , ∑ i = 1 N ( ϱ t , K , i − 1 ) ) ∀ ( a , Q ) ∈ S t . \operatorname{rank}(T_{t,Q}^{(a)\star}) \le\min\left( d_{k,t},\operatorname{rank}(H_{t,Q}^{\star}),\operatorname{rank}(H_{t,K}^{\star}),\sum_{i=1}^{N}(\varrho_{t,K,i}-1)\right)\quad\forall\,(a,Q)\in\mathcal{S}_{t}. rank ( T t , Q ( a ) ⋆ ) ≤ min ( d k , t , rank ( H t , Q ⋆ ) , rank ( H t , K ⋆ ) , i = 1 ∑ N ( ϱ t , K , i − 1 ) ) ∀ ( a , Q ) ∈ S t .
We evaluate in two regimes. We test a many-example regime where this estimate reaches the head cap d k , t d_{k,t} d k , t and a few-example regime where it is strictly smaller. For Qwen2.5-1.5B, d k , t = 128 d_{k,t}=128 d k , t = 128 .
Results. Figure 1 shows a consistent threshold effect: ranks below the estimate follow distinct, worse loss trajectories, while ranks at or above it nearly overlap and approach the full fine-tuning loss. This is consistent with the high-rank LoRA/full-fine-tuning alignment observed by Jang et al. [8 ] . We also observe that high-rank LoRA runs converge to almost the same point as the full fine-tuning baseline, although their training dynamics differ, consistent with the related phenomenon observed by Jang et al. [8 ] .
Figure 1. Training loss trajectories on GLUE classification tasks (MNLI, SST-2, and RTE) using Qwen2.5-1.5B across various LoRA ranks. The results demonstrate a clear threshold effect: ranks below the estimated threshold exhibit distinct, suboptimal convergence trajectories, whereas ranks at or above the threshold tightly cluster and converge to nearly indistinguishable final losses, closely matching the convergence point of full fine-tuning.
Conclusion
In this paper, we identify that LoRA operates via a finite-rank mechanism where frozen layers project updates into low-dimensional subspaces. Through this, trace-norm regularization selects a low-rank representative among all updates producing the same observed outputs. This yields global low-rank minimizers without NTK linearization and gives rank bounds controlled by bottleneck dimensions rather than by the ambient number of trainable parameters.
For attention modules, we showed that the observable subspaces are smaller than the naive frozen-feature image, row-wise softmax removes one key-column direction per sample, and value updates are observable only after attention-weighted pooling and the frozen output projection. These invariances give per-head rank certificates for query, key, and value updates at rank-minimal global minimizers. We also prove that small perturbations exclude full-rank stationary points in nonconvex LoRA while maintaining near-optimality.
Limitations. First, our rank theorems certify existence for trace-norm objectives, not convergence of gradient-based LoRA. Second, we can’t explicitly calculate bounds in Theorem 4.4 , which are based on global solutions. Finally, the attention bounds are a posteriori, and the generic rank-deficiency theorem is only a first-order statement for perturbed objectives; Proposition 4.9 shows that spurious local minima can remain.
Acknowledgments
The authors would like to thank anonymous reviewers and the Area Chair for their useful feedback to improve our work. This work is supported by grant VUNI.2526.CAIR.04 and Seed Fund 10000225 CECS.
Full list of notations and auxiliary definitions
We first summarize the key notation used throughout the paper. Table 2 summarizes the basic symbols for the LoRA fine-tuning setup, and Table 3 lists the additional notation used for attention layers and multi-stage linear mappings.
Table 2. Summary of notation and parameters for LoRA fine-tuning setup
Table 3. Summary of attention notation and multi-stage linear mapping
For a single block, we also write
F p r i m a l ( T ) : = g ( Φ ( T ) + s ) + λ ∥ T ∥ ⋆ , F l i f t ( Z ) : = g ( Φ ( Z ‾ ) + s ) + λ 2 tr ( Z ) , Z ⪰ 0. F_{\mathrm{primal}}(T) := g(\Phi(T) + s) + \lambda\lVert T\rVert_\star,\quad F_{\mathrm{lift}}(Z) := g(\Phi(\overline{Z}) + s) + \frac{\lambda}{2}\operatorname{tr}(Z),\ Z \succeq0. F primal ( T ) := g ( Φ ( T ) + s ) + λ ∥ T ∥ ⋆ , F lift ( Z ) := g ( Φ ( Z ) + s ) + 2 λ tr ( Z ) , Z ⪰ 0.
NTK linearization as a special case. Our setting also contains the NTK/lazy regime as a special case [7 , 10 ] . Let Δ θ ∈ R p \Delta\theta\in\mathbb{R}^{p} Δ θ ∈ R p denote the trainable parameter update; if the adapted parameter is a matrix, one may take Δ θ = vec ( T ) \Delta\theta= \operatorname{vec}(T) Δ θ = vec ( T ) . The first-order Taylor model around θ 0 \theta_0 θ 0 is
f ( x ; θ 0 + Δ θ ) ≈ f ( x ; θ 0 ) + J θ 0 ( x ) Δ θ , f(x;\theta_0 + \Delta\theta) \approx f(x;\theta_0) + J_{\theta_0}(x)\Delta\theta, f ( x ; θ 0 + Δ θ ) ≈ f ( x ; θ 0 ) + J θ 0 ( x ) Δ θ ,
where J θ 0 ( x ) ∈ R K × p J_{\theta_0}(x) \in\mathbb{R}^{K \times p} J θ 0 ( x ) ∈ R K × p is the Jacobian of the logits with respect to the trainable parameters at θ 0 \theta_0 θ 0 . Given training data { ( x i , y i ) } i = 1 N \{(x_i,y_i)\}_{i=1}^{N} {( x i , y i ) } i = 1 N , define
Φ N T K ( Δ θ ) : = ( J θ 0 ( x 1 ) Δ θ , … , J θ 0 ( x N ) Δ θ ) ∈ ( R K ) N \Phi_{\mathrm{NTK}}(\Delta\theta) := \bigl(J_{\theta_0}(x_1)\Delta\theta,\ldots,J_{\theta_0}(x_N)\Delta\theta\bigr) \in(\mathbb{R}^{K})^N Φ NTK ( Δ θ ) := ( J θ 0 ( x 1 ) Δ θ , … , J θ 0 ( x N ) Δ θ ) ∈ ( R K ) N
and
s N T K : = ( f ( x 1 ; θ 0 ) , … , f ( x N ; θ 0 ) ) ∈ ( R K ) N . s_{\mathrm{NTK}} := \bigl(f(x_1;\theta_0),\ldots,f(x_N;\theta_0)\bigr) \in(\mathbb{R}^{K})^N. s NTK := ( f ( x 1 ; θ 0 ) , … , f ( x N ; θ 0 ) ) ∈ ( R K ) N .
Then any empirical loss depending only on the vector of training predictions can be written as
L N T K ( Δ θ ) = g ( Φ N T K ( Δ θ ) + s N T K ) \mathcal{L}_{\mathrm{NTK}}(\Delta\theta) = g\bigl(\Phi_{\mathrm{NTK}}(\Delta\theta) + s_{\mathrm{NTK}}\bigr) L NTK ( Δ θ ) = g ( Φ NTK ( Δ θ ) + s NTK )
for a suitable map g : ( R K ) N → R g:(\mathbb{R}^{K})^N\to\mathbb{R} g : ( R K ) N → R . For a sample-wise loss, one may take
g ( z 1 , … , z N ) : = 1 N ∑ i = 1 N ℓ ( z i , y i ) . g(z_1,\ldots,z_N):=\frac{1}{N}\sum_{i=1}^{N}\ell(z_i,y_i). g ( z 1 , … , z N ) := N 1 i = 1 ∑ N ℓ ( z i , y i ) .
In the scalar-output case K = 1 K=1 K = 1 , this reduces to a map into R N \mathbb{R}^N R N . We now introduce several auxiliary concepts used later in the analysis.
Definition A.1 (C 1 C^1 C 1 -parameterizable subsets). Let V V V be a finite-dimensional real inner-product space and let G ⊆ V G\subseteq V G ⊆ V . We say that G G G is C 1 C^1 C 1 -parameterizable with dimension at most p p p if there exist finitely many open sets U j ⊆ R p j U_j\subseteq\mathbb{R}^{p_j} U j ⊆ R p j with p j ≤ p p_j\leq p p j ≤ p and C 1 C^1 C 1 maps f j : U j → V f_j:U_j\to V f j : U j → V such that
G ⊆ ⋃ j = 1 J f j ( U j ) . G\subseteq\bigcup_{j=1}^{J}f_j(U_j). G ⊆ j = 1 ⋃ J f j ( U j ) .
Definition A.2 (Reachable sets). For L L L -stage block-linear dependence, set Z L + 1 r e a c h : = Z L + 1 \mathcal{Z}^{\mathrm{reach}}_{L+1}:=\mathcal{Z}_{L+1} Z L + 1 reach := Z L + 1 . For t = L , L − 1 , … , 1 t=L,L-1,\ldots,1 t = L , L − 1 , … , 1 , define the reachable stage-t t t linear-output set
U t r e a c h : = { { Φ t , S ( T t , S , z t + 1 ) } S ∈ S t : z t + 1 ∈ Z t + 1 r e a c h , T t , S ∈ R m t , S × n t , S } ⊆ ∏ S ∈ S t E t , S , \mathcal{U}^{\mathrm{reach}}_t:=\left\{\left\{\Phi_{t,S}(T_{t,S},z_{t+1})\right\}_{S\in\mathcal{S}_t}:z_{t+1}\in\mathcal{Z}^{\mathrm{reach}}_{t+1},\ T_{t,S}\in\mathbb{R}^{m_{t,S}\times n_{t,S}}\right\}\subseteq\prod_{S\in\mathcal{S}_t}\mathcal{E}_{t,S}, U t reach := { { Φ t , S ( T t , S , z t + 1 ) } S ∈ S t : z t + 1 ∈ Z t + 1 reach , T t , S ∈ R m t , S × n t , S } ⊆ S ∈ S t ∏ E t , S , and the reachable stage-t t t state set
Z t r e a c h : = { g t ( { u t , S } S ∈ S t , z t + 1 ) : z t + 1 ∈ Z t + 1 r e a c h , u t , S ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ) } . \mathcal{Z}^{\mathrm{reach}}_t:=\left\{g_t\left(\{u_{t,S}\}_{S\in\mathcal{S}_t},z_{t+1}\right):z_{t+1}\in\mathcal{Z}^{\mathrm{reach}}_{t+1},\ u_{t,S}\in\mathcal{R}\left(\Phi_{t,S}(\cdot,z_{t+1})\right)\right\}. Z t reach := { g t ( { u t , S } S ∈ S t , z t + 1 ) : z t + 1 ∈ Z t + 1 reach , u t , S ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ) } .
Population and empirical risk. For a single update T ∈ R m × n T \in\mathbb{R}^{m \times n} T ∈ R m × n , define
L ( T ) : = E ( x , y ) ∼ P [ ℓ ( f W 0 + T ( x ) , y ) ] , L S d a t a v a n i l l a ( T ) : = 1 N ∑ i = 1 N ℓ ( f W 0 + T ( x i ) , y i ) . L(T) := \mathbb{E}_{(x,y)\sim\mathcal{P}}[\ell(f_{W_0+T}(x),y)], \qquad\mathcal{L}^{\mathrm{vanilla}}_{S_{\mathrm{data}}}(T) := \frac{1}{N}\sum_{i=1}^{N}\ell(f_{W_0+T}(x_i),y_i). L ( T ) := E ( x , y ) ∼ P [ ℓ ( f W 0 + T ( x ) , y )] , L S data vanilla ( T ) := N 1 i = 1 ∑ N ℓ ( f W 0 + T ( x i ) , y i ) .
For multiple adapted matrices { T S } \{T_S\} { T S } , we use the same notation blockwise. When the sample is fixed, we write L v a n i l l a \mathcal{L}^{\mathrm{vanilla}} L vanilla for L S d a t a v a n i l l a \mathcal{L}^{\mathrm{vanilla}}_{S_{\mathrm{data}}} L S data vanilla .
For D ≥ 0 D \ge0 D ≥ 0 , define the deterministic activation ball
Z a c t ( D , R h ) : = { z ∈ R m : ∥ z ∥ 2 ≤ ( ∥ W 0 ∥ 2 + D ) R h } . \mathcal{Z}_{\mathrm{act}}(D,R_h) := \{z \in\mathbb{R}^{m} : \|z\|_2 \le(\|W_0\|_2 + D)R_h\}. Z act ( D , R h ) := { z ∈ R m : ∥ z ∥ 2 ≤ ( ∥ W 0 ∥ 2 + D ) R h } .
To state our generalization bounds, we also recall the empirical Rademacher complexity.
Empirical Rademacher complexity. Let F \mathcal{F} F be a class of functions f : X × Y → R f : \mathcal{X} \times\mathcal{Y} \to\mathbb{R} f : X × Y → R . The (conditional) empirical Rademacher complexity of F \mathcal{F} F on S d a t a S_{\mathrm{data}} S data is
R S d a t a ( F ) : = E ϵ [ sup f ∈ F 1 N ∑ i = 1 N ϵ i f ( x i , y i ) ∣ S d a t a ] , \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}) := \mathbb{E}_{\epsilon}\left[\sup_{f\in\mathcal{F}}\frac{1}{N}\sum_{i=1}^{N}\epsilon_i f(x_i,y_i)\mathrel{\Big|} S_{\mathrm{data}}\right], R S data ( F ) := E ϵ [ f ∈ F sup N 1 i = 1 ∑ N ϵ i f ( x i , y i ) S data ] ,
where ϵ 1 , … , ϵ N \epsilon_1,\ldots,\epsilon_N ϵ 1 , … , ϵ N are i.i.d. uniform signs in { − 1 , 1 } \{-1,1\} { − 1 , 1 } . For D ≥ 0 D \ge0 D ≥ 0 , define the trace-norm ball
B ⋆ ( D ) : = { T ∈ R m × n : ∥ T ∥ ⋆ ≤ D } , \mathbb{B}_{\star}(D) := \{T \in\mathbb{R}^{m \times n} : \|T\|_{\star} \le D\}, B ⋆ ( D ) := { T ∈ R m × n : ∥ T ∥ ⋆ ≤ D } ,
and define the loss class induced by trace-norm-bounded updates
F D : = { ( x , y ) ↦ ℓ ( f W 0 + T ( x ) , y ) : T ∈ B ⋆ ( D ) } . \mathcal{F}_D := \{(x,y) \mapsto\ell(f_{W_0+T}(x),y) : T \in\mathbb{B}_{\star}(D)\}. F D := {( x , y ) ↦ ℓ ( f W 0 + T ( x ) , y ) : T ∈ B ⋆ ( D )} .
Miscellaneous. For Z ∈ S + m + n Z \in\mathbb{S}^{m+n}_{+} Z ∈ S + m + n write Z = [ Z 11 Z 12 Z 21 Z 22 ] Z = \begin{bmatrix} Z_{11} & Z_{12} \\ Z_{21} & Z_{22} \end{bmatrix} Z = [ Z 11 Z 21 Z 12 Z 22 ] with Z 12 ∈ R m × n Z_{12} \in\mathbb{R}^{m \times n} Z 12 ∈ R m × n and denote Z ‾ : = Z 12 \overline{Z} := Z_{12} Z := Z 12 .
For sets A A A , B B B in a vector space, the Minkowski sum is A + B : = { a + b : a ∈ A , b ∈ B } A+B := \{a+b : a \in A,b \in B\} A + B := { a + b : a ∈ A , b ∈ B } . We use blkdiag \operatorname{blkdiag} blkdiag for a block diagonal matrix. For a matrix M M M , let P M P_M P M denote the orthogonal projector onto the column space R ( M ) \mathcal{R}(M) R ( M ) . If M = 0 M=0 M = 0 , then P M = 0 P_M=0 P M = 0 .
Generalization bounds
We now state the assumptions used to derive the generalization bounds in this section.
Assumption 5 (Bounded and Lipschitz loss). The loss satisfies $0 \le \ell(\cdot,y) \le B$ and is L ℓ L_{\ell} L ℓ -Lipschitz (L ℓ > 0 L_{\ell} > 0 L ℓ > 0 ) with respect to ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 for every $y.
Assumption 6 (Almost-sure bounded frozen activations). For ( x , y ) ∼ P (x,y) \sim\mathcal{P} ( x , y ) ∼ P , the frozen input activation to the single adapted block satisfies ∥ h ( x ) ∥ 2 ≤ R h \|h(x)\|_2 \le R_h ∥ h ( x ) ∥ 2 ≤ R h almost surely. In the attention setting, for each S ∈ { Q , K , V } S \in\{Q,K,V\} S ∈ { Q , K , V } , the frozen token matrix satisfies ∥ H S ( x ) ∥ F ≤ R H , S \|H_S(x)\|_F \le R_{H,S} ∥ H S ( x ) ∥ F ≤ R H , S almost surely. Consequently, on every realized sample,
∥ h i ∥ 2 ≤ R h , ∥ H S , i ∥ F ≤ R H , S ( i ∈ [ N ] , S ∈ { Q , K , V } ) . \|h_i\|_2 \le R_h,\qquad\|H_{S,i}\|_F \le R_{H,S}\quad(i \in[N],\ S \in\{Q,K,V\}). ∥ h i ∥ 2 ≤ R h , ∥ H S , i ∥ F ≤ R H , S ( i ∈ [ N ] , S ∈ { Q , K , V }) .
Assumption 7 (Inter-stage token Lipschitzness for L L L consecutive attention modules). Fix trace-norm radii ( D t , S ) t ∈ [ L ] , S ∈ S t (D_{t,S})_{t\in[L],S\in\mathcal{S}_t} ( D t , S ) t ∈ [ L ] , S ∈ S t . For each t ∈ { 1 , … , L − 1 } t \in\{1,\ldots,L-1\} t ∈ { 1 , … , L − 1 } , assume there exists a finite constant L H , t ( D > t ) L_{H,t}(D_{>t}) L H , t ( D > t ) such that for all i ∈ [ N ] i \in[N] i ∈ [ N ] and all two upstream update collections { T τ , S } τ > t \{T_{\tau,S}\}_{\tau>t} { T τ , S } τ > t and { T τ , S ′ } τ > t \{T'_{\tau,S}\}_{\tau>t} { T τ , S ′ } τ > t satisfying ∥ T τ , S ∥ ⋆ , ∥ T τ , S ′ ∥ ⋆ ≤ D τ , S \|T_{\tau,S}\|_{\star},\|T'_{\tau,S}\|_{\star} \le D_{\tau,S} ∥ T τ , S ∥ ⋆ , ∥ T τ , S ′ ∥ ⋆ ≤ D τ , S ( τ > t ) (\tau>t) ( τ > t ) , we have
( ∑ S ∈ S t ∥ H t , S , i ( { T τ , S ′ } τ > t ) − H t , S , i ( { T τ , S ′ ′ } τ > t ) ∥ F 2 ) 1 / 2 ≤ L H , t ( D > t ) ( ∑ τ = t + 1 L ∑ S ∈ S τ ∥ T τ , S H τ , S , i ( { T ν , S ′ } ν > τ ) − T τ , S ′ H τ , S , i ( { T ν , S ′ ′ } ν > τ ) ∥ F 2 ) 1 / 2 . \left(\sum_{S\in\mathcal{S}_t}\left\|H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})-H_{t,S,i}(\{T'_{\tau,S'}\}_{\tau>t})\right\|_F^2\right)^{1/2}
\le L_{H,t}(D_{>t})\left(\sum_{\tau=t+1}^{L}\sum_{S\in\mathcal{S}_{\tau}}\left\|T_{\tau,S}H_{\tau,S,i}(\{T_{\nu,S'}\}_{\nu>\tau})-T'_{\tau,S}H_{\tau,S,i}(\{T'_{\nu,S'}\}_{\nu>\tau})\right\|_F^2\right)^{1/2}. ( S ∈ S t ∑ H t , S , i ({ T τ , S ′ } τ > t ) − H t , S , i ({ T τ , S ′ ′ } τ > t ) F 2 ) 1/2 ≤ L H , t ( D > t ) ( τ = t + 1 ∑ L S ∈ S τ ∑ T τ , S H τ , S , i ({ T ν , S ′ } ν > τ ) − T τ , S ′ H τ , S , i ({ T ν , S ′ ′ } ν > τ ) F 2 ) 1/2 .
Assumption 8 (Regular frozen suffix on reachable activation sets for L L L attention modules). Fix trace-norm radii ( D t , S ) (D_{t,S}) ( D t , S ) . Assume there exists a finite constant L a t t n ( D ) L_{\mathrm{attn}}(D) L attn ( D ) such that for all i ∈ [ N ] i \in[N] i ∈ [ N ] and all pairs of reachable collections { Z t , S , i } t , S , { Z t , S , i ′ } t , S \{Z_{t,S,i}\}_{t,S},\{Z'_{t,S,i}\}_{t,S} { Z t , S , i } t , S , { Z t , S , i ′ } t , S , one has
∥ ψ a t t n , i ( { Z t , S , i } t , S ) − ψ a t t n , i ( { Z t , S , i ′ } t , S ) ∥ 2 ≤ L a t t n ( D ) ( ∑ t = 1 L ∑ S ∈ S t ∥ Z t , S , i − Z t , S , i ′ ∥ F 2 ) 1 / 2 . \left\|\psi_{\mathrm{attn},i}(\{Z_{t,S,i}\}_{t,S})-\psi_{\mathrm{attn},i}(\{Z'_{t,S,i}\}_{t,S})\right\|_2
\le L_{\mathrm{attn}}(D)\left(\sum_{t=1}^{L}\sum_{S\in\mathcal{S}_t}\|Z_{t,S,i}-Z'_{t,S,i}\|_F^2\right)^{1/2}. ψ attn , i ({ Z t , S , i } t , S ) − ψ attn , i ({ Z t , S , i ′ } t , S ) 2 ≤ L attn ( D ) ( t = 1 ∑ L S ∈ S t ∑ ∥ Z t , S , i − Z t , S , i ′ ∥ F 2 ) 1/2 .
For the single-block case, let σ i ∈ { − 1 , 1 } m \sigma_i \in\{-1,1\}^{m} σ i ∈ { − 1 , 1 } m be an m m m -dimensional vector with independent Rademacher entries. Let E ∈ { − 1 , 1 } m × N E \in\{-1,1\}^{m \times N} E ∈ { − 1 , 1 } m × N be the matrix whose i i i -th column is σ i \sigma_i σ i . Then
∑ i = 1 N σ i h i = E H ⊤ . \sum_{i=1}^{N}\sigma_i h_i = EH^{\top}. i = 1 ∑ N σ i h i = E H ⊤ .
We also define
L ψ ( D , R h ) : = max i ∈ [ N ] sup z ∈ Z a c t ( D , R h ) ∥ D ψ i ( z ) ∥ 2 . L_{\psi}(D,R_h) := \max_{i\in[N]}\sup_{z\in\mathcal{Z}_{\mathrm{act}}(D,R_h)} \lVert D\psi_i(z)\rVert_2. L ψ ( D , R h ) := i ∈ [ N ] max z ∈ Z act ( D , R h ) sup ∥ D ψ i ( z ) ∥ 2 .
Assumption 9 (Regular single-block frozen suffix on reachable activation sets). Fix D ≥ 0 D \ge0 D ≥ 0 . Assume that for each i ∈ [ N ] i \in[N] i ∈ [ N ] , the map ψ i \psi_i ψ i is C 1 C^1 C 1 on an open neighborhood of Z a c t ( D , R h ) \mathcal{Z}_{\mathrm{act}}(D,R_h) Z act ( D , R h ) .
The set Z a c t ( D , R h ) \mathcal{Z}_{\mathrm{act}}(D,R_h) Z act ( D , R h ) is closed and bounded, hence compact. Since each D ψ i D\psi_i D ψ i is continuous, the supremum L ψ ( D , R h ) L_{\psi}(D,R_h) L ψ ( D , R h ) is finite.
The next two lemmas formalize the two “frozen suffix” decompositions used in the analysis. The first applies to a single adapted module, and the second extends the idea to L L L consecutive adapted attention modules.
Lemma B.1 (Single-cut suffix representation for one adapted module). Assume Assumption 2 holds. Let S \mathcal{S} S index adapted linear maps that lie behind one computation-graph cut, in the following sense: for every training sample i ∈ [ N ] i \in[N] i ∈ [ N ] , the input tensor H S , i H_{\mathcal{S},i} H S , i to each adapted map S \mathcal{S} S is computed before any map in S \mathcal{S} S is applied and is independent of the whole update tuple { T S ′ } \{T_{\mathcal{S}'}\} { T S ′ } . Then there exist deterministic maps
ψ i : ∏ S ∈ S R m S × ϱ S , i → R K \psi_i:\prod_{\mathcal{S}\in\mathcal{S}}\mathbb{R}^{m_{\mathcal{S}}\times\varrho_{\mathcal{S},i}} \to\mathbb{R}^{K} ψ i : S ∈ S ∏ R m S × ϱ S , i → R K depending only on x i x_i x i and the frozen weights such that, for all updates { T S } \{T_{\mathcal{S}}\} { T S } ,
f { W S , 0 + T S } ( x i ) = ψ i ( { ( W S , 0 + T S ) H S , i } ) = ψ i ( { W S , 0 H S , i + T S H S , i } ) . f_{\{W_{\mathcal{S},0}+T_{\mathcal{S}}\}}(x_i)
= \psi_i\left(\left\{(W_{\mathcal{S},0}+T_{\mathcal{S}})H_{\mathcal{S},i}\right\}\right)
= \psi_i\left(\left\{W_{\mathcal{S},0}H_{\mathcal{S},i}+T_{\mathcal{S}}H_{\mathcal{S},i}\right\}\right). f { W S , 0 + T S } ( x i ) = ψ i ( { ( W S , 0 + T S ) H S , i } ) = ψ i ( { W S , 0 H S , i + T S H S , i } ) . Consequently, for one adapted module, the empirical loss depends on each block T S T_{\mathcal{S}} T S only through the bottleneck outputs { T S H S , i } i = 1 N \{T_{\mathcal{S}}H_{\mathcal{S},i}\}_{i=1}^{N} { T S H S , i } i = 1 N , or equivalently through the stacked linear operator Φ S ( T S ) = T S H S \Phi_{\mathcal{S}}(T_{\mathcal{S}})=T_{\mathcal{S}}H_{\mathcal{S}} Φ S ( T S ) = T S H S .
Proof of Lemma B.1. Fix i ∈ [ N ] i \in[N] i ∈ [ N ] . By the single-cut assumption, the tensors { H S , i } \{H_{\mathcal{S},i}\} { H S , i } and all cached quantities produced before the adapted module are fixed functions of x i x_i x i and the frozen weights. Cut the computation graph at the outputs Z S , i : = ( W S , 0 + T S ) H S , i Z_{\mathcal{S},i}:=(W_{\mathcal{S},0}+T_{\mathcal{S}})H_{\mathcal{S},i} Z S , i := ( W S , 0 + T S ) H S , i , S ∈ S \mathcal{S}\in\mathcal{S} S ∈ S . After this cut, all remaining operations, residual connections, attention operations, nonlinearities, and output layers are frozen and deterministic. Therefore, the downstream computation defines a deterministic suffix map
ψ i : ∏ S ∈ S R m S × ϱ S , i → R K \psi_i:\prod_{\mathcal{S}\in\mathcal{S}}\mathbb{R}^{m_{\mathcal{S}}\times\varrho_{\mathcal{S},i}} \to\mathbb{R}^{K} ψ i : S ∈ S ∏ R m S × ϱ S , i → R K
from the collection { Z S , i } \{Z_{\mathcal{S},i}\} { Z S , i } to the logits. Substituting Z S , i = W S , 0 H S , i + T S H S , i Z_{\mathcal{S},i}=W_{\mathcal{S},0}H_{\mathcal{S},i}+T_{\mathcal{S}}H_{\mathcal{S},i} Z S , i = W S , 0 H S , i + T S H S , i gives the displayed representation. Thus, changing T S T_{\mathcal{S}} T S while keeping the rest of the module fixed can affect the empirical loss only through the finite collection { T S H S , i } i = 1 N \{T_{\mathcal{S}}H_{\mathcal{S},i}\}_{i=1}^{N} { T S H S , i } i = 1 N .
Lemma B.2 (Multi-cut decomposition for L L L consecutive adapted attention modules). Assume Assumption 2 holds. Consider L L L consecutive attention modules, indexed so that larger t t t is more upstream. For each stage t t t , the token matrix H t , S , i ( { T t ′ , S ′ } t ′ > t ) H_{t,\mathcal{S},i}(\{T_{t',\mathcal{S}'}\}_{t'>t}) H t , S , i ({ T t ′ , S ′ } t ′ > t ) entering block S \mathcal{S} S is allowed to depend on upstream updates, but not on the current block update T t , S T_{t,\mathcal{S}} T t , S . Then there exist deterministic maps
ψ L a t t n , i : ∏ t = 1 L ∏ S ∈ S t R m t , S × ϱ t , S , i → R K \psi_{\mathrm{Lattn},i}:\prod_{t=1}^{L}\prod_{\mathcal{S}\in\mathcal{S}_t}\mathbb{R}^{m_{t,\mathcal{S}}\times\varrho_{t,\mathcal{S},i}}\to\mathbb{R}^{K} ψ Lattn , i : t = 1 ∏ L S ∈ S t ∏ R m t , S × ϱ t , S , i → R K such that, for all update collections { T t , S } \{T_{t,\mathcal{S}}\} { T t , S } ,
f { W t , S , 0 + T t , S } ( x i ) = ψ L a t t n , i ( { ( W t , S , 0 + T t , S ) H t , S , i ( { T t ′ , S ′ } t ′ > t ) } t ∈ [ L ] , S ∈ S t ) . f_{\{W_{t,\mathcal{S},0}+T_{t,\mathcal{S}}\}}(x_i)
=\psi_{\mathrm{Lattn},i}\left(\left\{(W_{t,\mathcal{S},0}+T_{t,\mathcal{S}})H_{t,\mathcal{S},i}(\{T_{t',\mathcal{S}'}\}_{t'>t})\right\}_{t\in[L],\mathcal{S}\in\mathcal{S}_t}\right). f { W t , S , 0 + T t , S } ( x i ) = ψ Lattn , i ( { ( W t , S , 0 + T t , S ) H t , S , i ({ T t ′ , S ′ } t ′ > t ) } t ∈ [ L ] , S ∈ S t ) . Proof of Lemma B.2. Fix i ∈ [ N ] i \in[N] i ∈ [ N ] . We cut the computation graph once at each adapted attention module. At the most upstream adapted stage L L L , the inputs to the adapted projections are fixed by the frozen prefix, so the stage outputs have the form*
Z L , S , i = ( W L , S , 0 + T L , S ) H L , S , i . Z_{L,\mathcal{S},i}=(W_{L,\mathcal{S},0}+T_{L,\mathcal{S}})H_{L,\mathcal{S},i}. Z L , S , i = ( W L , S , 0 + T L , S ) H L , S , i .
Because the computation between stage L L L and stage L − 1 L-1 L − 1 is deterministic, the token matrices entering stage L − 1 L-1 L − 1 are deterministic functions of the stage-L L L cut outputs. Thus, the stage-L − 1 L-1 L − 1 adapted outputs have the form
Z L − 1 , S , i = ( W L − 1 , S , 0 + T L − 1 , S ) H L − 1 , S , i ( { T τ , S ′ } τ > L − 1 ) . Z_{L-1,S,i} = (W_{L-1,S,0} + T_{L-1,S})H_{L-1,S,i}(\{T_{\tau,S'}\}_{\tau>L-1}). Z L − 1 , S , i = ( W L − 1 , S , 0 + T L − 1 , S ) H L − 1 , S , i ({ T τ , S ′ } τ > L − 1 ) .
Repeating this argument down to stage 1 shows that every adapted output can be written as
Z t , S , i = ( W t , S , 0 + T t , S ) H t , S , i ( { T τ , S ′ } τ > t ) . Z_{t,S,i} = (W_{t,S,0} + T_{t,S})H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t}). Z t , S , i = ( W t , S , 0 + T t , S ) H t , S , i ({ T τ , S ′ } τ > t ) .
After all these cuts, the remaining suffix from the collection { Z t , S , i } t , S \{Z_{t,S,i}\}_{t,S} { Z t , S , i } t , S to the logits is frozen and deterministic. This suffix is the map ψ L a t t n , i \psi_{\mathrm{Lattn},i} ψ Lattn , i . The displayed formula follows by substituting the expressions for the cut outputs.
The Rademacher-complexity proof below requires applying vector contraction to functions whose natural domains are not all of Euclidean space, but only the reachable activation sets induced by trace-norm-bounded updates. To avoid introducing any extension assumption, we use the classical real-valued Lipschitz extension formula of McShane (1934). The lemma is stated in the metric-space form needed later.
Lemma B.3 (McShane extension theorem). Let ( X , d ) (X,d) ( X , d ) be a metric space, let ∅ ≠ A ⊆ X \varnothing\ne A \subseteq X ∅ = A ⊆ X , and let f : A → R f:A \to\mathbb{R} f : A → R be L L L -Lipschitz. Define
f ˉ ( x ) : = inf a ∈ A ( f ( a ) + L d ( x , a ) ) , x ∈ X . \bar{f}(x) := \inf_{a\in A}\left(f(a) + Ld(x,a)\right), \quad x\in X. f ˉ ( x ) := a ∈ A inf ( f ( a ) + L d ( x , a ) ) , x ∈ X . Then f ˉ : X → R \bar{f}:X \to\mathbb{R} f ˉ : X → R is an L L L -Lipschitz extension of f f f .
Proof of Lemma B.3. Fix x ∈ A x\in A x ∈ A . For every a ∈ A a\in A a ∈ A , the L L L -Lipschitz property of f f f gives
f ( a ) + L d ( x , a ) ≥ f ( x ) . f(a) + Ld(x,a) \ge f(x). f ( a ) + L d ( x , a ) ≥ f ( x ) .
Taking a = x a=x a = x shows equality is attained, hence f ˉ ( x ) = f ( x ) \bar{f}(x)=f(x) f ˉ ( x ) = f ( x ) . Thus f ˉ \bar{f} f ˉ extends f f f .
Now fix x , y ∈ X x,y\in X x , y ∈ X . For every $a\in A,
f ˉ ( x ) ≤ f ( a ) + L d ( x , a ) ≤ f ( a ) + L d ( y , a ) + L d ( x , y ) . \bar{f}(x) \le f(a) + Ld(x,a) \le f(a) + Ld(y,a) + Ld(x,y). f ˉ ( x ) ≤ f ( a ) + L d ( x , a ) ≤ f ( a ) + L d ( y , a ) + L d ( x , y ) .
Taking the infimum over a ∈ A a\in A a ∈ A yields f ˉ ( x ) ≤ f ˉ ( y ) + L d ( x , y ) \bar{f}(x) \le\bar{f}(y) + Ld(x,y) f ˉ ( x ) ≤ f ˉ ( y ) + L d ( x , y ) . Exchanging the roles of x x x and y y y gives f ˉ ( y ) ≤ f ˉ ( x ) + L d ( x , y ) \bar{f}(y) \le\bar{f}(x) + Ld(x,y) f ˉ ( y ) ≤ f ˉ ( x ) + L d ( x , y ) . Therefore ∣ f ˉ ( x ) − f ˉ ( y ) ∣ ≤ L d ( x , y ) |\bar{f}(x)-\bar{f}(y)| \le Ld(x,y) ∣ f ˉ ( x ) − f ˉ ( y ) ∣ ≤ L d ( x , y ) , so f ˉ \bar{f} f ˉ is L L L -Lipschitz.
We next show the standard Rademacher deviation bound that converts a complexity estimate into a high-probability generalization guarantee.
Lemma B.4 (Uniform deviation from conditional Rademacher complexity). Let F \mathcal{F} F be a class of functions f : X × Y → [ 0 , B ] f:\mathcal{X}\times\mathcal{Y}\to[0,B] f : X × Y → [ 0 , B ] . Let S d a t a = { ( x i , y i ) } i = 1 N S_{\mathrm{data}}=\{(x_i,y_i)\}_{i=1}^{N} S data = {( x i , y i ) } i = 1 N be i.i.d. from P \mathcal{P} P and write
E ^ S f : = 1 N ∑ i = 1 N f ( x i , y i ) , E f : = E ( x , y ) ∼ P [ f ( x , y ) ] . \widehat{\mathbb{E}}_S f := \frac{1}{N}\sum_{i=1}^{N}f(x_i,y_i), \qquad\mathbb{E}f := \mathbb{E}_{(x,y)\sim\mathcal{P}}[f(x,y)]. E S f := N 1 i = 1 ∑ N f ( x i , y i ) , E f := E ( x , y ) ∼ P [ f ( x , y )] . Define the (conditional) empirical Rademacher complexity
R S d a t a ( F ) : = E ϵ [ sup f ∈ F 1 N ∑ i = 1 N ϵ i f ( x i , y i ) ∣ S d a t a ] , \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}) := \mathbb{E}_{\epsilon}\left[\sup_{f\in\mathcal{F}}\frac{1}{N}\sum_{i=1}^{N}\epsilon_i f(x_i,y_i)\mid S_{\mathrm{data}}\right], R S data ( F ) := E ϵ [ f ∈ F sup N 1 i = 1 ∑ N ϵ i f ( x i , y i ) ∣ S data ] , where ϵ 1 , … , ϵ N \epsilon_1,\ldots,\epsilon_N ϵ 1 , … , ϵ N are i.i.d. uniform signs. Then for every η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) , with probability at least 1 − η 1-\eta 1 − η over S ∼ P N S\sim\mathcal{P}^{N} S ∼ P N ,
sup f ∈ F ( E f − E ^ S f ) ≤ 2 R S d a t a ( F ) + 3 B log ( 2 / η ) 2 N . \sup_{f\in\mathcal{F}}\left(\mathbb{E}f-\widehat{\mathbb{E}}_S f\right) \le2\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F})+3B\sqrt{\frac{\log(2/\eta)}{2N}}. f ∈ F sup ( E f − E S f ) ≤ 2 R S data ( F ) + 3 B 2 N log ( 2/ η ) .
Proof of Lemma B.4. Define
g ( S ) : = sup f ∈ F ( E f − E ^ S f ) . g(S):=\sup_{f\in\mathcal{F}}\left(\mathbb{E}f-\widehat{\mathbb{E}}_S f\right). g ( S ) := f ∈ F sup ( E f − E S f ) .
If we replace ( x i , y i ) (x_i,y_i) ( x i , y i ) by an independent copy ( x i ′ , y i ′ ) (x_i',y_i') ( x i ′ , y i ′ ) while keeping all other samples fixed, then for any fixed f f f ,
∣ E ^ S f − E ^ S d a t a ( i ) f ∣ = 1 N ∣ f ( x i , y i ) − f ( x i ′ , y i ′ ) ∣ ≤ B N . \left|\widehat{\mathbb{E}}_S f-\widehat{\mathbb{E}}_{S_{\mathrm{data}}^{(i)}}f\right|=\frac{1}{N}\left|f(x_i,y_i)-f(x_i',y_i')\right|\leq\frac{B}{N}. E S f − E S data ( i ) f = N 1 ∣ f ( x i , y i ) − f ( x i ′ , y i ′ ) ∣ ≤ N B .
Taking the supremum over f f f implies ∣ g ( S ) − g ( S d a t a ( i ) ) ∣ ≤ B / N |g(S)-g(S_{\mathrm{data}}^{(i)})|\leq B/N ∣ g ( S ) − g ( S data ( i ) ) ∣ ≤ B / N . McDiarmid’s inequality therefore yields: with probability at least 1 − η / 2 1-\eta/2 1 − η /2 ,
g ( S ) ≤ E [ g ( S ) ] + B log ( 2 / η ) 2 N . g(S)\leq\mathbb{E}[g(S)]+B\sqrt{\frac{\log(2/\eta)}{2N}}. g ( S ) ≤ E [ g ( S )] + B 2 N log ( 2/ η ) .
Let S ′ = { ( x i ′ , y i ′ ) } i = 1 N S'=\{(x_i',y_i')\}_{i=1}^{N} S ′ = {( x i ′ , y i ′ ) } i = 1 N be an i.i.d. ghost sample from P N \mathcal{P}^{N} P N , independent of S S S . Then
E [ g ( S ) ] = E S [ sup f ∈ F ( E ^ S ′ E ^ S ′ f − E ^ S f ) ] ≤ E S , S ′ [ sup f ∈ F ( E ^ S ′ f − E ^ S f ) ] . \mathbb{E}[g(S)]=\mathbb{E}_{S}\left[\sup_{f\in\mathcal{F}}\left(\widehat{\mathbb{E}}_{S'}\widehat{\mathbb{E}}_{S'}f-\widehat{\mathbb{E}}_{S}f\right)\right]\leq\mathbb{E}_{S,S'}\left[\sup_{f\in\mathcal{F}}\left(\widehat{\mathbb{E}}_{S'}f-\widehat{\mathbb{E}}_{S}f\right)\right]. E [ g ( S )] = E S [ f ∈ F sup ( E S ′ E S ′ f − E S f ) ] ≤ E S , S ′ [ f ∈ F sup ( E S ′ f − E S f ) ] .
Introduce i.i.d. signs ϵ 1 , … , ϵ N \epsilon_1,\ldots,\epsilon_N ϵ 1 , … , ϵ N , independent of ( S , S ′ ) (S,S') ( S , S ′ ) , and use that ( S , S ′ ) (S,S') ( S , S ′ ) and ( S ′ , S ) (S',S) ( S ′ , S ) have the same distribution to obtain
E S , S ′ [ sup f 1 N ∑ i = 1 N ( f ( x i ′ , y i ′ ) − f ( x i , y i ) ) ] = E S , S ′ , ϵ [ sup f 1 N ∑ i = 1 N ϵ i ( f ( x i ′ , y i ′ ) − f ( x i , y i ) ) ] . \mathbb{E}_{S,S'}\left[\sup_{f}\frac{1}{N}\sum_{i=1}^{N}\left(f(x_i',y_i')-f(x_i,y_i)\right)\right]=\mathbb{E}_{S,S',\epsilon}\left[\sup_{f}\frac{1}{N}\sum_{i=1}^{N}\epsilon_i\left(f(x_i',y_i')-f(x_i,y_i)\right)\right]. E S , S ′ [ f sup N 1 i = 1 ∑ N ( f ( x i ′ , y i ′ ) − f ( x i , y i ) ) ] = E S , S ′ , ϵ [ f sup N 1 i = 1 ∑ N ϵ i ( f ( x i ′ , y i ′ ) − f ( x i , y i ) ) ] .
By the triangle inequality and symmetry,
E [ g ( S ) ] ≤ 2 E S [ E ϵ [ sup f ∈ F 1 N ∑ i = 1 N ϵ i f ( x i , y i ) | S d a t a ] ] = 2 E S [ R S d a t a ( F ) ] . \mathbb{E}[g(S)]\leq2\mathbb{E}_{S}\left[\mathbb{E}_{\epsilon}\left[\sup_{f\in\mathcal{F}}\frac{1}{N}\sum_{i=1}^{N}\epsilon_i f(x_i,y_i)\,\middle|\,S_{\mathrm{data}}\right]\right]=2\mathbb{E}_{S}[\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F})]. E [ g ( S )] ≤ 2 E S [ E ϵ [ f ∈ F sup N 1 i = 1 ∑ N ϵ i f ( x i , y i ) S data ] ] = 2 E S [ R S data ( F )] .
Fix ϵ \epsilon ϵ and define
h ϵ ( S ) : = sup f ∈ F 1 N ∑ i = 1 N ϵ i f ( x i , y i ) . h_{\epsilon}(S):=\sup_{f\in\mathcal{F}}\frac{1}{N}\sum_{i=1}^{N}\epsilon_i f(x_i,y_i). h ϵ ( S ) := f ∈ F sup N 1 i = 1 ∑ N ϵ i f ( x i , y i ) .
Replacing one sample changes h ϵ ( S ) h_{\epsilon}(S) h ϵ ( S ) by at most B / N B/N B / N , hence R S d a t a ( F ) = E ϵ [ h ϵ ( S ) ∣ S ] \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F})=\mathbb{E}_{\epsilon}[h_{\epsilon}(S)\mid S] R S data ( F ) = E ϵ [ h ϵ ( S ) ∣ S ] also satisfies bounded differences with constant B / N B/N B / N . McDiarmid implies: with probability at least 1 − η / 2 1-\eta/2 1 − η /2 ,
E S [ R S d a t a ( F ) ] ≤ R S d a t a ( F ) + B log ( 2 / η ) 2 N . \mathbb{E}_{S}[\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F})]\leq\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F})+B\sqrt{\frac{\log(2/\eta)}{2N}}. E S [ R S data ( F )] ≤ R S data ( F ) + B 2 N log ( 2/ η ) .
On the intersection of the two 1 − η / 2 1-\eta/2 1 − η /2 events and using E [ g ( S ) ] ≤ 2 E [ R S d a t a ( F ) ] \mathbb{E}[g(S)]\leq2\mathbb{E}[\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F})] E [ g ( S )] ≤ 2 E [ R S data ( F )] , we get
g ( S ) ≤ 2 R S d a t a ( F ) + 3 B log ( 2 / η ) 2 N . g(S)\leq2\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F})+3B\sqrt{\frac{\log(2/\eta)}{2N}}. g ( S ) ≤ 2 R S data ( F ) + 3 B 2 N log ( 2/ η ) .
as desired.
Combining these results gives the main single-block generalization result. The bound reduces uniform generalization over the trace-norm ball to the empirical Rademacher complexity of the induced loss class, and then to the spectral norm of a signed feature matrix.
Theorem B.5 (Uniform generalization for trace-norm-bounded updates in frozen networks). Assume Assumptions 2, 5, 6, and 9 hold. Then with probability at least 1 − η 1-\eta 1 − η over S d a t a ∼ P N S_{\mathrm{data}}\sim\mathcal{P}^{N} S data ∼ P N ,
sup T ∈ B ⋆ ( D ) ( L ( T ) − L S d a t a v a n i l l a ( T ) ) ≤ 2 R S d a t a ( F D ) + 3 B log ( 2 / η ) 2 N . \sup_{T\in\mathbb{B}_{\star}(D)}\left(L(T)-\mathcal{L}_{S_{\mathrm{data}}}^{\mathrm{vanilla}}(T)\right)\leq2\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}_{D})+3B\sqrt{\frac{\log(2/\eta)}{2N}}. T ∈ B ⋆ ( D ) sup ( L ( T ) − L S data vanilla ( T ) ) ≤ 2 R S data ( F D ) + 3 B 2 N log ( 2/ η ) . Moreover,
R S d a t a ( F D ) ≤ 2 L ℓ L ψ ( D , R h ) D N E σ [ ∥ E H T ∥ 2 ∣ S d a t a ] . (2) \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}_{D})\leq\frac{\sqrt{2}L_{\ell}L_{\psi}(D,R_h)D}{N}\mathbb{E}_{\sigma}\left[\left\|EH^{\mathsf{T}}\right\|_2\mid S_{\mathrm{data}}\right]. \tag*{(2)} R S data ( F D ) ≤ N 2 L ℓ L ψ ( D , R h ) D E σ [ E H T 2 ∣ S data ] . (2)
Theorem B.5 reduces uniform generalization over the trace-norm ball B ⋆ ( D ) \mathbb{B}_{\star}(D) B ⋆ ( D ) to the spectral norm of a random signed feature matrix E H T EH^{\mathsf{T}} E H T .
Proof of Theorem B.5. The first display follows immediately from Lemma B.4 applied to the bounded class F D \mathcal{F}_D F D . It remains to prove (B.1). For each i ∈ [ N ] i \in[N] i ∈ [ N ] , define
K i ( D ) : = { T h i : ∥ T ∥ ⋆ ≤ D } ⊆ R m , ϕ i ( z ) : = ℓ ( ψ i ( W 0 h i + z ) , y i ) , z ∈ K i ( D ) . K_i(D) := \{T h_i : \|T\|_{\star} \le D\} \subseteq\mathbb{R}^m,\quad\phi_i(z) := \ell(\psi_i(W_0h_i+z),y_i),\ z \in K_i(D). K i ( D ) := { T h i : ∥ T ∥ ⋆ ≤ D } ⊆ R m , ϕ i ( z ) := ℓ ( ψ i ( W 0 h i + z ) , y i ) , z ∈ K i ( D ) .
Whenever z = T h i z=Th_i z = T h i and z ′ = T ′ h i z'=T'h_i z ′ = T ′ h i with ∥ T ∥ ⋆ , ∥ T ′ ∥ ⋆ ≤ D \|T\|_{\star},\|T'\|_{\star}\le D ∥ T ∥ ⋆ , ∥ T ′ ∥ ⋆ ≤ D , ∣ ϕ i ( z ) − ϕ i ( z ′ ) ∣ ≤ L ℓ L ψ ( D , R h ) ∥ z − z ′ ∥ 2 |\phi_i(z)-\phi_i(z')|\le L_{\ell}L_{\psi}(D,R_h)\|z-z'\|_2 ∣ ϕ i ( z ) − ϕ i ( z ′ ) ∣ ≤ L ℓ L ψ ( D , R h ) ∥ z − z ′ ∥ 2 . Define ϕ ~ i ( z ) : = ϕ i ( z ) − ϕ i ( 0 ) \widetilde{\phi}_i(z):=\phi_i(z)-\phi_i(0) ϕ i ( z ) := ϕ i ( z ) − ϕ i ( 0 ) on K i ( D ) K_i(D) K i ( D ) . Then ϕ ~ i ( 0 ) = 0 \widetilde{\phi}_i(0)=0 ϕ i ( 0 ) = 0 , and ϕ ~ i \widetilde{\phi}_i ϕ i is L ℓ L ψ ( D , R h ) L_{\ell}L_{\psi}(D,R_h) L ℓ L ψ ( D , R h ) -Lipschitz on K i ( D ) K_i(D) K i ( D ) .
Moreover, for every realization of ε \varepsilon ε ,
sup ∥ T ∥ ⋆ ≤ D 1 N ∑ i = 1 N ε i ϕ i ( T h i ) = sup ∥ T ∥ ⋆ ≤ D 1 N ∑ i = 1 N ε i ϕ ~ i ( T h i ) + 1 N ∑ i = 1 N ε i ϕ i ( 0 ) . \sup_{\|T\|_{\star}\le D}\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i\phi_i(Th_i)
=
\sup_{\|T\|_{\star}\le D}\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i\widetilde{\phi}_i(Th_i)
+
\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i\phi_i(0). ∥ T ∥ ⋆ ≤ D sup N 1 i = 1 ∑ N ε i ϕ i ( T h i ) = ∥ T ∥ ⋆ ≤ D sup N 1 i = 1 ∑ N ε i ϕ i ( T h i ) + N 1 i = 1 ∑ N ε i ϕ i ( 0 ) .
Taking the conditional expectation and using
E ε [ ∑ i = 1 N ε i ϕ i ( 0 ) ∣ S d a t a ] = 0 \mathbb{E}_{\varepsilon}\left[\left.\sum_{i=1}^{N}\varepsilon_i\phi_i(0)\right|S_{\mathrm{data}}\right]=0 E ε [ i = 1 ∑ N ε i ϕ i ( 0 ) S data ] = 0
gives
R S d a t a ( F D ) = E ε [ sup ∥ T ∥ ⋆ ≤ D 1 N ∑ i = 1 N ε i ϕ ~ i ( T h i ) ∣ S d a t a ] . \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}_D)
=
\mathbb{E}_{\varepsilon}\left[\left.
\sup_{\|T\|_{\star}\le D}\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i\widetilde{\phi}_i(Th_i)
\right|S_{\mathrm{data}}\right]. R S data ( F D ) = E ε [ ∥ T ∥ ⋆ ≤ D sup N 1 i = 1 ∑ N ε i ϕ i ( T h i ) S data ] .
By Lemma B.3, each ϕ ~ i \widetilde{\phi}_i ϕ i extends from K i ( D ) K_i(D) K i ( D ) to a function ϕ ‾ i : R m → R \overline{\phi}_i:\mathbb{R}^m\to\mathbb{R} ϕ i : R m → R with the same Lipschitz constant L ℓ L ψ ( D , R h ) L_{\ell}L_{\psi}(D,R_h) L ℓ L ψ ( D , R h ) . Since T h i ∈ K i ( D ) Th_i\in K_i(D) T h i ∈ K i ( D ) for every feasible T T T , we may write
R S d a t a ( F D ) = E ε [ sup ∥ T ∥ ⋆ ≤ D 1 N ∑ i = 1 N ε i ϕ ‾ i ( T h i ) ∣ S d a t a ] . \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}_D)
=
\mathbb{E}_{\varepsilon}\left[\left.
\sup_{\|T\|_{\star}\le D}\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i\overline{\phi}_i(Th_i)
\right|S_{\mathrm{data}}\right]. R S data ( F D ) = E ε [ ∥ T ∥ ⋆ ≤ D sup N 1 i = 1 ∑ N ε i ϕ i ( T h i ) S data ] .
Applying Maurer’s vector contraction inequality Maurer [14 ] gives
R S d a t a ( F D ) ≤ 2 L ℓ L ψ ( D , R h ) N E σ [ sup ∥ T ∥ ⋆ ≤ D ∑ i = 1 N ⟨ σ i , T h i ⟩ ∣ S d a t a ] . \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}_D)
\le
\frac{\sqrt{2}L_{\ell}L_{\psi}(D,R_h)}{N}
\mathbb{E}_{\sigma}\left[\left.
\sup_{\|T\|_{\star}\le D}\sum_{i=1}^{N}\langle\sigma_i,Th_i\rangle
\right|S_{\mathrm{data}}\right]. R S data ( F D ) ≤ N 2 L ℓ L ψ ( D , R h ) E σ [ ∥ T ∥ ⋆ ≤ D sup i = 1 ∑ N ⟨ σ i , T h i ⟩ S data ] .
Stacking the σ i \sigma_i σ i as columns of E E E gives
∑ i = 1 N ⟨ σ i , T h i ⟩ = ⟨ ∑ i = 1 N σ i h i ⊤ , T ⟩ F = ⟨ E H ⊤ , T ⟩ F . \sum_{i=1}^{N}\langle\sigma_i,Th_i\rangle
=
\left\langle\sum_{i=1}^{N}\sigma_i h_i^{\top},T\right\rangle_F
=
\langle EH^{\top},T\rangle_F. i = 1 ∑ N ⟨ σ i , T h i ⟩ = ⟨ i = 1 ∑ N σ i h i ⊤ , T ⟩ F = ⟨ E H ⊤ , T ⟩ F .
Therefore, by trace/spectral duality,
sup ∥ T ∥ ⋆ ≤ D ∑ i = 1 N ⟨ σ i , T h i ⟩ = D ∥ E H ⊤ ∥ 2 . \sup_{\|T\|_{\star}\le D}\sum_{i=1}^{N}\langle\sigma_i,Th_i\rangle
=
D\|EH^{\top}\|_2. ∥ T ∥ ⋆ ≤ D sup i = 1 ∑ N ⟨ σ i , T h i ⟩ = D ∥ E H ⊤ ∥ 2 .
Substituting this into the previous display yields (B.1). □ \square □
Why generalization depends on ∥ E H ⊤ ∥ 2 \|EH^{\top}\|_2 ∥ E H ⊤ ∥ 2 . The trace norm is the dual of the operator norm: sup ∥ T ∥ ⋆ ≤ D ⟨ T , M ⟩ = D ∥ M ∥ 2 \sup_{\|T\|_{\star}\le D}\langle T,M\rangle=D\|M\|_2 sup ∥ T ∥ ⋆ ≤ D ⟨ T , M ⟩ = D ∥ M ∥ 2 . After symmetrization and vector contraction, the complexity of the frozen-network loss class reduces to such a dual pairing with M = ∑ i σ i h i ⊤ = E H ⊤ M=\sum_i\sigma_i h_i^{\top}=EH^{\top} M = ∑ i σ i h i ⊤ = E H ⊤ . Thus, the effective capacity is governed by the spectral geometry of frozen features, not by the ambient parameter dimension m n mn mn .
From here, for each ( t , S ) (t,S) ( t , S ) , let E t , S E_{t,S} E t , S be an independent Rademacher matrix in { − 1 , 1 } m t , S × N \{-1,1\}^{m_{t,S}\times N} { − 1 , 1 } m t , S × N , ∑ i = 1 N ϱ t , S , i \sum_{i=1}^{N}\varrho_{t,S,i} ∑ i = 1 N ϱ t , S , i
Standing notation for the L L L -stage generalization argument. Fix a realized sample S d a t a = { ( x i , y i ) } i = 1 N S_{\mathrm{data}} = \{(x_i,y_i)\}_{i=1}^{N} S data = {( x i , y i ) } i = 1 N and radii D = ( D t , S ) t ∈ [ L ] , S ∈ S t D = (D_{t,S})_{t\in[L],S\in\mathcal{S}_t} D = ( D t , S ) t ∈ [ L ] , S ∈ S t . Define
Θ D : = { { T t , S } t ∈ [ L ] , S ∈ S t : ∥ T t , S ∥ ⋆ ≤ D t , S ∀ t ∈ [ L ] , S ∈ S t } . \Theta_D := \left\{\{T_{t,S}\}_{t\in[L],S\in\mathcal{S}_t} : \|T_{t,S}\|_{\star} \le D_{t,S}\ \forall t\in[L],\ S\in\mathcal{S}_t\right\}. Θ D := { { T t , S } t ∈ [ L ] , S ∈ S t : ∥ T t , S ∥ ⋆ ≤ D t , S ∀ t ∈ [ L ] , S ∈ S t } .
For each ( t , S ) (t,S) ( t , S ) let ϱ t , S max : = max i ∈ [ N ] ϱ t , S , i \varrho^{\max}_{t,S} := \max_{i\in[N]} \varrho_{t,S,i} ϱ t , S m a x := max i ∈ [ N ] ϱ t , S , i , and define the Hilbert product space
H : = ⨁ t = 1 L ⨁ S ∈ S t R m t , S × ϱ t , S max , \mathcal{H} := \bigoplus_{t=1}^{L}\bigoplus_{S\in\mathcal{S}_t}\mathbb{R}^{m_{t,S}\times\varrho^{\max}_{t,S}}, H := t = 1 ⨁ L S ∈ S t ⨁ R m t , S × ϱ t , S m a x ,
equipped with the product Frobenius norm, where each R m t , S × ϱ t , S , i \mathbb{R}^{m_{t,S}\times\varrho_{t,S,i}} R m t , S × ϱ t , S , i is identified with its zero-padded copy inside R m t , S × ϱ t , S max \mathbb{R}^{m_{t,S}\times\varrho^{\max}_{t,S}} R m t , S × ϱ t , S m a x .
For each i ∈ [ N ] i\in[N] i ∈ [ N ] , define the reachable observed-output set
K i ( D ) : = { ( { T t , S H t , S , i ( { T τ , S ′ } τ > t ) } S ∈ S t ) t = 1 L : { T t , S } ∈ Θ D } ⊆ H . K_i(D) := \left\{\left(\{T_{t,S}H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})\}_{S\in\mathcal{S}_t}\right)_{t=1}^{L} : \{T_{t,S}\}\in\Theta_D\right\}\subseteq\mathcal{H}. K i ( D ) := { ( { T t , S H t , S , i ({ T τ , S ′ } τ > t ) } S ∈ S t ) t = 1 L : { T t , S } ∈ Θ D } ⊆ H .
Since the all-zero update tuple belongs to Θ D \Theta_D Θ D , we have 0 ∈ K i ( D ) 0\in K_i(D) 0 ∈ K i ( D ) .
For z ∈ K i ( D ) z\in K_i(D) z ∈ K i ( D ) , choose any feasible tuple { T t , S } ∈ Θ D \{T_{t,S}\}\in\Theta_D { T t , S } ∈ Θ D producing z z z and define
ϕ i ( z ) : = ℓ ( f { W t , S , 0 + T t , S } ( x i ) , y i ) , ϕ ~ i ( z ) : = ϕ i ( z ) − ϕ i ( 0 ) . \phi_i(z):=\ell(f_{\{W_{t,S,0}+T_{t,S}\}}(x_i),y_i),\qquad\widetilde{\phi}_i(z):=\phi_i(z)-\phi_i(0). ϕ i ( z ) := ℓ ( f { W t , S , 0 + T t , S } ( x i ) , y i ) , ϕ i ( z ) := ϕ i ( z ) − ϕ i ( 0 ) .
This is well-defined by Lemma B.2 , because the observed-output tuple determines all full adapted outputs stage by stage and therefore determines the logits uniquely.
Finally, define
Ψ i ( { T t , S } ) : = ( { T t , S H t , S , i ( { T τ , S ′ } τ > t ) } S ∈ S t ) t = 1 L ∈ H . \Psi_i(\{T_{t,S}\}):=\left(\{T_{t,S}H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})\}_{S\in\mathcal{S}_t}\right)_{t=1}^{L}\in\mathcal{H}. Ψ i ({ T t , S }) := ( { T t , S H t , S , i ({ T τ , S ′ } τ > t ) } S ∈ S t ) t = 1 L ∈ H .
Define the corresponding loss class
F D ( L ) : = { ( x , y ) ↦ ℓ ( f { W t , S , 0 + T t , S } ( x ) , y ) : { T t , S } ∈ Θ D } . \mathcal{F}^{(L)}_D:=\left\{(x,y)\mapsto\ell(f_{\{W_{t,S,0}+T_{t,S}\}}(x),y):\{T_{t,S}\}\in\Theta_D\right\}. F D ( L ) := { ( x , y ) ↦ ℓ ( f { W t , S , 0 + T t , S } ( x ) , y ) : { T t , S } ∈ Θ D } .
The multi-stage attention bound has one additional difficulty compared with the single-block bound: perturbing an upstream observed output can change the token matrices seen by downstream adapted blocks. The next deterministic lemma isolates this recursive amplification.
Lemma B.6 (Recursive triangular amplification). Fix nonnegative numbers ρ 1 , … , ρ L − 1 \rho_1,\ldots,\rho_{L-1} ρ 1 , … , ρ L − 1 . Set
Γ L : = 1 , Γ t : = ∥ [ 1 ρ t 0 Γ t + 1 ] ∥ 2 ( t = L − 1 , … , 1 ) . \Gamma_L:=1,\qquad\Gamma_t:=\left\|\begin{bmatrix}1&\rho_t\\0&\Gamma_{t+1}\end{bmatrix}\right\|_2\qquad(t=L-1,\ldots,1). Γ L := 1 , Γ t := [ 1 0 ρ t Γ t + 1 ] 2 ( t = L − 1 , … , 1 ) . If nonnegative numbers a t , b t a_t,b_t a t , b t satisfy
b L ≤ a L , b t ≤ a t + ρ t ( ∑ τ = t + 1 L a τ 2 ) 1 / 2 ( t = 1 , … , L − 1 ) , b_L\le a_L,\qquad b_t\le a_t+\rho_t\left(\sum_{\tau=t+1}^{L}a_\tau^2\right)^{1/2}\qquad(t=1,\ldots,L-1), b L ≤ a L , b t ≤ a t + ρ t ( τ = t + 1 ∑ L a τ 2 ) 1/2 ( t = 1 , … , L − 1 ) , then for every t ∈ [ L ] t\in[L] t ∈ [ L ] ,
( ∑ τ = t L b τ 2 ) 1 / 2 ≤ Γ t ( ∑ τ = t L a τ 2 ) 1 / 2 . \left(\sum_{\tau=t}^{L}b_\tau^2\right)^{1/2}\le\Gamma_t\left(\sum_{\tau=t}^{L}a_\tau^2\right)^{1/2}. ( τ = t ∑ L b τ 2 ) 1/2 ≤ Γ t ( τ = t ∑ L a τ 2 ) 1/2 .
Proof of Lemma B.6. We argue by backward induction on t t t . The case t = L t=L t = L is immediate. Assume the claim holds for t + 1 t+1 t + 1 and set c : = ( ∑ τ = t + 1 L a τ 2 ) 1 / 2 c:=\left(\sum_{\tau=t+1}^{L}a_\tau^2\right)^{1/2} c := ( ∑ τ = t + 1 L a τ 2 ) 1/2 . Then
( ∑ τ = t L b τ 2 ) 1 / 2 ≤ ( a t + ρ t c ) 2 + Γ t + 1 2 c 2 = ∥ [ 1 ρ t 0 Γ t + 1 ] [ a t c ] ∥ 2 ≤ Γ t a t 2 + c 2 , \left(\sum_{\tau=t}^{L}b_\tau^2\right)^{1/2}\le\sqrt{(a_t+\rho_tc)^2+\Gamma_{t+1}^2c^2}
=\left\|\begin{bmatrix}1&\rho_t\\0&\Gamma_{t+1}\end{bmatrix}\begin{bmatrix}a_t\\c\end{bmatrix}\right\|_2
\le\Gamma_t\sqrt{a_t^2+c^2}, ( τ = t ∑ L b τ 2 ) 1/2 ≤ ( a t + ρ t c ) 2 + Γ t + 1 2 c 2 = [ 1 0 ρ t Γ t + 1 ] [ a t c ] 2 ≤ Γ t a t 2 + c 2 ,
which is exactly the desired bound. ∎
The previous lemma provides the recursive norm-growth factor that controls how perturbations accumulate across consecutive stages. Combining it with the observed output Lipschitz bound yields the following estimate for the full L L L -module attention.
Lemma B.7 (Observed-output Lipschitzness for L L L consecutive attention modules). Assume Assumptions 2, 5, 7, and 8 hold. For t = 1 , … , L − 1 t=1,\ldots,L-1 t = 1 , … , L − 1 , define
∥ W t , 0 ∥ S t : = ( ∑ S ∈ S t ∥ W t , S , 0 ∥ 2 2 ) 1 / 2 , ρ t ( D ) : = ∥ W t , 0 ∥ S t L H , t ( D > t ) , \lVert W_{t,0}\rVert_{\mathcal{S}_t}:=\left(\sum_{S\in\mathcal{S}_t}\lVert W_{t,S,0}\rVert_2^2\right)^{1/2},\qquad\rho_t(D):=\lVert W_{t,0}\rVert_{\mathcal{S}_t}L_{H,t}(D_{>t}), ∥ W t , 0 ∥ S t := ( S ∈ S t ∑ ∥ W t , S , 0 ∥ 2 2 ) 1/2 , ρ t ( D ) := ∥ W t , 0 ∥ S t L H , t ( D > t ) , and let Γ t ( D ) \Gamma_t(D) Γ t ( D ) be the constants from Lemma B.6. Then for any two update collections in the product trace-norm ball and every i ∈ [ N ] i\in[N] i ∈ [ N ] ,
∣ ℓ ( f { W t , S , 0 + T t , S } S ∈ S t ( x i ) , y i ) − ℓ ( f { W t , S , 0 + T t , S ′ } S ∈ S t ( x i ) , y i ) ∣ \left|\ell\left(f_{\{W_{t,S,0}+T_{t,S}\}_{S\in\mathcal{S}_t}}(x_i),y_i\right)-\ell\left(f_{\{W_{t,S,0}+T'_{t,S}\}_{S\in\mathcal{S}_t}}(x_i),y_i\right)\right| ℓ ( f { W t , S , 0 + T t , S } S ∈ S t ( x i ) , y i ) − ℓ ( f { W t , S , 0 + T t , S ′ } S ∈ S t ( x i ) , y i ) ≤ L ℓ L a t t n ( D ) Γ 1 ( D ) ( ∑ t = 1 L ∑ S ∈ S t ∥ T t , S H t , S , i ( { T τ , S ′ } τ > t ) − T t , S ′ H t , S , i ( { T τ , S ′ ′ } τ > t ) ∥ F 2 ) 1 / 2 . \le L_\ell L_{\mathrm{attn}}(D)\Gamma_1(D)\left(\sum_{t=1}^{L}\sum_{S\in\mathcal{S}_t}\left\lVert T_{t,S}H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})-T'_{t,S}H_{t,S,i}(\{T'_{\tau,S'}\}_{\tau>t})\right\rVert_F^2\right)^{1/2}. ≤ L ℓ L attn ( D ) Γ 1 ( D ) ( t = 1 ∑ L S ∈ S t ∑ T t , S H t , S , i ({ T τ , S ′ } τ > t ) − T t , S ′ H t , S , i ({ T τ , S ′ ′ } τ > t ) F 2 ) 1/2 .
Proof of Lemma B.7. Fix i i i . Define
a t : = ( ∑ S ∈ S t ∥ T t , S H t , S , i ( { T τ , S ′ } τ > t ) − T t , S ′ H t , S , i ( { T τ , S ′ ′ } τ > t ) ∥ F 2 ) 1 / 2 . a_t:=\left(\sum_{S\in\mathcal{S}_t}\left\lVert T_{t,S}H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})-T'_{t,S}H_{t,S,i}(\{T'_{\tau,S'}\}_{\tau>t})\right\rVert_F^2\right)^{1/2}. a t := ( S ∈ S t ∑ T t , S H t , S , i ({ T τ , S ′ } τ > t ) − T t , S ′ H t , S , i ({ T τ , S ′ ′ } τ > t ) F 2 ) 1/2 .
Also, define the actual adapted outputs
Z t , S , i : = ( W t , S , 0 + T t , S ) H t , S , i ( { T τ , S ′ } τ > t ) , Z_{t,S,i}:=(W_{t,S,0}+T_{t,S})H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t}), Z t , S , i := ( W t , S , 0 + T t , S ) H t , S , i ({ T τ , S ′ } τ > t ) ,
Z t , S , i ′ : = ( W t , S , 0 + T t , S ′ ) H t , S , i ( { T τ , S ′ ′ } τ > t ) , Z'_{t,S,i}:=(W_{t,S,0}+T'_{t,S})H_{t,S,i}(\{T'_{\tau,S'}\}_{\tau>t}), Z t , S , i ′ := ( W t , S , 0 + T t , S ′ ) H t , S , i ({ T τ , S ′ ′ } τ > t ) ,
and
b t : = ( ∑ S ∈ S t ∥ Z t , S , i − Z t , S , i ′ ∥ F 2 ) 1 / 2 . b_t:=\left(\sum_{S\in\mathcal{S}_t}\lVert Z_{t,S,i}-Z'_{t,S,i}\rVert_F^2\right)^{1/2}. b t := ( S ∈ S t ∑ ∥ Z t , S , i − Z t , S , i ′ ∥ F 2 ) 1/2 .
For the top stage L L L , the token matrices are frozen, hence b L ≤ a L b_L\le a_L b L ≤ a L . For t < L t<L t < L , write
Δ H t , S , i : = H t , S , i ( { T τ , S ′ } τ > t ) − H t , S , i ( { T τ , S ′ ′ } τ > t ) , \Delta H_{t,S,i}:=H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})-H_{t,S,i}(\{T'_{\tau,S'}\}_{\tau>t}), Δ H t , S , i := H t , S , i ({ T τ , S ′ } τ > t ) − H t , S , i ({ T τ , S ′ ′ } τ > t ) ,
so that
Z t , S , i − Z t , S , i ′ = W t , S , 0 Δ H t , S , i + ( T t , S H t , S , i ( { T τ , S ′ } τ > t ) − T t , S ′ H t , S , i ( { T τ , S ′ ′ } τ > t ) ) . Z_{t,S,i}-Z'_{t,S,i}=W_{t,S,0}\Delta H_{t,S,i}+\left(T_{t,S}H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})-T'_{t,S}H_{t,S,i}(\{T'_{\tau,S'}\}_{\tau>t})\right). Z t , S , i − Z t , S , i ′ = W t , S , 0 Δ H t , S , i + ( T t , S H t , S , i ({ T τ , S ′ } τ > t ) − T t , S ′ H t , S , i ({ T τ , S ′ ′ } τ > t ) ) .
Therefore
b t ≤ a t + ( ∑ S ∈ S t ∥ W t , S , 0 Δ H t , S , i ∥ F 2 ) 1 / 2 . b_t\le a_t+\left(\sum_{S\in\mathcal{S}_t}\lVert W_{t,S,0}\Delta H_{t,S,i}\rVert_F^2\right)^{1/2}. b t ≤ a t + ( S ∈ S t ∑ ∥ W t , S , 0 Δ H t , S , i ∥ F 2 ) 1/2 .
Using ∥ A B ∥ F ≤ ∥ A ∥ 2 ∥ B ∥ F \lVert AB\rVert_F\le\lVert A\rVert_2\lVert B\rVert_F ∥ A B ∥ F ≤ ∥ A ∥ 2 ∥ B ∥ F and Cauchy–Schwarz across S S S ,
b t ≤ a t + ∥ W t , 0 ∥ S t ( ∑ S ∈ S t ∥ Δ H t , S , i ∥ F 2 ) 1 / 2 . b_t\le a_t+\lVert W_{t,0}\rVert_{\mathcal{S}_t}\left(\sum_{S\in\mathcal{S}_t}\lVert\Delta H_{t,S,i}\rVert_F^2\right)^{1/2}. b t ≤ a t + ∥ W t , 0 ∥ S t ( S ∈ S t ∑ ∥ Δ H t , S , i ∥ F 2 ) 1/2 .
Assumption 7 gives
b t ≤ a t + ρ t ( D ) ( ∑ τ = t + 1 L a τ 2 ) 1 / 2 . b_t\le a_t+\rho_t(D)\left(\sum_{\tau=t+1}^{L}a_\tau^2\right)^{1/2}. b t ≤ a t + ρ t ( D ) ( τ = t + 1 ∑ L a τ 2 ) 1/2 .
Applying Lemma B.6,
( ∑ t = 1 L b t 2 ) 1 / 2 ≤ Γ 1 ( D ) ( ∑ t = 1 L a t 2 ) 1 / 2 . \left(\sum_{t=1}^{L}b_t^2\right)^{1/2}\le\Gamma_1(D)\left(\sum_{t=1}^{L}a_t^2\right)^{1/2}. ( t = 1 ∑ L b t 2 ) 1/2 ≤ Γ 1 ( D ) ( t = 1 ∑ L a t 2 ) 1/2 .
Finally, Assumption 8 and the L ℓ L_\ell L ℓ -Lipschitz property of ℓ ( ⋅ , y i ) \ell(\cdot,y_i) ℓ ( ⋅ , y i ) imply
∣ ℓ ( f T ( x i ) , y i ) − ℓ ( f T ′ ( x i ) , y i ) ∣ ≤ L ℓ L a t t n ( D ) ( ∑ t = 1 L b t 2 ) 1 / 2 , \left|\ell(f_T(x_i),y_i)-\ell(f_{T'}(x_i),y_i)\right|\le L_\ell L_{\mathrm{attn}}(D)\left(\sum_{t=1}^{L}b_t^2\right)^{1/2}, ∣ ℓ ( f T ( x i ) , y i ) − ℓ ( f T ′ ( x i ) , y i ) ∣ ≤ L ℓ L attn ( D ) ( t = 1 ∑ L b t 2 ) 1/2 ,
which, combined with the previous display, proves the claim.
With this estimate, we can state the multi-stage generalization bound. The result is the attention analogue of the single-block theorem above, with the random signed matrices appearing stage by stage.
Theorem B.8 (Uniform generalization for trace-norm-bounded updates in L L L consecutive attention modules). *Assume Assumptions 2,5,7, and 8 hold. Fix radii ( D t , S ) t ∈ [ L ] , S ∈ S t (D_{t,S})_{t\in[L],S\in\mathcal{S}_t} ( D t , S ) t ∈ [ L ] , S ∈ S t . For each ( t , S ) (t,S) ( t , S ) , let E t , S E_{t,S} E t , S be an independent Rademacher matrix of the same shape as the stacked stage-t t t , block-S S S feature matrix. Then with probability at least 1 − η 1-\eta 1 − η over S d a t a ∼ P N S_{\mathrm{data}}\sim P^N S data ∼ P N ,
sup ∥ T t , S ∥ ⋆ ≤ D t , S ( L ( { T t , S } ) − L S d a t a v a n i l l a ( { T t , S } ) ) ≤ 2 2 L ℓ L a t t n ( D ) Γ 1 ( D ) N ∑ t = 1 L ∑ S ∈ S t D t , S E [ sup ∥ T τ , S ′ ∥ ⋆ ≤ D τ , S ′ ( τ > t ) ∥ E t , S H t , S ( { T τ , S ′ } τ > t ) ⊤ ∥ 2 | S d a t a ] \sup_{\lVert T_{t,S}\rVert_{\star}\leq D_{t,S}}\left(\mathcal{L}(\{T_{t,S}\})-\mathcal{L}^{\mathrm{vanilla}}_{S_{\mathrm{data}}}(\{T_{t,S}\})\right)
\leq\frac{2\sqrt{2L\ell L_{\mathrm{attn}}(D)\Gamma_1(D)}}{N}\sum_{t=1}^{L}\sum_{S\in\mathcal{S}_t}D_{t,S}\mathbb{E}\left[\sup_{\lVert T_{\tau,S'}\rVert_{\star}\leq D_{\tau,S'}\,(\tau>t)}\left\lVert E_{t,S}H_{t,S}(\{T_{\tau,S'}\}_{\tau>t})^{\top}\right\rVert_2\middle|S_{\mathrm{data}}\right] ∥ T t , S ∥ ⋆ ≤ D t , S sup ( L ({ T t , S }) − L S data vanilla ({ T t , S }) ) ≤ N 2 2 L ℓ L attn ( D ) Γ 1 ( D ) t = 1 ∑ L S ∈ S t ∑ D t , S E [ ∥ T τ , S ′ ∥ ⋆ ≤ D τ , S ′ ( τ > t ) sup E t , S H t , S ({ T τ , S ′ } τ > t ) ⊤ 2 S data ] + 3 B log ( 2 / η ) 2 N . +3B\sqrt{\frac{\log(2/\eta)}{2N}}. + 3 B 2 N log ( 2/ η ) . Here, the supremum over ( τ > t ) (\tau>t) ( τ > t ) is interpreted as vacuous when t = L t=L t = L .
The same statement applies verbatim to multi-head attention by taking the block family at stage t t t to be a head-indexed collection such as A t × { Q , K , V } \mathcal{A}_t\times\{Q,K,V\} A t × { Q , K , V } , or the corresponding shared-block collection in grouped-query or multi-query attention.
Proof of Theorem B.8. By Lemma B.4 , it suffices to bound R S d a t a ( F D ( L ) ) \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}^{(L)}_D) R S data ( F D ( L ) ) . Fix a realized sample S d a t a S_{\mathrm{data}} S data . By Lemma B.7 ,
∣ ϕ ~ i ( z ) − ϕ ~ i ( z ′ ) ∣ ≤ L ℓ L a t t n ( D ) Γ 1 ( D ) ∥ z − z ′ ∥ H ∀ z , z ′ ∈ K i ( D ) . \left|\widetilde{\phi}_i(z)-\widetilde{\phi}_i(z')\right|\leq L_{\ell}L_{\mathrm{attn}}(D)\Gamma_1(D)\lVert z-z'\rVert_{\mathcal{H}}\quad\forall z,z'\in K_i(D). ϕ i ( z ) − ϕ i ( z ′ ) ≤ L ℓ L attn ( D ) Γ 1 ( D ) ∥ z − z ′ ∥ H ∀ z , z ′ ∈ K i ( D ) .
Because 0 ∈ K i ( D ) 0\in K_i(D) 0 ∈ K i ( D ) and ϕ ~ i ( 0 ) = 0 \widetilde{\phi}_i(0)=0 ϕ i ( 0 ) = 0 , Lemma B.3 yields an extension ϕ ‾ i : H → R \overline{\phi}_i:\mathcal{H}\to\mathbb{R} ϕ i : H → R such that
ϕ ‾ i ∣ K i ( D ) = ϕ ~ i \left.\overline{\phi}_i\right|_{K_i(D)}=\widetilde{\phi}_i ϕ i K i ( D ) = ϕ i
and
∣ ϕ ‾ i ( z ) − ϕ ‾ i ( z ′ ) ∣ ≤ L ℓ L a t t n ( D ) Γ 1 ( D ) ∥ z − z ′ ∥ H ∀ z , z ′ ∈ H . \left|\overline{\phi}_i(z)-\overline{\phi}_i(z')\right|\leq L_{\ell}L_{\mathrm{attn}}(D)\Gamma_1(D)\lVert z-z'\rVert_{\mathcal{H}}\quad\forall z,z'\in\mathcal{H}. ϕ i ( z ) − ϕ i ( z ′ ) ≤ L ℓ L attn ( D ) Γ 1 ( D ) ∥ z − z ′ ∥ H ∀ z , z ′ ∈ H .
Therefore
R S d a t a ( F D ( L ) ) = E ε [ sup { T t , S } ∈ Θ D 1 N ∑ i = 1 N ε i ϕ ~ i ( Ψ i ( { T t , S } ) ) | S d a t a ] = E ε [ sup { T t , S } ∈ Θ D 1 N ∑ i = 1 N ε i ϕ ‾ i ( Ψ i ( { T t , S } ) ) | S d a t a ] . \begin{aligned}
\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}^{(L)}_D)=\mathbb{E}_{\varepsilon}\left[\sup_{\{T_{t,S}\}\in\Theta_D}\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i\widetilde{\phi}_i\left(\Psi_i(\{T_{t,S}\})\right)\middle|S_{\mathrm{data}}\right] \\
&=\mathbb{E}_{\varepsilon}\left[\sup_{\{T_{t,S}\}\in\Theta_D}\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i\overline{\phi}_i\left(\Psi_i(\{T_{t,S}\})\right)\middle|S_{\mathrm{data}}\right].
\end{aligned} R S data ( F D ( L ) ) = E ε [ { T t , S } ∈ Θ D sup N 1 i = 1 ∑ N ε i ϕ i ( Ψ i ({ T t , S }) ) S data ] = E ε [ { T t , S } ∈ Θ D sup N 1 i = 1 ∑ N ε i ϕ i ( Ψ i ({ T t , S }) ) S data ] .
Applying Maurer [14 ] ’s vector contraction inequality to the class { i ↦ Ψ i ( { T t , S } ) : { T t , S } ∈ Θ D } \{i\mapsto\Psi_i(\{T_{t,S}\}):\{T_{t,S}\}\in\Theta_D\} { i ↦ Ψ i ({ T t , S }) : { T t , S } ∈ Θ D } gives
R S d a t a ( F D ( L ) ) ≤ C E [ sup { T t , S } ∈ Θ D 1 N ∑ t = 1 L ∑ S ∈ S t ∑ i = 1 N ⟨ Σ t , S , i , T t , S H t , S , i ( { T τ , S ′ } τ > t ) ⟩ F | S d a t a ] , \mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}^{(L)}_D)\leq C\mathbb{E}\left[\sup_{\{T_{t,S}\}\in\Theta_D}\frac{1}{N}\sum_{t=1}^{L}\sum_{S\in\mathcal{S}_t}\sum_{i=1}^{N}\left\langle\Sigma_{t,S,i},T_{t,S}H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})\right\rangle_F\middle|S_{\mathrm{data}}\right], R S data ( F D ( L ) ) ≤ C E [ { T t , S } ∈ Θ D sup N 1 t = 1 ∑ L S ∈ S t ∑ i = 1 ∑ N ⟨ Σ t , S , i , T t , S H t , S , i ({ T τ , S ′ } τ > t ) ⟩ F S data ] ,
where the Σ t , S , i \Sigma_{t,S,i} Σ t , S , i are independent Rademacher sign matrices of matching size and
C : = 2 L ℓ L a t t n ( D ) Γ 1 ( D ) N . C:=\frac{\sqrt{2L\ell L_{\mathrm{attn}}(D)\Gamma_1(D)}}{N}. C := N 2 L ℓ L attn ( D ) Γ 1 ( D ) .
Stacking the Σ t , S , i \Sigma_{t,S,i} Σ t , S , i across i i i gives
∑ i = 1 N ⟨ Σ t , S , i , T t , S H t , S , i ( { T τ , S ′ } τ > t ) ⟩ F = ⟨ E t , S H t , S ( { T τ , S ′ } τ > t ) ⊤ , T t , S ⟩ F . \sum_{i=1}^{N}\left\langle\Sigma_{t,S,i},T_{t,S}H_{t,S,i}(\{T_{\tau,S'}\}_{\tau>t})\right\rangle_F=\left\langle E_{t,S}H_{t,S}(\{T_{\tau,S'}\}_{\tau>t})^{\top},T_{t,S}\right\rangle_F. i = 1 ∑ N ⟨ Σ t , S , i , T t , S H t , S , i ({ T τ , S ′ } τ > t ) ⟩ F = ⟨ E t , S H t , S ({ T τ , S ′ } τ > t ) ⊤ , T t , S ⟩ F .
Therefore, using sup x ∑ j a j ( x ) ≤ ∑ j sup x a j ( x ) \sup_x \sum_j a_j(x) \le\sum_j \sup_x a_j(x) sup x ∑ j a j ( x ) ≤ ∑ j sup x a j ( x ) and trace/spectral duality,
R S d a t a ( F D ( L ) ) ≤ C ∑ t = 1 L ∑ S ∈ S t E [ sup ∥ T t , S ∥ ⋆ ≤ D t , S ∥ T τ , S ′ ∥ ⋆ ≤ D τ , S ′ ( τ > t ) ⟨ E t , S H t , S ( { T τ , S ′ } τ > t ) ⊤ , T t , S ⟩ F ] = C ∑ t = 1 L ∑ S ∈ S t D t , S E [ sup ∥ T τ , S ′ ∥ ⋆ ≤ D τ , S ′ ( τ > t ) ∥ E t , S H t , S ( { T τ , S ′ } τ > t ) ⊤ ∥ 2 ∣ S d a t a ] . \begin{aligned}
\mathfrak{R}_{S_{\mathrm{data}}}(\mathcal{F}^{(L)}_D) \le C \sum_{t=1}^{L} \sum_{S\in\mathcal{S}_t} \mathbb{E}\left[\sup_{\substack{\|T_{t,S}\|_\star\le D_{t,S}\\ \|T_{\tau,S'}\|_\star\le D_{\tau,S'}\;(\tau>t)}} \left\langle E_{t,S}H_{t,S}(\{T_{\tau,S'}\}_{\tau>t})^\top,T_{t,S}\right\rangle_F\right] \\
&= C \sum_{t=1}^{L} \sum_{S\in\mathcal{S}_t} D_{t,S}\mathbb{E}\left[\sup_{\|T_{\tau,S'}\|_\star\le D_{\tau,S'}\;(\tau>t)} \left\|E_{t,S}H_{t,S}(\{T_{\tau,S'}\}_{\tau>t})^\top\right\|_2\mid S_{\mathrm{data}}\right].
\end{aligned} R S data ( F D ( L ) ) ≤ C t = 1 ∑ L S ∈ S t ∑ E ∥ T t , S ∥ ⋆ ≤ D t , S ∥ T τ , S ′ ∥ ⋆ ≤ D τ , S ′ ( τ > t ) sup ⟨ E t , S H t , S ({ T τ , S ′ } τ > t ) ⊤ , T t , S ⟩ F = C t = 1 ∑ L S ∈ S t ∑ D t , S E [ ∥ T τ , S ′ ∥ ⋆ ≤ D τ , S ′ ( τ > t ) sup E t , S H t , S ({ T τ , S ′ } τ > t ) ⊤ 2 ∣ S data ] .
Applying Lemma B.4 to the bounded class F D ( L ) \mathcal{F}^{(L)}_D F D ( L ) yields the stated uniform generalization bound. □
Proofs for Main results
We begin with two reductions that are used repeatedly. The first reduction says that if a loss is invariant under projecting an update onto fixed left and right subspaces, then nuclear-norm regularization never prefers the unprojected representative. The second reduction rewrites the nuclear norm via a positive-semidefinite lift, which we can apply the Barvinok–Pataki rank bound [16 , 2 ] .
Proofs for Section 4.1
Lemma C.1 (Projector compression for nuclear-norm regularization). Let P L ∈ R m × m P_L\in\mathbb{R}^{m\times m} P L ∈ R m × m and P R ∈ R n × n P_R\in\mathbb{R}^{n\times n} P R ∈ R n × n be orthogonal projectors. Let Φ : R m × n → E \Phi:\mathbb{R}^{m\times n}\to\mathcal{E} Φ : R m × n → E be any map such that Φ ( T ) = Φ ( P L T P R ) \Phi(T)=\Phi(P_LTP_R) Φ ( T ) = Φ ( P L T P R ) ∀ T ∈ R m × n \forall T\in\mathbb{R}^{m\times n} ∀ T ∈ R m × n . Then ∥ P L T P R ∥ ⋆ ≤ ∥ T ∥ ⋆ \|P_LTP_R\|_\star\le\|T\|_\star ∥ P L T P R ∥ ⋆ ≤ ∥ T ∥ ⋆ ∀ T ∈ R m × n \forall T\in\mathbb{R}^{m\times n} ∀ T ∈ R m × n . Consequently, if
T ↦ g ( Φ ( T ) + s ) + λ ∥ T ∥ ⋆ , λ > 0 , T\mapsto g(\Phi(T)+s)+\lambda\|T\|_\star,\quad\lambda>0, T ↦ g ( Φ ( T ) + s ) + λ ∥ T ∥ ⋆ , λ > 0 , admits a global minimizer, then for every global minimizer T ~ \widetilde{T} T , the compressed matrix T ⋆ : = P L T ~ P R T^\star:=P_L\widetilde{T}P_R T ⋆ := P L T P R is also a global minimizer and satisfies
rank ( T ⋆ ) ≤ min { rank ( P L ) , rank ( P R ) } . \operatorname{rank}(T^\star)\le\min\{\operatorname{rank}(P_L),\operatorname{rank}(P_R)\}. rank ( T ⋆ ) ≤ min { rank ( P L ) , rank ( P R )} .
Proof of Lemma C.1. Since P L P_L P L and P R P_R P R are orthogonal projectors, we have ∥ P L ∥ 2 ≤ 1 \|P_L\|_2\le1 ∥ P L ∥ 2 ≤ 1 , ∥ P R ∥ 2 ≤ 1 \|P_R\|_2\le1 ∥ P R ∥ 2 ≤ 1 . By Schatten-norm submultiplicativity,
∥ P L T P R ∥ ⋆ ≤ ∥ P L ∥ 2 ∥ T ∥ ⋆ ∥ P R ∥ 2 ≤ ∥ T ∥ ⋆ . \|P_LTP_R\|_\star\le\|P_L\|_2\|T\|_\star\|P_R\|_2\le\|T\|_\star. ∥ P L T P R ∥ ⋆ ≤ ∥ P L ∥ 2 ∥ T ∥ ⋆ ∥ P R ∥ 2 ≤ ∥ T ∥ ⋆ .
Let T ~ \widetilde{T} T be any global minimizer and define T ⋆ : = P L T ~ P R T^\star:=P_L\widetilde{T}P_R T ⋆ := P L T P R . By hypothesis, Φ ( T ⋆ ) = Φ ( T ~ ) \Phi(T^\star)=\Phi(\widetilde{T}) Φ ( T ⋆ ) = Φ ( T ) , so the loss term is unchanged. The nuclear norm does not increase, so
g ( Φ ( T ⋆ ) + s ) + λ ∥ T ⋆ ∥ ⋆ ≤ g ( Φ ( T ~ ) + s ) + λ ∥ T ~ ∥ ⋆ . g(\Phi(T^\star)+s)+\lambda\|T^\star\|_\star\le g(\Phi(\widetilde{T})+s)+\lambda\|\widetilde{T}\|_\star. g ( Φ ( T ⋆ ) + s ) + λ ∥ T ⋆ ∥ ⋆ ≤ g ( Φ ( T ) + s ) + λ ∥ T ∥ ⋆ .
Since T ~ \widetilde{T} T is globally optimal, equality must hold, and T ⋆ T^\star T ⋆ is also globally optimal. We also have
rank ( T ⋆ ) ≤ min { rank ( P L ) , rank ( P R ) } . \operatorname{rank}(T^\star)\le\min\{\operatorname{rank}(P_L),\operatorname{rank}(P_R)\}. rank ( T ⋆ ) ≤ min { rank ( P L ) , rank ( P R )} .
□
Lemma C.2 (Nuclear-norm lifting: equivalence of infima). Let g : E → R g:\mathcal{E}\to\mathbb{R} g : E → R be any function and fix s ∈ E s\in\mathcal{E} s ∈ E . Then
min T ∈ R m × n F p r i m a l ( T ) = min Z ⪰ 0 F l i f t ( Z ) . \min_{T\in\mathbb{R}^{m\times n}} F_{\mathrm{primal}}(T)=\min_{Z\succeq0}F_{\mathrm{lift}}(Z). T ∈ R m × n min F primal ( T ) = Z ⪰ 0 min F lift ( Z ) . Moreover:
For every T T T there exists Z ⪰ 0 Z\succeq0 Z ⪰ 0 with Z ‾ = T \overline{Z}=T Z = T and F l i f t ( Z ) = F p r i m a l ( T ) F_{\mathrm{lift}}(Z)=F_{\mathrm{primal}}(T) F lift ( Z ) = F primal ( T ) .
For every Z ⪰ 0 Z\succeq0 Z ⪰ 0 , F p r i m a l ( Z ‾ ) ≤ F l i f t ( Z ) F_{\mathrm{primal}}(\overline{Z})\le F_{\mathrm{lift}}(Z) F primal ( Z ) ≤ F lift ( Z ) .
If Z ⋆ Z^\star Z ⋆ minimizes F l i f t F_{\mathrm{lift}} F lift , then T ⋆ = Z ‾ ⋆ T^\star=\overline{Z}^\star T ⋆ = Z ⋆ minimizes F p r i m a l F_{\mathrm{primal}} F primal .
Proof of Lemma C.2. Fix Z ⪰ 0 Z \succeq0 Z ⪰ 0 and factor Z = R R T Z = R R^{\mathsf{T}} Z = R R T with R = [ U V ] R = \begin{bmatrix} U \\ V \end{bmatrix} R = [ U V ] . Then Z ˉ = U V T \bar{Z} = U V^{\mathsf{T}} Z ˉ = U V T and
∥ U V T ∥ ⋆ ≤ 1 2 ( ∥ U ∥ F 2 + ∥ V ∥ F 2 ) = 1 2 tr ( Z ) , \lVert U V^{\mathsf{T}}\rVert_{\star} \le\frac{1}{2}\left(\lVert U\rVert_{F}^{2} + \lVert V\rVert_{F}^{2}\right) = \frac{1}{2}\operatorname{tr}(Z), ∥ U V T ∥ ⋆ ≤ 2 1 ( ∥ U ∥ F 2 + ∥ V ∥ F 2 ) = 2 1 tr ( Z ) ,
so F p r i m a l ( Z ˉ ) ≤ F l i f t ( Z ) F_{\mathrm{primal}}(\bar{Z}) \le F_{\mathrm{lift}}(Z) F primal ( Z ˉ ) ≤ F lift ( Z ) .
Conversely, fix T ∈ R m × n T \in\mathbb{R}^{m \times n} T ∈ R m × n and let T = U T Σ V T T T = U_T \Sigma V_T^{\mathsf{T}} T = U T Σ V T T be a compact SVD. Set U : = U T Σ 1 / 2 ∈ R m × r U := U_T \Sigma^{1/2} \in\mathbb{R}^{m \times r} U := U T Σ 1/2 ∈ R m × r and V : = V T Σ 1 / 2 ∈ R n × r V := V_T \Sigma^{1/2} \in\mathbb{R}^{n \times r} V := V T Σ 1/2 ∈ R n × r , where r = rank ( T ) r = \operatorname{rank}(T) r = rank ( T ) . Then U V T = T U V^{\mathsf{T}} = T U V T = T and ∥ U ∥ F 2 = ∥ V ∥ F 2 = tr ( Σ ) = ∥ T ∥ ⋆ \lVert U\rVert_{F}^{2} = \lVert V\rVert_{F}^{2} = \operatorname{tr}(\Sigma) = \lVert T\rVert_{\star} ∥ U ∥ F 2 = ∥ V ∥ F 2 = tr ( Σ ) = ∥ T ∥ ⋆ .
Define
Z : = [ U U T U V T V U T V V T ] = [ U V ] [ U V ] T ⪰ 0 , Z := \begin{bmatrix} U U^{\mathsf{T}} & U V^{\mathsf{T}} \\ V U^{\mathsf{T}} & V V^{\mathsf{T}} \end{bmatrix}
= \begin{bmatrix} U \\ V \end{bmatrix}
\begin{bmatrix} U \\ V \end{bmatrix}^{\mathsf{T}}
\succeq0, Z := [ U U T V U T U V T V V T ] = [ U V ] [ U V ] T ⪰ 0 ,
so Z ˉ = U V T = T \bar{Z} = U V^{\mathsf{T}} = T Z ˉ = U V T = T and tr ( Z ) = ∥ U ∥ F 2 + ∥ V ∥ F 2 = 2 ∥ T ∥ ⋆ \operatorname{tr}(Z) = \lVert U\rVert_{F}^{2} + \lVert V\rVert_{F}^{2} = 2\lVert T\rVert_{\star} tr ( Z ) = ∥ U ∥ F 2 + ∥ V ∥ F 2 = 2 ∥ T ∥ ⋆ . Hence F l i f t ( Z ) = F p r i m a l ( T ) F_{\mathrm{lift}}(Z) = F_{\mathrm{primal}}(T) F lift ( Z ) = F primal ( T ) . Taking infima yields equality, and the minimizer mapping follows.
Proof of Theorem 4.1. Define
F ( T ) : = g ( Φ ( T ) + s ) + λ ∥ T ∥ ⋆ . F(T) := g(\Phi(T) + s) + \lambda\lVert T\rVert_{\star}. F ( T ) := g ( Φ ( T ) + s ) + λ ∥ T ∥ ⋆ .
Since λ > 0 \lambda> 0 λ > 0 , the map T ↦ λ ∥ T ∥ ⋆ T \mapsto\lambda\lVert T\rVert_{\star} T ↦ λ ∥ T ∥ ⋆ is coercive. Since g g g is bounded below and lower semicontinuous, and since Φ \Phi Φ is continuous, the objective F F F is bounded below, lower semicontinuous, and coercive on the finite-dimensional space R m × n \mathbb{R}^{m \times n} R m × n . Therefore, F F F attains a global minimum.
Fix a global minimizer T ~ \widetilde{T} T and let T ~ = A B T \widetilde{T} = A B^{\mathsf{T}} T = A B T be a balanced factorization obtained from a compact singular value decomposition, so that
1 2 ( ∥ A ∥ F 2 + ∥ B ∥ F 2 ) = ∥ T ~ ∥ ⋆ . \frac{1}{2}\left(\lVert A\rVert_{F}^{2} + \lVert B\rVert_{F}^{2}\right) = \lVert\widetilde{T}\rVert_{\star}. 2 1 ( ∥ A ∥ F 2 + ∥ B ∥ F 2 ) = ∥ T ∥ ⋆ .
Define
Z ~ : = [ A B ] [ A B ] T . \widetilde{Z} := \begin{bmatrix} A \\ B \end{bmatrix}
\begin{bmatrix} A \\ B \end{bmatrix}^{\mathsf{T}}. Z := [ A B ] [ A B ] T .
Then Z ~ ⪰ 0 \widetilde{Z} \succeq0 Z ⪰ 0 , Z ~ ˉ = T ~ \bar{\widetilde{Z}} = \widetilde{T} Z ˉ = T , and tr ( Z ~ ) = 2 ∥ T ~ ∥ ⋆ \operatorname{tr}(\widetilde{Z}) = 2\lVert\widetilde{T}\rVert_{\star} tr ( Z ) = 2 ∥ T ∥ ⋆ .
Consider the conditional trace-minimization problem
min Z ⪰ 0 tr ( Z ) subject to Φ ( Z ˉ ) = Φ ( T ~ ) . \min_{Z \succeq0} \operatorname{tr}(Z) \quad\text{subject to} \quad\Phi(\bar{Z}) = \Phi(\widetilde{T}). Z ⪰ 0 min tr ( Z ) subject to Φ ( Z ˉ ) = Φ ( T ) .
This feasible set is nonempty because it contains Z ~ \widetilde{Z} Z . The objective is bounded below by 0 0 0 . Moreover, every trace sublevel set is compact, because Z ⪰ 0 Z \succeq0 Z ⪰ 0 implies ∥ Z ∥ F ≤ tr ( Z ) \lVert Z\rVert_{F} \le\operatorname{tr}(Z) ∥ Z ∥ F ≤ tr ( Z ) . Hence, the conditional trace-minimization problem attains its minimum. Let v ⋆ v^{\star} v ⋆ denote its minimum value.
The set C ⋆ : = { Z ⪰ 0 : Φ ( Z ˉ ) = Φ ( T ~ ) , tr ( Z ) = v ⋆ } C^{\star} := \{Z \succeq0 : \Phi(\bar{Z}) = \Phi(\widetilde{T}), \operatorname{tr}(Z) = v^{\star}\} C ⋆ := { Z ⪰ 0 : Φ ( Z ˉ ) = Φ ( T ) , tr ( Z ) = v ⋆ } is nonempty and compact. Choose an extreme point Z ⋆ Z^{\star} Z ⋆ of C ⋆ C^{\star} C ⋆ . Since C ⋆ C^{\star} C ⋆ is the set of minimizers of the linear functional Z ↦ tr ( Z ) Z \mapsto\operatorname{tr}(Z) Z ↦ tr ( Z ) over the spectrahedron C : = { Z ⪰ 0 : Φ ( Z ˉ ) = Φ ( T ~ ) } C := \{Z \succeq0 : \Phi(\bar{Z}) = \Phi(\widetilde{T})\} C := { Z ⪰ 0 : Φ ( Z ˉ ) = Φ ( T )} , the set C ⋆ C^{\star} C ⋆ is a face of C C C . Therefore, Z ⋆ Z^{\star} Z ⋆ is also an extreme point of C C C .
Let e 1 , … , e d Φ e_1, \ldots, e_{d_{\Phi}} e 1 , … , e d Φ be an orthonormal basis of R ( Φ ) \mathcal{R}(\Phi) R ( Φ ) . For ℓ = 1 , … , d Φ \ell= 1, \ldots, d_{\Phi} ℓ = 1 , … , d Φ , define
A ℓ : = 1 2 [ 0 Φ ⋆ ( e ℓ ) Φ ⋆ ( e ℓ ) T 0 ] . A_{\ell} := \frac{1}{2}
\begin{bmatrix}
0 & \Phi^{\star}(e_{\ell}) \\
\Phi^{\star}(e_{\ell})^{\mathsf{T}} & 0
\end{bmatrix}. A ℓ := 2 1 [ 0 Φ ⋆ ( e ℓ ) T Φ ⋆ ( e ℓ ) 0 ] .
Then the constraint Φ ( Z ˉ ) = Φ ( T ~ ) \Phi(\bar{Z}) = \Phi(\widetilde{T}) Φ ( Z ˉ ) = Φ ( T ) is equivalent to the independent affine constraints
⟨ A ℓ , Z ⟩ = ⟨ e ℓ , Φ ( T ~ ) ⟩ E , ℓ = 1 , … , d Φ . \langle A_{\ell}, Z\rangle= \langle e_{\ell}, \Phi(\widetilde{T})\rangle_{\mathcal{E}}, \quad\ell= 1, \ldots, d_{\Phi}. ⟨ A ℓ , Z ⟩ = ⟨ e ℓ , Φ ( T ) ⟩ E , ℓ = 1 , … , d Φ .
The independence follows because Φ ⋆ \Phi^{\star} Φ ⋆ is injective on R ( Φ ) \mathcal{R}(\Phi) R ( Φ ) .
By the Barvinok-Pataki rank bound for extreme points of spectrahedra with d Φ d_{\Phi} d Φ independent affine constraints ([16 , 2 ] ), Z ⋆ Z^{\star} Z ⋆ satisfies
rank ( Z ⋆ ) ( rank ( Z ⋆ ) + 1 ) 2 ≤ d Φ . \frac{\operatorname{rank}(Z^{\star})\left(\operatorname{rank}(Z^{\star}) + 1\right)}{2} \le d_{\Phi}. 2 rank ( Z ⋆ ) ( rank ( Z ⋆ ) + 1 ) ≤ d Φ .
Set T ⋆ : = Z ‾ ⋆ T^\star:= \overline{Z}^\star T ⋆ := Z ⋆ . Then Φ ( T ⋆ ) = Φ ( Z ‾ ⋆ ) = Φ ( T ~ ) \Phi(T^\star) = \Phi(\overline{Z}^\star) = \Phi(\widetilde{T}) Φ ( T ⋆ ) = Φ ( Z ⋆ ) = Φ ( T ) . Furthermore, Lemma C.2 gives
F ( T ⋆ ) ≤ g ( Φ ( T ~ ) + s ) + λ 2 tr ( Z ⋆ ) ≤ g ( Φ ( T ~ ) + s ) + λ 2 tr ( Z ~ ) = F ( T ~ ) . F(T^\star) \le g(\Phi(\widetilde{T}) + s) + \frac{\lambda}{2}\operatorname{tr}(Z^\star) \le g(\Phi(\widetilde{T}) + s) + \frac{\lambda}{2}\operatorname{tr}(\widetilde{Z}) = F(\widetilde{T}). F ( T ⋆ ) ≤ g ( Φ ( T ) + s ) + 2 λ tr ( Z ⋆ ) ≤ g ( Φ ( T ) + s ) + 2 λ tr ( Z ) = F ( T ) .
Since T ~ \widetilde{T} T is globally optimal, T ⋆ T^\star T ⋆ is also globally optimal. Finally, rank ( T ⋆ ) ≤ rank ( Z ⋆ ) \operatorname{rank}(T^\star) \le\operatorname{rank}(Z^\star) rank ( T ⋆ ) ≤ rank ( Z ⋆ ) , so
rank ( T ⋆ ) ( rank ( T ⋆ ) + 1 ) 2 ≤ d Φ . \frac{\operatorname{rank}(T^\star)(\operatorname{rank}(T^\star)+1)}{2} \le d_{\Phi}. 2 rank ( T ⋆ ) ( rank ( T ⋆ ) + 1 ) ≤ d Φ .
Solving this quadratic inequality gives the displayed rank bound.
Proof of Theorem 4.3. Let
F ( { T t , S } ) : = L ( { T t , S } ) + ∑ t = 1 L ∑ S ∈ S t λ t , S ∥ T t , S ∥ ⋆ . F(\{T_{t,S}\}) := \mathcal{L}(\{T_{t,S}\}) + \sum_{t=1}^{L}\sum_{S\in\mathcal{S}_t}\lambda_{t,S}\lVert T_{t,S}\rVert_\star. F ({ T t , S }) := L ({ T t , S }) + t = 1 ∑ L S ∈ S t ∑ λ t , S ∥ T t , S ∥ ⋆ .
Because all λ t , S > 0 \lambda_{t,S} > 0 λ t , S > 0 , the regularizer is coercive on the finite-dimensional product space ∏ t , S R m t , S × n t , S \prod_{t,S}\mathbb{R}^{m_{t,S}\times n_{t,S}} ∏ t , S R m t , S × n t , S . By Assumption 1, each stage map in the recursion of Definition 3.2 is continuous, hence the map from the parameter tuple to z 1 z_1 z 1 is continuous; since g 0 g_0 g 0 is bounded below and lower semicontinuous, L = g 0 ( z 1 ) \mathcal{L} = g_0(z_1) L = g 0 ( z 1 ) is bounded below and lower semicontinuous. Therefore, F F F attains a global minimum.
Fix any global minimizer of F F F . We now process the stages from upstream to downstream, i.e. in the order t = L , L − 1 , … , 1 t = L, L - 1, \ldots, 1 t = L , L − 1 , … , 1 .
Assume that all stages u > t u > t u > t have already been fixed once and for all. Then the corresponding state z ~ t + 1 ⋆ \widetilde{z}_{t+1}^\star z t + 1 ⋆ is fixed and will not change in the subsequent steps. Now fix a block S ∈ S t S \in\mathcal{S}_t S ∈ S t and hold every block other than T t , S T_{t,S} T t , S fixed. The resulting conditional objective has the form
T ↦ g ~ t , S ( Φ t , S ( T , z ~ t + 1 ⋆ ) ) + λ t , S ∥ T ∥ ⋆ , T \mapsto\widetilde{g}_{t,S}(\Phi_{t,S}(T,\widetilde{z}_{t+1}^\star)) + \lambda_{t,S}\lVert T\rVert_\star, T ↦ g t , S ( Φ t , S ( T , z t + 1 ⋆ )) + λ t , S ∥ T ∥ ⋆ ,
where g ~ t , S \widetilde{g}_{t,S} g t , S is obtained by fixing all companion inputs and composing the remaining stage maps with g 0 g_0 g 0 . Because the companion blocks and z ~ t + 1 ⋆ \widetilde{z}_{t+1}^\star z t + 1 ⋆ are fixed, all maps below stage t t t are continuous functions of the single argument, and g 0 g_0 g 0 is lower semicontinuous and bounded below. Thus g ~ t , S \widetilde{g}_{t,S} g t , S is bounded below and lower semicontinuous on the relevant finite-dimensional space.
Since the current value of T t , S T_{t,S} T t , S is already globally optimal for this conditional problem with all companion blocks fixed, Theorem 4.1 allows us to replace it by another conditional minimizer T t , S ⋆ T_{t,S}^\star T t , S ⋆ with the same Φ t , S \Phi_{t,S} Φ t , S satisfying
r t , S ( r t , S + 1 ) 2 ≤ d t , S ( z ~ t + 1 ⋆ ) , r t , S : = rank ( T t , S ⋆ ) . \frac{r_{t,S}(r_{t,S}+1)}{2} \le d_{t,S}(\widetilde{z}_{t+1}^\star), \qquad r_{t,S} := \operatorname{rank}(T_{t,S}^\star). 2 r t , S ( r t , S + 1 ) ≤ d t , S ( z t + 1 ⋆ ) , r t , S := rank ( T t , S ⋆ ) .
The equality of the resulting values implies that the stage-t t t state z t z_t z t and all downstream states z t − 1 , … , z 1 z_{t-1},\ldots,z_1 z t − 1 , … , z 1 are unchanged by the replacement. Thus, the global objective value is preserved, and all blocks already processed remain valid.
Proceeding inductively through t = L , L − 1 , … , 1 t = L, L - 1, \ldots, 1 t = L , L − 1 , … , 1 yields a global minimizer { T t , S ⋆ } \{T_{t,S}^\star\} { T t , S ⋆ } such that for every t ∈ [ L ] t \in[L] t ∈ [ L ] and S ∈ S t S \in\mathcal{S}_t S ∈ S t ,
r t , S ( r t , S + 1 ) 2 ≤ d t , S ( z ~ t + 1 ⋆ ) ≤ d t , S max . \frac{r_{t,S}(r_{t,S}+1)}{2} \le d_{t,S}(\widetilde{z}_{t+1}^\star) \le d_{t,S}^{\max}. 2 r t , S ( r t , S + 1 ) ≤ d t , S ( z t + 1 ⋆ ) ≤ d t , S m a x .
The final scalar bound follows from the implication
r ( r + 1 ) 2 ≤ d ⟹ r ≤ ⌊ 8 d + 1 − 1 2 ⌋ . \frac{r(r+1)}{2} \le d \Longrightarrow r \le\left\lfloor\frac{\sqrt{8d+1}-1}{2}\right\rfloor. 2 r ( r + 1 ) ≤ d ⟹ r ≤ ⌊ 2 8 d + 1 − 1 ⌋ .
Proofs for Section 4.2
The proof of Theorem 4.4 has three ingredients: row-wise score centering for softmax, a blockwise projector-compression principle at a rank-minimal global minimizer, and explicit identification of the query, key, and value subspaces that remain observable after the frozen attention computation.
Lemma C.3 (Row-wise softmax depends only on the centered score matrix). Let L ′ ∈ N L' \in\mathbb{N} L ′ ∈ N and let P 1 L ′ ⊥ = I L ′ − 1 L ′ 1 L ′ 1 L ′ T P_{1_{L'}}^\perp= I_{L'} - \frac{1}{L'}1_{L'}1_{L'}^{\mathsf{T}} P 1 L ′ ⊥ = I L ′ − L ′ 1 1 L ′ 1 L ′ T . Let C ∈ ( R ∪ { − ∞ } ) a × L ′ C \in(\mathbb{R} \cup\{-\infty\})^{a \times L'} C ∈ ( R ∪ { − ∞ } ) a × L ′ be such that each row of C C C has at least one finite entry. Then for every M ∈ R a × L ′ M \in\mathbb{R}^{a \times L'} M ∈ R a × L ′ ,
softmax r o w ( M + C ) = softmax r o w ( M P 1 L ′ ⊥ + C ) . \operatorname{softmax}_{\mathrm{row}}(M+C)=\operatorname{softmax}_{\mathrm{row}}(MP_{1_{L'}}^\perp+C). softmax row ( M + C ) = softmax row ( M P 1 L ′ ⊥ + C ) . In particular, when C = 0 C=0 C = 0 , softmax r o w ( M ) = softmax r o w ( M P 1 L ′ ⊥ ) \operatorname{softmax}_{\mathrm{row}}(M)=\operatorname{softmax}_{\mathrm{row}}(MP_{1_{L'}}^\perp) softmax row ( M ) = softmax row ( M P 1 L ′ ⊥ ) .
Proof of Lemma C.3. Fix a row index i ∈ [ a ] i \in[a] i ∈ [ a ] and define J i : = { j ∈ [ L ′ ] : C i j > − ∞ } J_i := \{j \in[L'] : C_{ij} > -\infty\} J i := { j ∈ [ L ′ ] : C ij > − ∞ } . By assumption, J i ≠ ∅ J_i \ne\varnothing J i = ∅ . Set c i : = 1 L ′ ∑ j = 1 L ′ M i j c_i := \frac{1}{L'}\sum_{j=1}^{L'} M_{ij} c i := L ′ 1 ∑ j = 1 L ′ M ij . Then, for every j ∈ J i j \in J_i j ∈ J i , ( M P 1 L ′ ⊥ + C ) i j = ( M + C ) i j − c i (MP_{1_{L'}}^\perp+C)_{ij}=(M+C)_{ij}-c_i ( M P 1 L ′ ⊥ + C ) ij = ( M + C ) ij − c i . Hence
exp ( ( M P 1 L ′ ⊥ + C ) i j ) ∑ k ∈ J i exp ( ( M P 1 L ′ ⊥ + C ) i k ) = e − c i exp ( ( M + C ) i j ) e − c i ∑ k ∈ J i exp ( ( M + C ) i k ) = exp ( ( M + C ) i j ) ∑ k ∈ J i exp ( ( M + C ) i k ) . \frac{\exp((MP_{1_{L'}}^\perp+C)_{ij})}{\sum_{k\in J_i}\exp((MP_{1_{L'}}^\perp+C)_{ik})}
=
\frac{e^{-c_i}\exp((M+C)_{ij})}{e^{-c_i}\sum_{k\in J_i}\exp((M+C)_{ik})}
=
\frac{\exp((M+C)_{ij})}{\sum_{k\in J_i}\exp((M+C)_{ik})}. ∑ k ∈ J i exp (( M P 1 L ′ ⊥ + C ) ik ) exp (( M P 1 L ′ ⊥ + C ) ij ) = e − c i ∑ k ∈ J i exp (( M + C ) ik ) e − c i exp (( M + C ) ij ) = ∑ k ∈ J i exp (( M + C ) ik ) exp (( M + C ) ij ) .
For j ∉ J i j \notin J_i j ∈ / J i , both ( M + C ) i j (M+C)_{ij} ( M + C ) ij and ( M P 1 L ′ ⊥ + C ) i j (MP_{1_{L'}}^\perp+C)_{ij} ( M P 1 L ′ ⊥ + C ) ij are equal to − ∞ -\infty − ∞ , so both rows assign zero softmax mass to coordinate j j j . Therefore, the i i i -th row of the two row-wise softmax matrices coincide. Since i i i was arbitrary, the matrices are equal. The special case C = 0 C=0 C = 0 is immediate.
:::
Theorem C.4 (Self-consistent projector compression). Let λ S > 0 \lambda_S>0 λ S > 0 for every S ∈ S S\in\mathcal{S} S ∈ S , and define
F ( { T S } S ∈ S ) : = L ( { T S } S ∈ S ) + ∑ S ∈ S λ S ∥ T S ∥ ⋆ . F(\{T_S\}_{S\in\mathcal{S}}):=\mathcal{L}(\{T_S\}_{S\in\mathcal{S}})+\sum_{S\in\mathcal{S}}\lambda_S\lVert T_S\rVert_\star. F ({ T S } S ∈ S ) := L ({ T S } S ∈ S ) + S ∈ S ∑ λ S ∥ T S ∥ ⋆ . Assume that F F F admits a global minimizer. Suppose that for each block S S S and every fixed companion tuple { T S ′ } S ′ ≠ S \{T_{S'}\}_{S'\ne S} { T S ′ } S ′ = S there exist orthogonal projectors P S L ( { T S ′ } S ′ ≠ S ) ∈ R m S × m S P_S^{\mathrm{L}}(\{T_{S'}\}_{S'\ne S})\in\mathbb{R}^{m_S\times m_S} P S L ({ T S ′ } S ′ = S ) ∈ R m S × m S , P S R ( { T S ′ } S ′ ≠ S ) ∈ R n S × n S P_S^{\mathrm{R}}(\{T_{S'}\}_{S'\ne S})\in\mathbb{R}^{n_S\times n_S} P S R ({ T S ′ } S ′ = S ) ∈ R n S × n S , such that
L ( { T S ′ } S ′ ≠ S , T S ) = L ( { T S ′ } S ′ ≠ S , P S L ( { T S ′ } S ′ ≠ S ) T S P S R ( { T S ′ } S ′ ≠ S ) ) ∀ T S . \mathcal{L}(\{T_{S'}\}_{S'\ne S},T_S)
=
\mathcal{L}\left(\{T_{S'}\}_{S'\ne S},P_S^{\mathrm{L}}(\{T_{S'}\}_{S'\ne S})T_SP_S^{\mathrm{R}}(\{T_{S'}\}_{S'\ne S})\right)
\quad\forall T_S. L ({ T S ′ } S ′ = S , T S ) = L ( { T S ′ } S ′ = S , P S L ({ T S ′ } S ′ = S ) T S P S R ({ T S ′ } S ′ = S ) ) ∀ T S . Then there exists a global minimizer { T S ⋆ } \{T_S^\star\} { T S ⋆ } minimizing
∑ S ∈ S rank ( T S ) \sum_{S\in\mathcal{S}}\operatorname{rank}(T_S) S ∈ S ∑ rank ( T S ) over the global minimizer set, and every such minimizer satisfies
rank ( T S ⋆ ) ≤ min { rank ( P S L ( { T S ′ ⋆ } S ′ ≠ S ) ) , rank ( P S R ( { T S ′ ⋆ } S ′ ≠ S ) ) } . \operatorname{rank}(T_S^\star)\le\min\left\{\operatorname{rank}\left(P_S^{\mathrm{L}}(\{T_{S'}^\star\}_{S'\ne S})\right),\operatorname{rank}\left(P_S^{\mathrm{R}}(\{T_{S'}^\star\}_{S'\ne S})\right)\right\}. rank ( T S ⋆ ) ≤ min { rank ( P S L ({ T S ′ ⋆ } S ′ = S ) ) , rank ( P S R ({ T S ′ ⋆ } S ′ = S ) ) } .
Proof of Theorem C.4. Since rank ( T S ) ∈ { 0 , 1 , … , min ( m S , n S ) } \operatorname{rank}(T_S)\in\{0,1,\ldots,\min(m_S,n_S)\} rank ( T S ) ∈ { 0 , 1 , … , min ( m S , n S )} for each block S S S , the set of possible values of ∑ S rank ( T S ) \sum_S\operatorname{rank}(T_S) ∑ S rank ( T S ) is finite, and there exists a global minimizer { T S ⋆ } ∈ argmin F \{T_S^\star\}\in\operatorname*{argmin} F { T S ⋆ } ∈ argmin F for which ∑ S rank ( T S ) \sum_S\operatorname{rank}(T_S) ∑ S rank ( T S ) is minimal.
Fix a block S ∈ S S\in\mathcal{S} S ∈ S and abbreviate P S L , ⋆ : = P S L ( { T S ′ ⋆ } S ′ ≠ S ) P_S^{\mathrm{L},\star}:=P_S^{\mathrm{L}}(\{T_{S'}^\star\}_{S'\ne S}) P S L , ⋆ := P S L ({ T S ′ ⋆ } S ′ = S ) , P S R , ⋆ : = P S R ( { T S ′ ⋆ } S ′ ≠ S ) P_S^{\mathrm{R},\star}:=P_S^{\mathrm{R}}(\{T_{S'}^\star\}_{S'\ne S}) P S R , ⋆ := P S R ({ T S ′ ⋆ } S ′ = S ) . Define T ^ S : = P S L , ⋆ T S ⋆ P S R , ⋆ \widehat{T}_S:=P_S^{\mathrm{L},\star}T_S^\star P_S^{\mathrm{R},\star} T S := P S L , ⋆ T S ⋆ P S R , ⋆ . By the assumed invariance of the loss,
L ( { T S ′ ⋆ } S ′ ≠ S , T ^ S ) = L ( { T S ′ ⋆ } S ′ ≠ S , T S ⋆ ) . \mathcal{L}(\{T_{S'}^\star\}_{S'\ne S},\widehat{T}_S)
=
\mathcal{L}(\{T_{S'}^\star\}_{S'\ne S},T_S^\star). L ({ T S ′ ⋆ } S ′ = S , T S ) = L ({ T S ′ ⋆ } S ′ = S , T S ⋆ ) .
By Lemma C.1, ∥ T ^ S ∥ ⋆ ≤ ∥ T S ⋆ ∥ ⋆ \lVert\widehat{T}_S\rVert_\star\le\lVert T_S^\star\rVert_\star ∥ T S ∥ ⋆ ≤ ∥ T S ⋆ ∥ ⋆ . Hence F ( { T S ′ ⋆ } S ′ ≠ S , T ^ S ) ≤ F ( { T S ⋆ } ) F(\{T_{S'}^\star\}_{S'\ne S},\widehat{T}_S)\le F(\{T_S^\star\}) F ({ T S ′ ⋆ } S ′ = S , T S ) ≤ F ({ T S ⋆ }) . Since { T S ⋆ } \{T_S^\star\} { T S ⋆ } is already globally optimal, the new tuple is also a global minimizer.
If rank ( T S ⋆ ) > min { rank ( P S L , ⋆ ) , rank ( P S R , ⋆ ) } \operatorname{rank}(T_S^\star)>\min\{\operatorname{rank}(P_S^{\mathrm{L},\star}),\operatorname{rank}(P_S^{\mathrm{R},\star})\} rank ( T S ⋆ ) > min { rank ( P S L , ⋆ ) , rank ( P S R , ⋆ )} , then
rank ( T ^ S ) ≤ min { rank ( P S L , ⋆ ) , rank ( P S R , ⋆ ) } < rank ( T S ⋆ ) , \operatorname{rank}(\widehat{T}_S)\le\min\{\operatorname{rank}(P_S^{\mathrm{L},\star}),\operatorname{rank}(P_S^{\mathrm{R},\star})\}<\operatorname{rank}(T_S^\star), rank ( T S ) ≤ min { rank ( P S L , ⋆ ) , rank ( P S R , ⋆ )} < rank ( T S ⋆ ) ,
so, the new global minimizer would have strictly smaller total rank, contradicting the choice of { T S ⋆ } \{T_S^\star\} { T S ⋆ } . Therefore, the displayed rank bound must hold for every block S S S .
:::
Proof of Theorem 4.4. By Lemma B.2, for each sample i i i the network output is a deterministic function of the collection
{ ( ( W t , S , 0 ( a ) + T t , S ( a ) ) H t , S , i , { T τ , S ′ ( a ′ ) } τ > t ) ( t , a , S ) ∈ S ~ } . \left\{ \left((W_{t,S,0}^{(a)}+T_{t,S}^{(a)})H_{t,S,i},\{T_{\tau,S'}^{(a')}\}_{\tau>t}\right)_{(t,a,S)\in\widetilde{\mathcal{S}}} \right\}. { ( ( W t , S , 0 ( a ) + T t , S ( a ) ) H t , S , i , { T τ , S ′ ( a ′ ) } τ > t ) ( t , a , S ) ∈ S } .
Therefore the empirical loss exhibits L L L -stage blockwise linear transformation dependence in the sense of Definition 3.2, with linear mappings Φ t , ( a , S ) ( T , z ) = T H t , S ( z ) \Phi_{t,(a,S)}(T,z)=TH_{t,S}(z) Φ t , ( a , S ) ( T , z ) = T H t , S ( z ) . Assumption 1 lets us invoke Theorem 4.3, so the corresponding trace-norm regularized objective admits a global minimizer. We now verify the projector-invariance hypothesis of Theorem C.4 for the same objective, viewed as a multi-block problem indexed by S ~ \widetilde{\mathcal{S}} S .
Fix an adapted block ( t , a , S ) ∈ S ~ (t,a,S)\in\widetilde{\mathcal{S}} ( t , a , S ) ∈ S and hold all companion blocks fixed. These companions determine a fixed upstream state z t + 1 z_{t+1} z t + 1 and therefore fixed samplewise input matrices H t , Q , i , H t , K , i , H t , V , i H_{t,Q,i},H_{t,K,i},H_{t,V,i} H t , Q , i , H t , K , i , H t , V , i ( i ∈ [ N ] ) (i\in[N]) ( i ∈ [ N ]) .
First, consider the case S = Q S=Q S = Q . Hold all blocks other than T t , Q ( a ) T_{t,Q}^{(a)} T t , Q ( a ) fixed. Define
K ~ t , i ( a ) : = ( W t , K , 0 ( a ) + T t , K ( a ) ) H t , K , i , V ~ t , i ( a ) : = ( W t , V , 0 ( a ) + T t , V ( a ) ) H t , V , i , \widetilde{K}_{t,i}^{(a)}:=\left(W_{t,K,0}^{(a)}+T_{t,K}^{(a)}\right)H_{t,K,i},\qquad\widetilde{V}_{t,i}^{(a)}:=\left(W_{t,V,0}^{(a)}+T_{t,V}^{(a)}\right)H_{t,V,i}, K t , i ( a ) := ( W t , K , 0 ( a ) + T t , K ( a ) ) H t , K , i , V t , i ( a ) := ( W t , V , 0 ( a ) + T t , V ( a ) ) H t , V , i ,
and for T ∈ R d k , t × n t , Q T\in\mathbb{R}^{d_{k,t}\times n_{t,Q}} T ∈ R d k , t × n t , Q set
A t , Q , i ( a ) ( T ) : = softmax r o w ( ( ( W t , Q , 0 ( a ) + T ) H t , Q , i ) T K ~ t , i ( a ) d k , t + C t , i M ) . A_{t,Q,i}^{(a)}(T):=\operatorname{softmax}_{\mathrm{row}}\left(\frac{\left((W_{t,Q,0}^{(a)}+T)H_{t,Q,i}\right)^{\mathsf T}\widetilde{K}_{t,i}^{(a)}}{\sqrt{d_{k,t}}}+C_{t,i}^{M}\right). A t , Q , i ( a ) ( T ) := softmax row d k , t ( ( W t , Q , 0 ( a ) + T ) H t , Q , i ) T K t , i ( a ) + C t , i M .
Since every other head, every residual contribution, and every downstream stage is fixed, the full stage-t t t output on sample i i i is equal to a fixed matrix plus
W t , O ( a ) V ~ t , i ( a ) A t , Q , i ( a ) ( T ) T . W_{t,O}^{(a)}\widetilde{V}_{t,i}^{(a)}A_{t,Q,i}^{(a)}(T)^{\mathsf T}. W t , O ( a ) V t , i ( a ) A t , Q , i ( a ) ( T ) T .
Hence, the conditional loss depends on T T T only through the collection
{ W t , O ( a ) V ~ t , i ( a ) A t , Q , i ( a ) ( T ) T } i = 1 N . \left\{W_{t,O}^{(a)}\widetilde{V}_{t,i}^{(a)}A_{t,Q,i}^{(a)}(T)^{\mathsf T}\right\}_{i=1}^{N}. { W t , O ( a ) V t , i ( a ) A t , Q , i ( a ) ( T ) T } i = 1 N .
Define
P t , i ⊥ : = I ϱ t , K , i − 1 ϱ t , K , i 1 ϱ t , K , i 1 ϱ t , K , i T , K ~ t , c t r ( a ) : = [ K ~ t , 1 ( a ) P t , 1 ⊥ , … , K ~ t , N ( a ) P t , N ⊥ ] ; T ^ t , Q ( a ) : = P K t , c t r ( a ) T t , Q ( a ) P H t , Q . P_{t,i}^{\perp}:=I_{\varrho_{t,K,i}}-\frac{1}{\varrho_{t,K,i}}\mathbf{1}_{\varrho_{t,K,i}}\mathbf{1}_{\varrho_{t,K,i}}^{\mathsf T},\qquad
\widetilde{K}_{t,\mathrm{ctr}}^{(a)}:=\left[\widetilde{K}_{t,1}^{(a)}P_{t,1}^{\perp},\ldots,\widetilde{K}_{t,N}^{(a)}P_{t,N}^{\perp}\right];
\qquad
\widehat{T}_{t,Q}^{(a)}:=P_{K_{t,\mathrm{ctr}}^{(a)}}T_{t,Q}^{(a)}P_{H_{t,Q}}. P t , i ⊥ := I ϱ t , K , i − ϱ t , K , i 1 1 ϱ t , K , i 1 ϱ t , K , i T , K t , ctr ( a ) := [ K t , 1 ( a ) P t , 1 ⊥ , … , K t , N ( a ) P t , N ⊥ ] ; T t , Q ( a ) := P K t , ctr ( a ) T t , Q ( a ) P H t , Q .
By Lemma C.3, for every i i i , the matrix A t , Q , i ( a ) ( T ) A_{t,Q,i}^{(a)}(T) A t , Q , i ( a ) ( T ) depends only on
( ( ( W t , Q , 0 ( a ) + T ) H t , Q , i ) T K ~ t , i ( a ) d k , t ) P t , i ⊥ . \left(\frac{\left((W_{t,Q,0}^{(a)}+T)H_{t,Q,i}\right)^{\mathsf T}\widetilde{K}_{t,i}^{(a)}}{\sqrt{d_{k,t}}}\right)P_{t,i}^{\perp}. d k , t ( ( W t , Q , 0 ( a ) + T ) H t , Q , i ) T K t , i ( a ) P t , i ⊥ .
Moreover, ( H t , Q , i ) T ( T ^ t , Q ( a ) ) T K ~ t , i ( a ) P t , i ⊥ = ( H t , Q , i ) T ( T t , Q ( a ) ) T K ~ t , i ( a ) P t , i ⊥ (H_{t,Q,i})^{\mathsf T}( \widehat{T}_{t,Q}^{(a)})^{\mathsf T}\widetilde{K}_{t,i}^{(a)}P_{t,i}^{\perp}=(H_{t,Q,i})^{\mathsf T}(T_{t,Q}^{(a)})^{\mathsf T}\widetilde{K}_{t,i}^{(a)}P_{t,i}^{\perp} ( H t , Q , i ) T ( T t , Q ( a ) ) T K t , i ( a ) P t , i ⊥ = ( H t , Q , i ) T ( T t , Q ( a ) ) T K t , i ( a ) P t , i ⊥ , because ( H t , Q , i ) T P H t , Q = ( H t , Q , i ) T (H_{t,Q,i})^{\mathsf T}P_{H_{t,Q}}=(H_{t,Q,i})^{\mathsf T} ( H t , Q , i ) T P H t , Q = ( H t , Q , i ) T and P K t , c t r ( a ) K ~ t , i ( a ) P t , i ⊥ = K ~ t , i ( a ) P t , i ⊥ P_{K_{t,\mathrm{ctr}}^{(a)}}\widetilde{K}_{t,i}^{(a)}P_{t,i}^{\perp}=\widetilde{K}_{t,i}^{(a)}P_{t,i}^{\perp} P K t , ctr ( a ) K t , i ( a ) P t , i ⊥ = K t , i ( a ) P t , i ⊥ . Therefore A t , Q , i ( a ) ( T ^ t , Q ( a ) ) = A t , Q , i ( a ) ( T t , Q ( a ) ) ∀ i A_{t,Q,i}^{(a)}(\widehat{T}_{t,Q}^{(a)})=A_{t,Q,i}^{(a)}(T_{t,Q}^{(a)})\ \forall i A t , Q , i ( a ) ( T t , Q ( a ) ) = A t , Q , i ( a ) ( T t , Q ( a ) ) ∀ i , so the conditional loss is invariant under
T t , Q ( a ) ↦ P K t , c t r ( a ) T t , Q ( a ) P H t , Q . T_{t,Q}^{(a)}\mapsto P_{K_{t,\mathrm{ctr}}^{(a)}}T_{t,Q}^{(a)}P_{H_{t,Q}}. T t , Q ( a ) ↦ P K t , ctr ( a ) T t , Q ( a ) P H t , Q .
Next, consider the case S = K S=K S = K . Hold all blocks other than T t , K ( a ) T_{t,K}^{(a)} T t , K ( a ) fixed and define
Q ~ t , i ( a ) : = ( W t , Q , 0 ( a ) + T t , Q ( a ) ) H t , Q , i , V ~ t , i ( a ) : = ( W t , V , 0 ( a ) + T t , V ( a ) ) H t , V , i . \widetilde{Q}_{t,i}^{(a)}:=\left(W_{t,Q,0}^{(a)}+T_{t,Q}^{(a)}\right)H_{t,Q,i},\qquad\widetilde{V}_{t,i}^{(a)}:=\left(W_{t,V,0}^{(a)}+T_{t,V}^{(a)}\right)H_{t,V,i}. Q t , i ( a ) := ( W t , Q , 0 ( a ) + T t , Q ( a ) ) H t , Q , i , V t , i ( a ) := ( W t , V , 0 ( a ) + T t , V ( a ) ) H t , V , i .
For T ∈ R d k , t × n t , K T\in\mathbb{R}^{d_{k,t}\times n_{t,K}} T ∈ R d k , t × n t , K , set
A t , K , i ( a ) ( T ) : = softmax r o w ( ( Q ~ t , i ( a ) ) T ( W t , K , 0 ( a ) + T ) H t , K , i d k , t + C t , i M ) . A_{t,K,i}^{(a)}(T):=\operatorname{softmax}_{\mathrm{row}}\left(\frac{\left(\widetilde{Q}_{t,i}^{(a)}\right)^{\mathsf T}\left(W_{t,K,0}^{(a)}+T\right)H_{t,K,i}}{\sqrt{d_{k,t}}}+C_{t,i}^{M}\right). A t , K , i ( a ) ( T ) := softmax row d k , t ( Q t , i ( a ) ) T ( W t , K , 0 ( a ) + T ) H t , K , i + C t , i M .
As above, the conditional loss depends on T T T only through
{ W t , O ( a ) V ~ t , i ( a ) A t , K , i ( a ) ( T ) T } i = 1 N . \left\{W_{t,O}^{(a)}\widetilde{V}_{t,i}^{(a)}A_{t,K,i}^{(a)}(T)^{\mathsf T}\right\}_{i=1}^{N}. { W t , O ( a ) V t , i ( a ) A t , K , i ( a ) ( T ) T } i = 1 N .
By Lemma C.3, A t , K , i ( a ) ( T ) A^{(a)}_{t,K,i}(T) A t , K , i ( a ) ( T ) depends only on
( Q ~ t , i ( a ) ) ⊤ W t , K , 0 ( a ) H t , K , i P t , i ⊥ + ( Q ~ t , i ( a ) ) ⊤ T H t , K , i P t , i ⊥ d k , t . \frac{(\widetilde{Q}^{(a)}_{t,i})^{\top}W^{(a)}_{t,K,0}H_{t,K,i}P^{\perp}_{t,i}+(\widetilde{Q}^{(a)}_{t,i})^{\top}TH_{t,K,i}P^{\perp}_{t,i}}{\sqrt{d_{k,t}}}. d k , t ( Q t , i ( a ) ) ⊤ W t , K , 0 ( a ) H t , K , i P t , i ⊥ + ( Q t , i ( a ) ) ⊤ T H t , K , i P t , i ⊥ .
Define
Q t , a l l ( a ) : = [ Q ~ t , 1 ( a ) , … , Q ~ t , N ( a ) ] , H t , K , c t r : = [ H t , K , 1 P t , 1 ⊥ , … , H t , K , N P t , N ⊥ ] , T ^ t , K ( a ) : = P Q t , a l l ( a ) T t , K ( a ) P H t , K , c t r . Q^{(a)}_{t,\mathrm{all}}:=[\widetilde{Q}^{(a)}_{t,1},\ldots,\widetilde{Q}^{(a)}_{t,N}],\quad H_{t,K,\mathrm{ctr}}:=[H_{t,K,1}P^{\perp}_{t,1},\ldots,H_{t,K,N}P^{\perp}_{t,N}],\quad\widehat{T}^{(a)}_{t,K}:=P_{Q^{(a)}_{t,\mathrm{all}}}T^{(a)}_{t,K}P_{H_{t,K,\mathrm{ctr}}}. Q t , all ( a ) := [ Q t , 1 ( a ) , … , Q t , N ( a ) ] , H t , K , ctr := [ H t , K , 1 P t , 1 ⊥ , … , H t , K , N P t , N ⊥ ] , T t , K ( a ) := P Q t , all ( a ) T t , K ( a ) P H t , K , ctr .
Then ( Q ~ t , i ( a ) ) ⊤ T ^ t , K ( a ) H t , K , i P t , i ⊥ = ( Q ~ t , i ( a ) ) ⊤ T t , K ( a ) H t , K , i P t , i ⊥ (\widetilde{Q}^{(a)}_{t,i})^{\top}\widehat{T}^{(a)}_{t,K}H_{t,K,i}P^{\perp}_{t,i}=(\widetilde{Q}^{(a)}_{t,i})^{\top}T^{(a)}_{t,K}H_{t,K,i}P^{\perp}_{t,i} ( Q t , i ( a ) ) ⊤ T t , K ( a ) H t , K , i P t , i ⊥ = ( Q t , i ( a ) ) ⊤ T t , K ( a ) H t , K , i P t , i ⊥ , because ( Q ~ t , i ( a ) ) ⊤ P Q t , a l l ( a ) = ( Q ~ t , i ( a ) ) ⊤ (\widetilde{Q}^{(a)}_{t,i})^{\top}P_{Q^{(a)}_{t,\mathrm{all}}}=(\widetilde{Q}^{(a)}_{t,i})^{\top} ( Q t , i ( a ) ) ⊤ P Q t , all ( a ) = ( Q t , i ( a ) ) ⊤ , P H t , K , c t r H t , K , i P t , i ⊥ = H t , K , i P t , i ⊥ P_{H_{t,K,\mathrm{ctr}}}H_{t,K,i}P^{\perp}_{t,i}=H_{t,K,i}P^{\perp}_{t,i} P H t , K , ctr H t , K , i P t , i ⊥ = H t , K , i P t , i ⊥ . Thus A t , K , i ( a ) ( T ^ t , K ( a ) ) = A t , K , i ( a ) ( T t , K ( a ) ) A^{(a)}_{t,K,i}(\widehat{T}^{(a)}_{t,K})=A^{(a)}_{t,K,i}(T^{(a)}_{t,K}) A t , K , i ( a ) ( T t , K ( a ) ) = A t , K , i ( a ) ( T t , K ( a ) ) ∀ i \forall i ∀ i , and the conditional loss is invariant under
T t , K ( a ) ↦ P Q t , a l l ( a ) T t , K ( a ) P H t , K , c t r . T^{(a)}_{t,K}\mapsto P_{Q^{(a)}_{t,\mathrm{all}}}T^{(a)}_{t,K}P_{H_{t,K,\mathrm{ctr}}}. T t , K ( a ) ↦ P Q t , all ( a ) T t , K ( a ) P H t , K , ctr .
Finally, consider the case S = V S=V S = V . Hold all blocks other than T t , V ( a ) T^{(a)}_{t,V} T t , V ( a ) fixed. Then all attention matrices
A t , i ( a ) : = softmax r o w ( ( Q ~ t , i ( a ) ) ⊤ K ~ t , i ( a ) d k , t + C t , i M ) A^{(a)}_{t,i}:=\operatorname{softmax}_{\mathrm{row}}\left(\frac{(\widetilde{Q}^{(a)}_{t,i})^{\top}\widetilde{K}^{(a)}_{t,i}}{\sqrt{d_{k,t}}}+C^{M}_{t,i}\right) A t , i ( a ) := softmax row ( d k , t ( Q t , i ( a ) ) ⊤ K t , i ( a ) + C t , i M )
are fixed, and the conditional loss depends on T T T only through
{ W t , O ( a ) ( W t , V , 0 ( a ) + T ) H t , V , i ( A t , i ( a ) ) ⊤ } i = 1 N . \left\{W^{(a)}_{t,O}(W^{(a)}_{t,V,0}+T)H_{t,V,i}(A^{(a)}_{t,i})^{\top}\right\}_{i=1}^{N}. { W t , O ( a ) ( W t , V , 0 ( a ) + T ) H t , V , i ( A t , i ( a ) ) ⊤ } i = 1 N .
Define H t , V , A ( a ) : = [ H t , V , 1 ( A t , 1 ( a ) ) ⊤ , … , H t , V , N ( A t , N ( a ) ) ⊤ ] H^{(a)}_{t,V,A}:=[H_{t,V,1}(A^{(a)}_{t,1})^{\top},\ldots,H_{t,V,N}(A^{(a)}_{t,N})^{\top}] H t , V , A ( a ) := [ H t , V , 1 ( A t , 1 ( a ) ) ⊤ , … , H t , V , N ( A t , N ( a ) ) ⊤ ] and set T ^ t , V ( a ) : = P W t , O ( a ) T t , V ( a ) P H t , V , A ( a ) \widehat{T}^{(a)}_{t,V}:=P_{W^{(a)}_{t,O}}T^{(a)}_{t,V}P_{H^{(a)}_{t,V,A}} T t , V ( a ) := P W t , O ( a ) T t , V ( a ) P H t , V , A ( a ) .
Then for every i i i , W t , O ( a ) T ^ t , V ( a ) H t , V , i ( A t , i ( a ) ) ⊤ = W t , O ( a ) T t , V ( a ) H t , V , i ( A t , i ( a ) ) ⊤ W^{(a)}_{t,O}\widehat{T}^{(a)}_{t,V}H_{t,V,i}(A^{(a)}_{t,i})^{\top}=W^{(a)}_{t,O}T^{(a)}_{t,V}H_{t,V,i}(A^{(a)}_{t,i})^{\top} W t , O ( a ) T t , V ( a ) H t , V , i ( A t , i ( a ) ) ⊤ = W t , O ( a ) T t , V ( a ) H t , V , i ( A t , i ( a ) ) ⊤ , because W t , O ( a ) P W t , O ( a ) = W t , O ( a ) W^{(a)}_{t,O}P_{W^{(a)}_{t,O}}=W^{(a)}_{t,O} W t , O ( a ) P W t , O ( a ) = W t , O ( a ) , P H t , V , A ( a ) H t , V , i ( A t , i ( a ) ) ⊤ = H t , V , i ( A t , i ( a ) ) ⊤ P_{H^{(a)}_{t,V,A}}H_{t,V,i}(A^{(a)}_{t,i})^{\top}=H_{t,V,i}(A^{(a)}_{t,i})^{\top} P H t , V , A ( a ) H t , V , i ( A t , i ( a ) ) ⊤ = H t , V , i ( A t , i ( a ) ) ⊤ . Hence, the conditional loss is invariant under
T t , V ( a ) ↦ P W t , O ( a ) T t , V ( a ) P H t , V , A ( a ) . T^{(a)}_{t,V}\mapsto P_{W^{(a)}_{t,O}}T^{(a)}_{t,V}P_{H^{(a)}_{t,V,A}}. T t , V ( a ) ↦ P W t , O ( a ) T t , V ( a ) P H t , V , A ( a ) .
We have therefore verified the hypotheses of Theorem C.4 for the flattened block family S ~ \widetilde{\mathcal{S}} S . Hence, there exists a global minimizer minimizing total rank over the global minimizer set, and for that minimizer, the rank of each adapted block is bounded by the ranks of the corresponding projector pairs.
Evaluating the projector ranks at this rank-minimal global minimizer gives the following intermediate bounds:
rank ( T t , Q ( a ) ⋆ ) ≤ min { rank ( K t , c t r ( a ) ⋆ ) , rank ( H t , Q ⋆ ) } \operatorname{rank}(T^{(a)\star}_{t,Q})\leq\min\{\operatorname{rank}(K^{(a)\star}_{t,\mathrm{ctr}}),\operatorname{rank}(H^\star_{t,Q})\} rank ( T t , Q ( a ) ⋆ ) ≤ min { rank ( K t , ctr ( a ) ⋆ ) , rank ( H t , Q ⋆ )}
rank ( T t , K ( a ) ⋆ ) ≤ min { rank ( Q t , a l l ( a ) ⋆ ) , rank ( H t , K , c t r ⋆ ) } , \operatorname{rank}(T^{(a)\star}_{t,K})\leq\min\{\operatorname{rank}(Q^{(a)\star}_{t,\mathrm{all}}),\operatorname{rank}(H^\star_{t,K,\mathrm{ctr}})\}, rank ( T t , K ( a ) ⋆ ) ≤ min { rank ( Q t , all ( a ) ⋆ ) , rank ( H t , K , ctr ⋆ )} ,
and
rank ( T t , V ( a ) ⋆ ) ≤ min { rank ( W t , O ( a ) ) , rank ( H t , V , A ( a ) ⋆ ) } . \operatorname{rank}(T^{(a)\star}_{t,V})\leq\min\{\operatorname{rank}(W^{(a)}_{t,O}),\operatorname{rank}(H^{(a)\star}_{t,V,A})\}. rank ( T t , V ( a ) ⋆ ) ≤ min { rank ( W t , O ( a ) ) , rank ( H t , V , A ( a ) ⋆ )} .
Here
K t , c t r ( a ) ⋆ : = [ K ~ t , 1 ( a ) ⋆ P t , 1 ⊥ , … , K ~ t , N ( a ) ⋆ P t , N ⊥ ] , Q t , a l l ( a ) ⋆ : = [ Q ~ t , 1 ( a ) ⋆ , … , Q ~ t , N ( a ) ⋆ ] , K^{(a)\star}_{t,\mathrm{ctr}}:=[\widetilde{K}^{(a)\star}_{t,1}P^{\perp}_{t,1},\ldots,\widetilde{K}^{(a)\star}_{t,N}P^{\perp}_{t,N}],\quad Q^{(a)\star}_{t,\mathrm{all}}:=[\widetilde{Q}^{(a)\star}_{t,1},\ldots,\widetilde{Q}^{(a)\star}_{t,N}], K t , ctr ( a ) ⋆ := [ K t , 1 ( a ) ⋆ P t , 1 ⊥ , … , K t , N ( a ) ⋆ P t , N ⊥ ] , Q t , all ( a ) ⋆ := [ Q t , 1 ( a ) ⋆ , … , Q t , N ( a ) ⋆ ] ,
and
H t , V , A ( a ) ⋆ : = [ H t , V , 1 ⋆ ( A t , 1 ( a ) ⋆ ) ⊤ , … , H t , V , N ⋆ ( A t , N ( a ) ⋆ ) ⊤ ] , H t , K , c t r ⋆ : = [ H t , K , 1 ⋆ P t , 1 ⊥ , … , H t , K , N ⋆ P t , N ⊥ ] . H^{(a)\star}_{t,V,A}:=[H^\star_{t,V,1}(A^{(a)\star}_{t,1})^{\top},\ldots,H^\star_{t,V,N}(A^{(a)\star}_{t,N})^{\top}],\quad H^\star_{t,K,\mathrm{ctr}}:=[H^\star_{t,K,1}P^{\perp}_{t,1},\ldots,H^\star_{t,K,N}P^{\perp}_{t,N}]. H t , V , A ( a ) ⋆ := [ H t , V , 1 ⋆ ( A t , 1 ( a ) ⋆ ) ⊤ , … , H t , V , N ⋆ ( A t , N ( a ) ⋆ ) ⊤ ] , H t , K , ctr ⋆ := [ H t , K , 1 ⋆ P t , 1 ⊥ , … , H t , K , N ⋆ P t , N ⊥ ] .
We now bound the ranks of these matrices. Since
K t , c t r ( a ) ⋆ = [ K ~ t , 1 ( a ) ⋆ , … , K ~ t , N ( a ) ⋆ ] blkdiag ( P t , 1 ⊥ , … , P t , N ⊥ ) , K^{(a)\star}_{t,\mathrm{ctr}}=[\widetilde{K}^{(a)\star}_{t,1},\ldots,\widetilde{K}^{(a)\star}_{t,N}]\operatorname{blkdiag}(P^{\perp}_{t,1},\ldots,P^{\perp}_{t,N}), K t , ctr ( a ) ⋆ = [ K t , 1 ( a ) ⋆ , … , K t , N ( a ) ⋆ ] blkdiag ( P t , 1 ⊥ , … , P t , N ⊥ ) ,
we have
rank ( K t , c t r ( a ) ⋆ ) ≤ min { rank ( [ K ~ t , 1 ( a ) ⋆ , … , K ~ t , N ( a ) ⋆ ] ) , ∑ i = 1 N rank ( P t , i ⊥ ) } . \operatorname{rank}(K^{(a)\star}_{t,\mathrm{ctr}})\leq\min\left\{\operatorname{rank}([\widetilde{K}^{(a)\star}_{t,1},\ldots,\widetilde{K}^{(a)\star}_{t,N}]),\sum_{i=1}^{N}\operatorname{rank}(P^{\perp}_{t,i})\right\}. rank ( K t , ctr ( a ) ⋆ ) ≤ min { rank ([ K t , 1 ( a ) ⋆ , … , K t , N ( a ) ⋆ ]) , i = 1 ∑ N rank ( P t , i ⊥ ) } .
Because rank ( P t , i ⊥ ) = ϱ t , K , i − 1 \operatorname{rank}(P_{t,i}^{\perp})=\varrho_{t,K,i}-1 rank ( P t , i ⊥ ) = ϱ t , K , i − 1 and [ K ~ t , 1 ( a ) ⋆ , … , K ~ t , N ( a ) ⋆ ] = ( W t , K , 0 ( a ) + T t , K ( a ) ⋆ ) H t , K ⋆ [\widetilde{K}_{t,1}^{(a)\star},\ldots,\widetilde{K}_{t,N}^{(a)\star}]=(W_{t,K,0}^{(a)}+T_{t,K}^{(a)\star})H_{t,K}^{\star} [ K t , 1 ( a ) ⋆ , … , K t , N ( a ) ⋆ ] = ( W t , K , 0 ( a ) + T t , K ( a ) ⋆ ) H t , K ⋆ , it follows that
rank ( K t , ctr ( a ) ⋆ ) ≤ min { d k , t , rank ( H t , K ⋆ ) , ∑ i = 1 N ( ϱ t , K , i − 1 ) } . \operatorname{rank}(K_{t,\operatorname{ctr}}^{(a)\star})\leq\min\left\{d_{k,t},\operatorname{rank}(H_{t,K}^{\star}),\sum_{i=1}^{N}(\varrho_{t,K,i}-1)\right\}. rank ( K t , ctr ( a ) ⋆ ) ≤ min { d k , t , rank ( H t , K ⋆ ) , i = 1 ∑ N ( ϱ t , K , i − 1 ) } .
Combining this with
rank ( T t , Q ( a ) ⋆ ) ≤ min { rank ( K t , ctr ( a ) ⋆ ) , rank ( H t , Q ⋆ ) } \operatorname{rank}(T_{t,Q}^{(a)\star})\leq\min\{\operatorname{rank}(K_{t,\operatorname{ctr}}^{(a)\star}),\operatorname{rank}(H_{t,Q}^{\star})\} rank ( T t , Q ( a ) ⋆ ) ≤ min { rank ( K t , ctr ( a ) ⋆ ) , rank ( H t , Q ⋆ )}
gives the stated query bound.
For the key bound, first observe that rank ( Q t , all ( a ) ⋆ ) ≤ min { d k , t , rank ( H t , Q ⋆ ) } \operatorname{rank}(Q_{t,\operatorname{all}}^{(a)\star})\leq\min\{d_{k,t},\operatorname{rank}(H_{t,Q}^{\star})\} rank ( Q t , all ( a ) ⋆ ) ≤ min { d k , t , rank ( H t , Q ⋆ )} , because Q t , all ( a ) ⋆ = ( W t , Q , 0 ( a ) + T t , Q ( a ) ⋆ ) H t , Q ⋆ Q_{t,\operatorname{all}}^{(a)\star}=(W_{t,Q,0}^{(a)}+T_{t,Q}^{(a)\star})H_{t,Q}^{\star} Q t , all ( a ) ⋆ = ( W t , Q , 0 ( a ) + T t , Q ( a ) ⋆ ) H t , Q ⋆ . Also, H t , K , ctr ⋆ = H t , K ⋆ blkdiag ( P t , 1 ⊥ , … , P t , N ⊥ ) H_{t,K,\operatorname{ctr}}^{\star}=H_{t,K}^{\star}\operatorname{blkdiag}(P_{t,1}^{\perp},\ldots,P_{t,N}^{\perp}) H t , K , ctr ⋆ = H t , K ⋆ blkdiag ( P t , 1 ⊥ , … , P t , N ⊥ ) , so
rank ( H t , K , ctr ⋆ ) ≤ min { rank ( H t , K ⋆ ) , ∑ i = 1 N ( ϱ t , K , i − 1 ) } . \operatorname{rank}(H_{t,K,\operatorname{ctr}}^{\star})\leq\min\left\{\operatorname{rank}(H_{t,K}^{\star}),\sum_{i=1}^{N}(\varrho_{t,K,i}-1)\right\}. rank ( H t , K , ctr ⋆ ) ≤ min { rank ( H t , K ⋆ ) , i = 1 ∑ N ( ϱ t , K , i − 1 ) } .
Combining these two estimates with
rank ( T t , K ( a ) ⋆ ) ≤ min { rank ( Q t , all ( a ) ⋆ ) , rank ( H t , K , ctr ⋆ ) } \operatorname{rank}(T_{t,K}^{(a)\star})\leq\min\{\operatorname{rank}(Q_{t,\operatorname{all}}^{(a)\star}),\operatorname{rank}(H_{t,K,\operatorname{ctr}}^{\star})\} rank ( T t , K ( a ) ⋆ ) ≤ min { rank ( Q t , all ( a ) ⋆ ) , rank ( H t , K , ctr ⋆ )}
gives the stated key bound.
For the value bound, we have H t , V , A ( a ) ⋆ = H t , V ⋆ blkdiag ( ( A t , 1 ( a ) ⋆ ) T , … , ( A t , N ( a ) ⋆ ) T ) H_{t,V,A}^{(a)\star}=H_{t,V}^{\star}\operatorname{blkdiag}((A_{t,1}^{(a)\star})^{\mathsf{T}},\ldots,(A_{t,N}^{(a)\star})^{\mathsf{T}}) H t , V , A ( a ) ⋆ = H t , V ⋆ blkdiag (( A t , 1 ( a ) ⋆ ) T , … , ( A t , N ( a ) ⋆ ) T ) , and hence
rank ( H t , V , A ( a ) ⋆ ) ≤ min { rank ( H t , V ⋆ ) , ∑ i = 1 N rank ( A t , i ( a ) ⋆ ) } . \operatorname{rank}(H_{t,V,A}^{(a)\star})\leq\min\left\{\operatorname{rank}(H_{t,V}^{\star}),\sum_{i=1}^{N}\operatorname{rank}(A_{t,i}^{(a)\star})\right\}. rank ( H t , V , A ( a ) ⋆ ) ≤ min { rank ( H t , V ⋆ ) , i = 1 ∑ N rank ( A t , i ( a ) ⋆ ) } .
Since W t , O ( a ) W_{t,O}^{(a)} W t , O ( a ) has at most d v , t d_{v,t} d v , t columns, rank ( W t , O ( a ) ) ≤ d v , t \operatorname{rank}(W_{t,O}^{(a)})\leq d_{v,t} rank ( W t , O ( a ) ) ≤ d v , t , so combining this with
rank ( T t , V ( a ) ⋆ ) ≤ min { rank ( W t , O ( a ) ) , rank ( H t , V , A ( a ) ⋆ ) } \operatorname{rank}(T_{t,V}^{(a)\star})\leq\min\{\operatorname{rank}(W_{t,O}^{(a)}),\operatorname{rank}(H_{t,V,A}^{(a)\star})\} rank ( T t , V ( a ) ⋆ ) ≤ min { rank ( W t , O ( a ) ) , rank ( H t , V , A ( a ) ⋆ )}
gives the stated value bound. □
The adapted-output extension follows from the same blockwise compression, but not from a simultaneous linear observation map for the full Q / K / V / O Q/K/V/O Q / K / V / O tuple.
Proof of Theorem 4.5. Let I \mathcal{I} I index all selected query, key, value, and full output-projection updates. By the assumed continuity of the forward computation, the empirical loss is lower semicontinuous and bounded below. Since every adapted block has a strictly positive nuclear-norm coefficient, the objective is coercive on the finite-dimensional product space ∏ j ∈ I R m j × n j \prod_{j\in\mathcal{I}}\mathbb{R}^{m_j\times n_j} ∏ j ∈ I R m j × n j and therefore attains a global minimum.
Since the sum of matrix ranks is integer valued and bounded above by ∑ j ∈ I min { m j , n j } \sum_{j\in\mathcal{I}}\min\{m_j,n_j\} ∑ j ∈ I min { m j , n j } , there is a global minimizer { T j ⋆ } \{T_j^\star\} { T j ⋆ } minimizing that sum over all global minimizers. We establish the hypotheses of Theorem C.4 by fixing all companion updates each time a block is compressed.
For any query update, the input matrices H t , Q , i H_{t,Q,i} H t , Q , i and adapted keys K ~ t , i ( a ) \widetilde{K}_{t,i}^{(a)} K t , i ( a ) do not depend on that query update. Row-wise score centering and the argument in the proof of Theorem 4.4 therefore show that replacing T t , Q ( a ) T_{t,Q}^{(a)} T t , Q ( a ) by P K t , ctr ( a ) T t , Q ( a ) P H t , Q P_{K_{t,\operatorname{ctr}}^{(a)}}T_{t,Q}^{(a)}P_{H_{t,Q}} P K t , ctr ( a ) T t , Q ( a ) P H t , Q preserves every corresponding attention matrix. The adapted output projection is held fixed during this replacement, so it does not change that invariance. Similarly, for a key update, replacing T t , K ( a ) T_{t,K}^{(a)} T t , K ( a ) by
P Q t , all ( a ) T t , K ( a ) P H t , K , ctr P_{Q_{t,\operatorname{all}}^{(a)}}T_{t,K}^{(a)}P_{H_{t,K,\operatorname{ctr}}} P Q t , all ( a ) T t , K ( a ) P H t , K , ctr
preserves the centered score products and the attention matrices. Thus the query and key projector bounds from Theorem 4.4 remain valid.
For an adapted value block, all attention matrices and the companion output update T t , O T_{t,O} T t , O are fixed. Write the adapted output projection in head blocks as
W ~ t , O : = W t , O , 0 + T t , O = [ W ~ t , O ( a ) ] a ∈ A t , W ~ t , O ( a ) ∈ R d o u t , t × d v , t , \widetilde{W}_{t,O} := W_{t,O,0} + T_{t,O} = [\widetilde{W}^{(a)}_{t,O}]_{a\in\mathcal{A}_t}, \qquad\widetilde{W}^{(a)}_{t,O} \in\mathbb{R}^{d_{\mathrm{out},t}\times d_{v,t}}, W t , O := W t , O , 0 + T t , O = [ W t , O ( a ) ] a ∈ A t , W t , O ( a ) ∈ R d out , t × d v , t ,
and put
C t , i ( a ) = H t , V , i ( A t , i ( a ) ) ⊤ , C t ( a ) = [ C t , 1 ( a ) , … , C t , N ( a ) ] . C^{(a)}_{t,i} = H_{t,V,i}(A^{(a)}_{t,i})^{\top}, \qquad C^{(a)}_t = [C^{(a)}_{t,1}, \ldots, C^{(a)}_{t,N}]. C t , i ( a ) = H t , V , i ( A t , i ( a ) ) ⊤ , C t ( a ) = [ C t , 1 ( a ) , … , C t , N ( a ) ] .
Here C t , i ( a ) ∈ R n t , V × ϱ t , Q , i C^{(a)}_{t,i} \in\mathbb{R}^{n_{t,V}\times\varrho_{t,Q,i}} C t , i ( a ) ∈ R n t , V × ϱ t , Q , i . Set
P O ( a ) = P ( W ~ t , O ( a ) ) ⊤ , P C ( a ) = P C t ( a ) . P^{(a)}_O = P(\widetilde{W}^{(a)}_{t,O})^{\top}, \qquad P^{(a)}_C = P_{C^{(a)}_t}. P O ( a ) = P ( W t , O ( a ) ) ⊤ , P C ( a ) = P C t ( a ) .
For any T ∈ R d v , t × n t , V T \in\mathbb{R}^{d_{v,t}\times n_{t,V}} T ∈ R d v , t × n t , V and every sample i i i ,
W ~ t , O ( a ) P O ( a ) T P C ( a ) C t , i ( a ) = W ~ t , O ( a ) T C t , i ( a ) . \widetilde{W}^{(a)}_{t,O}P^{(a)}_O T P^{(a)}_C C^{(a)}_{t,i} = \widetilde{W}^{(a)}_{t,O}T C^{(a)}_{t,i}. W t , O ( a ) P O ( a ) T P C ( a ) C t , i ( a ) = W t , O ( a ) T C t , i ( a ) .
Thus the stage output and the downstream loss are unchanged by T ↦ P O ( a ) T P C ( a ) T \mapsto P^{(a)}_O T P^{(a)}_C T ↦ P O ( a ) T P C ( a ) . The corresponding nuclear norm does not increase by Lemma C.1.
For the full output projection, hold all Q / K / V Q/K/V Q / K / V updates and all other blocks fixed. For each training sample i i i , define
B t , i ( a ) : = V ~ t , i ( a ) ( A t , i ( a ) ) ⊤ ∈ R d v , t × ϱ t , Q , i , B t , i : = [ B t , i ( a 1 ) ⋮ B t , i ( a ∣ A t ∣ ) ] , H t , O : = [ B t , 1 , … , B t , N ] . B^{(a)}_{t,i} := \widetilde{V}^{(a)}_{t,i}(A^{(a)}_{t,i})^{\top} \in\mathbb{R}^{d_{v,t}\times\varrho_{t,Q,i}}, \qquad B_{t,i} := \begin{bmatrix} B^{(a_1)}_{t,i} \\ \vdots\\ B^{(a_{|\mathcal{A}_t|})}_{t,i} \end{bmatrix}, \qquad H_{t,O} := [B_{t,1},\ldots,B_{t,N}]. B t , i ( a ) := V t , i ( a ) ( A t , i ( a ) ) ⊤ ∈ R d v , t × ϱ t , Q , i , B t , i := B t , i ( a 1 ) ⋮ B t , i ( a ∣ A t ∣ ) , H t , O := [ B t , 1 , … , B t , N ] .
Every B t , i B_{t,i} B t , i is then fixed, and the varying part of the stage output equals T t , O B t , i T_{t,O}B_{t,i} T t , O B t , i . Since P H t , O B t , i = B t , i P_{H_{t,O}}B_{t,i}=B_{t,i} P H t , O B t , i = B t , i ,
( T t , O P H t , O ) B t , i = T t , O B t , i ( i ∈ [ N ] ) . (T_{t,O}P_{H_{t,O}})B_{t,i}=T_{t,O}B_{t,i} \qquad(i\in[N]). ( T t , O P H t , O ) B t , i = T t , O B t , i ( i ∈ [ N ]) .
The replacement T t , O ↦ T t , O P H t , O T_{t,O} \mapsto T_{t,O}P_{H_{t,O}} T t , O ↦ T t , O P H t , O therefore preserves the objective’s loss term and does not increase the nuclear norm.
All conditional compression projectors depend only on companion updates. Theorem C.4 applies to the finite block family I \mathcal{I} I and gives, at the chosen rank-minimal global minimizer,
rank ( T t , V ( a ) ⋆ ) ≤ min { rank ( W ~ t , O ( a ) ⋆ ) , rank ( C t ( a ) ⋆ ) } , rank ( T t , O ⋆ ) ≤ min { d o u t , t , rank ( H t , O ⋆ ) } . (3) \operatorname{rank}(T^{(a)\star}_{t,V}) \le\min\{\operatorname{rank}(\widetilde{W}^{(a)\star}_{t,O}),\operatorname{rank}(C^{(a)\star}_t)\}, \qquad\operatorname{rank}(T^\star_{t,O}) \le\min\{d_{\mathrm{out},t},\operatorname{rank}(H^\star_{t,O})\}. \tag*{(3)} rank ( T t , V ( a ) ⋆ ) ≤ min { rank ( W t , O ( a ) ⋆ ) , rank ( C t ( a ) ⋆ )} , rank ( T t , O ⋆ ) ≤ min { d out , t , rank ( H t , O ⋆ )} . (3)
It remains to bound the three ranks on the right-hand side of (C.1). First, W ~ t , O ( a ) ⋆ \widetilde{W}^{(a)\star}_{t,O} W t , O ( a ) ⋆ has d o u t , t d_{\mathrm{out},t} d out , t rows and d v , t d_{v,t} d v , t columns, so
rank ( W ~ t , O ( a ) ⋆ ) ≤ min { d o u t , t , d v , t } ≤ d v , t . \operatorname{rank}(\widetilde{W}^{(a)\star}_{t,O}) \le\min\{d_{\mathrm{out},t},d_{v,t}\} \le d_{v,t}. rank ( W t , O ( a ) ⋆ ) ≤ min { d out , t , d v , t } ≤ d v , t .
Second, the identity
C t ( a ) ⋆ = H t , V ⋆ blkdiag ( ( A t , 1 ( a ) ⋆ ) ⊤ , … , ( A t , N ( a ) ⋆ ) ⊤ ) C^{(a)\star}_t = H^\star_{t,V}\operatorname{blkdiag}((A^{(a)\star}_{t,1})^{\top},\ldots,(A^{(a)\star}_{t,N})^{\top}) C t ( a ) ⋆ = H t , V ⋆ blkdiag (( A t , 1 ( a ) ⋆ ) ⊤ , … , ( A t , N ( a ) ⋆ ) ⊤ )
implies
rank ( C t ( a ) ⋆ ) ≤ min { rank ( H t , V ⋆ ) , ∑ i = 1 N rank ( A t , i ( a ) ⋆ ) } . \operatorname{rank}(C^{(a)\star}_t) \le\min\left\{\operatorname{rank}(H^\star_{t,V}),\sum_{i=1}^{N}\operatorname{rank}(A^{(a)\star}_{t,i})\right\}. rank ( C t ( a ) ⋆ ) ≤ min { rank ( H t , V ⋆ ) , i = 1 ∑ N rank ( A t , i ( a ) ⋆ ) } .
Third, H t , O ⋆ ∈ R ( ∣ A t ∣ d v , t ) × ∑ i = 1 N ϱ t , Q , i H^\star_{t,O} \in\mathbb{R}^{(|\mathcal{A}_t|d_{v,t})\times\sum_{i=1}^{N}\varrho_{t,Q,i}} H t , O ⋆ ∈ R ( ∣ A t ∣ d v , t ) × ∑ i = 1 N ϱ t , Q , i by construction, so
rank ( H t , O ⋆ ) ≤ min { ∣ A t ∣ d v , t , ∑ i = 1 N ϱ t , Q , i } . \operatorname{rank}(H^\star_{t,O}) \le\min\left\{|\mathcal{A}_t|d_{v,t},\sum_{i=1}^{N}\varrho_{t,Q,i}\right\}. rank ( H t , O ⋆ ) ≤ min { ∣ A t ∣ d v , t , i = 1 ∑ N ϱ t , Q , i } .
Substituting these three estimates into (C.1) yields the two bounds stated in the theorem and completes the proof.
Proofs for Section 4.3
For the positive rank-deficiency statement, the first step is to write the first-order stationarity equations in a symmetric block form.
Lemma C.5 (Stationarity conditions for the perturbed block LoRA objective). In the setting of Definition 4.6, suppose L \mathcal{L} L is differentiable and { ( U S , V S ) } \{(U_S,V_S)\} {( U S , V S )} is a first-order stationary point of L { λ S } , { P S } l o r a \mathcal{L}^{\mathrm{lora}}_{\{\lambda_S\},\{P_S\}} L { λ S } , { P S } lora . Let T S : = U S V S ⊤ T_S := U_SV_S^{\top} T S := U S V S ⊤ and G S : = ∇ T S L ( { T S ′ } ) G_S := \nabla_{T_S}\mathcal{L}(\{T_{S'}\}) G S := ∇ T S L ({ T S ′ }) . Then for every S ∈ S S\in\mathcal{S} S ∈ S ,
( B m S , n S ( G S ) + λ S I m S + n S + P S ) R S = 0 , R S = [ U S V S ] . \left(B_{m_S,n_S}(G_S)+\lambda_S I_{m_S+n_S}+P_S\right)R_S=0,\qquad R_S=\begin{bmatrix}U_S\\V_S\end{bmatrix}. ( B m S , n S ( G S ) + λ S I m S + n S + P S ) R S = 0 , R S = [ U S V S ] .
Proof of Lemma C.5. Fix S S S and differentiate w.r.t. U S U_S U S and V S V_S V S using the chain rule: ∇ U S L ( { U S ′ V S ′ T } ) = G S V S \nabla_{U_S}\mathcal{L}(\{U_{S'}V_{S'}^{\mathsf{T}}\})=G_SV_S ∇ U S L ({ U S ′ V S ′ T }) = G S V S and ∇ V S L ( { U S ′ V S ′ T } ) = G S T U S \nabla_{V_S}\mathcal{L}(\{U_{S'}V_{S'}^{\mathsf{T}}\})=G_S^{\mathsf{T}}U_S ∇ V S L ({ U S ′ V S ′ T }) = G S T U S . The regularizer contributes λ S U S \lambda_SU_S λ S U S and λ S V S \lambda_SV_S λ S V S . The perturbation term satisfies ∇ R S ⟨ P S , R S R S T ⟩ = P S R S \nabla_{R_S}\langle P_S,R_SR_S^{\mathsf{T}}\rangle=P_SR_S ∇ R S ⟨ P S , R S R S T ⟩ = P S R S . Stacking the two stationarity equations yields the displayed KKT condition.
The genericity argument is a dimension-counting statement. We will need the following elementary measure-zero lemma:
Lemma C.6 (Measure-zero Minkowski sums with C 1 C^1 C 1 images). Let M \mathcal{M} M be an m m m -dimensional embedded C 1 C^1 C 1 submanifold of R d \mathbb{R}^d R d . Let U ⊆ R p U\subseteq\mathbb{R}^p U ⊆ R p be open, and let f : U → R d f:U\to\mathbb{R}^d f : U → R d be C 1 C^1 C 1 . If m + p < d m+p<d m + p < d , then
M + f ( U ) : = { x + f ( u ) : x ∈ M , u ∈ U } \mathcal{M}+f(U):=\{x+f(u):x\in\mathcal{M},u\in U\} M + f ( U ) := { x + f ( u ) : x ∈ M , u ∈ U } has Lebesgue measure zero in R d \mathbb{R}^d R d .
Proof of Lemma C.6. Since M ⊆ R d \mathcal{M}\subseteq\mathbb{R}^d M ⊆ R d is an embedded submanifold, it is second countable and admits a countable C 1 C^1 C 1 atlas. It is therefore enough to prove the claim on one coordinate patch.
Fix a chart φ : V → M \varphi:V\to\mathcal{M} φ : V → M , where V ⊆ R m V\subseteq\mathbb{R}^m V ⊆ R m is open. On this patch, { φ ( v ) + f ( u ) : v ∈ V , u ∈ U } \{\varphi(v)+f(u):v\in V,u\in U\} { φ ( v ) + f ( u ) : v ∈ V , u ∈ U } is the image of the C 1 C^1 C 1 map
F : V × U → R d , F ( v , u ) = φ ( v ) + f ( u ) . F:V\times U\to\mathbb{R}^d,\qquad F(v,u)=\varphi(v)+f(u). F : V × U → R d , F ( v , u ) = φ ( v ) + f ( u ) .
The domain V × U V\times U V × U is open in R m + p \mathbb{R}^{m+p} R m + p . Set k : = m + p k:=m+p k := m + p . By assumption, k < d k<d k < d .
Cover V × U V\times U V × U by countably many compact cubes Q ℓ Q_\ell Q ℓ whose interiors cover V × U V\times U V × U and whose closures are contained in V × U V\times U V × U . Fix one such cube Q Q Q . Since F F F is C 1 C^1 C 1 , its derivative is bounded on Q Q Q . Hence there exists L Q < ∞ L_Q<\infty L Q < ∞ such that
∥ F ( x ) − F ( y ) ∥ 2 ≤ L Q ∥ x − y ∥ 2 ∀ x , y ∈ Q . \|F(x)-F(y)\|_2\leq L_Q\|x-y\|_2\quad\forall x,y\in Q. ∥ F ( x ) − F ( y ) ∥ 2 ≤ L Q ∥ x − y ∥ 2 ∀ x , y ∈ Q .
Subdivide Q Q Q into N k N^k N k subcubes of side length proportional to N − 1 N^{-1} N − 1 . The image of each subcube is contained in a Euclidean ball in R d \mathbb{R}^d R d of radius at most C Q N − 1 C_QN^{-1} C Q N − 1 , where C Q C_Q C Q is independent of N N N . Therefore the d d d -dimensional Lebesgue outer measure of F ( Q ) F(Q) F ( Q ) is bounded by
N k ω d ( C Q N ) d = ω d C Q d N k − d , N^k\omega_d\left(\frac{C_Q}{N}\right)^d=\omega_dC_Q^dN^{k-d}, N k ω d ( N C Q ) d = ω d C Q d N k − d ,
where ω d \omega_d ω d is the volume of the unit ball in R d \mathbb{R}^d R d . Since k − d < 0 k-d<0 k − d < 0 , this upper bound converges to 0 0 0 as N → ∞ N\to\infty N → ∞ . Thus F ( Q ) F(Q) F ( Q ) has Lebesgue measure zero.
Taking the countable union over all cubes Q ℓ Q_\ell Q ℓ and then over the countable atlas of M \mathcal{M} M shows that M + f ( U ) \mathcal{M}+f(U) M + f ( U ) has Lebesgue measure zero in R d \mathbb{R}^d R d .
Proof of Theorem 4.7. If r > d r>d r > d , then every R ∈ R d × r R\in\mathbb{R}^{d\times r} R ∈ R d × r satisfies rank ( R ) ≤ d < r \operatorname{rank}(R)\leq d<r rank ( R ) ≤ d < r , so the conclusion is immediate. Assume from now on that r ≤ d r\leq d r ≤ d .
Let
D s y m : = dim S d = d ( d + 1 ) 2 . D_{\mathrm{sym}}:=\dim\mathbb{S}^d=\frac{d(d+1)}{2}. D sym := dim S d = 2 d ( d + 1 ) .
For s ∈ { r , r + 1 , … , d } s\in\{r,r+1,\ldots,d\} s ∈ { r , r + 1 , … , d } , define A s : = { M ∈ S d : dim N ( M ) = s } \mathcal{A}_s:=\{M\in\mathbb{S}^d:\dim\mathcal{N}(M)=s\} A s := { M ∈ S d : dim N ( M ) = s } . The set A s \mathcal{A}_s A s is an embedded C ∞ C^\infty C ∞ submanifold of S d \mathbb{S}^d S d with
dim ( A s ) = D s y m − s ( s + 1 ) 2 . \dim(\mathcal{A}_s)=D_{\mathrm{sym}}-\frac{s(s+1)}{2}. dim ( A s ) = D sym − 2 s ( s + 1 ) .
Indeed, A s \mathcal{A}_s A s is the set of symmetric matrices of rank d − s d-s d − s ; its codimension in S d \mathbb{S}^d S d is the dimension of S s \mathbb{S}^s S s , namely s ( s + 1 ) / 2 s(s+1)/2 s ( s + 1 ) /2 .
Suppose that for some P ∈ S d P\in\mathbb{S}^d P ∈ S d there exist G ∈ G G\in\mathcal{G} G ∈ G and R ∈ R d × r R\in\mathbb{R}^{d\times r} R ∈ R d × r such that rank ( R ) = r \operatorname{rank}(R)=r rank ( R ) = r and
( G + λ I d + P ) R = 0. (G+\lambda I_d+P)R=0. ( G + λ I d + P ) R = 0.
Set M : = G + λ I d + P M:=G+\lambda I_d+P M := G + λ I d + P . Since the columns of R R R are linearly independent and lie in N ( M ) \mathcal{N}(M) N ( M ) , we have dim N ( M ) ≥ r \dim\mathcal{N}(M)\geq r dim N ( M ) ≥ r . Therefore M ∈ ⋃ s = r d A s M\in\bigcup_{s=r}^{d}\mathcal{A}_s M ∈ ⋃ s = r d A s , and hence
P = M − λ I d − G ∈ ⋃ s = r d ( ( A s − λ I d ) − G ) . P=M-\lambda I_d-G\in\bigcup_{s=r}^{d}\bigl((\mathcal{A}_s-\lambda I_d)-\mathcal{G}\bigr). P = M − λ I d − G ∈ s = r ⋃ d ( ( A s − λ I d ) − G ) .
It remains to show that each set ( A s − λ I d ) − G (\mathcal{A}_s-\lambda I_d)-\mathcal{G} ( A s − λ I d ) − G has Lebesgue measure zero in S d \mathbb{S}^d S d . Since − G -\mathcal{G} − G is contained in a finite union of C 1 C^1 C 1 images of dimension at most p p p , there exist open sets U j ⊆ R p j U_j\subseteq\mathbb{R}^{p_j} U j ⊆ R p j with p j ≤ p p_j\leq p p j ≤ p and C 1 C^1 C 1 maps f j : U j → S d f_j:U_j\to\mathbb{S}^d f j : U j → S d , j = 1 , … , J j=1,\ldots,J j = 1 , … , J , such that
− G ⊆ ⋃ j = 1 J f j ( U j ) . -\mathcal{G}\subseteq\bigcup_{j=1}^{J}f_j(U_j). − G ⊆ j = 1 ⋃ J f j ( U j ) .
Fix s ∈ { r , … , d } s\in\{r,\ldots,d\} s ∈ { r , … , d } and j ∈ [ J ] j\in[J] j ∈ [ J ] . The translated manifold A s − λ I d \mathcal{A}_s-\lambda I_d A s − λ I d is an embedded C ∞ C^\infty C ∞ submanifold of S d \mathbb{S}^d S d with dimension D s y m − s ( s + 1 ) 2 D_{\mathrm{sym}}-\frac{s(s+1)}{2} D sym − 2 s ( s + 1 ) . Because s ≥ r s\geq r s ≥ r and p j ≤ p p_j\leq p p j ≤ p ,
( D s y m − s ( s + 1 ) 2 ) + p j ≤ D s y m − r ( r + 1 ) 2 + p < D s y m . \left(D_{\mathrm{sym}}-\frac{s(s+1)}{2}\right)+p_j\leq D_{\mathrm{sym}}-\frac{r(r+1)}{2}+p<D_{\mathrm{sym}}. ( D sym − 2 s ( s + 1 ) ) + p j ≤ D sym − 2 r ( r + 1 ) + p < D sym .
Lemma C.6, applied inside the Euclidean space S d \mathbb{S}^d S d , implies that ( A s − λ I d ) + f j ( U j ) (\mathcal{A}_s-\lambda I_d)+f_j(U_j) ( A s − λ I d ) + f j ( U j ) has Lebesgue measure zero in S d \mathbb{S}^d S d . Taking the finite union over j j j gives that ( A s − λ I d ) − G (\mathcal{A}_s-\lambda I_d)-\mathcal{G} ( A s − λ I d ) − G has Lebesgue measure zero in S d \mathbb{S}^d S d .
Finally, taking the finite union over s = r , … , d s=r,\ldots,d s = r , … , d gives a Lebesgue-null exceptional set. Thus, for every P P P outside this exceptional set, there do not exist G ∈ G G\in\mathcal{G} G ∈ G and R ∈ R d × r R\in\mathbb{R}^{d\times r} R ∈ R d × r with rank ( R ) = r \operatorname{rank}(R)=r rank ( R ) = r satisfying
( G + λ I d + P ) R = 0. (G+\lambda I_d+P)R=0. ( G + λ I d + P ) R = 0.
Intersecting this full-measure set with S + d \mathbb{S}^{d}_{+} S + d proves the conclusion for Lebesgue-a.e. positive semidefinite perturbation.
The next lemma connects the stagewise observation model to the KKT matrices above.
Lemma C.7 (Stagewise gradient range under L L L -stage linear map dependence). Assume L \mathcal{L} L from Definition 3.2 is differentiable. Fix a parameter tuple and the corresponding states z t z_t z t . Then for every t ∈ [ L ] t\in[L] t ∈ [ L ] and S ∈ S t S\in\mathcal{S}_t S ∈ S t , we have ∇ T t , S L ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) \nabla_{T_{t,S}}\mathcal{L}\in\mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1})^*) ∇ T t , S L ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) .
Proof of Lemma C.7. Fix t ∈ [ L ] t\in[L] t ∈ [ L ] and S ∈ S t S\in\mathcal{S}_t S ∈ S t , and fix all blocks except T t , S T_{t,S} T t , S . Because all upstream blocks are fixed, the state z t + 1 z_{t+1} z t + 1 is fixed. Define the conditional loss
F t , S ( T ) : = L ( { T u , A } ( u , A ) ≠ ( t , S ) , T t , S = T ) . F_{t,S}(T):=\mathcal{L}\left(\{T_{u,A}\}_{(u,A)\neq(t,S)},T_{t,S}=T\right). F t , S ( T ) := L ( { T u , A } ( u , A ) = ( t , S ) , T t , S = T ) .
By Definition 3.2, the function F t , S F_{t,S} F t , S depends on T T T only through the linear map T ↦ Φ t , S ( T , z t + 1 ) T\mapsto\Phi_{t,S}(T,z_{t+1}) T ↦ Φ t , S ( T , z t + 1 ) . Therefore, for every Δ ∈ N ( Φ t , S ( ⋅ , z t + 1 ) ) \Delta\in\mathcal{N}(\Phi_{t,S}(\cdot,z_{t+1})) Δ ∈ N ( Φ t , S ( ⋅ , z t + 1 )) and every T T T , F t , S ( T + Δ ) = F t , S ( T ) F_{t,S}(T+\Delta)=F_{t,S}(T) F t , S ( T + Δ ) = F t , S ( T ) . Since F F F is differentiable, this implies
0 = D F t , S ( T ) [ Δ ] = ⟨ ∇ T t , S L , Δ ⟩ F ∀ Δ ∈ N ( Φ t , S ( ⋅ , z t + 1 ) ) . 0=DF_{t,S}(T)[\Delta]=\langle\nabla_{T_{t,S}}\mathcal{L},\Delta\rangle_F\quad\forall\Delta\in\mathcal{N}(\Phi_{t,S}(\cdot,z_{t+1})). 0 = D F t , S ( T ) [ Δ ] = ⟨ ∇ T t , S L , Δ ⟩ F ∀Δ ∈ N ( Φ t , S ( ⋅ , z t + 1 )) .
Hence ∇ T t , S L ∈ N ( Φ t , S ( ⋅ , z t + 1 ) ) ⊥ \nabla_{T_{t,S}}\mathcal{L}\in\mathcal{N}(\Phi_{t,S}(\cdot,z_{t+1}))^\perp ∇ T t , S L ∈ N ( Φ t , S ( ⋅ , z t + 1 ) ) ⊥ . In finite dimensions, N ( Φ t , S ( ⋅ , z t + 1 ) ) ⊥ = R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) \mathcal{N}(\Phi_{t,S}(\cdot,z_{t+1}))^\perp=\mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1})^*) N ( Φ t , S ( ⋅ , z t + 1 ) ) ⊥ = R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) , so ∇ T t , S L ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) \nabla_{T_{t,S}}\mathcal{L}\in\mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1})^*) ∇ T t , S L ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) .
To apply Theorem 4.7 , we must bound the dimension of the union of these moving ranges over all reachable upstream states. The next lemma provides this bound by recursively parameterizing the reachable observed-output sets.
Lemma C.8 (Recursive parameterization of reachable states and moving adjoint ranges). Assume Assumption 4 holds. Then the following statements hold:
For each t ∈ [ L + 1 ] t\in[L+1] t ∈ [ L + 1 ] , the reachable state set Z t r e a c h \mathcal{Z}^{\mathrm{reach}}_t Z t reach is C 1 C^1 C 1 -parameterizable with dimension at most β t \beta_t β t .
For each t ∈ [ L ] t\in[L] t ∈ [ L ] and S ∈ S t S\in\mathcal{S}_t S ∈ S t , define
G t , S : = ⋃ z t + 1 ∈ Z t + 1 r e a c h { B m t , S , n t , S ( G ) : G ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) } ⊆ S m t , S + n t , S . \mathcal{G}_{t,S}:=\bigcup_{z_{t+1}\in\mathcal{Z}^{\mathrm{reach}}_{t+1}}\left\{\mathcal{B}_{m_{t,S},n_{t,S}}(G):G\in\mathcal{R}\left(\Phi_{t,S}(\cdot,z_{t+1})^*\right)\right\}\subseteq\mathbb{S}^{m_t,S+n_{t,S}}. G t , S := z t + 1 ∈ Z t + 1 reach ⋃ { B m t , S , n t , S ( G ) : G ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) } ⊆ S m t , S + n t , S . Then G t , S \mathcal{G}_{t,S} G t , S is C 1 C^1 C 1 -parameterizable with dimension at most β t + 1 + d t , S max \beta_{t+1}+d^{\max}_{t,S} β t + 1 + d t , S m a x .
Proof of Lemma C.8. For each t ∈ [ L ] t \in[L] t ∈ [ L ] , let O t O_t O t be an open neighborhood of the reachable argument set
A t a r g : = { ( { u S } S ∈ S t , z t + 1 ) : z t + 1 ∈ Z t + 1 r e a c h , u S ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ) } \mathcal{A}^{\mathrm{arg}}_t := \left\{(\{u_S\}_{S\in\mathcal{S}_t},z_{t+1}): z_{t+1}\in\mathcal{Z}^{\mathrm{reach}}_{t+1},\ u_S\in\mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1}))\right\} A t arg := { ({ u S } S ∈ S t , z t + 1 ) : z t + 1 ∈ Z t + 1 reach , u S ∈ R ( Φ t , S ( ⋅ , z t + 1 )) }
on which g t g_t g t is C 1 C^1 C 1 . For each pair ( t , S ) (t,S) ( t , S ) , let O t , S Φ O^\Phi_{t,S} O t , S Φ be an open neighborhood of Z t + 1 r e a c h \mathcal{Z}^{\mathrm{reach}}_{t+1} Z t + 1 reach on which the coordinate maps
z t + 1 ↦ Φ t , S ( B , z t + 1 ) and z t + 1 ↦ Φ t , S ( ⋅ , z t + 1 ) ∗ ( e ) z_{t+1}\mapsto\Phi_{t,S}(B,z_{t+1})\quad\text{and}\quad z_{t+1}\mapsto\Phi_{t,S}(\cdot,z_{t+1})^*(e) z t + 1 ↦ Φ t , S ( B , z t + 1 ) and z t + 1 ↦ Φ t , S ( ⋅ , z t + 1 ) ∗ ( e )
are C 1 C^1 C 1 for every basis matrix B B B of R m t , S × n t , S \mathbb{R}^{m_{t,S}\times n_{t,S}} R m t , S × n t , S and every basis vector e e e of E t , S \mathcal{E}_{t,S} E t , S . These open sets exist by Assumption 4.
We first prove the reachable-state claim by backward induction on t t t . For t = L + 1 t=L+1 t = L + 1 , the set Z L + 1 r e a c h \mathcal{Z}^{\mathrm{reach}}_{L+1} Z L + 1 reach is a singleton, hence it is C 1 C^1 C 1 -parameterizable with dimension 0 = β L + 1 0=\beta_{L+1} 0 = β L + 1 .
Assume that Z t + 1 r e a c h \mathcal{Z}^{\mathrm{reach}}_{t+1} Z t + 1 reach is C 1 C^1 C 1 -parameterizable with dimension at most β t + 1 \beta_{t+1} β t + 1 . Choose finitely many C 1 C^1 C 1 maps z t + 1 ( j ) : Θ j → Z t + 1 r e a c h z^{(j)}_{t+1}:\Theta_j\to\mathcal{Z}^{\mathrm{reach}}_{t+1} z t + 1 ( j ) : Θ j → Z t + 1 reach , j = 1 , … , J j=1,\ldots,J j = 1 , … , J , whose images cover Z t + 1 r e a c h \mathcal{Z}^{\mathrm{reach}}_{t+1} Z t + 1 reach , where each Θ j \Theta_j Θ j is open and has dimension at most β t + 1 \beta_{t+1} β t + 1 . For each block S ∈ S t S\in\mathcal{S}_t S ∈ S t , choose a basis B S , 1 , … , B S , p S B_{S,1},\ldots,B_{S,p_S} B S , 1 , … , B S , p S of R m t , S × n t , S \mathbb{R}^{m_{t,S}\times n_{t,S}} R m t , S × n t , S , and set k S : = d t , S max k_S:=d^{\max}_{t,S} k S := d t , S m a x . For every j j j and every tuple of index sets I = ( I S ) S ∈ S t I=(I_S)_{S\in\mathcal{S}_t} I = ( I S ) S ∈ S t , I S = { i S , 1 , … , i S , q S } ⊆ [ p S ] I_S=\{i_{S,1},\ldots,i_{S,q_S}\}\subseteq[p_S] I S = { i S , 1 , … , i S , q S } ⊆ [ p S ] , q S ≤ k S q_S\le k_S q S ≤ k S , define the preliminary argument map
Q j , I ( θ , { α S } S ∈ S t ) : = ( { ∑ ℓ = 1 q S α S , ℓ Φ t , S ( B S , i S , ℓ , z t + 1 ( j ) ( θ ) ) } S ∈ S t , z t + 1 ( j ) ( θ ) ) . Q_{j,I}(\theta,\{\alpha_S\}_{S\in\mathcal{S}_t})
:=\left(\left\{\sum_{\ell=1}^{q_S}\alpha_{S,\ell}\Phi_{t,S}(B_{S,i_{S,\ell}},z^{(j)}_{t+1}(\theta))\right\}_{S\in\mathcal{S}_t},z^{(j)}_{t+1}(\theta)\right). Q j , I ( θ , { α S } S ∈ S t ) := { ℓ = 1 ∑ q S α S , ℓ Φ t , S ( B S , i S , ℓ , z t + 1 ( j ) ( θ )) } S ∈ S t , z t + 1 ( j ) ( θ ) .
Here α S ∈ R q S \alpha_S\in\mathbb{R}^{q_S} α S ∈ R q S . Restrict the domain to the open set Ω j , I : = Q j , I − 1 ( O t ) \Omega_{j,I}:=Q_{j,I}^{-1}(O_t) Ω j , I := Q j , I − 1 ( O t ) . On Ω j , I \Omega_{j,I} Ω j , I , define
F j , I ( θ , { α S } S ∈ S t ) : = g t ( { ∑ ℓ = 1 q S α S , ℓ Φ t , S ( B S , i S , ℓ , z t + 1 ( j ) ( θ ) ) } S ∈ S t , z t + 1 ( j ) ( θ ) ) . F_{j,I}(\theta,\{\alpha_S\}_{S\in\mathcal{S}_t})
:=g_t\left(\left\{\sum_{\ell=1}^{q_S}\alpha_{S,\ell}\Phi_{t,S}(B_{S,i_{S,\ell}},z^{(j)}_{t+1}(\theta))\right\}_{S\in\mathcal{S}_t},z^{(j)}_{t+1}(\theta)\right). F j , I ( θ , { α S } S ∈ S t ) := g t { ℓ = 1 ∑ q S α S , ℓ Φ t , S ( B S , i S , ℓ , z t + 1 ( j ) ( θ )) } S ∈ S t , z t + 1 ( j ) ( θ ) .
The restriction to Ω j , I \Omega_{j,I} Ω j , I is the important point: it ensures that the argument of g t g_t g t lies in the neighborhood on which g t g_t g t is assumed to be C 1 C^1 C 1 . Since each z t + 1 ( j ) ( θ ) z^{(j)}_{t+1}(\theta) z t + 1 ( j ) ( θ ) lies in Z t + 1 r e a c h \mathcal{Z}^{\mathrm{reach}}_{t+1} Z t + 1 reach , and since the coordinate maps of Φ t , S \Phi_{t,S} Φ t , S are C 1 C^1 C 1 on O t , S Φ O^\Phi_{t,S} O t , S Φ , the map F j , I F_{j,I} F j , I is C 1 C^1 C 1 on Ω j , I \Omega_{j,I} Ω j , I .
The dimension of the domain of F j , I F_{j,I} F j , I is at most
β t + 1 + ∑ S ∈ S t q S ≤ β t + 1 + ∑ S ∈ S t d t , S max = β t . \beta_{t+1}+\sum_{S\in\mathcal{S}_t}q_S\le\beta_{t+1}+\sum_{S\in\mathcal{S}_t}d^{\max}_{t,S}=\beta_t. β t + 1 + S ∈ S t ∑ q S ≤ β t + 1 + S ∈ S t ∑ d t , S m a x = β t .
It remains to check that these finitely many maps cover Z t r e a c h \mathcal{Z}^{\mathrm{reach}}_t Z t reach . Take any z t ∈ Z t r e a c h z_t\in\mathcal{Z}^{\mathrm{reach}}_t z t ∈ Z t reach . By definition, there exist z t + 1 ∈ Z t + 1 r e a c h z_{t+1}\in\mathcal{Z}^{\mathrm{reach}}_{t+1} z t + 1 ∈ Z t + 1 reach , u S ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ) u_S\in\mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1})) u S ∈ R ( Φ t , S ( ⋅ , z t + 1 )) such that
z t = g t ( { u S } S ∈ S t , z t + 1 ) . z_t=g_t(\{u_S\}_{S\in\mathcal{S}_t},z_{t+1}). z t = g t ({ u S } S ∈ S t , z t + 1 ) .
Choose j j j and θ ∈ Θ j \theta\in\Theta_j θ ∈ Θ j with z t + 1 = z t + 1 ( j ) ( θ ) z_{t+1}=z^{(j)}_{t+1}(\theta) z t + 1 = z t + 1 ( j ) ( θ ) . For each S S S , the range R ( Φ t , S ( ⋅ , z t + 1 ) ) \mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1})) R ( Φ t , S ( ⋅ , z t + 1 )) has dimension at most k S k_S k S and is spanned by the finite list
{ Φ t , S ( B S , a , z t + 1 ) } a = 1 p S . \{\Phi_{t,S}(B_{S,a},z_{t+1})\}_{a=1}^{p_S}. { Φ t , S ( B S , a , z t + 1 ) } a = 1 p S .
Therefore, u S u_S u S can be expressed as a linear combination of at most k S k_S k S vectors from this list. This gives some tuple I I I and coefficients { α S } \{\alpha_S\} { α S } such that z t = F j , I ( θ , { α S } S ∈ S t ) z_t=F_{j,I}(\theta,\{\alpha_S\}_{S\in\mathcal{S}_t}) z t = F j , I ( θ , { α S } S ∈ S t ) . Moreover, the corresponding argument belongs to A t a r g ⊆ O t \mathcal{A}^{\mathrm{arg}}_t\subseteq O_t A t arg ⊆ O t , so the parameter point lies in Ω j , I \Omega_{j,I} Ω j , I . Thus Z t r e a c h \mathcal{Z}^{\mathrm{reach}}_t Z t reach is covered by finitely many C 1 C^1 C 1 images of dimension at most β t \beta_t β t . This completes the induction and proves the first claim.
We now prove the state-dependent adjoint ranges claim. Fix t ∈ [ L ] t\in[L] t ∈ [ L ] and S ∈ S t S\in\mathcal{S}_t S ∈ S t . By the first claim, Z t + 1 r e a c h \mathcal{Z}^{\mathrm{reach}}_{t+1} Z t + 1 reach is covered by finitely many C 1 C^1 C 1 parameterizations z t + 1 ( j ) : Θ j → Z t + 1 r e a c h z^{(j)}_{t+1}:\Theta_j\to\mathcal{Z}^{\mathrm{reach}}_{t+1} z t + 1 ( j ) : Θ j → Z t + 1 reach , with domain dimension at most β t + 1 \beta_{t+1} β t + 1 . Let e 1 , … , e q t , S e_1,\ldots,e_{q_{t,S}} e 1 , … , e q t , S be a basis of E t , S \mathcal{E}_{t,S} E t , S , and set k : = d t , S max k:=d^{\max}_{t,S} k := d t , S m a x . For every j j j and every index set I = { i 1 , … , i q } ⊆ [ q t , S ] I=\{i_1,\ldots,i_q\}\subseteq[q_{t,S}] I = { i 1 , … , i q } ⊆ [ q t , S ] , q ≤ k q\le k q ≤ k , define
M j , I ( θ , α ) : = ∑ ℓ = 1 q α ℓ B m t , S , n t , S ( Φ t , S ( ⋅ , z t + 1 ( j ) ( θ ) ) ∗ ( e i ℓ ) ) , M_{j,I}(\theta,\alpha):=\sum_{\ell=1}^{q}\alpha_\ell\mathcal{B}_{m_{t,S},n_{t,S}}\left(\Phi_{t,S}(\cdot,z^{(j)}_{t+1}(\theta))^*(e_{i_\ell})\right), M j , I ( θ , α ) := ℓ = 1 ∑ q α ℓ B m t , S , n t , S ( Φ t , S ( ⋅ , z t + 1 ( j ) ( θ ) ) ∗ ( e i ℓ ) ) ,
where α ∈ R q \alpha\in\mathbb{R}^{q} α ∈ R q . As before, one may restrict the domain to the open preimage
Ω j , I Φ : = { ( θ , α ) : z t + 1 ( j ) ( θ ) ∈ O t , S Φ } . \Omega_{j,I}^{\Phi}:=\left\{(\theta,\alpha):z_{t+1}^{(j)}(\theta)\in\mathcal{O}_{t,S}^{\Phi}\right\}. Ω j , I Φ := { ( θ , α ) : z t + 1 ( j ) ( θ ) ∈ O t , S Φ } .
This restriction contains all parameter points needed to cover the reachable set, because z t + 1 ( j ) ( Θ j ) ⊆ Z t + 1 r e a c h ⊆ O t , S Φ z_{t+1}^{(j)}(\Theta_j)\subseteq\mathcal{Z}_{t+1}^{\mathrm{reach}}\subseteq\mathcal{O}_{t,S}^{\Phi} z t + 1 ( j ) ( Θ j ) ⊆ Z t + 1 reach ⊆ O t , S Φ . By Assumption 4, each map M j , I M_{j,I} M j , I is C 1 C^1 C 1 on Ω j , I Φ \Omega_{j,I}^{\Phi} Ω j , I Φ . Its domain dimension is at most β t + 1 + q ≤ β t + 1 + d t , S max \beta_{t+1}+q\leq\beta_{t+1}+d_{t,S}^{\max} β t + 1 + q ≤ β t + 1 + d t , S m a x .
It remains to verify coverage. Take any element of G t , S \mathcal{G}_{t,S} G t , S . It has the form B m t , S , n t , S ( G ) \mathcal{B}_{m_{t,S},n_{t,S}}(G) B m t , S , n t , S ( G ) , G ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) G\in\mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1})^*) G ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) for some z t + 1 ∈ Z t + 1 r e a c h z_{t+1}\in\mathcal{Z}_{t+1}^{\mathrm{reach}} z t + 1 ∈ Z t + 1 reach . Choose j j j and θ \theta θ such that z t + 1 = z t + 1 ( j ) ( θ ) z_{t+1}=z_{t+1}^{(j)}(\theta) z t + 1 = z t + 1 ( j ) ( θ ) . The space R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) \mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1})^*) R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) has dimension at most d t , S max = k d_{t,S}^{\max}=k d t , S m a x = k and is spanned by
{ Φ t , S ( ⋅ , z t + 1 ) ∗ ( e a ) } a = 1 q t , S . \left\{\Phi_{t,S}(\cdot,z_{t+1})^*(e_a)\right\}_{a=1}^{q_{t,S}}. { Φ t , S ( ⋅ , z t + 1 ) ∗ ( e a ) } a = 1 q t , S .
Hence G G G can be written as a linear combination of at most k k k vectors from this list. Therefore B m t , S , n t , S ( G ) \mathcal{B}_{m_{t,S},n_{t,S}}(G) B m t , S , n t , S ( G ) belongs to one of the images of M j , I M_{j,I} M j , I . Thus G t , S \mathcal{G}_{t,S} G t , S is C 1 C^1 C 1 -parameterizable with dimension at most β t + 1 + d t , S max \beta_{t+1}+d_{t,S}^{\max} β t + 1 + d t , S m a x .
We now combine the KKT identity, the gradient-range lemma, and the parameterization lemma. The result is the stagewise version of the master genericity theorem:
Theorem C.9 (Generic rank-deficiency for L L L -stage perturbed LoRA). Assume the objective has L L L -stage blockwise linear map dependence, is differentiable, and Assumption 4 holds. Fix factor ranks r t , S r_{t,S} r t , S and regularization parameters λ t , S > 0 \lambda_{t,S}>0 λ t , S > 0 . If
r t , S ( r t , S + 1 ) 2 > β t + 1 + d t , S max ∀ t ∈ [ L ] , S ∈ S t , \frac{r_{t,S}(r_{t,S}+1)}{2}>\beta_{t+1}+d_{t,S}^{\max}\quad\forall t\in[L],\ S\in\mathcal{S}_t, 2 r t , S ( r t , S + 1 ) > β t + 1 + d t , S m a x ∀ t ∈ [ L ] , S ∈ S t , then for Lebesgue-a.e. choice of PSD perturbations { P t , S } \{P_{t,S}\} { P t , S } , every first-order stationary point of the perturbed LoRA objective satisfies
rank ( R t , S ) < r t , S ∀ t ∈ [ L ] , S ∈ S t . \operatorname{rank}(R_{t,S})<r_{t,S}\quad\forall t\in[L],\ S\in\mathcal{S}_t. rank ( R t , S ) < r t , S ∀ t ∈ [ L ] , S ∈ S t .
Proof of Theorem C.9. Fix a stage/block ( t , S ) (t,S) ( t , S ) . At any first-order stationary point, Lemma C.5 gives
( B m t , S , n t , S ( G t , S ) + λ t , S I + P t , S ) R t , S = 0. \left(\mathcal{B}_{m_{t,S},n_{t,S}}(G_{t,S})+\lambda_{t,S}I+P_{t,S}\right)R_{t,S}=0. ( B m t , S , n t , S ( G t , S ) + λ t , S I + P t , S ) R t , S = 0.
where G t , S = ∇ T t , S L G_{t,S}=\nabla_{T_{t,S}}\mathcal{L} G t , S = ∇ T t , S L . By Lemma C.7 , G t , S ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) G_{t,S}\in\mathcal{R}(\Phi_{t,S}(\cdot,z_{t+1})^*) G t , S ∈ R ( Φ t , S ( ⋅ , z t + 1 ) ∗ ) for the reachable state z t + 1 z_{t+1} z t + 1 induced by the stationary point. Hence B m t , S , n t , S ( G t , S ) ∈ G t , S \mathcal{B}_{m_{t,S},n_{t,S}}(G_{t,S})\in\mathcal{G}_{t,S} B m t , S , n t , S ( G t , S ) ∈ G t , S . By Lemma C.8 , the family G t , S \mathcal{G}_{t,S} G t , S is C 1 C^1 C 1 -parameterizable with dimension at most β t + 1 + d t , S max \beta_{t+1}+d_{t,S}^{\max} β t + 1 + d t , S m a x . For the block ( t , S ) (t,S) ( t , S ) , define d t , S b l k : = m t , S + n t , S d_{t,S}^{\mathrm{blk}}:=m_{t,S}+n_{t,S} d t , S blk := m t , S + n t , S . By Lemma C.8 , the family G t , S \mathcal{G}_{t,S} G t , S is contained in a finite union of C 1 C^1 C 1 images of dimension at most β t + 1 + d t , S max \beta_{t+1}+d_{t,S}^{\max} β t + 1 + d t , S m a x . Since
r t , S ( r t , S + 1 ) 2 > β t + 1 + d t , S max , \frac{r_{t,S}(r_{t,S}+1)}{2}>\beta_{t+1}+d_{t,S}^{\max}, 2 r t , S ( r t , S + 1 ) > β t + 1 + d t , S m a x ,
Theorem 4.7 , applied with d = d t , S b l k d=d_{t,S}^{\mathrm{blk}} d = d t , S blk , r = r t , S r=r_{t,S} r = r t , S , λ = λ t , S \lambda=\lambda_{t,S} λ = λ t , S , and G = G t , S \mathcal{G}=\mathcal{G}_{t,S} G = G t , S , gives a full-measure set Ω t , S ⊆ S + m t , S + n t , S \Omega_{t,S}\subseteq\mathbb{S}_{+}^{m_{t,S}+n_{t,S}} Ω t , S ⊆ S + m t , S + n t , S such that, whenever P t , S ∈ Ω t , S P_{t,S}\in\Omega_{t,S} P t , S ∈ Ω t , S , no matrix R t , S ∈ R ( m t , S + n t , S ) × r t , S R_{t,S}\in\mathbb{R}^{(m_{t,S}+n_{t,S})\times r_{t,S}} R t , S ∈ R ( m t , S + n t , S ) × r t , S with rank ( R t , S ) = r t , S \operatorname{rank}(R_{t,S})=r_{t,S} rank ( R t , S ) = r t , S can satisfy the block KKT equation
( B m t , S , n t , S ( G ) + λ t , S I + P t , S ) R t , S = 0 \left(\mathcal{B}_{m_{t,S},n_{t,S}}(G)+\lambda_{t,S}I+P_{t,S}\right)R_{t,S}=0 ( B m t , S , n t , S ( G ) + λ t , S I + P t , S ) R t , S = 0
for any B m t , S , n t , S ( G ) ∈ G t , S \mathcal{B}_{m_{t,S},n_{t,S}}(G)\in\mathcal{G}_{t,S} B m t , S , n t , S ( G ) ∈ G t , S .
Taking the product of the full-measure sets Ω t , S \Omega_{t,S} Ω t , S over the finite block family preserves full measure. For any perturbation tuple in this product set, every first-order stationary point satisfies the displayed KKT equation for each block by Lemma C.5 , with its gradient block belonging to G t , S \mathcal{G}_{t,S} G t , S by Lemma C.7 . Therefore every first-order stationary point satisfies rank ( R t , S ) < r t , S \operatorname{rank}(R_{t,S})<r_{t,S} rank ( R t , S ) < r t , S for every t ∈ [ L ] t\in[L] t ∈ [ L ] and S ∈ S t S\in\mathcal{S}_t S ∈ S t .
The attention corollary is obtained by evaluating the bottleneck dimension for the concrete map T ↦ T H t , S ( z t + 1 ) T\mapsto T H_{t,S}(z_{t+1}) T ↦ T H t , S ( z t + 1 ) .
Proof of Corollary 4.8. For an adapted attention projection block α = ( a , S ) \alpha=(a,S) α = ( a , S ) , the stagewise linear map has the form Φ t , α ( T , z t + 1 ) = T H t , S ( z t + 1 ) \Phi_{t,\alpha}(T,z_{t+1})=TH_{t,S}(z_{t+1}) Φ t , α ( T , z t + 1 ) = T H t , S ( z t + 1 ) . Therefore
d t , α ( z t + 1 ) = dim R ( Φ t , α ( ⋅ , z t + 1 ) ) = m t , α rank ( H t , S ( z t + 1 ) ) . d_{t,\alpha}(z_{t+1})=\dim\mathcal{R}(\Phi_{t,\alpha}(\cdot,z_{t+1}))=m_{t,\alpha}\operatorname{rank}(H_{t,S}(z_{t+1})). d t , α ( z t + 1 ) = dim R ( Φ t , α ( ⋅ , z t + 1 )) = m t , α rank ( H t , S ( z t + 1 )) .
Taking the supremum over z t + 1 ∈ Z t + 1 r e a c h z_{t+1}\in Z_{t+1}^{\mathrm{reach}} z t + 1 ∈ Z t + 1 reach gives d ˉ t , α = m t , α r H , t , α \bar d_{t,\alpha}=m_{t,\alpha}r_{H,t,\alpha} d ˉ t , α = m t , α r H , t , α . Substituting this identity into Theorem C.9 gives the stated result.
Now we prove the limitation result from § 4.3 .
Proof of Proposition 4.9. Choose any constant a > λ + ε a>\lambda+\varepsilon a > λ + ε and set
g ( t ) : = a ( t 2 − 1 ) 2 . g(t):=a(t^2-1)^2. g ( t ) := a ( t 2 − 1 ) 2 .
Then g g g is smooth and nonconvex, g ( 0 ) = a g(0)=a g ( 0 ) = a , g ( 1 ) = 0 g(1)=0 g ( 1 ) = 0 , and g ′ ( 0 ) = 0 g'(0)=0 g ′ ( 0 ) = 0 . Write R = [ U V ] R=\begin{bmatrix}U\\V\end{bmatrix} R = [ U V ] with U = ( u 1 , … , u r ) U=(u_1,\ldots,u_r) U = ( u 1 , … , u r ) and V = ( v 1 , … , v r ) V=(v_1,\ldots,v_r) V = ( v 1 , … , v r ) . Let
t ( R ) : = U V ⊤ = ∑ k = 1 r u k v k . t(R):=UV^\top=\sum_{k=1}^{r}u_kv_k. t ( R ) := U V ⊤ = k = 1 ∑ r u k v k .
Under the column-wise vectorization r ′ = vec ( R ) ∈ R 2 r r'=\operatorname{vec}(R)\in\mathbb{R}^{2r} r ′ = vec ( R ) ∈ R 2 r , the chain rule gives
∇ r ′ 2 ( g ∘ t ) ( 0 ) = g ′ ′ ( t ( 0 ) ) ∇ t ( 0 ) ∇ t ( 0 ) ⊤ + g ′ ( t ( 0 ) ) ∇ 2 t ( 0 ) = 0 , \nabla_{r'}^2(g\circ t)(0)=g''(t(0))\nabla t(0)\nabla t(0)^\top+g'(t(0))\nabla^2t(0)=0, ∇ r ′ 2 ( g ∘ t ) ( 0 ) = g ′′ ( t ( 0 )) ∇ t ( 0 ) ∇ t ( 0 ) ⊤ + g ′ ( t ( 0 )) ∇ 2 t ( 0 ) = 0 ,
because t ( 0 ) = 0 t(0)=0 t ( 0 ) = 0 , ∇ t ( 0 ) = 0 \nabla t(0)=0 ∇ t ( 0 ) = 0 , and g ′ ( 0 ) = 0 g'(0)=0 g ′ ( 0 ) = 0 . Moreover,
∇ r ′ 2 ( λ 2 ∥ R ∥ F 2 ) = λ I 2 r , ∇ r ′ 2 ( 1 2 ⟨ P , R R ⊤ ⟩ ) = I r ⊗ P . \nabla_{r'}^2\left(\frac{\lambda}{2}\lVert R\rVert_F^2\right)=\lambda I_{2r},\qquad
\nabla_{r'}^2\left(\frac{1}{2}\langle P,RR^\top\rangle\right)=I_r\otimes P. ∇ r ′ 2 ( 2 λ ∥ R ∥ F 2 ) = λ I 2 r , ∇ r ′ 2 ( 2 1 ⟨ P , R R ⊤ ⟩ ) = I r ⊗ P .
The perturbed objective is
F P ( R ) : = g ( t ( R ) ) + λ 2 ∥ R ∥ F 2 + 1 2 ⟨ P , R R ⊤ ⟩ . F_P(R):=g(t(R))+\frac{\lambda}{2}\lVert R\rVert_F^2+\frac{1}{2}\langle P,RR^\top\rangle. F P ( R ) := g ( t ( R )) + 2 λ ∥ R ∥ F 2 + 2 1 ⟨ P , R R ⊤ ⟩ .
Therefore ∇ r ′ 2 F P ( 0 ) = λ I 2 r + I r ⊗ P ⪰ λ I 2 r ≻ 0 \nabla_{r'}^2F_P(0)=\lambda I_{2r}+I_r\otimes P\succeq\lambda I_{2r}\succ0 ∇ r ′ 2 F P ( 0 ) = λ I 2 r + I r ⊗ P ⪰ λ I 2 r ≻ 0 , so R = 0 R=0 R = 0 is a strict local minimum. Now set
R 1 : = [ 1 0 ⋯ 0 1 0 ⋯ 0 ] . R_1:=\begin{bmatrix}1&0&\cdots&0\\1&0&\cdots&0\end{bmatrix}. R 1 := [ 1 1 0 0 ⋯ ⋯ 0 0 ] .
Then
t ( R 1 ) = 1 , ∥ R 1 ∥ F 2 = 2 , R 1 R 1 ⊤ = [ 1 1 1 1 ] , ∥ R 1 R 1 ⊤ ∥ F = 2. t(R_1)=1,\ \lVert R_1\rVert_F^2=2,\ R_1R_1^\top=\begin{bmatrix}1&1\\1&1\end{bmatrix},\ \lVert R_1R_1^\top\rVert_F=2. t ( R 1 ) = 1 , ∥ R 1 ∥ F 2 = 2 , R 1 R 1 ⊤ = [ 1 1 1 1 ] , ∥ R 1 R 1 ⊤ ∥ F = 2.
Hence
F P ( R 1 ) = g ( 1 ) + λ 2 ∥ R 1 ∥ F 2 + 1 2 ⟨ P , R 1 R 1 ⊤ ⟩ = λ + 1 2 ⟨ P , R 1 R 1 ⊤ ⟩ ≤ λ + 1 2 ∥ P ∥ F ∥ R 1 R 1 ⊤ ∥ F = λ + ∥ P ∥ F ≤ λ + ε < a = F P ( 0 ) . \begin{aligned}
F_P(R_1)&=g(1)+\frac{\lambda}{2}\lVert R_1\rVert_F^2+\frac{1}{2}\langle P,R_1R_1^\top\rangle\\
&=\lambda+\frac{1}{2}\langle P,R_1R_1^\top\rangle\\
&\leq\lambda+\frac{1}{2}\lVert P\rVert_F\lVert R_1R_1^\top\rVert_F\\
&=\lambda+\lVert P\rVert_F\\
&\leq\lambda+\varepsilon<a=F_P(0).
\end{aligned} F P ( R 1 ) = g ( 1 ) + 2 λ ∥ R 1 ∥ F 2 + 2 1 ⟨ P , R 1 R 1 ⊤ ⟩ = λ + 2 1 ⟨ P , R 1 R 1 ⊤ ⟩ ≤ λ + 2 1 ∥ P ∥ F ∥ R 1 R 1 ⊤ ∥ F = λ + ∥ P ∥ F ≤ λ + ε < a = F P ( 0 ) .
Thus, R = 0 R=0 R = 0 is not global.
Proofs for Section 4.4
Lemma C.10 ([17 Lemma 5.1] ). For any T ∈ R m × n T\in\mathbb{R}^{m\times n} T ∈ R m × n and any integer r ≥ rank ( T ) r\geq\operatorname{rank}(T) r ≥ rank ( T ) ,
∥ T ∥ ∗ = 1 2 min U ∈ R m × r , V ∈ R n × r U V ⊤ = T { ∥ U ∥ F 2 + ∥ V ∥ F 2 } . \lVert T\rVert_*=\frac{1}{2}\min_{\substack{U\in\mathbb{R}^{m\times r},V\in\mathbb{R}^{n\times r}\\UV^\top=T}}\left\{\lVert U\rVert_F^2+\lVert V\rVert_F^2\right\}. ∥ T ∥ ∗ = 2 1 U ∈ R m × r , V ∈ R n × r U V ⊤ = T min { ∥ U ∥ F 2 + ∥ V ∥ F 2 } . Equivalently,
∥ T ∥ ∗ = 1 2 min U ∈ R m × r , V ∈ R n × r r ∈ N , U V ⊤ = T { ∥ U ∥ F 2 + ∥ V ∥ F 2 } . \lVert T\rVert_*=\frac{1}{2}\min_{\substack{U\in\mathbb{R}^{m\times r},V\in\mathbb{R}^{n\times r}\\r\in\mathbb{N},UV^\top=T}}\left\{\lVert U\rVert_F^2+\lVert V\rVert_F^2\right\}. ∥ T ∥ ∗ = 2 1 U ∈ R m × r , V ∈ R n × r r ∈ N , U V ⊤ = T min { ∥ U ∥ F 2 + ∥ V ∥ F 2 } .
We finally prove that small PSD perturbations do not substantially change the trace-norm objective value attained by a global factorized solution.
Proof of Theorem 4.10 . The perturbed LoRA objective is lower semicontinuous because L \mathcal{L} L is lower semicontinuous and the maps ( U j , V j ) ↦ U j V j ⊤ (U_j,V_j) \mapsto U_jV_j^{\top} ( U j , V j ) ↦ U j V j ⊤ are continuous. It is coercive because
∑ j ∈ I λ j 2 ( ∥ U j ∥ F 2 + ∥ V j ∥ F 2 ) \sum_{j\in\mathcal{I}} \frac{\lambda_j}{2}\left(\lVert U_j\rVert_F^2+\lVert V_j\rVert_F^2\right) j ∈ I ∑ 2 λ j ( ∥ U j ∥ F 2 + ∥ V j ∥ F 2 )
is coercive, and each perturbation term is nonnegative. Since L \mathcal{L} L is bounded below, the perturbed LoRA objective attains a global minimum.
For each S S S , write
R ^ j ≔ [ U ^ j V ^ j ] . \widehat{R}_j \coloneqq\begin{bmatrix}\widehat{U}_j\\ \widehat{V}_j\end{bmatrix}. R j : = [ U j V j ] .
By Lemma C.10, ∥ T ^ j ∥ ⋆ ≤ 1 2 ( ∥ U ^ j ∥ F 2 + ∥ V ^ j ∥ F 2 ) \lVert\widehat{T}_j\rVert_{\star} \le\frac{1}{2}\left(\lVert\widehat{U}_j\rVert_F^2+\lVert\widehat{V}_j\rVert_F^2\right) ∥ T j ∥ ⋆ ≤ 2 1 ( ∥ U j ∥ F 2 + ∥ V j ∥ F 2 ) . Since P j ⪰ 0 P_j \succeq0 P j ⪰ 0 , ⟨ P j , R ^ j R ^ j ⊤ ⟩ ≥ 0 \langle P_j,\widehat{R}_j\widehat{R}_j^{\top}\rangle\ge0 ⟨ P j , R j R j ⊤ ⟩ ≥ 0 . Therefore
L ( { T ^ j } ) + ∑ j ∈ I λ j ∥ T ^ j ∥ ⋆ ≤ L { λ j } , { P j } l o r a ( { U ^ j , V ^ j } ) . \mathcal{L}(\{\widehat{T}_j\})+\sum_{j\in\mathcal{I}}\lambda_j\lVert\widehat{T}_j\rVert_{\star}\le\mathcal{L}^{\mathrm{lora}}_{\{\lambda_j\},\{P_j\}}(\{\widehat{U}_j,\widehat{V}_j\}). L ({ T j }) + j ∈ I ∑ λ j ∥ T j ∥ ⋆ ≤ L { λ j } , { P j } lora ({ U j , V j }) .
For each S S S , since rank ( T j ⋆ ) ≤ r j \operatorname{rank}(T_j^\star)\le r_j rank ( T j ⋆ ) ≤ r j , choose a balanced SVD factorization padded with zero columns if necessary, so that
T j ⋆ = U j ⋆ V j ⋆ ⊤ and ∥ T j ⋆ ∥ ⋆ = 1 2 ( ∥ U j ⋆ ∥ F 2 + ∥ V j ⋆ ∥ F 2 ) . T_j^\star=U_j^\star V_j^{\star\top}\quad\text{and}\quad\lVert T_j^\star\rVert_{\star}=\frac{1}{2}\left(\lVert U_j^\star\rVert_F^2+\lVert V_j^\star\rVert_F^2\right). T j ⋆ = U j ⋆ V j ⋆ ⊤ and ∥ T j ⋆ ∥ ⋆ = 2 1 ( ∥ U j ⋆ ∥ F 2 + ∥ V j ⋆ ∥ F 2 ) .
By global optimality of { ( U ^ j , V ^ j ) } \{(\widehat{U}_j,\widehat{V}_j)\} {( U j , V j )} for the perturbed factorized objective,
L { λ j } , { P j } l o r a ( { U ^ j , V ^ j } ) ≤ L { λ j } , { P j } l o r a ( { U j ⋆ , V j ⋆ } ) . \mathcal{L}^{\mathrm{lora}}_{\{\lambda_j\},\{P_j\}}(\{\widehat{U}_j,\widehat{V}_j\})\le\mathcal{L}^{\mathrm{lora}}_{\{\lambda_j\},\{P_j\}}(\{U_j^\star,V_j^\star\}). L { λ j } , { P j } lora ({ U j , V j }) ≤ L { λ j } , { P j } lora ({ U j ⋆ , V j ⋆ }) .
The right-hand side equals
L ( { T j ⋆ } ) + ∑ j ∈ I λ j ∥ T j ⋆ ∥ ⋆ + 1 2 ∑ j ∈ I ⟨ P j , R j ⋆ R j ⋆ ⊤ ⟩ . \mathcal{L}(\{T_j^\star\})+\sum_{j\in\mathcal{I}}\lambda_j\lVert T_j^\star\rVert_{\star}+\frac{1}{2}\sum_{j\in\mathcal{I}}\langle P_j,R_j^\star R_j^{\star\top}\rangle. L ({ T j ⋆ }) + j ∈ I ∑ λ j ∥ T j ⋆ ∥ ⋆ + 2 1 j ∈ I ∑ ⟨ P j , R j ⋆ R j ⋆ ⊤ ⟩ .
Since { T j ⋆ } \{T_j^\star\} { T j ⋆ } is a global minimizer of the nuclear-norm-regularized problem, combining the preceding displays proves the first inequality.
For the final bound, use P j ⪰ 0 P_j\succeq0 P j ⪰ 0 , ∥ P j ∥ 2 ≤ ε j \lVert P_j\rVert_2\le\varepsilon_j ∥ P j ∥ 2 ≤ ε j , and
tr ( R j ⋆ R j ⋆ ⊤ ) = ∥ U j ⋆ ∥ F 2 + ∥ V j ⋆ ∥ F 2 = 2 ∥ T j ⋆ ∥ ⋆ . \operatorname{tr}(R_j^\star R_j^{\star\top})=\lVert U_j^\star\rVert_F^2+\lVert V_j^\star\rVert_F^2=2\lVert T_j^\star\rVert_{\star}. tr ( R j ⋆ R j ⋆ ⊤ ) = ∥ U j ⋆ ∥ F 2 + ∥ V j ⋆ ∥ F 2 = 2 ∥ T j ⋆ ∥ ⋆ .
Then
1 2 ⟨ P j , R j ⋆ R j ⋆ ⊤ ⟩ ≤ 1 2 ∥ P j ∥ 2 tr ( R j ⋆ R j ⋆ ⊤ ) ≤ ε j ∥ T j ⋆ ∥ ⋆ . \frac{1}{2}\langle P_j,R_j^\star R_j^{\star\top}\rangle\le\frac{1}{2}\lVert P_j\rVert_2\operatorname{tr}(R_j^\star R_j^{\star\top})\le\varepsilon_j\lVert T_j^\star\rVert_{\star}. 2 1 ⟨ P j , R j ⋆ R j ⋆ ⊤ ⟩ ≤ 2 1 ∥ P j ∥ 2 tr ( R j ⋆ R j ⋆ ⊤ ) ≤ ε j ∥ T j ⋆ ∥ ⋆ .
Summing over j ∈ I j\in\mathcal{I} j ∈ I gives the claimed perturbation bound.
Experiment details
Proxy estimation. The proxy rank is chosen large enough to exactly represent the selected head block; for Qwen2.5-1.5B, this is 128, matching the query-head dimension d k d_k d k .
The proxy is trained with the same regularized objective as the LoRA sweep, using λ = 10 − 4 \lambda=10^{-4} λ = 1 0 − 4 , AdamW learning rate 5 × 10 − 4 5\times10^{-4} 5 × 1 0 − 4 , and best-checkpoint selection by the lowest training L λ l o r a \mathcal{L}^{\mathrm{lora}}_{\lambda} L λ lora . After training, the LoRA update is merged into the base linear weights, and the model is run in evaluation mode over the training split. We then estimate the downstream activation rank using rank ( H ) = rank ( H H ⊤ ) \operatorname{rank}(H)=\operatorname{rank}(HH^{\top}) rank ( H ) = rank ( H H ⊤ ) to lower the computation by considering the rank of a smaller matrix. The numerical rank is computed by the number of eigenvalues of H H ⊤ HH^{\top} H H ⊤ that are larger than the eigenvalue threshold 10 − 4 10^{-4} 1 0 − 4 .
Let r ^ u p , S \widehat{r}_{\mathrm{up},S} r up , S denote the numerical rank of the upstream frozen-feature subspace for block S S S , and let r ^ d o w n , S \widehat{r}_{\mathrm{down},S} r down , S denote the numerical rank of the downstream observable subspace estimated after the trace-norm proxy optimization. The final predicted threshold is
r t h r e s h o l d = max S min { r ^ u p , S , r ^ d o w n , S } . r_{\mathrm{threshold}}=\max_S\min\{\widehat{r}_{\mathrm{up},S},\widehat{r}_{\mathrm{down},S}\}. r threshold = S max min { r up , S , r down , S } .
Table 4. Hyperparameters for LoRA fine-tuning and full fine-tuning on NLP tasks.
This proxy approximates the optimizer-dependent quantity appearing in Theorem 4.4 .
Before the LoRA sweep, the classification head is warmed up for 10 epochs, then frozen. For the high-rank bound case, we use 100 samples, and for the low-rank bound case, we use 4 samples for training.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [N/A].
[N/A] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide a short (1–2 sentence) justification right after your answer (even for [N/A]).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will also be asked to include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While [Yes] is generally preferable to [No], it is perfectly acceptable to answer [No] provided a proper justification is given (e.g., error bars are not reported because it would be too computationally expensive” or “we were unable to find the license for the dataset we used”). In general, answering [No] or [N/A] is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading “NeurIPS Paper Checklist”,
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: We provided technical proof and experimental evidence to support our claims.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation, as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in Section 6. Guidelines:
The answer [N/A] means that the paper has no limitation, while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper provided the full set of assumptions and complete proofs in Section 3, Section 4, and Appendix.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provided experiment descriptions in the main paper and appendix.
Guidelines:
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The experiments use GLUE tasks and an open-source Qwen2.5 checkpoint, but the current paper does not yet include released code.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: We provided details for experiments in the paper.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The current figures show single optimization trajectories without error bars.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [No]
Justification: The draft does not report GPU type, memory, runtime per run, and total compute.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: Our paper follows the NeurIPS Code of Ethics.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper identifies several phenomena in general parameter-efficient fine-tuning methods that deepen our understanding of LoRA and may inform the development of more capable small models. Advances in these methods could have both positive and negative societal impacts.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A] Justification: Our work does not pose such risks.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited papers and resources used in our experiment.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification: Our paper does not release new assets.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A]
Justification: We do not have experiments involving crowdsourcing or research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: We do not involve crowdsourcing or research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: LLMs have been employed solely for writing, editing, and formatting purposes. Their use does not affect the core methodology, scientific rigor, or originality of the research.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.