Introduction
Foundation models have transformed artificial intelligence across numerous domains. However, deploying these models in real-world applications typically requires fine-tuning on specialized datasets to meet accuracy, safety, and alignment requirements. Traditional full fine-tuning, which updates all parameters of a pre-trained model, incurs substantial computational cost, including large memory footprints, long training times, and significant storage demands. Low-Rank Adaptation (LoRA) [20 ] addresses this challenge by augmenting pre-trained weight matrices W W W with low-rank adapters B B B , A A A while keeping W W W frozen. Empirically, LoRA matches the performance of full fine-tuning while training only a small fraction of the parameters.1
40th Conference on Neural Information Processing Systems (NeurIPS 2026).
While LoRA was originally introduced for its computational efficiency, practitioners have since identified two striking empirical properties.
Robustness to catastrophic forgetting. In fine-tuning and continual learning, the degradation of performance on previously learned tasks after adapting to new ones (a.k.a. catastrophic forgetting) is a fundamental challenge [37 , 24 , 40 ] . Surprisingly, recent studies show that LoRA exhibits strong robustness to catastrophic forgetting, even after substantial adaptation [52 , 3 , 38 ] . This robustness makes LoRA particularly appealing for continual learning.
Effective adapter merging. In multi-task settings, maintaining separate fine-tuned models for each task is storage-intensive. Remarkably, LoRA adapters trained independently on different tasks can be merged (e.g., via simple weighted averaging) into a single adapter that performs well across all tasks [21 , 55 , 49 , 56 , 61 ] . This property enables flexible and efficient deployment.
Despite these empirical benefits, the theoretical mechanisms underlying LoRA’s robustness to catastrophic forgetting and its adapter merging capabilities remain poorly understood. In this work, we provide a unified theoretical explanation for both phenomena through a margin-based perspective.
Paper contributions. We study LoRA in multiclass linear classification under a near-orthogonal-task regime, in which fine-tuning samples are nearly orthogonal to one another and to the pre-training samples. In this setting, we characterize the global minimizer of the LoRA objective and use this characterization to analyze forgetting and adapter merging. Compared with prior theoretical work on LoRA (see Appendix § 8 ), our work makes the following contributions:
Characterization of the optimal solution. We characterize the global minimizer of LoRA across different regularization regimes (Theorem 3.1). In the high-penalty regime, the optimal adapter is zero. In the intermediate regime, the adapter aligns with the max-margin direction on the fine-tuning data. In the low-penalty regime, the adapter aligns with a non-max-margin classifier whose direction generally does not admit a simple closed form. Nevertheless, we show that this direction converges to an interpretable limit as the regularization parameter tends to zero.
Theory for reduced forgetting. We define forgetting as the reduction in margin on the pre-training data after fine-tuning, and derive closed-form expressions for the normalized margins on the pre-training data, the fine-tuning data, and their union (Theorem 3.2 ). Our analysis shows that these margins are governed by the regularization parameter and satisfy a precise trade-off: decreasing the regularization parameter increases the fine-tuning margin but also increases the Frobenius-norm ratio of the adapter to the pre-trained weights, leading to greater forgetting. We further identify the regularization parameter that maximizes the margin on the union of pre-training and fine-tuning tasks, thereby optimally balancing retention and adaptation.
Theory for adapter merging. Under the intermediate-regime assumption for each task, we derive closed-form expressions for the margin of the merged model on every task (Theorem 3.3 ). We show that the merged model attains strictly positive margin on each task, providing a theoretical explanation for the effectiveness of adapter merging. We also characterize, in closed form, the mixing coefficients that maximize the margin of the merged model on the union of all tasks, which naturally leads to margin-based LoRA merging (MaLoRA), a theoretically motivated adapter-merging method. This contrasts with many existing model-merging approaches, which rely primarily on heuristic prescriptions for choosing how to combine task-specific models.
Numerical validation. Experiments on real datasets and modern architectures support the theory in three ways: (i) in the linear probing setting, the empirically optimal regularization parameter agrees with the value predicted by our analysis; (ii) in the same setting, the empirically optimal adapter mixing weights agree with our closed-form coefficients; and (iii) in LoRA adapter merging, MaLoRA achieves performance comparable to, and in several cases better than, strong heuristic adapter-merging baselines across vision and language classification tasks.
Preliminaries
In this section, we first introduce the multiclass linear classification problem and its corresponding hard-margin SVM formulation. We then describe the specific learning problem considered in this work, which consists of a pre-training and fine-tuning phase. Finally, we state the key assumptions that underpin our analysis, and discuss their motivation and implications.
Background on multiclass linear classification
We first review the standard K K K -class linear classification problem. Given a dataset D \mathcal{D} D , the goal is to learn a weight matrix W ∈ R K × d W \in\mathbb{R}^{K \times d} W ∈ R K × d that minimizes the mean empirical cross-entropy loss:
min W ∈ R K × d L ( W ; D ) : = 1 ∣ D ∣ ∑ ( x , y ) ∈ D L C E ( y , W x ) , (1) \min_{W \in\mathbb{R}^{K \times d}} L(W;\mathcal{D}) := \frac{1}{|\mathcal{D}|} \sum_{(x,y)\in\mathcal{D}} \mathcal{L}_{\mathrm{CE}}(y,Wx), \tag*{(1)} W ∈ R K × d min L ( W ; D ) := ∣ D ∣ 1 ( x , y ) ∈ D ∑ L CE ( y , W x ) , (1)
where x ∈ R d x \in\mathbb{R}^{d} x ∈ R d is the input and y ∈ R K y \in\mathbb{R}^{K} y ∈ R K is a one-hot label vector. The cross-entropy loss is given by:
L C E ( y , y ^ ) : = − ∑ c = 1 K y c log ( exp ( y ^ c ) ∑ i = 1 K exp ( y ^ i ) ) , y ^ = W x . (2) \mathcal{L}_{\mathrm{CE}}(y,\hat{y}) := -\sum_{c=1}^{K} y_c \log\left(\frac{\exp(\hat{y}_c)}{\sum_{i=1}^{K}\exp(\hat{y}_i)}\right), \qquad\hat{y}=Wx. \tag*{(2)} L CE ( y , y ^ ) := − c = 1 ∑ K y c log ( ∑ i = 1 K exp ( y ^ i ) exp ( y ^ c ) ) , y ^ = W x . (2)
Recent studies have shown that when the data are linearly separable, gradient descent (GD) implicitly biases the solution of (1) toward the max-margin classifier [46 , 34 , 41 ] . Specifically, the iterates of GD converge in direction to the max-margin solution of the following hard-margin SVM problem2 :
W S V M ( D ) : = arg min W ∈ R K × d ∥ W ∥ F 2 subject to C ( W ; D ) ≥ 1 , (3) W^{\mathrm{SVM}}(\mathcal{D}) := \mathop{\arg\min}_{W\in\mathbb{R}^{K\times d}} \lVert W\rVert_{F}^{2}\quad\text{subject to}\quad C(W;\mathcal{D})\ge1, \tag*{(3)} W SVM ( D ) := arg min W ∈ R K × d ∥ W ∥ F 2 subject to C ( W ; D ) ≥ 1 , (3)
where C ( W ; D ) C(W;\mathcal{D}) C ( W ; D ) is the minimum margin over the dataset:
C ( W ; D ) : = min ( x , y ) ∈ D min k ≠ c , y c = 1 ( w c T x − w k T x ) , with W = [ w 1 ; … ; w K ] T . (4) C(W;\mathcal{D}) := \min_{(x,y)\in\mathcal{D}}\min_{\substack{k\ne c,y_c=1}}(w_c^{\mathsf{T}}x-w_k^{\mathsf{T}}x),\qquad\text{with } W=[w_1;\ldots;w_K]^{\mathsf{T}}. \tag*{(4)} C ( W ; D ) := ( x , y ) ∈ D min k = c , y c = 1 min ( w c T x − w k T x ) , with W = [ w 1 ; … ; w K ] T . (4)
The hard-margin SVM problem in (3) seeks a weight matrix of minimum norm that separates all examples with at least unit margin between the correct class and the nearest competing class.
Intuitively, classification decisions depend primarily on the direction of the weight matrix rather than its scale. To enable comparisons across different scales of W W W , we adopt the normalized margin used in [9 ] , i.e., γ ( W ; D ) : = C ( W ; D ) ∥ W ∥ F \gamma(W;\mathcal{D}):=\frac{C(W;\mathcal{D})}{\lVert W\rVert_F} γ ( W ; D ) := ∥ W ∥ F C ( W ; D ) for W ≠ 0 W\ne0 W = 0 . This definition has been used in recent theoretical studies [34 , 41 ] . We additionally set γ ( 0 ; D ) : = 0 \gamma(0;\mathcal{D}):=0 γ ( 0 ; D ) := 0 by convention.
Pre-training and fine-tuning setup
We now describe the specific pre-training and fine-tuning framework used in our theoretical analysis.
Pre-training stage. Let D p r e \mathcal{D}_{\mathrm{pre}} D pre be a labeled dataset with K K K classes, and let W p r e ∈ R K × d W_{\mathrm{pre}}\in\mathbb{R}^{K\times d} W pre ∈ R K × d denote a pre-trained linear classifier obtained by GD minimizing the cross-entropy loss L ( W ; D p r e ) L(W;\mathcal{D}_{\mathrm{pre}}) L ( W ; D pre ) . When D p r e \mathcal{D}_{\mathrm{pre}} D pre is linearly separable, this objective admits no finite minimizer. The GD iterates instead diverge in norm while converging in direction to the max-margin solution [34 , 46 ] . We formalize this directional structure in Assumption 2.4.
Fine-tuning stage. We adapt the pre-trained model to a downstream dataset D f t \mathcal{D}_{\mathrm{ft}} D ft consisting of n n n additional samples from each of 2 ≤ K ˉ ≤ K 2\le\bar{K}\le K 2 ≤ K ˉ ≤ K classes selected from the original K K K pre-training classes. This restriction applies throughout the paper and appendix, including to every task in adapter merging. This setup captures practical scenarios such as domain-incremental learning [12 ] , domain adaptation [Zohrisadeh et al., 2019, Zhang et al., 2022], and bias-rebalancing fine-tuning [30 , 48 ] .
When LoRA is applied to adapt the pre-trained model to the fine-tuning task, we introduce a low-rank update to the weights, parameterized by matrices B ∈ R K × r B\in\mathbb{R}^{K\times r} B ∈ R K × r and A ∈ R r × d A\in\mathbb{R}^{r\times d} A ∈ R r × d :
min B , A L ( W p r e + B A ; D f t ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) . (5) \min_{B,A}\ L(W_{\mathrm{pre}}+BA;\mathcal{D}_{\mathrm{ft}})+\frac{\lambda}{2}\left(\lVert B\rVert_{F}^{2}+\lVert A\rVert_{F}^{2}\right). \tag*{(5)} B , A min L ( W pre + B A ; D ft ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) . (5)
Here, the Frobenius-norm penalties explicitly constrain the adapters in weight space, limiting the deviation of the fine-tuned model from the pretrained initialization.
Although some empirical studies of LoRA do not include this penalty explicitly in the loss, they train the adapters using the AdamW optimizer [33 ] , whose decoupled weight decay induces an effect analogous to L 2 L_2 L 2 regularization on B B B and A A A . Thus, the objective in (5) captures the regularization that is, explicitly or implicitly, used in standard LoRA fine-tuning practice [20 , 52 , 3 ] .
Our goal is to understand the structure of the optimal solution to (5) , and how it explains both LoRA’s robustness to catastrophic forgetting and its effectiveness in enabling adapter merging.
Assumptions and their implications
Our theoretical analysis uses the following assumptions. Let N f t = K ˉ n N_{\mathrm{ft}}=\bar{K}n N ft = K ˉ n be the number of fine-tuning samples, write X f t = [ x ˉ 1 , … , x ˉ N f t ] X_{\mathrm{ft}}=[\bar{x}_1,\ldots,\bar{x}_{N_{\mathrm{ft}}}] X ft = [ x ˉ 1 , … , x ˉ N ft ] , and let X p r e X_{\mathrm{pre}} X pre collect the pre-training feature vectors as columns.
Assumption 2.1 (Task geometry). The combined pre-training and fine-tuning datasets are linearly separable. There exist ϵ 1 , ϵ 2 ∈ [ 0 , 1 ) \epsilon_1,\epsilon_2\in[0,1) ϵ 1 , ϵ 2 ∈ [ 0 , 1 ) such that, for all i , j ∈ [ N f t ] i,j\in[N_{\mathrm{ft}}] i , j ∈ [ N ft ] and x ∈ D p r e x\in\mathcal{D}_{\mathrm{pre}} x ∈ D pre ,
(a) ∥ x ˉ i ∥ 2 = ∥ x ∥ 2 = 1 , (b) ∣ x ˉ i T x ˉ j ∣ ≤ ϵ 1 ( i ≠ j ) , (c) ∣ x ˉ i T x ∣ ≤ ϵ 2 . (6) \begin{aligned}
\text{(a)}\quad&\lVert\bar{x}_i\rVert_2=\lVert x\rVert_2=1,\qquad
\text{(b)}\quad\lvert\bar{x}_i^{\mathsf{T}}\bar{x}_j\rvert\le\epsilon_1\ (i\ne j),\qquad
\text{(c)}\quad\lvert\bar{x}_i^{\mathsf{T}}x\rvert\le\epsilon_2. \tag*{(6)}
\end{aligned} (a) ∥ x ˉ i ∥ 2 = ∥ x ∥ 2 = 1 , (b) ∣ x ˉ i T x ˉ j ∣ ≤ ϵ 1 ( i = j ) , (c) ∣ x ˉ i T x ∣ ≤ ϵ 2 . (6)
Assumption 2.2 (Dimension). In the strict-orthogonal case ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 , the input dimension satisfies d ≥ N f t + rank ( X p r e ) d\ge N_{\mathrm{ft}}+\operatorname{rank}(X_{\mathrm{pre}}) d ≥ N ft + rank ( X pre ) . For near-orthogonal data, we assume ( N f t − 1 ) ϵ 1 < 1 (N_{\mathrm{ft}}-1)\epsilon_1<1 ( N ft − 1 ) ϵ 1 < 1 .
Assumption 2.3 (LoRA rank). The LoRA rank parameter satisfies r ≥ K ˉ r\ge\bar{K} r ≥ K ˉ , where K ˉ \bar{K} K ˉ is the number of fine-tuning classes.
Assumption 2.4 (Pre-trained classifier). The pre-trained classifier W p r e W_{\mathrm{pre}} W pre is a scaled version of the hard-margin SVM solution on D p r e \mathcal{D}_{\mathrm{pre}} D pre :
W p r e = ρ p r e ⋅ W p r e S V M ∥ W p r e S V M ∥ F , W_{\mathrm{pre}}=\rho_{\mathrm{pre}}\cdot\frac{W_{\mathrm{pre}}^{\mathrm{SVM}}}{\lVert W_{\mathrm{pre}}^{\mathrm{SVM}}\rVert_F}, W pre = ρ pre ⋅ ∥ W pre SVM ∥ F W pre SVM , for some scalar ρ p r e > 0 \rho_{\mathrm{pre}}>0 ρ pre > 0 , where W p r e S V M = W S V M ( D p r e ) W_{\mathrm{pre}}^{\mathrm{SVM}}=W^{\mathrm{SVM}}(\mathcal{D}_{\mathrm{pre}}) W pre SVM = W SVM ( D pre ) .
Assumption justification. The near-orthogonal task Assumption 2.1 is arguably a strong assumption in practice, but reflects the realistic geometric structure of fine-tuning data, and similar orthogonality-type assumptions are commonly used in theoretical analyses to enable closed-form characterizations [6 , 5 , 16 , 26 ] . The feature-correlation analyses on CIFAR-100 and six multi-domain benchmarks in Appendices § 14 .1.1 and § 14 .§ 2.1 provide empirical motivation for studying this reference geometry. Assumption 2.2 makes the geometric feasibility and conditioning requirements explicit. The strict-case bound accounts for both the fine-tuning sample count and the pre-training feature span, specifying a high-dimensional, sample-limited setting. Assumption 2.3 requires only r ≥ K ˉ r\ge\bar{K} r ≥ K ˉ , meaning the LoRA rank needs only to be at least the number of fine-tuning classes. Finally, Assumption 2.4 is motivated by recent theoretical results on the implicit bias of GD in multiclass classification [35 , 41 ] , as discussed in §2.1. ρ p r e \rho_{\mathrm{pre}} ρ pre captures the (unspecified) scale at which pre-training was stopped, which arises naturally in implicit-bias analyses where the GD iterate diverges along the SVM direction.
For clarity of exposition, all theorems in the main paper are stated under the strict-orthogonal special case (ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 ) of Assumption 2.1. The general versions of these results for ϵ 1 , ϵ 2 > 0 \epsilon_1,\epsilon_2>0 ϵ 1 , ϵ 2 > 0 , together with their proofs, are deferred to the Appendix D–F.
Implications. The assumptions introduced above lead to a clean closed-form expression for the K ˉ \bar{K} K ˉ -class hard-margin SVM solution for the fine-tuning task, and also imply near-orthogonality between the pre-trained weights W p r e W_{\mathrm{pre}} W pre and the fine-tuning dataset D f t \mathcal{D}_{\mathrm{ft}} D ft , which underpins our subsequent theoretical analysis in §3. To formalize these implications, we begin by introducing the concept of a simplex equiangular tight frame (simplex ETF), which characterizes the geometry of the hard-margin SVM solution in our setting.
Definition 2.1 (Simplex ETF). For m ≥ 2 m\ge2 m ≥ 2 , an m m m -simplex ETF is a collection of m m m vectors in R p \mathbb{R}^{p} R p for some p ≥ m p\ge m p ≥ m , given by the columns of
M m : = m m − 1 P ( I m − 1 m 1 m 1 m T ) , M_m:=\sqrt{\frac{m}{m-1}}P\left(I_m-\frac{1}{m}\mathbf{1}_m\mathbf{1}_m^{\mathsf{T}}\right), M m := m − 1 m P ( I m − m 1 1 m 1 m T ) , where P ∈ R p × m P\in\mathbb{R}^{p\times m} P ∈ R p × m satisfies P T P = I m P^{\mathsf{T}}P=I_m P T P = I m .
Throughout, we use the canonical choice p = m p=m p = m and P = I m P=I_m P = I m , so M m ∈ R m × m M_m\in\mathbb{R}^{m\times m} M m ∈ R m × m . Using this definition, we now present a proposition that, under the strict-orthogonal specialization, characterizes the structure of the hard-margin SVM solution for the fine-tuning data (D f t \mathcal{D}_{\mathrm{ft}} D ft ) and the orthogonality between the pre-trained weights (W p r e W_{\mathrm{pre}} W pre ) and the fine-tuning data.
Proposition 2.1. Let K ˉ ≥ 2 \bar{K} \ge2 K ˉ ≥ 2 . Under Assumption 2.1 with ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 and Assumption 2.4, the following hold:
(i) The minimum-norm SVM solution W f t S V M ∈ R K ˉ × d W_{\mathrm{ft}}^{\mathrm{SVM}} \in\mathbb{R}^{\bar{K}\times d} W ft SVM ∈ R K ˉ × d for D f t \mathcal{D}_{\mathrm{ft}} D ft , defined by (3) with K ˉ \bar{K} K ˉ output classes, admits the closed-form solution
W f t S V M = K ˉ − 1 K ˉ ( M K ˉ ⊗ 1 n T ) X f t T , W_{\mathrm{ft}}^{\mathrm{SVM}}=\sqrt{\frac{\bar{K}-1}{\bar{K}}}\left(M_{\bar{K}}\otimes\mathbf{1}_{n}^{\mathsf{T}}\right)X_{\mathrm{ft}}^{\mathsf{T}}, W ft SVM = K ˉ K ˉ − 1 ( M K ˉ ⊗ 1 n T ) X ft T , where x ˉ i , j \bar{x}_{i,j} x ˉ i , j is the j j j -th sample from class i i i , and X f t = [ x ˉ 1 , 1 , … , x ˉ 1 , n , … , x ˉ K ˉ , n ] X_{\mathrm{ft}}=[\bar{x}_{1,1},\ldots,\bar{x}_{1,n},\ldots,\bar{x}_{\bar{K},n}] X ft = [ x ˉ 1 , 1 , … , x ˉ 1 , n , … , x ˉ K ˉ , n ] is the fine-tuning data matrix.
(ii) The pre-trained weights are orthogonal to the fine-tuning data: W p r e x ˉ = 0 W_{\mathrm{pre}}\bar{x}=0 W pre x ˉ = 0 for all x ˉ ∈ D f t \bar{x}\in\mathcal{D}_{\mathrm{ft}} x ˉ ∈ D ft .
Remark on Proposition 2.1. Proposition 2.1 reveals that the hard-margin SVM solution for the fine-tuning task admits a compact closed-form expression: W f t S V M = K ˉ − 1 K ˉ ( M K ˉ ⊗ 1 n T ) X f t T W_{\mathrm{ft}}^{\mathrm{SVM}}=\sqrt{\frac{\bar{K}-1}{\bar{K}}}\left(M_{\bar{K}}\otimes\mathbf{1}_{n}^{\mathsf{T}}\right)X_{\mathrm{ft}}^{\mathsf{T}} W ft SVM = K ˉ K ˉ − 1 ( M K ˉ ⊗ 1 n T ) X ft T . However, this form does not immediately reveal how the classifier aggregates information from the fine-tuning data. To provide insight, we explicitly derive the expression for the first row of W f t S V M W_{\mathrm{ft}}^{\mathrm{SVM}} W ft SVM (the remaining rows follow symmetrically), yielding
( K ˉ − 1 K ˉ ∑ i = 1 n x ˉ 1 , i − 1 K ˉ ∑ k = 2 K ˉ ∑ j = 1 n x ˉ k , j ) T . \left(\frac{\bar{K}-1}{\bar{K}}\sum_{i=1}^{n}\bar{x}_{1,i}-\frac{1}{\bar{K}}\sum_{k=2}^{\bar{K}}\sum_{j=1}^{n}\bar{x}_{k,j}\right)^{\mathsf{T}}. K ˉ K ˉ − 1 i = 1 ∑ n x ˉ 1 , i − K ˉ 1 k = 2 ∑ K ˉ j = 1 ∑ n x ˉ k , j T .
Intuitively, each row of W f t S V M W_{\mathrm{ft}}^{\mathrm{SVM}} W ft SVM encodes a direction that emphasizes its corresponding class mean while uniformly suppressing the influence of all other classes, leading to class separation in max-margin classification. The second result in Proposition 2.1 follows from the representer theorem for SVMs, which implies that the hard-margin SVM solution on pre-training data lies in the span of D p r e \mathcal{D}_{\mathrm{pre}} D pre . By Assumption 2.4, W p r e W_{\mathrm{pre}} W pre inherits this property; combined with condition (c) of Assumption 2.1 at ϵ 2 = 0 \epsilon_2=0 ϵ 2 = 0 , each row of W p r e W_{\mathrm{pre}} W pre is therefore orthogonal to the fine-tuning data.
Building upon these assumptions and propositions, we develop a rigorous theoretical framework that provides a principled explanation for LoRA’s empirical advantages in the subsequent section. Our analysis reveals properties that enable precise quantification of LoRA’s benefits in terms of robustness to catastrophic forgetting and adapter merging performance.
Main Results
In this section, we first analyze the optimization landscape of the LoRA objective and characterize its global minimum in §3.1. We then derive closed-form expressions for the margins on the union of pre-training and fine-tuning datasets, explaining why LoRA mitigates catastrophic forgetting in §3.2. Finally, in §3.3, we extend our margin-based analysis to the multi-task setting, explaining the effectiveness of adapter merging and deriving optimal mixing coefficients that maximize the margin on the union of all tasks. Moreover, building on these closed-form coefficients, we translate them into a simple training-free merging rule, margin-based LoRA merging (MaLoRA, Algorithm 1).
Global minimum of the LoRA objective
In this subsection, we present our main theoretical result characterizing the global minimizer of the LoRA objective in (5). Interestingly, under suitable regularization, we find that part of the optimal LoRA adapter aligns exactly with the K ˉ \bar{K} K ˉ -class hard-margin SVM solution (W f t S V M W_{\mathrm{ft}}^{\mathrm{SVM}} W ft SVM ) for the fine-tuning data (see Proposition 2.1 for its definition). While our analysis neither assumes nor relies on this alignment, its emergence reveals a geometric structure that partially explains LoRA’s effectiveness.
Theorem 3.1. Let K K K and K ˉ ≤ K \bar{K}\le K K ˉ ≤ K be the number of pre-training and fine-tuning classes, respectively. There exist scalar functions a λ a_\lambda a λ , b λ b_\lambda b λ , c λ c_\lambda c λ , Θ λ \Theta_\lambda Θ λ of λ \lambda λ such that for any global minimizer ( B λ ∗ , A λ ∗ ) (B_\lambda^*,A_\lambda^*) ( B λ ∗ , A λ ∗ ) of (5) with λ > 0 \lambda>0 λ > 0 , we have
B λ ∗ A λ ∗ = ( ( ( a λ + b λ ) I K ˉ − b λ 1 K ˉ 1 K ˉ T ) ⊗ 1 n T ) X f t T , ∥ B λ ∗ A λ ∗ ∥ F = Θ λ . (7) B_\lambda^*A_\lambda^*=\left(\left((a_\lambda+b_\lambda)I_{\bar{K}}-b_\lambda\mathbf{1}_{\bar{K}}\mathbf{1}_{\bar{K}}^{\mathsf{T}}\right)\otimes\mathbf{1}_{n}^{\mathsf{T}}\right)X_{\mathrm{ft}}^{\mathsf{T}},\qquad\lVert B_\lambda^*A_\lambda^*\rVert_F=\Theta_\lambda. \tag*{(7)} B λ ∗ A λ ∗ = ( ( ( a λ + b λ ) I K ˉ − b λ 1 K ˉ 1 K ˉ T ) ⊗ 1 n T ) X ft T , ∥ B λ ∗ A λ ∗ ∥ F = Θ λ . (7) When K ˉ ≥ 2 \bar{K}\ge2 K ˉ ≥ 2 , there exists a critical regularization weight λ c r i t ∈ [ 0 , 1 K n ) \lambda_{\mathrm{crit}}\in\left[0,\frac{1}{K\sqrt{n}}\right) λ crit ∈ [ 0 , K n 1 ) such that the scalar functions a λ a_\lambda a λ , b λ b_\lambda b λ , c λ c_\lambda c λ , Θ λ \Theta_\lambda Θ λ are characterized as follows:
(i) High-penalty regime ( λ ≥ 1 K n ) \left(\lambda\ge\frac{1}{K\sqrt{n}}\right) ( λ ≥ K n 1 ) : a λ = b λ = c λ = Θ λ = 0 a_\lambda=b_\lambda=c_\lambda=\Theta_\lambda=0 a λ = b λ = c λ = Θ λ = 0 , thus B λ ∗ A λ ∗ = 0 K × d B_\lambda^*A_\lambda^*=0_{K\times d} B λ ∗ A λ ∗ = 0 K × d .
(ii) Intermediate regime λ c r i t < λ < 1 K n \lambda_{\mathrm{crit}} < \lambda< \frac{1}{K\sqrt{n}} λ crit < λ < K n 1 : a λ = Θ λ K ‾ − 1 K n a_\lambda= \frac{\Theta_\lambda\sqrt{\overline{K}-1}}{K\sqrt{n}} a λ = K n Θ λ K − 1 , b λ = Θ λ K n ( K ‾ − 1 ) b_\lambda= \frac{\Theta_\lambda}{K\sqrt{n}(\overline{K}-1)} b λ = K n ( K − 1 ) Θ λ , c λ = 0 c_\lambda= 0 c λ = 0 , and Θ λ \Theta_\lambda Θ λ is the unique root of a nonlinear equation (see (29) in Appendix D), thus the minimizer is
B λ ∗ A λ ∗ = Θ λ K n ( ( M K ( K ‾ ) 0 ( K − K ‾ ) × K ‾ ) ⊗ 1 n T ) X f t T . (8) B_\lambda^*A_\lambda^* = \frac{\Theta_\lambda}{\sqrt{K n}}\left(\begin{pmatrix} M_K^{(\overline{K})} \\ 0_{(K-\overline{K})\times\overline{K}} \end{pmatrix}\otimes\mathbf{1}_n^{\mathsf{T}}\right)X_{\mathrm{ft}}^{\mathsf{T}}. \tag*{(8)} B λ ∗ A λ ∗ = K n Θ λ ( ( M K ( K ) 0 ( K − K ) × K ) ⊗ 1 n T ) X ft T . (8)
(iii) Low-penalty regime 0 < λ ≤ λ c r i t 0 < \lambda\le\lambda_{\mathrm{crit}} 0 < λ ≤ λ crit : a λ a_\lambda a λ , b λ b_\lambda b λ , c λ c_\lambda c λ , Θ λ \Theta_\lambda Θ λ are non-negative.
Moreover, the small-regularization limit satisfies
lim λ → 0 + Θ λ = ∞ , lim λ → 0 + B λ ∗ A λ ∗ ∥ B λ ∗ A λ ∗ ∥ F = 1 n K ( M K ( K ‾ ) ⊗ 1 n T ) X f t T , (9) \lim_{\lambda\to0^+}\Theta_\lambda=\infty,\qquad\lim_{\lambda\to0^+}\frac{B_\lambda^*A_\lambda^*}{\lVert B_\lambda^*A_\lambda^*\rVert_F}=\sqrt{\frac{1}{nK}}\left(M_K^{(\overline{K})}\otimes\mathbf{1}_n^{\mathsf{T}}\right)X_{\mathrm{ft}}^{\mathsf{T}}, \tag*{(9)} λ → 0 + lim Θ λ = ∞ , λ → 0 + lim ∥ B λ ∗ A λ ∗ ∥ F B λ ∗ A λ ∗ = n K 1 ( M K ( K ) ⊗ 1 n T ) X ft T , (9)
where M K ( K ‾ ) M_K^{(\overline{K})} M K ( K ) is the first K ‾ \overline{K} K columns of the K K K -ETF matrix M K M_K M K .
Remark 3.1. For K ‾ ≥ 2 \overline{K}\ge2 K ≥ 2 , the critical weight satisfies λ c r i t = 0 \lambda_{\mathrm{crit}}=0 λ crit = 0 if and only if K ‾ = K \overline{K}=K K = K . In this case, the intermediate regime covers all 0 < λ < 1 / ( K n ) 0<\lambda<1/(K\sqrt{n}) 0 < λ < 1/ ( K n ) , and the low-penalty regime contains no positive regularization weights. The limits in (9) remain valid as λ → 0 + \lambda\to0^+ λ → 0 + , approached through the intermediate regime.
The proof of Theorem 3.1 is provided in Appendix D. The theorem shows that the optimal LoRA update B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ has the unified form in (7) , and identifies up to three regimes based on the regularization parameter λ \lambda λ . We make the following remarks:
Effect of regularization and connection with max-margin classifier. The theorem shows that the regularization parameter governs the structure of the optimal LoRA adapters. In the high-penalty regime , the regularization term dominates and drives the optimal adapters to zero. In the intermediate regime , the adapters balance cross-entropy and regularization, yielding the explicit form in (8) . In particular, the first K ‾ \overline{K} K rows of B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ align with the K ‾ \overline{K} K -class hard-margin SVM solution on the fine-tuning data (Proposition 2.1), which gives a clear max-margin interpretation of the learned solution. In the low-penalty regime , explicit formulas for a λ a_\lambda a λ and b λ b_\lambda b λ are no longer available. Nevertheless, as λ → 0 + \lambda\to0^+ λ → 0 + , the cross-entropy term dominates and Θ λ → ∞ \Theta_\lambda\to\infty Θ λ → ∞ , while our asymptotic analysis still characterizes the limiting direction of B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ in (9) . This direction coincides with the hard-margin SVM solution only when K ‾ = K \overline{K}=K K = K .
Implication for the fine-tuned model. The structured form of the optimal LoRA adapters in (7) naturally enables W L o R A λ : = W p r e + B λ ∗ A λ ∗ W_{\mathrm{LoRA}}^\lambda:=W_{\mathrm{pre}}+B_\lambda^*A_\lambda^* W LoRA λ := W pre + B λ ∗ A λ ∗ to perform well on both pre-training and fine-tuning data. Under Assumptions 2.1 and 2.4, Proposition 2.1 and Theorem 3.1 imply that W p r e x ‾ = 0 W_{\mathrm{pre}}\overline{x}=0 W pre x = 0 for all x ‾ ∈ D f t \overline{x}\in\mathcal{D}_{\mathrm{ft}} x ∈ D ft and B λ ∗ A λ ∗ x = 0 B_\lambda^*A_\lambda^*x=0 B λ ∗ A λ ∗ x = 0 for all x ∈ D p r e x\in\mathcal{D}_{\mathrm{pre}} x ∈ D pre . That is, the two components operate independently: W p r e W_{\mathrm{pre}} W pre classifies the pre-training data, while B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ adapts to the fine-tuning data:
∀ x ∈ D p r e ∪ D f t , W L o R A λ x = { W p r e x if x ∈ D p r e , B λ ∗ A λ ∗ x if x ∈ D f t . holds ∀ λ > 0. (10) \forall x\in\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}},\quad W_{\mathrm{LoRA}}^\lambda x=
\begin{cases}
W_{\mathrm{pre}}x & \text{if }x\in\mathcal{D}_{\mathrm{pre}},\\
B_\lambda^*A_\lambda^*x & \text{if }x\in\mathcal{D}_{\mathrm{ft}}.
\end{cases}
\quad\text{holds }\forall\lambda>0. \tag*{(10)} ∀ x ∈ D pre ∪ D ft , W LoRA λ x = { W pre x B λ ∗ A λ ∗ x if x ∈ D pre , if x ∈ D ft . holds ∀ λ > 0. (10)
This clean separation lets us decompose the normalized margin of W L o R A λ W_{\mathrm{LoRA}}^\lambda W LoRA λ on each component dataset into a contribution from the relevant component (either W p r e W_{\mathrm{pre}} W pre or B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ ) and a relative-norm factor:
γ ( W L o R A λ ; D ) = γ ( W ; D ) ∥ W ∥ F ∥ W L o R A λ ∥ F , ( W , D ) ∈ { ( W p r e , D p r e ) , ( B λ ∗ A λ ∗ , D f t ) } . (11) \gamma(W_{\mathrm{LoRA}}^\lambda;\mathcal{D})=\gamma(W;\mathcal{D})\frac{\lVert W\rVert_F}{\lVert W_{\mathrm{LoRA}}^\lambda\rVert_F},\qquad(W,\mathcal{D})\in\{(W_{\mathrm{pre}},\mathcal{D}_{\mathrm{pre}}),(B_\lambda^*A_\lambda^*,\mathcal{D}_{\mathrm{ft}})\}. \tag*{(11)} γ ( W LoRA λ ; D ) = γ ( W ; D ) ∥ W LoRA λ ∥ F ∥ W ∥ F , ( W , D ) ∈ {( W pre , D pre ) , ( B λ ∗ A λ ∗ , D ft )} . (11)
The expression in (11) shows that the normalized margin on the combined dataset depends not only on the individual normalized margins of W p r e W_{\mathrm{pre}} W pre and B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ on the D p r e \mathcal{D}_{\mathrm{pre}} D pre and D f t \mathcal{D}_{\mathrm{ft}} D ft respectively, but also on the relative Frobenius norms of these components. In the following sections, we use (11) to derive closed-form expressions for the normalized margin on the combined dataset and to compute the optimal mixing coefficients for adapter merging . For clarity of presentation, we focus on the case K ‾ = K \overline{K}=K K = K and refer the reader to Appendix § 11 and Appendix F for the full version of our results.
Margin trade-off between forgetting and fine-tuning under LoRA adaptation
We now analyze catastrophic forgetting through the normalized margin. Our goal is to characterize how the LoRA fine-tuned model trades off performance on the pre-training data and the fine-tuning data as the regularization parameter varies, and to identify the regularization level that maximizes the margin on their union. For convenience, define γ p r e : = γ ( W p r e ; D p r e ) \gamma_{\mathrm{pre}} := \gamma(W_{\mathrm{pre}}; \mathcal{D}_{\mathrm{pre}}) γ pre := γ ( W pre ; D pre ) , γ f t , λ : = γ ( B λ ∗ A λ ∗ ; D f t ) \gamma_{\mathrm{ft},\lambda} := \gamma(B_{\lambda}^{*}A_{\lambda}^{*}; \mathcal{D}_{\mathrm{ft}}) γ ft , λ := γ ( B λ ∗ A λ ∗ ; D ft ) . Moreover, we define the adaptation to the fine-tuning data and forgetting on the pre-training data by
Adapt ( λ ) = γ ( W L o R A λ ; D f t ) − γ ( W p r e , D f t ) , Forget ( λ ) : = γ p r e − γ ( W L o R A λ ; D p r e ) . (12) \operatorname{Adapt}(\lambda)=\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{ft}})-\gamma(W_{\mathrm{pre}},\mathcal{D}_{\mathrm{ft}}),\qquad\operatorname{Forget}(\lambda):=\gamma_{\mathrm{pre}}-\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}). \tag*{(12)} Adapt ( λ ) = γ ( W LoRA λ ; D ft ) − γ ( W pre , D ft ) , Forget ( λ ) := γ pre − γ ( W LoRA λ ; D pre ) . (12)
Based on these definitions, we now present the theorem.
Theorem 3.2. Adopt the setup of Theorem 3.1 under strict orthogonality, and assume K ˉ = K ≥ 2 \bar{K}=K\ge2 K ˉ = K ≥ 2 . Let γ f t ∗ \gamma_{\mathrm{ft}}^{*} γ ft ∗ be the max normalized margin any linear classifier can obtain on the fine-tuning data D f t \mathcal{D}_{\mathrm{ft}} D ft , and recall the scalar ρ p r e \rho_{\mathrm{pre}} ρ pre from Assumption 2.4 . Then, Adapt ( λ ) \operatorname{Adapt}(\lambda) Adapt ( λ ) and Forget ( λ ) \operatorname{Forget}(\lambda) Forget ( λ ) can be characterized uniformly over all λ \lambda λ as follows:
Adapt ( λ ) = γ f t , λ Θ λ Θ λ 2 + ρ p r e 2 , Forget ( λ ) = γ p r e ( 1 − ρ p r e Θ λ 2 + ρ p r e 2 ) . (13) \operatorname{Adapt}(\lambda)=\gamma_{\mathrm{ft},\lambda}\frac{\Theta_{\lambda}}{\sqrt{\Theta_{\lambda}^{2}+\rho_{\mathrm{pre}}^{2}}},\qquad\operatorname{Forget}(\lambda)=\gamma_{\mathrm{pre}}\left(1-\frac{\rho_{\mathrm{pre}}}{\sqrt{\Theta_{\lambda}^{2}+\rho_{\mathrm{pre}}^{2}}}\right). \tag*{(13)} Adapt ( λ ) = γ ft , λ Θ λ 2 + ρ pre 2 Θ λ , Forget ( λ ) = γ pre 1 − Θ λ 2 + ρ pre 2 ρ pre . (13) Moreover, the effect of the regularization parameter λ \lambda λ is characterized by two regimes:
(i) High-penalty regime: if λ ≥ 1 K n \lambda\ge\frac{1}{K\sqrt{n}} λ ≥ K n 1 , then Θ λ = 0 \Theta_{\lambda}=0 Θ λ = 0 , and hence Forget ( λ ) = Adapt ( λ ) = 0 \operatorname{Forget}(\lambda)=\operatorname{Adapt}(\lambda)=0 Forget ( λ ) = Adapt ( λ ) = 0 .
(ii) Intermediate and low-penalty regimes: if λ < 1 K n \lambda<\frac{1}{K\sqrt{n}} λ < K n 1 , then γ f t , λ = γ f t ∗ \gamma_{\mathrm{ft},\lambda}=\gamma_{\mathrm{ft}}^{*} γ ft , λ = γ ft ∗ , and Θ λ \Theta_{\lambda} Θ λ is decreasing in λ \lambda λ . Therefore, Adapt ( λ ) \operatorname{Adapt}(\lambda) Adapt ( λ ) and Forget ( λ ) \operatorname{Forget}(\lambda) Forget ( λ ) are both decreasing in λ \lambda λ . In particular,
lim λ → 0 + ( Adapt ( λ ) , Forget ( λ ) ) = ( γ f t ∗ , γ p r e ) , lim λ → 1 K n ( Adapt ( λ ) , Forget ( λ ) ) = ( 0 , 0 ) . (14) \lim_{\lambda\to0^{+}}\left(\operatorname{Adapt}(\lambda),\operatorname{Forget}(\lambda)\right)=\left(\gamma_{\mathrm{ft}}^{*},\gamma_{\mathrm{pre}}\right),\qquad\lim_{\lambda\to\frac{1}{K\sqrt{n}}}\left(\operatorname{Adapt}(\lambda),\operatorname{Forget}(\lambda)\right)=(0,0). \tag*{(14)} λ → 0 + lim ( Adapt ( λ ) , Forget ( λ ) ) = ( γ ft ∗ , γ pre ) , λ → K n 1 lim ( Adapt ( λ ) , Forget ( λ ) ) = ( 0 , 0 ) . (14)
Remark 3.2. Notably, the intermediate and low-penalty regimes coincide when K ˉ = K \bar{K}=K K ˉ = K . This merging is specific to the case K ˉ = K \bar{K}=K K ˉ = K ; when K ˉ < K \bar{K}<K K ˉ < K , the two regimes remain distinct.
Balancing adaptation and forgetting. Theorem 3.2 shows that the same quantity Θ λ \Theta_{\lambda} Θ λ controls both adaptation and forgetting. As λ \lambda λ decreases, the adapter scale Θ λ \Theta_{\lambda} Θ λ increases, improving Adapt ( λ ) \operatorname{Adapt}(\lambda) Adapt ( λ ) on the fine-tuning data while also increasing Forget ( λ ) \operatorname{Forget}(\lambda) Forget ( λ ) on the pre-training data. Thus, LoRA fine-tuning induces a direct margin trade-off: stronger adaptation comes at the cost of larger pre-training margin degradation. The natural question is how to choose λ \lambda λ to balance these effects. In the following Corollary 3.1, we quantify this trade-off by studying the normalized margin on the union of the pre-training and fine-tuning datasets, which leads to an explicit optimal regularization level.
Corollary 3.1 (Optimal balance between adaptation and forgetting). Under the conditions of Theorem 3.2 , with K ˉ = K \bar{K}=K K ˉ = K , the normalized margin of W L o R A λ W_{\mathrm{LoRA}}^{\lambda} W LoRA λ on the union D p r e ∪ D f t \mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}} D pre ∪ D ft is
γ ( W L o R A λ ; D p r e ∪ D f t ) = min { γ p r e − Forget ( λ ) , Adapt ( λ ) } . (15) \gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}})=\min\{\gamma_{\mathrm{pre}}-\operatorname{Forget}(\lambda),\operatorname{Adapt}(\lambda)\}. \tag*{(15)} γ ( W LoRA λ ; D pre ∪ D ft ) = min { γ pre − Forget ( λ ) , Adapt ( λ )} . (15) This margin is uniquely maximized at the balancing point λ ∗ ∈ ( 0 , 1 / ( K n ) ) \lambda^{*}\in(0,1/(K\sqrt{n})) λ ∗ ∈ ( 0 , 1/ ( K n )) , with
max λ > 0 γ ( W L o R A λ ; D p r e ∪ D f t ) = ( 1 ( γ f t ∗ ) 2 + 1 γ p r e 2 ) − 1 / 2 . (16) \max_{\lambda>0}\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}})=\left(\frac{1}{(\gamma_{\mathrm{ft}}^{*})^{2}}+\frac{1}{\gamma_{\mathrm{pre}}^{2}}\right)^{-1/2}. \tag*{(16)} λ > 0 max γ ( W LoRA λ ; D pre ∪ D ft ) = ( ( γ ft ∗ ) 2 1 + γ pre 2 1 ) − 1/2 . (16)
Margin characterization of merged LoRA models and optimal mixing coefficients
We now extend the margin-based analysis to the multi-task setting, where a single pre-trained model is independently adapted by T T T LoRA adapters that we wish to merge into a unified model. Through the same margin lens, we derive closed-form expressions for the merged model’s margin on each task and identify mixing coefficients that maximize the margin on the union of all tasks (Theorem 3.3). Building on this characterization, we design a practical LoRA merging algorithm margin-based LoRA merging (MaLoRA, Algorithm 1).
Let ( B i ∗ , A i ∗ ) (B_{i}^{*},A_{i}^{*}) ( B i ∗ , A i ∗ ) denote the LoRA adapter trained on dataset D i \mathcal{D}_{i} D i , for i ∈ [ T ] i\in[T] i ∈ [ T ] . We merge the adapters with the pre-trained weight matrix as follows:
W L o R A ( α ) : = W p r e + ∑ i = 1 T α i B i ∗ A i ∗ , α 1 , … , α T ≥ 0 , (17) W_{\mathrm{LoRA}}(\boldsymbol{\alpha}):=W_{\mathrm{pre}}+\sum_{i=1}^{T}\alpha_{i}B_{i}^{*}A_{i}^{*},\qquad\alpha_{1},\ldots,\alpha_{T}\ge0, \tag*{(17)} W LoRA ( α ) := W pre + i = 1 ∑ T α i B i ∗ A i ∗ , α 1 , … , α T ≥ 0 , (17)
where α = ( α 1 , … , α T ) \boldsymbol{\alpha}=(\alpha_{1},\ldots,\alpha_{T}) α = ( α 1 , … , α T ) are mixing coefficients. Moreover, we generalize Assumption 2.1 to the setting of learning multiple tasks as follows.
Algorithm 1 Margin-based LoRA Merging (MaLoRA)
Require: $W_{\mathrm{pre}}$, $\{(B_i^*, A_i^*)\}_{i=1}^{T}$, $\mathcal{D}_{\mathrm{pre}}^{\mathrm{val}}$, $\{\mathcal{D}_i^{\mathrm{val}}\}_{i=1}^{T}$, $S \subset \mathbb{R}_{>0}$
1: $\bar{C}_{\mathrm{pre}} \leftarrow \bar{C}(W_{\mathrm{pre}};\mathcal{D}_{\mathrm{pre}})$
2: **for** $i = 1,\ldots,T$ **do**
3: $\bar{C}_i \leftarrow \bar{C}(W_{\mathrm{pre}} + B_i^* A_i^*;\mathcal{D}_i^{\mathrm{val}}), \quad \beta_i \leftarrow \begin{cases} \bar{C}_{\mathrm{pre}}/\bar{C}_i, & \bar{C}_i > 0, \\ 0, & \text{otherwise} \end{cases}$
4: **end for**
5: $s^* \in \arg\max_{s\in S}\operatorname{ValAcc}\left(W_{\mathrm{pre}} + s\sum_{i=1}^{T}\beta_i B_i^* A_i^*\right)$
6: **return** $W_{\mathrm{pre}} + s^*\sum_{i=1}^{T}\beta_i B_i^* A_i^*$
Assumption 3.1. There exist ϵ 1 , ϵ 2 ∈ [ 0 , 1 ) \epsilon_1,\epsilon_2 \in[0,1) ϵ 1 , ϵ 2 ∈ [ 0 , 1 ) such that
(a) Every sample has unit norm: ∥ x ∥ 2 = 1 \lVert x\rVert_2 = 1 ∥ x ∥ 2 = 1 for every x ∈ ⋃ i = 1 T D i ∪ D p r e x \in\bigcup_{i=1}^{T}\mathcal{D}_i \cup\mathcal{D}_{\mathrm{pre}} x ∈ ⋃ i = 1 T D i ∪ D pre .
(b) Within-task: for each i ∈ [ T ] i \in[T] i ∈ [ T ] and each pair of distinct x ˉ , x ˉ ′ ∈ D i \bar{x},\bar{x}' \in\mathcal{D}_i x ˉ , x ˉ ′ ∈ D i , ∣ x ˉ T x ˉ ′ ∣ ≤ ϵ 1 \lvert\bar{x}^{\mathsf{T}}\bar{x}'\rvert\le\epsilon_1 ∣ x ˉ T x ˉ ′ ∣ ≤ ϵ 1 .
(c) Cross-task and pre-training: ∀ x ˉ ∈ D i \forall\bar{x} \in\mathcal{D}_i ∀ x ˉ ∈ D i , x ˉ ′ ∈ D j \bar{x}' \in\mathcal{D}_j x ˉ ′ ∈ D j , i ≠ j i \ne j i = j , or x ∈ D p r e x \in\mathcal{D}_{\mathrm{pre}} x ∈ D pre , then ∣ x ˉ T x ˉ ′ ∣ , ∣ x ˉ T x ∣ ≤ ϵ 2 \lvert\bar{x}^{\mathsf{T}}\bar{x}'\rvert,\ \lvert\bar{x}^{\mathsf{T}}x\rvert\le\epsilon_2 ∣ x ˉ T x ˉ ′ ∣ , ∣ x ˉ T x ∣ ≤ ϵ 2 .
The following theorem characterizes the optimal mixing coefficients in the strict orthogonal case (ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 ). We refer the reader to Appendix F for the proof.
Theorem 3.3. Under Assumption 3.1, suppose each fine–tuning task D i \mathcal{D}_i D i (i = 1 , … , T i=1,\ldots,T i = 1 , … , T ) is trained with a regularization parameter in the intermediate or low-penalty regime. Let Θ λ , i = ∥ B i ∗ A i ∗ ∥ F \Theta_{\lambda,i}=\lVert B_i^* A_i^*\rVert_F Θ λ , i = ∥ B i ∗ A i ∗ ∥ F and γ i = γ ( B i ∗ A i ∗ ; D i ) \gamma_i=\gamma(B_i^* A_i^*;\mathcal{D}_i) γ i = γ ( B i ∗ A i ∗ ; D i ) . For any nonnegative coefficient vector α = ( α 1 , … , α T ) ∈ R ≥ 0 T \alpha=(\alpha_1,\ldots,\alpha_T)\in\mathbb{R}_{\ge0}^{T} α = ( α 1 , … , α T ) ∈ R ≥ 0 T , the merged model (17) achieves the normalized margins on each task as follows
γ ( W L o R A ( α ) ; D p r e ) = γ p r e ρ p r e ρ p r e 2 + ∑ j = 1 T α j 2 Θ λ , j 2 , γ ( W L o R A ( α ) ; D i ) = γ i α i Θ λ , i ρ p r e 2 + ∑ j = 1 T α j 2 Θ λ , j 2 , \gamma(W_{\mathrm{LoRA}}(\alpha);\mathcal{D}_{\mathrm{pre}})=\frac{\gamma_{\mathrm{pre}}\rho_{\mathrm{pre}}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}\alpha_j^2\Theta_{\lambda,j}^2}},\qquad
\gamma(W_{\mathrm{LoRA}}(\alpha);\mathcal{D}_i)=\frac{\gamma_i\alpha_i\Theta_{\lambda,i}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}\alpha_j^2\Theta_{\lambda,j}^2}}, γ ( W LoRA ( α ) ; D pre ) = ρ pre 2 + ∑ j = 1 T α j 2 Θ λ , j 2 γ pre ρ pre , γ ( W LoRA ( α ) ; D i ) = ρ pre 2 + ∑ j = 1 T α j 2 Θ λ , j 2 γ i α i Θ λ , i , γ ( W L o R A ( α ) ; D p r e ∪ { D i } i = 1 T ) = min { γ ( W L o R A ( α ) ; D p r e ) , γ ( W L o R A ( α ) ; D i ) , i ∈ [ T ] } . (18) \gamma(W_{\mathrm{LoRA}}(\alpha);\mathcal{D}_{\mathrm{pre}}\cup\{\mathcal{D}_i\}_{i=1}^{T})=\min\left\{\gamma(W_{\mathrm{LoRA}}(\alpha);\mathcal{D}_{\mathrm{pre}}),\gamma(W_{\mathrm{LoRA}}(\alpha);\mathcal{D}_i),i\in[T]\right\}. \tag*{(18)} γ ( W LoRA ( α ) ; D pre ∪ { D i } i = 1 T ) = min { γ ( W LoRA ( α ) ; D pre ) , γ ( W LoRA ( α ) ; D i ) , i ∈ [ T ] } . (18) Choosing the weights α i = C ( W p r e ; D p r e ) C ( B i A i ; D i ) \alpha_i=\frac{C(W_{\mathrm{pre}};\mathcal{D}_{\mathrm{pre}})}{C(B_iA_i;\mathcal{D}_i)} α i = C ( B i A i ; D i ) C ( W pre ; D pre ) , i ∈ [ T ] i\in[T] i ∈ [ T ] , maximizes the margin on the union of all tasks:
max α ∈ R ≥ 0 T γ ( W L o R A ( α ) ; D p r e ∪ { D i } i = 1 T ) = ( 1 γ p r e 2 + ∑ j = 1 T 1 γ j 2 ) − 1 / 2 . \max_{\alpha\in\mathbb{R}_{\ge0}^{T}}\gamma(W_{\mathrm{LoRA}}(\alpha);\mathcal{D}_{\mathrm{pre}}\cup\{\mathcal{D}_i\}_{i=1}^{T})=\left(\frac{1}{\gamma_{\mathrm{pre}}^2}+\sum_{j=1}^{T}\frac{1}{\gamma_j^2}\right)^{-1/2}. α ∈ R ≥ 0 T max γ ( W LoRA ( α ) ; D pre ∪ { D i } i = 1 T ) = ( γ pre 2 1 + j = 1 ∑ T γ j 2 1 ) − 1/2 .
Theorem 3.3 characterizes the margins on each task as well as on the union of tasks as functions of the mixing coefficients. In particular, it shows that the optimal mixing coefficients are given by the ratio between the unnormalized margin of the pre-trained model on D p r e \mathcal{D}_{\mathrm{pre}} D pre and that of each fine-tuned adapter on its own task. The unnormalized margin measures how confidently a classifier separates classes on its dataset, with smaller values indicating more challenging classification problems. This formula admits a direct interpretation: an adapter that achieves a smaller margin on its task is assigned a larger mixing weight, increasing its contribution so as to preserve a positive margin for that task in the merged model.
Towards practical margin-based adapter mixing The preceding analysis suggests choosing adapter mixing coefficients according to how confidently each adapter separates its task data, as measured by the unnormalized margin. In practice, however, datasets are often not perfectly separable, so the minimum margin C ( ⋅ ; ⋅ ) C(\cdot;\cdot) C ( ⋅ ; ⋅ ) can be negative and overly sensitive to hard examples. We therefore use the average margin C ˉ ( ⋅ ; D ) \bar{C}(\cdot;\mathcal{D}) C ˉ ( ⋅ ; D ) as a more stable proxy for adapter performance. Under the strict-orthogonal assumptions of Theorem 3.3, the per-sample margins of each optimal adapter are equal on its own fine-tuning dataset. Consequently, minimum and average margins coincide on these fine-tuning datasets. In our practical algorithm, we therefore use average margins as empirical proxies for the minimum margins appearing in the theoretical mixing rule.
Algorithm 1: Margin-based LoRA merging (MaLoRA). Motivated by Theorem 3.3, we propose a practical LoRA merging algorithm using the average-margin proxies C ˉ ( W ; D ) \bar{C}(W;\mathcal{D}) C ˉ ( W ; D ) . When a task has non-positive average margin C ˉ i \bar{C}_i C ˉ i , the corresponding coefficient would be negative and no longer consistent with the regime motivated by the theorem. Algorithm 1 therefore drops such tasks by setting their coefficients to zero. We also tune a global scaling factor s ∗ s^{*} s ∗ on a held-out validation set, following Ilharco et al. [ [22 ] , and find that this further improves performance. In § 4.2 , we compare the resulting method with other popular adapter-merging baselines.
Experiments
Our experiments aim to validate the theoretical predictions in Section 3 and assess how well they transfer beyond the linear setting, rather than to establish state-of-the-art performance. Since our analysis gives closed-form predictions for optimal regularization and adapter mixing coefficients, we test whether these quantities match empirical behavior in modern architectures. Specifically, we focus on two questions: (i) whether the mixing coefficients derived from our margin analysis agree with empirically optimal coefficients, and (ii) whether the resulting theory-driven merging rule, MaLoRA, remains competitive with adapter-merging baselines in nonlinear models. Experiments on the effect of regularization are deferred to Appendix § 14 .1.2.
Setup and metric. We evaluate our theory and method in two complementary settings. First, in linear probing on CIFAR-100 [27 ] (§ 4.1 ), we use four frozen pre-trained backbones (ResNet-50, ViT-B/16, ConvNeXt, and CLIP) as feature extractors [17 , 11 , 39 , 32 ] , apply LoRA only to the final linear classifier [25 ] , and construct tasks from the CIFAR-100 superclass hierarchy. Second, in multi-domain adapter merging (§ 4.2 ), we evaluate ViT-B/16 on PACS, VLCS, and TerraIncognita [14 , 2 ] (Li et al. 2017), and XLM-RoBERTa Base on MNLI, Amazon Reviews, and CivilComments [60 , 51 , 4 , 8 ] . Although our theory is stated in terms of normalized margin, we report classification accuracy in experiments, since real-world classification tasks often do not achieve perfect separation and can have negative normalized margins. Additional details are given in Appendix § 14 .
Optimal mixing coefficients for adapter merging: theory vs. grid search
We fine-tune LoRA adapters on two disjoint tasks (D 1 \mathcal{D}_1 D 1 , D 2 \mathcal{D}_2 D 2 ), then merge them with the pre-trained weights using mixing coefficients ( α 1 , α 2 ) (\alpha_1, \alpha_2) ( α 1 , α 2 ) . We evaluate the merged model’s performance on the combined dataset D p r e ∪ D 1 ∪ D 2 \mathcal{D}_{\mathrm{pre}} \cup\mathcal{D}_1 \cup\mathcal{D}_2 D pre ∪ D 1 ∪ D 2 . As shown in Figure 1 , our predicted mixing coefficients closely match the grid-searched optimum, confirming the practical accuracy of our merging theory.
Figure 1. Merged-model accuracy across architectures. Each panel shows the accuracy of the merged model evaluated on the combined dataset, across a 50 × 50 50 \times50 50 × 50 grid of mixing coefficients ( α 1 , α 2 ) ∈ ( 0 , 1 ) 2 (\alpha_1, \alpha_2) \in(0,1)^2 ( α 1 , α 2 ) ∈ ( 0 , 1 ) 2 . For each architecture, the red star indicates the theoretically predicted optimal coefficients from Theorem 3.3 , while the red triangle marks the empirically optimal coefficients.
Comparison between MaLoRA and other training-free baselines
Following Ilharco et al. [ [22 ] , we report average normalized accuracy (per-task merged accuracy divided by per-task finetuned accuracy, averaged across tasks) and its complement, relative accuracy drop , in Table 1 . The aggregate performance of the methods is comparable, with average normalized accuracy ranging from 92.87% to 93.32%. MaLoRA lies at the top of this range, achieving an average normalized accuracy of 93.32% and the smallest relative accuracy drop of 6.68%. We emphasize that although MaLoRA is competitive with baselines, our purpose here is not to achieve state-of-the-art results but to demonstrate the efficacy of our theoretically-grounded method.
Table 1. Average normalized accuracy and relative accuracy drop, following Ilharco et al. [ [22 ] . Results are averaged over 10 seeds. Bold indicates the best result in each column.
Computational overhead. MaLoRA computes task margins once and reuses them throughout the scale search. Under a separate matched 49-configuration timing protocol, the mean runtime is 972.6 s versus 950.3 s for TA, approximately 2.35% overhead across the six benchmarks. Detailed timings and the measurement protocol are in Appendix § 14 .§ 2.2 .
Conclusion, Limitations, and Future Work
In this work, we study LoRA fine-tuning through the normalized margin under a near-orthogonal task regime. We characterize the global minimizer of the regularized LoRA objective across three regularization regimes (Theorem 3.1), derive closed-form expressions for the margins on the pretraining and fine-tuning data, and identify the regularization level that maximizes the margin on their union (Theorem 3.2 ). We further extend the analysis to multi-task settings by deriving closed-form optimal mixing coefficients for adapter merging (Theorem 3.3 ). Based on this theory, we propose margin-based LoRA merging (MaLoRA, Algorithm 1), which matches or outperforms several adapter-merging baselines on a range of vision and language benchmarks. Across architectures and tasks, the empirical results closely align with the theoretical predictions.
Limitations and future work. First, our analysis relies on simplifying assumptions, most notably near-orthogonal task structure (Assumption 2.1) and balanced class sizes in the fine-tuning data. These assumptions enable a sharp closed-form characterization, but they also limit the scope of the theory. An important direction for future work is to relax them and study LoRA under more general data distributions and pre-training conditions. Second, experiments applying MaLoRA to nonlinear models provide empirical support for the practical usefulness of our merging rule beyond multiclass linear classification. These findings motivate extending the margin-based theoretical analysis to nonlinear adaptation. Third, our theory characterizes margins on the training datasets; establishing corresponding generalization or test-risk guarantees requires further analysis.
Acknowledgments and Disclosure of Funding
Ziqing Xu, Lachlan Ewen MacDonald, Salma Tarmoun, and René Vidal acknowledge support from NSF grant 2031985, Simons Foundation grant 814201 (Theory), ONR MURI 503405071, and University of Pennsylvania startup funds. Enrique Mallada acknowledges support from NSF Global Centers grant 2330450 and DOE Office of Science (ASCR) award 826565. Hancheng Min acknowledges support from the Science and Technology Commission of Shanghai Municipality through the Shanghai Pujiang Programme (grant 25PJА070) and Shanghai Jiao Tong University startup funds.
LLM Usage
In this work, we use LLM to revise the writing, and help build standard training baselines of the experiments, such as dataset downloading, data processing, and pre-trained model loading.
Related work
Theory of LoRA. There are many works theoretically studying the expressiveness, characterizing the loss landscape, and understanding the learning dynamics of LoRA. Zeng and Lee [ [57 ] prove that, under mild assumptions, LoRA can approximate any deep linear, feed-forward, or transformer network. Within the NTK regime, Malladi et al. [ [36 ] characterize the conditions under which one can study LoRA in the NTK regime, while Jang et al. [ [23 ] show that when the LoRA rank is r i ≳ N r_i \gtrsim\sqrt{N} r i ≳ N , where N N N is the number of samples, the optimization landscape of LoRA has no spurious local minima, and GD can find O ( N ) O(\sqrt{N}) O ( N ) -rank solutions that generalize well. Moreover, Xu et al. [ [54 ] study the learning dynamics of LoRA in the context of matrix factorization, and show that smaller initialization leads to longer training time and lower training error. However, none of these studies theoretically characterize why LoRA reduces catastrophic forgetting and support effective adapter merging .
LoRA merging. A variety of techniques have been proposed to merge task-specific LoRA adapters by forming a weighted average of their parameters. For example, LoRA-Hub [21 ] learns per-task mixing coefficients first, and then applies a weighted average of LoRA weight matrices B B B and A A A separately. LoRA-Flow [49 ] introduces token-level gates that dynamically assign merging coefficients to each LoRA adapter before taking weighted averaging. LoRA-Retriever [61 ] first retrieves the most relevant adapters for each input, and then averages the selected LoRA adapters. Beyond weighted averaging, another line of work attempts to address redundancy and conflicting updates directly. TIES [55 ] proceeds in three steps: trimming redundant parameters, resolving sign conflicts into an aggregate vector, and averaging only the parameters consistent with the aggregate sign, thereby mitigating degradation from redundant or conflicting updates. DARE [56 ] can be used as a preprocessing step for other merging methods, where parameters are randomly dropped according to a specified rate and the remaining ones are rescaled, reducing redundancy and potential interference among merged adapters. However, all these methods remain largely heuristic and lack theoretical guarantees.
Regularization and task orthogonality help reduce forgetting. A large body of work in fine-tuning and continual learning shows that controlling deviation of the fine-tuned model from the pretrained model reduces catastrophic forgetting. Concretely, these approaches impose weight-space penalties that restrict the magnitude and direction of parameter updates away from the pretrained solution. Weight-anchoring and importance-aware penalties exemplify this idea: Kirkpatrick et al. [ [24 ] penalize movement along directions identified by the Fisher information matrix, while Zenke et al. [ [58 ] accumulate path-wise importance during training. Schwarz et al. [ [45 ] extend this idea to long sequences of tasks. Related approaches estimate parameter sensitivity [1 ] , apply output-level distillation to preserve prior behavior [31 ] , or explicitly regularize parameters toward the pretrained anchor [29 ] . Although these approaches differ in how importance or anchoring is computed, the common mechanism is the same: weight-space regularizers constrain updates during fine-tuning relative to the pretrained model to mitigate catastrophic forgetting. Another complementary perspective highlights the role of task relations. When task representations are sufficiently decorrelated, interference between tasks is naturally reduced and forgetting is alleviated. In linear and kernelized models, it has been shown that when tasks are orthogonal or nearly orthogonal, cross-task gradient interference vanishes in expectation, implying negligible forgetting [10 , 13 ] . These insights resonate with our analysis of LoRA, where we explicitly study a regularized objective on low-rank adapters and make the simplifying assumption that the fine-tuning data are orthogonal to the pretraining data. This assumption directly connects to prior findings on task orthogonality and provides a tractable setting in which we can isolate and analyze the role of regularization in mitigating forgetting.
Characterization of Hard-margin SVM Solution for Orthogonal data
In this section, we characterize the hard-margin SVM solution for K ˉ \bar{K} K ˉ -class classification problem for orthogonal data. We show the solution is closely related to the K ˉ \bar{K} K ˉ -ETF simplex.
Problem Formulation. Consider a K ˉ \bar{K} K ˉ -class classification problem with data { ( x i , j , y i ) } i = 1 , j = 1 K ˉ , n \{(x_{i,j},y_i)\}_{i=1,j=1}^{\bar{K},n} {( x i , j , y i ) } i = 1 , j = 1 K ˉ , n where:
K ˉ ≥ 2 \bar{K} \ge2 K ˉ ≥ 2 is the number of classes in the dataset
n n n is the number of samples per class
y i ∈ [ K ˉ ] y_i \in[\bar{K}] y i ∈ [ K ˉ ] denotes the class label for samples x i , j x_{i,j} x i , j
All data vectors x i , j x_{i,j} x i , j have unit norm: ∥ x i , j ∥ = 1 \lVert x_{i,j}\rVert= 1 ∥ x i , j ∥ = 1
All data vectors are mutually orthogonal: x i , j T x k , l = 0 x_{i,j}^{\mathsf{T}}x_{k,l}=0 x i , j T x k , l = 0 for ( i , j ) ≠ ( k , l ) (i,j) \ne(k,l) ( i , j ) = ( k , l )
Let W ∈ R K ˉ × d W \in\mathbb{R}^{\bar{K}\times d} W ∈ R K ˉ × d be a weight matrix for a linear classifier, where the predicted class for input x x x is argmax k ( W k x ) \operatorname{argmax}_k(W_kx) argmax k ( W k x ) . For simplicity, we use the following notation to denote all the data points: X = [ x 1 , 1 , ⋯ , x K ˉ , n ] X=[x_{1,1},\cdots,x_{\bar{K},n}] X = [ x 1 , 1 , ⋯ , x K ˉ , n ] . We use the canonical ETF M K ˉ : = K ˉ K ˉ − 1 ( I K ˉ − 1 K ˉ 1 K ˉ T / K ˉ ) M_{\bar{K}}:=\sqrt{\frac{\bar{K}}{\bar{K}-1}}\left(I_{\bar{K}}-\mathbf{1}_{\bar{K}}\mathbf{1}_{\bar{K}}^{\mathsf{T}}/\bar{K}\right) M K ˉ := K ˉ − 1 K ˉ ( I K ˉ − 1 K ˉ 1 K ˉ T / K ˉ ) , corresponding to P = I K ˉ P=I_{\bar{K}} P = I K ˉ in Definition 2.1.
Now, we are ready to present the main result.
Theorem C.1 (Maximum Margin with Frobenius Norm). Under the above data assumptions, the largest normalized margin any nonzero linear classifier can achieve is:
γ ∗ = 1 n ( K ˉ − 1 ) (19) \gamma^*=\frac{1}{\sqrt{n(\bar{K}-1)}} \tag*{(19)} γ ∗ = n ( K ˉ − 1 ) 1 (19)
The unique unit-margin minimum-Frobenius-norm SVM solution is
W S V M = K ˉ − 1 K ˉ ( M K ˉ ⊗ 1 n T ) X T . W^{\mathrm{SVM}}=\sqrt{\frac{\bar{K}-1}{\bar{K}}}\left(M_{\bar{K}}\otimes\mathbf{1}_n^{\mathsf{T}}\right)X^{\mathsf{T}}. W SVM = K ˉ K ˉ − 1 ( M K ˉ ⊗ 1 n T ) X T .
The normalized margin is maximized precisely by W = c W S V M W=cW^{\mathrm{SVM}} W = c W SVM with c > 0 c>0 c > 0 .
Proof. We first solve the unit-margin minimum-norm SVM problem (3) with K ˉ \bar{K} K ˉ output classes. By Proposition G.1, its solution has the form W S V M = A ∗ X T W^{\mathrm{SVM}}=A^*X^{\mathsf{T}} W SVM = A ∗ X T for some A ∗ ∈ R K ˉ × K ˉ n A^*\in\mathbb{R}^{\bar{K}\times\bar{K}n} A ∗ ∈ R K ˉ × K ˉ n . Since X T X = I K ˉ n X^{\mathsf{T}}X=I_{\bar{K}n} X T X = I K ˉ n , the logits satisfy W S V M X = A ∗ W^{\mathrm{SVM}}X=A^* W SVM X = A ∗ and ∥ W S V M ∥ F = ∥ A ∗ ∥ F \lVert W^{\mathrm{SVM}}\rVert_F=\lVert A^*\rVert_F ∥ W SVM ∥ F = ∥ A ∗ ∥ F . Thus, A ∗ A^* A ∗ solves
A ∗ = arg min A ∈ R K ˉ × K ˉ n 1 2 ∥ A ∥ F 2 A^*=\underset{A\in\mathbb{R}^{\bar{K}\times\bar{K}n}}{\operatorname{arg\,min}}\frac{1}{2}\lVert A\rVert_F^2 A ∗ = A ∈ R K ˉ × K ˉ n arg min 2 1 ∥ A ∥ F 2
subject to
A i , ( i − 1 ) n + j − A k , ( i − 1 ) n + j ≥ 1 , i , k ∈ [ K ˉ ] , i ≠ k , j ∈ [ n ] . (20) A_{i,(i-1)n+j}-A_{k,(i-1)n+j}\ge1,\qquad i,k\in[\bar{K}],\ i\ne k,\ j\in[n]. \tag*{(20)} A i , ( i − 1 ) n + j − A k , ( i − 1 ) n + j ≥ 1 , i , k ∈ [ K ˉ ] , i = k , j ∈ [ n ] . (20)
This feasible problem has a strictly convex objective and a convex constraint set, so its solution is unique [7 ] . We use the permutation invariances of this fixed-scale problem to characterize A ∗ A^* A ∗ . We first decompose A ∗ A^* A ∗ into block matrices, i.e., A ∗ = [ A 1 ∗ ⋯ A K ˉ ∗ ] A^*=[A_1^*\ \cdots\ A_{\bar{K}}^*] A ∗ = [ A 1 ∗ ⋯ A K ˉ ∗ ] where A i ∗ ∈ R K ˉ × n A_i^*\in\mathbb{R}^{\bar{K}\times n} A i ∗ ∈ R K ˉ × n , i ∈ [ K ˉ ] i\in[\bar{K}] i ∈ [ K ˉ ] .
Invariance under column permutation within block matrices. Permuting columns within a class block preserves both the constraints and the objective in (20) . Thus, for any permutation matrices Π i ∈ R n × n \Pi_i\in\mathbb{R}^{n\times n} Π i ∈ R n × n , the matrix [ A 1 ∗ Π 1 ⋯ A K ˉ ∗ Π K ˉ ] [A_1^*\Pi_1\ \cdots\ A_{\bar{K}}^*\Pi_{\bar{K}}] [ A 1 ∗ Π 1 ⋯ A K ˉ ∗ Π K ˉ ] is also optimal. Uniqueness gives A i ∗ Π i = A i ∗ A_i^*\Pi_i=A_i^* A i ∗ Π i = A i ∗ for every Π i \Pi_i Π i , so all columns of each block are equal: A i ∗ = v i 1 n T A_i^*=v_i\mathbf{1}_n^{\mathsf{T}} A i ∗ = v i 1 n T for some v i ∈ R K ˉ v_i\in\mathbb{R}^{\bar{K}} v i ∈ R K ˉ .
Invariance under permutation across block matrices. Let V : = [ v 1 , … , v K ˉ ] V:=[v_1,\ldots,v_{\bar{K}}] V := [ v 1 , … , v K ˉ ] . Simultaneously permuting output rows and the corresponding class blocks preserves the constraints and objective in (20) . Uniqueness therefore gives Π V Π T = V \Pi V\Pi^{\mathsf{T}}=V Π V Π T = V for every permutation matrix Π ∈ R K ˉ × K ˉ \Pi\in\mathbb{R}^{\bar{K}\times\bar{K}} Π ∈ R K ˉ × K ˉ . By Lemma G.5, V = a I K ˉ − b 1 K ˉ 1 K ˉ T / K ˉ V=aI_{\bar{K}}-b\mathbf{1}_{\bar{K}}\mathbf{1}_{\bar{K}}^{\mathsf{T}}/\bar{K} V = a I K ˉ − b 1 K ˉ 1 K ˉ T / K ˉ for some a , b ∈ R a,b\in\mathbb{R} a , b ∈ R .
Scalar optimization. The unit-margin constraints reduce to a ≥ 1 a \ge1 a ≥ 1 , and
∥ A ∗ ∥ F 2 = n K ˉ ( a − b ) 2 + n K ˉ ( K ˉ − 1 ) b 2 = n [ ( K ˉ − 1 ) a 2 + K ˉ 2 ( b − a K ˉ ) 2 ] . \lVert A^* \rVert_F^2 = n\bar{K}(a-b)^2+n\bar{K}(\bar{K}-1)b^2
= n\left[(\bar{K}-1)a^2+\bar{K}^2\left(b-\frac{a}{\bar{K}}\right)^2\right]. ∥ A ∗ ∥ F 2 = n K ˉ ( a − b ) 2 + n K ˉ ( K ˉ − 1 ) b 2 = n [ ( K ˉ − 1 ) a 2 + K ˉ 2 ( b − K ˉ a ) 2 ] .
The unique minimum is attained at a = 1 a=1 a = 1 and b = 1 / K ˉ b=1/\bar{K} b = 1/ K ˉ . Hence,
W S V M = [ ( I K ˉ − 1 K ˉ 1 K ˉ 1 K ˉ T ) ⊗ 1 n T ] X T , ∥ W S V M ∥ F = n ( K ˉ − 1 ) . W^{\mathrm{SVM}}=\left[\left(I_{\bar{K}}-\frac{1}{\bar{K}}\mathbf{1}_{\bar{K}}\mathbf{1}_{\bar{K}}^{\mathsf{T}}\right)\otimes\mathbf{1}_{n}^{\mathsf{T}}\right]X^{\mathsf{T}},\qquad\lVert W^{\mathrm{SVM}}\rVert_F=\sqrt{n(\bar{K}-1)}. W SVM = [ ( I K ˉ − K ˉ 1 1 K ˉ 1 K ˉ T ) ⊗ 1 n T ] X T , ∥ W SVM ∥ F = n ( K ˉ − 1 ) .
which gives the stated ETF formula and has C ( W S V M ; D ) = 1 C(W^{\mathrm{SVM}};\mathcal{D})=1 C ( W SVM ; D ) = 1 .
Finally, any classifier W W W with C ( W ; D ) > 0 C(W;\mathcal{D})>0 C ( W ; D ) > 0 can be rescaled to the feasible unit-margin classifier W / C ( W ; D ) W/C(W;\mathcal{D}) W / C ( W ; D ) . Minimum-norm optimality yields
γ ( W ; D ) = 1 ∥ W / C ( W ; D ) ∥ F ≤ 1 ∥ W S V M ∥ F = 1 n ( K ˉ − 1 ) . \gamma(W;\mathcal{D})=\frac{1}{\lVert W/C(W;\mathcal{D})\rVert_F}\leq\frac{1}{\lVert W^{\mathrm{SVM}}\rVert_F}=\frac{1}{\sqrt{n(\bar{K}-1)}}. γ ( W ; D ) = ∥ W / C ( W ; D ) ∥ F 1 ≤ ∥ W SVM ∥ F 1 = n ( K ˉ − 1 ) 1 .
Equality holds precisely when W / C ( W ; D ) = W S V M W/C(W;\mathcal{D})=W^{\mathrm{SVM}} W / C ( W ; D ) = W SVM , by uniqueness of the unit-margin minimum-norm solution. Thus all normalized-margin maximizers are exactly the positive multiples of W S V M W^{\mathrm{SVM}} W SVM . Classifiers with nonpositive margin cannot attain this positive optimum.
Proof of Theorem 3.1
In this section, we first present the proof of Theorem 3.1. Then, under additional assumptions on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 , we show that the same conclusion still holds, with additional error terms that depend on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 .
Proof of Theorem 3.1
For the case 2 ≤ K ˉ < K 2\leq\bar{K}<K 2 ≤ K ˉ < K , we first define the following functions used in the proof of Theorem 3.1. The balanced case K ˉ = K \bar{K}=K K ˉ = K is treated separately below.
g N ( x ) : = ρ ( K − K ˉ ) K ˉ ( K − K ˉ ) x 2 + K K ˉ ( 1 − x ) 2 , g a ( x ) : = exp ( g N ( x ) ρ K n x 2 + K K − K ˉ ( 1 − x ) 2 [ x 2 + K K − K ˉ ( 1 − x ) 2 ] ) , g b ( x ) : = exp ( g N ( x ) ρ K n x 2 + K K − K ˉ ( 1 − x ) 2 [ − x 2 K ˉ − 1 + K K − K ˉ ( 1 − x ) 2 ] ) , g c ( x ) : = exp ( g N ( x ) ρ K n x 2 + K K − K ˉ ( 1 − x ) 2 [ − K ˉ K − K ˉ K K − K ˉ ( 1 − x ) 2 ] ) . (21) \begin{aligned}
g_N(x)&:=\sqrt{\frac{\rho(K-\bar{K})}{\bar{K}(K-\bar{K})x^2+K\bar{K}(1-x)^2}},\\
g_a(x)&:=\exp\left(\frac{g_N(x)\sqrt{\frac{\rho}{K n}}}{\sqrt{x^2+\frac{K}{K-\bar{K}}(1-x)^2}}\left[x^2+\sqrt{\frac{K}{K-\bar{K}}}(1-x)^2\right]\right),\\
g_b(x)&:=\exp\left(\frac{g_N(x)\sqrt{\frac{\rho}{K n}}}{\sqrt{x^2+\frac{K}{K-\bar{K}}(1-x)^2}}\left[-\frac{x^2}{\bar{K}-1}+\sqrt{\frac{K}{K-\bar{K}}}(1-x)^2\right]\right),\\
g_c(x)&:=\exp\left(\frac{g_N(x)\sqrt{\frac{\rho}{K n}}}{\sqrt{x^2+\frac{K}{K-\bar{K}}(1-x)^2}}\left[-\frac{\bar{K}}{K-\bar{K}}\sqrt{\frac{K}{K-\bar{K}}}(1-x)^2\right]\right). \tag*{(21)}
\end{aligned} g N ( x ) g a ( x ) g b ( x ) g c ( x ) := K ˉ ( K − K ˉ ) x 2 + K K ˉ ( 1 − x ) 2 ρ ( K − K ˉ ) , := exp x 2 + K − K ˉ K ( 1 − x ) 2 g N ( x ) K n ρ [ x 2 + K − K ˉ K ( 1 − x ) 2 ] , := exp x 2 + K − K ˉ K ( 1 − x ) 2 g N ( x ) K n ρ [ − K ˉ − 1 x 2 + K − K ˉ K ( 1 − x ) 2 ] , := exp x 2 + K − K ˉ K ( 1 − x ) 2 g N ( x ) K n ρ [ − K − K ˉ K ˉ K − K ˉ K ( 1 − x ) 2 ] . (21)
Moreover, let x 0 ∈ [ 0 , 1 ] x_0\in[0,1] x 0 ∈ [ 0 , 1 ] be the root of the equation
g b ( x ) g c ( x ) = 1 K K − K ˉ + 1 . \frac{g_b(x)}{g_c(x)}=\frac{1}{\sqrt{\frac{K}{K-\bar{K}}+1}}. g c ( x ) g b ( x ) = K − K ˉ K + 1 1 .
Remark D.1. Our proof technique follows Fang et al. [ [15 ] , and the readers can see the same definition of g a ( x ) g_a(x) g a ( x ) , g b ( x ) g_b(x) g b ( x ) , g c ( x ) g_c(x) g c ( x ) and x 0 x_0 x 0 in Fang et al. [ [15 ] . Moreover, the authors prove the existence of x 0 x_0 x 0 under certain conditions.
Now, we begin the proof of Theorem 3.1.
Theorem D.1 (Restatement of Theorem 3.1). Let ( B λ ∗ , A λ ∗ ) (B_\lambda^*, A_\lambda^*) ( B λ ∗ , A λ ∗ ) be any minimizer of (5). There exist scalar functions a λ a_\lambda a λ , b λ b_\lambda b λ , c λ c_\lambda c λ , Θ λ \Theta_\lambda Θ λ of λ \lambda λ such that for any global minimizer ( B λ ∗ , A λ ∗ ) (B_\lambda^*, A_\lambda^*) ( B λ ∗ , A λ ∗ ) of (5) with λ > 0 \lambda>0 λ > 0 , we have
B λ ∗ A λ ∗ = ( ( ( a λ + b λ ) I K ˉ − b λ 1 K ˉ 1 K ˉ T ) ⊗ 1 n T ) X f t T , ∥ B λ ∗ A λ ∗ ∥ F = Θ λ . (22) B_\lambda^* A_\lambda^* = \left(\left((a_\lambda+b_\lambda)\mathbf{I}_{\bar K}-b_\lambda\mathbf{1}_{\bar K}\mathbf{1}_{\bar K}^{\mathsf{T}}\right)\otimes\mathbf{1}_n^{\mathsf{T}}\right)X_{\mathrm{ft}}^{\mathsf{T}},\quad\lVert B_\lambda^*A_\lambda^*\rVert_F=\Theta_\lambda. \tag*{(22)} B λ ∗ A λ ∗ = ( ( ( a λ + b λ ) I K ˉ − b λ 1 K ˉ 1 K ˉ T ) ⊗ 1 n T ) X ft T , ∥ B λ ∗ A λ ∗ ∥ F = Θ λ . (22) When K ˉ ≥ 2 \bar K\ge2 K ˉ ≥ 2 , there exists a critical regularization weight λ c r i t ∈ [ 0 , 1 K n ) \lambda_{\mathrm{crit}}\in\left[0,\frac{1}{K\sqrt{n}}\right) λ crit ∈ [ 0 , K n 1 ) , with λ c r i t = 0 \lambda_{\mathrm{crit}}=0 λ crit = 0 if and only if K ˉ = K \bar K=K K ˉ = K , such that the scalar functions a λ a_\lambda a λ , b λ b_\lambda b λ , c λ c_\lambda c λ , Θ λ \Theta_\lambda Θ λ are characterized as follows:
(i) High-penalty regime ( λ ≥ 1 K n ) \left(\lambda\ge\frac{1}{K\sqrt{n}}\right) ( λ ≥ K n 1 ) : a λ = b λ = c λ = Θ λ = 0 a_\lambda=b_\lambda=c_\lambda=\Theta_\lambda=0 a λ = b λ = c λ = Θ λ = 0 , thus B λ ∗ A λ ∗ = 0 K × d B_\lambda^*A_\lambda^*=\mathbf{0}_{K\times d} B λ ∗ A λ ∗ = 0 K × d .
(ii) Intermediate regime ( λ c r i t < λ < 1 K n ) \left(\lambda_{\mathrm{crit}}<\lambda<\frac{1}{K\sqrt{n}}\right) ( λ crit < λ < K n 1 ) : a λ = Θ λ K ˉ − 1 K n a_\lambda=\frac{\Theta_\lambda\sqrt{\bar K-1}}{K\sqrt{n}} a λ = K n Θ λ K ˉ − 1 , b λ = Θ λ K n ( K ˉ − 1 ) b_\lambda=\frac{\Theta_\lambda}{K\sqrt{n(\bar K-1)}} b λ = K n ( K ˉ − 1 ) Θ λ , c λ = 0 c_\lambda=0 c λ = 0 , and Θ λ \Theta_\lambda Θ λ is the unique root of a nonlinear equation (see (29) in Appendix D), thus the minimizer is
B λ ∗ A λ ∗ = Θ λ K K ˉ n ( M K ˉ 0 ( K − K ˉ ) × K ˉ ) ⊗ 1 n T X f t T . (23) B_\lambda^*A_\lambda^*=\frac{\Theta_\lambda}{\sqrt{K\bar K n}}\left(\begin{array}{c}M_{\bar K}\\\mathbf{0}_{(K-\bar K)\times\bar K}\end{array}\right)\otimes\mathbf{1}_n^{\mathsf{T}}X_{\mathrm{ft}}^{\mathsf{T}}. \tag*{(23)} B λ ∗ A λ ∗ = K K ˉ n Θ λ ( M K ˉ 0 ( K − K ˉ ) × K ˉ ) ⊗ 1 n T X ft T . (23) (iii) Low-penalty regime ( 0 < λ ≤ λ c r i t ) (0<\lambda\le\lambda_{\mathrm{crit}}) ( 0 < λ ≤ λ crit ) : a λ a_\lambda a λ , b λ b_\lambda b λ , c λ c_\lambda c λ , Θ λ \Theta_\lambda Θ λ are non-negative.
Moreover, the small-regularization limit satisfies
lim λ → 0 + Θ λ = ∞ , lim λ → 0 + B λ ∗ A λ ∗ ∥ B λ ∗ A λ ∗ ∥ F = 1 K ˉ K ( M K ( K ˉ ) ⊗ 1 n T ) X f t T , (24) \lim_{\lambda\to0^+}\Theta_\lambda=\infty,\qquad\lim_{\lambda\to0^+}\frac{B_\lambda^*A_\lambda^*}{\lVert B_\lambda^*A_\lambda^*\rVert_F}=\sqrt{\frac{1}{\bar K K}}\left(M_K^{(\bar K)}\otimes\mathbf{1}_n^{\mathsf{T}}\right)X_{\mathrm{ft}}^{\mathsf{T}}, \tag*{(24)} λ → 0 + lim Θ λ = ∞ , λ → 0 + lim ∥ B λ ∗ A λ ∗ ∥ F B λ ∗ A λ ∗ = K ˉ K 1 ( M K ( K ˉ ) ⊗ 1 n T ) X ft T , (24) where M K ( K ˉ ) M_K^{(\bar K)} M K ( K ˉ ) is the first K ˉ \bar K K ˉ columns of the K K K -ETF matrix M K M_K M K .
Proof. We first show that one can simplify the objective in (5) based on the following Lemma.
Lemma D.1. Under Assumptions 2.1 and 2.4 with ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 , the original fine-tuning problem in (5) is equivalent to the following symmetric formulation:
min B , A ~ L ( B , A ~ ) : = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , B A ~ e ~ ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ~ ∥ F 2 ) , (25) \min_{B,\widetilde A}\mathcal{L}(B,\widetilde A):=\frac{1}{\bar K n}\sum_{c=1}^{\bar K}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,B\widetilde A\widetilde e_{(c-1)n+j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert\widetilde A\rVert_F^2\right), \tag*{(25)} B , A min L ( B , A ) := K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , B A e ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) , (25) where X f t = [ x ˉ 1 , 1 , … , x ˉ 1 , n , … , x ˉ K ˉ , 1 , … , x ˉ K ˉ , n ] X_{\mathrm{ft}}=[\bar x_{1,1},\ldots,\bar x_{1,n},\ldots,\bar x_{\bar K,1},\ldots,\bar x_{\bar K,n}] X ft = [ x ˉ 1 , 1 , … , x ˉ 1 , n , … , x ˉ K ˉ , 1 , … , x ˉ K ˉ , n ] , B ∈ R K × r B\in\mathbb{R}^{K\times r} B ∈ R K × r , A ~ = A X f t ∈ R r × K ˉ n \widetilde A=AX_{\mathrm{ft}}\in\mathbb{R}^{r\times\bar K n} A = A X ft ∈ R r × K ˉ n .
Let X f t , ⊥ X_{\mathrm{ft},\perp} X ft , ⊥ be an orthonormal complement of X f t X_{\mathrm{ft}} X ft . Then:
If ( B ∗ , A ∗ ) (B^*,A^*) ( B ∗ , A ∗ ) is any global minimizer of (5), then ( B ∗ , A ~ ∗ ) (B^*,\widetilde A^*) ( B ∗ , A ∗ ) with A ~ ∗ = A ∗ X f t \widetilde A^*=A^*X_{\mathrm{ft}} A ∗ = A ∗ X ft is a global minimizer of (25).
Conversely, if ( B ∗ , A ~ ∗ ) (B^*,\widetilde A^*) ( B ∗ , A ∗ ) is any global minimizer of (25), then ( B ∗ , A ~ ∗ X f t T ) (B^*,\widetilde A^*X_{\mathrm{ft}}^{\mathsf{T}}) ( B ∗ , A ∗ X ft T ) yields a global minimizer of the original problem in (5).
We refer the readers to Appendix G.2 for the proof of Lemma D.1. Lemma D.1 indicates that to characterize the global minimizer of the objective in (5), one can instead characterize the global minimizer of the objective in (25).
We start with the following equations that characterize the global minimizer of (25)
∂ L ( B , A ~ ) ∂ B = λ B + 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n ∂ ∂ B A ~ ( L C E ( y c , B A ~ e ~ ( c − 1 ) n + j ) ) e ~ ( c − 1 ) n + j T A ~ T = 0 \frac{\partial\mathcal{L}(B,\widetilde A)}{\partial B}
=\lambda B+\frac{1}{\bar K n}\sum_{c=1}^{\bar K}\sum_{j=1}^{n}\frac{\partial}{\partial B\widetilde A}\left(\mathcal{L}_{\mathrm{CE}}\left(y_c,B\widetilde A\widetilde e_{(c-1)n+j}\right)\right)\widetilde e_{(c-1)n+j}^{\mathsf{T}}\widetilde A^{\mathsf{T}}=0 ∂ B ∂ L ( B , A ) = λ B + K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n ∂ B A ∂ ( L CE ( y c , B A e ( c − 1 ) n + j ) ) e ( c − 1 ) n + j T A T = 0
∂ L ( B , A ~ ) ∂ A ~ = λ A ~ + B T 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n ∂ ∂ B A ~ ( L C E ( y c , B A ~ e ~ ( c − 1 ) n + j ) ) e ~ ( c − 1 ) n + j T = 0. \frac{\partial\mathcal{L}(B,\widetilde A)}{\partial\widetilde A}
=\lambda\widetilde A+B^{\mathsf{T}}\frac{1}{\bar K n}\sum_{c=1}^{\bar K}\sum_{j=1}^{n}\frac{\partial}{\partial B\widetilde A}\left(\mathcal{L}_{\mathrm{CE}}\left(y_c,B\widetilde A\widetilde e_{(c-1)n+j}\right)\right)\widetilde e_{(c-1)n+j}^{\mathsf{T}}=0. ∂ A ∂ L ( B , A ) = λ A + B T K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n ∂ B A ∂ ( L CE ( y c , B A e ( c − 1 ) n + j ) ) e ( c − 1 ) n + j T = 0.
Based on the above equation, we can conclude that any minimizer ( B , A ~ ) (B,\widetilde{A}) ( B , A ) of (25) must satisfy
λ B T B = λ A ~ A ~ T ⇒ ∥ B ∥ F 2 = ∥ A ~ ∥ F 2 . (26) \lambda B^{\mathsf T}B=\lambda\widetilde{A}\widetilde{A}^{\mathsf T}\Rightarrow\|B\|_{F}^{2}=\|\widetilde{A}\|_{F}^{2}. \tag*{(26)} λ B T B = λ A A T ⇒ ∥ B ∥ F 2 = ∥ A ∥ F 2 . (26)
Our second step is to converge the problem into a constraint optimization problem. The following lemma characterizes this.
Lemma D.2. Let B ∈ R K × r B\in\mathbb{R}^{K\times r} B ∈ R K × r , A ~ ∈ R r × K ˉ n \widetilde{A}\in\mathbb{R}^{r\times\bar{K}n} A ∈ R r × K ˉ n , and assume r ≥ K ˉ r\geq\bar{K} r ≥ K ˉ as in Assumption 2.3. For each λ > 0 \lambda>0 λ > 0 , there exists a unique non-negative value ρ λ \rho_{\lambda} ρ λ such that the solution set for the following optimization problems are equivalent
min B , A ~ 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , B A ~ ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ~ ∥ F 2 ) , \min_{B,\widetilde{A}}\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,B\widetilde{A}_{(c-1)n+j}\right)+\frac{\lambda}{2}\left(\|B\|_{F}^{2}+\|\widetilde{A}\|_{F}^{2}\right), B , A min K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , B A ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) , (Problem one)
min B , A ~ 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , B A ~ ( c − 1 ) n + j ) s.t. ∥ B ∥ F 2 ≤ ρ λ , ∥ A ~ ∥ F 2 ≤ ρ λ . \min_{B,\widetilde{A}}\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,B\widetilde{A}_{(c-1)n+j}\right)\quad\text{s.t.}\ \|B\|_{F}^{2}\leq\rho_{\lambda},\ \|\widetilde{A}\|_{F}^{2}\leq\rho_{\lambda}. B , A min K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , B A ( c − 1 ) n + j ) s.t. ∥ B ∥ F 2 ≤ ρ λ , ∥ A ∥ F 2 ≤ ρ λ . (Problem two)
Moreover, the map λ ↦ ρ λ \lambda\mapsto\rho_{\lambda} λ ↦ ρ λ enjoys the following properties:
(i) Monotonicity and continuity. ρ λ \rho_{\lambda} ρ λ is continuous and non-increasing on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) .
(ii) Asymptotic behaviour.
lim λ → 0 + ρ λ = ∞ , lim λ → ∞ ρ λ = 0. \lim_{\lambda\to0^{+}}\rho_{\lambda}=\infty,\qquad\lim_{\lambda\to\infty}\rho_{\lambda}=0. λ → 0 + lim ρ λ = ∞ , λ → ∞ lim ρ λ = 0. (iii) Flat regions. For any 0 < λ 1 < λ 2 0<\lambda_{1}<\lambda_{2} 0 < λ 1 < λ 2 , ρ ( λ 1 ) = ρ ( λ 2 ) ⟺ ρ ( λ 1 ) = ρ ( λ 2 ) = 0 \rho(\lambda_{1})=\rho(\lambda_{2})\Longleftrightarrow\rho(\lambda_{1})=\rho(\lambda_{2})=0 ρ ( λ 1 ) = ρ ( λ 2 ) ⟺ ρ ( λ 1 ) = ρ ( λ 2 ) = 0 .
We refer the readers to Appendix G.3 for the proof.
Balanced case K ˉ = K ≥ 2 \bar{K}=K\geq2 K ˉ = K ≥ 2 . By Lemmas D.1 and D.2 , it suffices to solve the convex problem (Problem three). Let ℓ ( Z ) \ell(Z) ℓ ( Z ) denote its mean cross-entropy term and set C : = ( I K − 1 K 1 K T / K ) ⊗ 1 n T C:=(I_K-\mathbf{1}_K\mathbf{1}_K^{\mathsf T}/K)\otimes\mathbf{1}_n^{\mathsf T} C := ( I K − 1 K 1 K T / K ) ⊗ 1 n T . The K − 1 K-1 K − 1 nonzero singular values of C C C equal n \sqrt{n} n , so for t > 0 t>0 t > 0 ,
∇ ℓ ( t C ) = − C n ( e t + K − 1 ) , C n ∈ ∂ ∥ t C ∥ ∗ . \nabla\ell(tC)=-\frac{C}{n(e^t+K-1)},\qquad\frac{C}{\sqrt{n}}\in\partial\|tC\|_{*}. ∇ ℓ ( tC ) = − n ( e t + K − 1 ) C , n C ∈ ∂ ∥ tC ∥ ∗ .
Thus the global optimality condition 0 ∈ ∇ ℓ ( Z ) + λ ∂ ∥ Z ∥ ∗ 0\in\nabla\ell(Z)+\lambda\partial\|Z\|_{*} 0 ∈ ∇ ℓ ( Z ) + λ ∂ ∥ Z ∥ ∗ holds at Z = t C Z=tC Z = tC exactly when e t + K − 1 = 1 / ( λ n ) e^t+K-1=1/(\lambda\sqrt{n}) e t + K − 1 = 1/ ( λ n ) . At zero, ∥ ∇ ℓ ( 0 ) ∥ o p = 1 / ( K n ) \|\nabla\ell(0)\|_{\mathrm{op}}=1/(K\sqrt{n}) ∥∇ ℓ ( 0 ) ∥ op = 1/ ( K n ) , so zero is optimal exactly when λ ≥ 1 / ( K n ) \lambda\geq1/(K\sqrt{n}) λ ≥ 1/ ( K n ) . Uniqueness (Lemma G.6) therefore gives
Z λ ∗ = t λ C , t λ = { log ( 1 λ n − ( K − 1 ) ) , 0 < λ < 1 K n , 0 , λ ≥ 1 K n . Z_{\lambda}^{*}=t_{\lambda}C,\qquad
t_{\lambda}=
\begin{cases}
\log\left(\frac{1}{\lambda\sqrt{n}}-(K-1)\right), & 0<\lambda<\frac{1}{K\sqrt{n}},\\
0, & \lambda\geq\frac{1}{K\sqrt{n}}.
\end{cases} Z λ ∗ = t λ C , t λ = ⎩ ⎨ ⎧ log ( λ n 1 − ( K − 1 ) ) , 0 , 0 < λ < K n 1 , λ ≥ K n 1 .
Since rank ( Z λ ∗ ) ≤ K − 1 ≤ r \operatorname{rank}(Z_{\lambda}^{*})\leq K-1\leq r rank ( Z λ ∗ ) ≤ K − 1 ≤ r , this solution is feasible for the factorized problem. Writing A ˉ λ ∗ = A λ ∗ X f t \bar{A}_{\lambda}^{*}=A_{\lambda}^{*}Xf_t A ˉ λ ∗ = A λ ∗ X f t , every global minimizer satisfies B λ ∗ A ˉ λ ∗ = Z λ ∗ B_{\lambda}^{*}\bar{A}_{\lambda}^{*}=Z_{\lambda}^{*} B λ ∗ A ˉ λ ∗ = Z λ ∗ and B λ ∗ A λ ∗ = t λ C X T f t B_{\lambda}^{*}A_{\lambda}^{*}=t_{\lambda}CX^{\mathsf T}f_t B λ ∗ A λ ∗ = t λ C X T f t . Consequently, ρ λ = n ( K − 1 ) t λ \rho_{\lambda}=\sqrt{n}(K-1)t_{\lambda} ρ λ = n ( K − 1 ) t λ and Θ λ = n ( K − 1 ) t λ \Theta_{\lambda}=\sqrt{n}(K-1)t_{\lambda} Θ λ = n ( K − 1 ) t λ . These formulas prove the intermediate ETF form and both limits in (9). We set λ c r i t = 0 \lambda_{\mathrm{crit}}=0 λ crit = 0 and ρ c r i t = + ∞ \rho_{\mathrm{crit}}=+\infty ρ crit = + ∞ as limiting boundaries; no minimizer at λ = 0 \lambda=0 λ = 0 is asserted.
For the remainder of this strict-orthogonal proof, assume 2 ≤ K < K ˉ 2\leq K<\bar{K} 2 ≤ K < K ˉ . For simplicity, we will use ρ \rho ρ to denote ρ λ \rho_{\lambda} ρ λ for convenience.
Characterization to solution of (Problem two). (Problem two) can be viewed as a special case of the neural collapse phenomenon in the extreme setting of an imbalanced dataset: for the first K ˉ \bar{K} K ˉ classes we observe n n n data points per class, while all remaining classes have zero samples. In this context, we directly invoke the following result from Fang et al. [ [15 ] . Fang et al. [ [15 ] (see p. 26 in . Fang et al. [ [15 ] ).
Lemma D.3 (Neural collapse under extreme imbalance; [15 Page 26] ). Assume 2 ≤ K ˉ < K 2 \le\bar{K} < K 2 ≤ K ˉ < K . For any constants c 1 , c 2 , c 3 , ρ > 0 c_1,c_2,c_3,\rho> 0 c 1 , c 2 , c 3 , ρ > 0 , define
c 1 ′ = c 1 c 1 + ( K ˉ − 1 ) c 2 + ( K − K ˉ ) c 3 c'_1 = \frac{c_1}{c_1 + (\bar{K}-1)c_2 + (K-\bar{K})c_3} c 1 ′ = c 1 + ( K ˉ − 1 ) c 2 + ( K − K ˉ ) c 3 c 1 c 2 ′ = c 2 c 1 + ( K ˉ − 1 ) c 2 + ( K − K ˉ ) c 3 c'_2 = \frac{c_2}{c_1 + (\bar{K}-1)c_2 + (K-\bar{K})c_3} c 2 ′ = c 1 + ( K ˉ − 1 ) c 2 + ( K − K ˉ ) c 3 c 2 c 3 ′ = c 3 c 1 + ( K ˉ − 1 ) c 2 + ( K − K ˉ ) c 3 c'_3 = \frac{c_3}{c_1 + (\bar{K}-1)c_2 + (K-\bar{K})c_3} c 3 ′ = c 1 + ( K ˉ − 1 ) c 2 + ( K − K ˉ ) c 3 c 3 c 4 = − c 1 ′ log c 1 ′ − c 2 ′ ( K ˉ − 1 ) log c 2 ′ − ( K − K ˉ ) c 3 ′ log ( c 3 ′ ) c_4 = -c'_1 \log c'_1 - c'_2(\bar{K}-1)\log c'_2 - (K-\bar{K})c'_3\log(c'_3) c 4 = − c 1 ′ log c 1 ′ − c 2 ′ ( K ˉ − 1 ) log c 2 ′ − ( K − K ˉ ) c 3 ′ log ( c 3 ′ ) c 5 = K ˉ c 2 K ˉ c 2 + ( K − K ˉ ) c 3 c_5 = \frac{\bar{K}c_2}{\bar{K}c_2 + (K-\bar{K})c_3} c 5 = K ˉ c 2 + ( K − K ˉ ) c 3 K ˉ c 2 c 6 = ( K − K ˉ ) c 3 K ˉ c 2 + ( K − K ˉ ) c 3 c_6 = \frac{(K-\bar{K})c_3}{\bar{K}c_2 + (K-\bar{K})c_3} c 6 = K ˉ c 2 + ( K − K ˉ ) c 3 ( K − K ˉ ) c 3 c 7 = K ˉ c 2 + ( K − K ˉ ) c 3 c 1 + ( K ˉ − 1 ) c 2 + ( K − K ˉ ) c 3 c_7 = \frac{\bar{K}c_2 + (K-\bar{K})c_3}{c_1 + (\bar{K}-1)c_2 + (K-\bar{K})c_3} c 7 = c 1 + ( K ˉ − 1 ) c 2 + ( K − K ˉ ) c 3 K ˉ c 2 + ( K − K ˉ ) c 3 For any feasible solution ( B , A ~ ) (B,\tilde{A}) ( B , A ~ ) of (Problem two), the objective value of (Problem two) can be simplified to
1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , B A ~ e ( c − 1 ) n + j ) ≥ c 7 K ˉ ρ η ∑ i = 1 K ˉ ∥ c 5 w 1 + c 6 w 2 − b i ∥ 2 + c 4 \frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}(y_c,B\tilde{A}e_{(c-1)n+j}) \geq\frac{c_7}{\bar{K}}\sqrt{\frac{\rho}{\eta}}\sqrt{\sum_{i=1}^{\bar{K}}\lVert c_5w_1+c_6w_2-b_i\rVert^2+c_4} K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , B A ~ e ( c − 1 ) n + j ) ≥ K ˉ c 7 η ρ i = 1 ∑ K ˉ ∥ c 5 w 1 + c 6 w 2 − b i ∥ 2 + c 4 where w 1 = 1 K ˉ ∑ i = 1 K ˉ b i w_1=\frac{1}{\bar{K}}\sum_{i=1}^{\bar{K}}b_i w 1 = K ˉ 1 ∑ i = 1 K ˉ b i , w 2 = 1 K − K ˉ ∑ i = K ˉ + 1 K b i w_2=\frac{1}{K-\bar{K}}\sum_{i=\bar{K}+1}^{K}b_i w 2 = K − K ˉ 1 ∑ i = K ˉ + 1 K b i , and ≥ \geq ≥ becomes equality under certain choices of c 1 , c 2 , c 3 c_1,c_2,c_3 c 1 , c 2 , c 3 .
Readers are referred to Fang et al. [ [15 ] for the full proof. Note the following distinction between Lemma D.3 and its analogue in Fang et al. [ [15 ] : the inequality in Fang et al. [ [15 ] becomes tight only when all minority-class features are zero and certain choices of c 1 , c 2 , c 3 c_1,c_2,c_3 c 1 , c 2 , c 3 , whereas in our lemma, we do not require that all minority-class features are zero. This difference is resolved by our change of variables A ~ = A X f t \tilde{A}=AX_{\mathrm{ft}} A ~ = A X ft , which effectively confines A ~ \tilde{A} A ~ to the subspace of observed (majority-class) features and thus guarantees equality in Lemma D.3.
Now, we carefull pick the B B B , A ~ \tilde{A} A ~ and c 1 , c 2 , c 3 c_1,c_2,c_3 c 1 , c 2 , c 3 to achieve the global minimum of the objective in (Problem two). The following lemma characterizes this exactly.
Lemma D.4 (Lemma 5 in Fang et al. [ [15 ] ). Under the assumptions of Theorem 3.1, with 2 ≤ K ˉ < K 2 \leq\bar{K} < K 2 ≤ K ˉ < K , define
ρ c r i t = n K ˉ ( K ˉ − 1 ) log ( K K − K ˉ + 1 ) \rho_{\mathrm{crit}}=\sqrt{n}\bar{K}(\bar{K}-1)\log\left(\sqrt{\frac{K}{K-\bar{K}}}+1\right) ρ crit = n K ˉ ( K ˉ − 1 ) log ( K − K ˉ K + 1 ) such that the optimal value to (Problem two) is as follows
when K ˉ ≥ 2 \bar{K}\geq2 K ˉ ≥ 2 and ρ λ < ρ c r i t \rho_\lambda<\rho_{\mathrm{crit}} ρ λ < ρ crit , we choose c 1 = exp ( ρ λ K n ) c_1=\exp\left(\frac{\rho_\lambda}{K\sqrt{n}}\right) c 1 = exp ( K n ρ λ ) , c 2 = exp ( − ρ λ n K ˉ ( K ˉ − 1 ) ) c_2=\exp\left(-\frac{\rho_\lambda}{\sqrt{n}\bar{K}(\bar{K}-1)}\right) c 2 = exp ( − n K ˉ ( K ˉ − 1 ) ρ λ ) , c 3 = 1 c_3=1 c 3 = 1 , and (Problem two) attain its minimum
− ρ λ n K ˉ + log ( K − K ˉ + ( K ˉ − 1 ) exp ( − ρ λ n K ˉ ( K ˉ − 1 ) ) + exp ( ρ λ n K ˉ ) ) , -\frac{\rho_\lambda}{\sqrt{n}\bar{K}}+\log\left(K-\bar{K}+(\bar{K}-1)\exp\left(-\frac{\rho_\lambda}{\sqrt{n}\bar{K}(\bar{K}-1)}\right)+\exp\left(\frac{\rho_\lambda}{\sqrt{n}\bar{K}}\right)\right), − n K ˉ ρ λ + log ( K − K ˉ + ( K ˉ − 1 ) exp ( − n K ˉ ( K ˉ − 1 ) ρ λ ) + exp ( n K ˉ ρ λ ) ) , where B ∗ , A ~ ∗ B^*,\tilde{A}^* B ∗ , A ~ ∗ takes the following form
[ b 1 ∗ , b 2 ∗ , … , b K ˉ ∗ ] = n [ a ˉ 1 ∗ , a ˉ 2 ∗ , … , a ˉ K ˉ ∗ ] = ρ K ˉ P M K ˉ , [b_1^*,b_2^*,\ldots,b_{\bar{K}}^*]=\sqrt{n}[\bar{a}_1^*,\bar{a}_2^*,\ldots,\bar{a}_{\bar{K}}^*]=\sqrt{\frac{\rho}{\bar{K}}}PM_{\bar{K}}, [ b 1 ∗ , b 2 ∗ , … , b K ˉ ∗ ] = n [ a ˉ 1 ∗ , a ˉ 2 ∗ , … , a ˉ K ˉ ∗ ] = K ˉ ρ P M K ˉ , a i ∗ = a ˉ ⌈ i / n ⌉ ∗ , i = 1 , … , K ˉ n , a_i^*=\bar{a}_{\lceil i/n\rceil}^*,\qquad i=1,\ldots,\bar{K}n, a i ∗ = a ˉ ⌈ i / n ⌉ ∗ , i = 1 , … , K ˉ n , b j ∗ = 0 , j > K ˉ . b_j^*=0,\qquad j>\bar{K}. b j ∗ = 0 , j > K ˉ .
Here P ∈ R r × K ˉ P \in\mathbb{R}^{r\times\bar{K}} P ∈ R r × K ˉ satisfies P ⊤ P = I K ˉ P^{\top}P=I_{\bar{K}} P ⊤ P = I K ˉ , a i ∗ a_i^* a i ∗ is the i i i -th column of A ~ ∗ \widetilde{A}^* A ∗ , and a ˉ c ∗ \bar{a}_c^* a ˉ c ∗ is the common column for samples in class c c c . Writing A ˉ ∗ : = A ~ ∗ = A ∗ X f t \bar{A}^* := \widetilde{A}^* = A^*X_{\mathrm{ft}} A ˉ ∗ := A ∗ = A ∗ X ft for the data-coordinate factor, the product takes the following form
B ∗ A ˉ ∗ = ρ λ K n ( K ˉ − 1 ) ( M K ˉ 0 ( K − K ˉ ) × K ˉ ) ⊗ 1 n ⊤ . (27) B^*\bar{A}^*=\frac{\rho_\lambda}{\sqrt{K n(\bar{K}-1)}}\left(\begin{array}{c}M_{\bar{K}}\\0_{(K-\bar{K})\times\bar{K}}\end{array}\right)\otimes1_n^{\top}. \tag*{(27)} B ∗ A ˉ ∗ = K n ( K ˉ − 1 ) ρ λ ( M K ˉ 0 ( K − K ˉ ) × K ˉ ) ⊗ 1 n ⊤ . (27)
c 1 = g a ( x 0 ) , c 2 = g b ( x 0 ) , c 3 = g c ( x 0 ) , c_1=g_a(x_0),\qquad c_2=g_b(x_0),\qquad c_3=g_c(x_0), c 1 = g a ( x 0 ) , c 2 = g b ( x 0 ) , c 3 = g c ( x 0 ) ,
where g a ( x ) g_a(x) g a ( x ) , g b ( x ) g_b(x) g b ( x ) , g c ( x ) g_c(x) g c ( x ) and x 0 x_0 x 0 are defined in (21). Then, (Problem two) attain its minimum
log ( g a ( x 0 ) + ( K ˉ − 1 ) g b ( x 0 ) + ( K − K ˉ ) g c ( x 0 ) g a ( x 0 ) ) , \log\left(\frac{g_a(x_0)+(\bar{K}-1)g_b(x_0)+(K-\bar{K})g_c(x_0)}{g_a(x_0)}\right), log ( g a ( x 0 ) g a ( x 0 ) + ( K ˉ − 1 ) g b ( x 0 ) + ( K − K ˉ ) g c ( x 0 ) ) ,
where
b i ∗ = { g N ( x 0 ) P [ K ˉ x 0 K ˉ ( K ˉ − 1 ) y ˉ i + ( 1 − x 0 K ˉ − x 0 K ˉ ( K ˉ − 1 ) ) 1 K ˉ ] , i ≤ K ˉ , − g N ( x 0 ) K ˉ ( 1 − x 0 ) K − K ˉ P 1 K ˉ , i > K ˉ . b_i^*=
\begin{cases}
g_N(x_0)P\left[\dfrac{\bar{K}x_0}{\sqrt{\bar{K}(\bar{K}-1)}}\bar{y}_i+\left(\dfrac{1-x_0}{\sqrt{\bar{K}}}-\dfrac{x_0}{\sqrt{\bar{K}(\bar{K}-1)}}\right)1_{\bar{K}}\right], & i\le\bar{K},\\
-\dfrac{g_N(x_0)\sqrt{\bar{K}}(1-x_0)}{K-\bar{K}}P1_{\bar{K}}, & i>\bar{K}.
\end{cases} b i ∗ = ⎩ ⎨ ⎧ g N ( x 0 ) P [ K ˉ ( K ˉ − 1 ) K ˉ x 0 y ˉ i + ( K ˉ 1 − x 0 − K ˉ ( K ˉ − 1 ) x 0 ) 1 K ˉ ] , − K − K ˉ g N ( x 0 ) K ˉ ( 1 − x 0 ) P 1 K ˉ , i ≤ K ˉ , i > K ˉ .
a ˉ c ∗ = g N ( x 0 ) n P [ x 0 K ˉ K ˉ − 1 y ˉ c + ( ( 1 − x 0 ) K / K ˉ K ˉ − x 0 K ˉ ( K ˉ − 1 ) ) 1 K ˉ ] , \bar{a}_c^*=\frac{g_N(x_0)}{\sqrt{n}}P\left[x_0\sqrt{\frac{\bar{K}}{\bar{K}-1}}\bar{y}_c+\left(\frac{(1-x_0)\sqrt{K/\bar{K}}}{\sqrt{\bar{K}}}-\frac{x_0}{\sqrt{\bar{K}(\bar{K}-1)}}\right)1_{\bar{K}}\right], a ˉ c ∗ = n g N ( x 0 ) P x 0 K ˉ − 1 K ˉ y ˉ c + ( K ˉ ( 1 − x 0 ) K / K ˉ − K ˉ ( K ˉ − 1 ) x 0 ) 1 K ˉ ,
where y ˉ c ∈ R K ˉ \bar{y}_c\in\mathbb{R}^{\bar{K}} y ˉ c ∈ R K ˉ is a one-hot vector with the c c c -th entry equal to one (and y ˉ i \bar{y}_i y ˉ i is defined analogously with the i i i -th entry equal to one), and P ∈ R r × K ˉ P\in\mathbb{R}^{r\times\bar{K}} P ∈ R r × K ˉ is a partial orthogonal matrix such that P ⊤ P = I K ˉ P^{\top}P=I_{\bar{K}} P ⊤ P = I K ˉ . The sample columns satisfy a ( c − 1 ) n + j ∗ = a ˉ c ∗ a^*_{(c-1)n+j}=\bar{a}_c^* a ( c − 1 ) n + j ∗ = a ˉ c ∗ for c ∈ [ K ˉ ] c\in[\bar{K}] c ∈ [ K ˉ ] and j ∈ [ n ] j\in[n] j ∈ [ n ] . Moreover, B ∗ A ~ ∗ B^*\widetilde{A}^* B ∗ A ∗ takes the following form
B ∗ A ~ ∗ = ( ( log g a ( x 0 ) − log g b ( x 0 ) ) I K ˉ + log g b ( x 0 ) 1 K ˉ 1 K ˉ ⊤ log g c ( x 0 ) 1 K − K ˉ 1 K ˉ ⊤ ) ⊗ 1 n ⊤ . B^*\widetilde{A}^*=
\left(\begin{array}{c}
(\log g_a(x_0)-\log g_b(x_0))I_{\bar{K}}+\log g_b(x_0)1_{\bar{K}}1_{\bar{K}}^{\top}\\
\log g_c(x_0)1_{K-\bar{K}}1_{\bar{K}}^{\top}
\end{array}\right)\otimes1_n^{\top}. B ∗ A ∗ = ( ( log g a ( x 0 ) − log g b ( x 0 )) I K ˉ + log g b ( x 0 ) 1 K ˉ 1 K ˉ ⊤ log g c ( x 0 ) 1 K − K ˉ 1 K ˉ ⊤ ) ⊗ 1 n ⊤ .
Lemma D.4 is Lemma 5 in Fang et al. [ [15 ] . We refer the readers to Fang et al. [ [15 ] for the proof. Lemma D.4 exactly characterize the optimal solution and value of the objective in (Problem two).
Computation of the product of LoRA adapters. Based on Lemma D.4, we can characterize the solution to the LoRA objective in (5), then it suffices to compute the product of the minimizer to prove Theorem 3.1.
In the Intermediate regime , restricting B ∗ B^* B ∗ to its first K ˉ \bar{K} K ˉ rows gives
( B ∗ ) 1 : K ˉ , : [ a ˉ 1 ∗ , a ˉ 2 ∗ , … , a ˉ K ˉ ∗ ] = ρ λ K ˉ n M K ˉ P ⊤ P M K ˉ = ρ λ K ˉ n K ˉ K ˉ − 1 ( I K ˉ − 1 K ˉ 1 K ˉ 1 K ˉ ⊤ ) = ρ λ K n ( K − 1 ) M K ˉ , \begin{aligned}
\left(B^*\right)_{1:\bar{K},:}\left[\bar{a}_1^*,\bar{a}_2^*,\ldots,\bar{a}_{\bar{K}}^*\right]
=\frac{\rho_\lambda}{\bar{K}\sqrt{n}}M_{\bar{K}}P^{\top}PM_{\bar{K}} \\
&=\frac{\rho_\lambda}{\bar{K}\sqrt{n}}\frac{\bar{K}}{\bar{K}-1}\left(I_{\bar{K}}-\frac{1}{\bar{K}}1_{\bar{K}}1_{\bar{K}}^{\top}\right) \\
&=\frac{\rho_\lambda}{\sqrt{K n(K-1)}}M_{\bar{K}},
\end{aligned} ( B ∗ ) 1 : K ˉ , : [ a ˉ 1 ∗ , a ˉ 2 ∗ , … , a ˉ K ˉ ∗ ] = K ˉ n ρ λ M K ˉ P ⊤ P M K ˉ = K ˉ n ρ λ K ˉ − 1 K ˉ ( I K ˉ − K ˉ 1 1 K ˉ 1 K ˉ ⊤ ) = K n ( K − 1 ) ρ λ M K ˉ ,
and it leads to the product of final solution
B ∗ A ~ ∗ = ρ λ K n ( K − 1 ) ( M K ˉ 0 ( K − K ˉ ) × K ˉ ) ⊗ 1 n ⊤ . (28) B^*\widetilde{A}^*=\frac{\rho_\lambda}{\sqrt{K n(K-1)}}\left(\begin{array}{c}M_{\bar{K}}\\0_{(K-\bar{K})\times\bar{K}}\end{array}\right)\otimes1_n^{\top}. \tag*{(28)} B ∗ A ∗ = K n ( K − 1 ) ρ λ ( M K ˉ 0 ( K − K ˉ ) × K ˉ ) ⊗ 1 n ⊤ . (28)
In the Low penalty regime , in Fang et al. [ [15 ] , the authors further show that
( b i ∗ ) ⊤ a ˉ j ∗ = { log g a ( x 0 ) , i = j ≤ K ˉ , log g b ( x 0 ) , i ≠ j , i ≤ K ˉ , log g c ( x 0 ) , K ˉ < i ≤ K , i ∈ [ K ] , j ∈ [ K ˉ ] . (b_i^*)^{\top}\bar{a}_j^*=
\begin{cases}
\log g_a(x_0), & i=j\le\bar{K},\\
\log g_b(x_0), & i\ne j,\ i\le\bar{K},\\
\log g_c(x_0), & \bar{K}<i\le K,
\end{cases}
\qquad i\in[K],\ j\in[\bar{K}]. ( b i ∗ ) ⊤ a ˉ j ∗ = ⎩ ⎨ ⎧ log g a ( x 0 ) , log g b ( x 0 ) , log g c ( x 0 ) , i = j ≤ K ˉ , i = j , i ≤ K ˉ , K ˉ < i ≤ K , i ∈ [ K ] , j ∈ [ K ˉ ] .
Thus, one can show that
B ∗ A ~ ∗ = ( [ ( log g a ( x 0 ) − log g b ( x 0 ) ) I K ˉ + log g b ( x 0 ) 1 K ˉ 1 K ˉ T log g c ( x 0 ) 1 K − K ˉ 1 K ˉ T ] ⊗ 1 n T ) . B^* \widetilde{A}^* =
\left(
\begin{bmatrix}
(\log g_a(x_0)-\log g_b(x_0))I_{\bar K}+\log g_b(x_0)1_{\bar K}1_{\bar K}^{\mathsf T}\\
\log g_c(x_0)1_{K-\bar K}1_{\bar K}^{\mathsf T}
\end{bmatrix}
\otimes1_n^{\mathsf T}
\right). B ∗ A ∗ = ( [ ( log g a ( x 0 ) − log g b ( x 0 )) I K ˉ + log g b ( x 0 ) 1 K ˉ 1 K ˉ T log g c ( x 0 ) 1 K − K ˉ 1 K ˉ T ] ⊗ 1 n T ) .
Characterization of λ c r i t \lambda_{\mathrm{crit}} λ crit . In Lemma D.4 , when K ˉ ≥ 2 \bar K\ge2 K ˉ ≥ 2 , we see that the solution takes different form based on whether ρ λ < ρ c r i t \rho_\lambda<\rho_{\mathrm{crit}} ρ λ < ρ crit or ρ λ ≥ ρ c r i t \rho_\lambda\ge\rho_{\mathrm{crit}} ρ λ ≥ ρ crit . Moreover, based on Lemma D.2 , there must exist a unique λ c r i t \lambda_{\mathrm{crit}} λ crit such that ρ c r i t = ρ ( λ c r i t ) \rho_{\mathrm{crit}}=\rho(\lambda_{\mathrm{crit}}) ρ crit = ρ ( λ crit ) . Now, we characterize the range of λ c r i t \lambda_{\mathrm{crit}} λ crit here.
Due to Lemma D.4 , we know when λ > λ c r i t \lambda>\lambda_{\mathrm{crit}} λ > λ crit , the solution takes the form
B ∗ A ~ ∗ = ρ n ( K − 1 ) ( [ I K ˉ − 1 K ˉ 1 K ˉ 1 K ˉ T 0 ( K − K ˉ ) × K ˉ ] ⊗ 1 n T ) , B^*\widetilde{A}^*=
\frac{\rho}{\sqrt{n}(K-1)}
\left(
\begin{bmatrix}
I_{\bar K}-\frac{1}{\bar K}1_{\bar K}1_{\bar K}^{\mathsf T}\\
0_{(K-\bar K)\times\bar K}
\end{bmatrix}
\otimes1_n^{\mathsf T}
\right), B ∗ A ∗ = n ( K − 1 ) ρ ( [ I K ˉ − K ˉ 1 1 K ˉ 1 K ˉ T 0 ( K − K ˉ ) × K ˉ ] ⊗ 1 n T ) ,
and adding the penalty λ ρ \lambda\rho λ ρ to this constrained minimum cross entropy gives the scalar penalized objective
ψ ( ρ ) : = λ ρ − ρ n K ˉ + log ( K − K ˉ + ( K ˉ − 1 ) exp ( − ρ n K ˉ ( K ˉ − 1 ) ) + exp ( ρ n K ˉ ) ) . \psi(\rho):=\lambda\rho-\frac{\rho}{\sqrt{n\bar K}}+\log\left(K-\bar K+(\bar K-1)\exp\left(-\frac{\rho}{\sqrt{n\bar K}(\bar K-1)}\right)+\exp\left(\frac{\rho}{\sqrt{n\bar K}}\right)\right). ψ ( ρ ) := λ ρ − n K ˉ ρ + log ( K − K ˉ + ( K ˉ − 1 ) exp ( − n K ˉ ( K ˉ − 1 ) ρ ) + exp ( n K ˉ ρ ) ) .
For this ETF-form update, ρ = K ˉ − 1 Θ \rho=\sqrt{\bar K-1}\Theta ρ = K ˉ − 1 Θ . Differentiating ψ ( ρ ) \psi(\rho) ψ ( ρ ) and substituting this relation into the stationarity condition gives
d ψ ( ρ ) d ρ = 0 \frac{d\psi(\rho)}{d\rho}=0 d ρ d ψ ( ρ ) = 0
⟺ ( λ − 1 n K ˉ ) + 1 n K ˉ ⋅ exp ( ρ n K ˉ ) − exp ( − ρ n K ˉ ( K − 1 ) ) K − K ˉ + ( K ˉ − 1 ) exp ( − ρ n K ˉ ( K ˉ − 1 ) ) + exp ( ρ n K ˉ ) = 0 \Longleftrightarrow
\left(\lambda-\frac{1}{\sqrt{n\bar K}}\right)+\frac{1}{\sqrt{n\bar K}}\cdot
\frac{\exp\left(\frac{\rho}{\sqrt{n\bar K}}\right)-\exp\left(-\frac{\rho}{\sqrt{n\bar K}(K-1)}\right)}
{K-\bar K+(\bar K-1)\exp\left(-\frac{\rho}{\sqrt{n\bar K}(\bar K-1)}\right)+\exp\left(\frac{\rho}{\sqrt{n\bar K}}\right)}
=0 ⟺ ( λ − n K ˉ 1 ) + n K ˉ 1 ⋅ K − K ˉ + ( K ˉ − 1 ) exp ( − n K ˉ ( K ˉ − 1 ) ρ ) + exp ( n K ˉ ρ ) exp ( n K ˉ ρ ) − exp ( − n K ˉ ( K − 1 ) ρ ) = 0
⟺ ( λ − 1 n K ˉ ) + 1 n K ˉ ⋅ exp ( Θ n ( K ˉ − 1 ) ) − 1 ( K − K ˉ ) exp ( Θ K ˉ n ( K ˉ − 1 ) ) + ( K ˉ − 1 ) + exp ( Θ n ( K ˉ − 1 ) ) = 0 (29) \Longleftrightarrow
\left(\lambda-\frac{1}{\sqrt{n\bar K}}\right)+\frac{1}{\sqrt{n\bar K}}\cdot
\frac{\exp\left(\frac{\Theta}{\sqrt{n(\bar K-1)}}\right)-1}
{(K-\bar K)\exp\left(\frac{\Theta}{\bar K\sqrt{n(\bar K-1)}}\right)+(\bar K-1)+\exp\left(\frac{\Theta}{\sqrt{n(\bar K-1)}}\right)}
=0 \tag*{(29)} ⟺ ( λ − n K ˉ 1 ) + n K ˉ 1 ⋅ ( K − K ˉ ) exp ( K ˉ n ( K ˉ − 1 ) Θ ) + ( K ˉ − 1 ) + exp ( n ( K ˉ − 1 ) Θ ) exp ( n ( K ˉ − 1 ) Θ ) − 1 = 0 (29)
We argue by contradiction. Suppose λ c r i t ≥ 1 / n K ˉ \lambda_{\mathrm{crit}}\ge1/\sqrt{n\bar K} λ crit ≥ 1/ n K ˉ . For every λ > λ c r i t \lambda>\lambda_{\mathrm{crit}} λ > λ crit , the preceding ETF form (including the zero product) applies and (29) gives d ψ / d ρ > 0 d\psi/d\rho>0 d ψ / d ρ > 0 for ρ > 0 \rho>0 ρ > 0 , so ρ λ = 0 \rho_\lambda=0 ρ λ = 0 . Letting λ ↓ λ c r i t \lambda\downarrow\lambda_{\mathrm{crit}} λ ↓ λ crit and using continuity from Lemma D.2 yields ρ ( λ c r i t ) = 0 \rho(\lambda_{\mathrm{crit}})=0 ρ ( λ crit ) = 0 , contradicting its defining value ρ c r i t > 0 \rho_{\mathrm{crit}}>0 ρ crit > 0 . This covers both equality and strict inequality in the contradiction hypothesis. Therefore
λ c r i t < 1 n K ˉ . \lambda_{\mathrm{crit}}<\frac{1}{\sqrt{n\bar K}}. λ crit < n K ˉ 1 .
Characterization of ρ λ \rho_\lambda ρ λ . In this part, we characterize the ρ λ \rho_\lambda ρ λ when λ > λ c r i t \lambda>\lambda_{\mathrm{crit}} λ > λ crit . By Lemma D.4 , in this ETF regime ψ ( ρ ) \psi(\rho) ψ ( ρ ) equals the constrained minimum cross entropy at budget ρ \rho ρ , plus the penalty λ ρ \lambda\rho λ ρ . Thus, ρ λ \rho_\lambda ρ λ must attain the minimum of ψ ( ρ ) \psi(\rho) ψ ( ρ ) . Based on (29), one can first see that if λ ≥ 1 n K ˉ \lambda\ge\frac{1}{\sqrt{n\bar K}} λ ≥ n K ˉ 1 , d ψ ( ρ ) / d ρ > 0 d\psi(\rho)/d\rho>0 d ψ ( ρ ) / d ρ > 0 , and the minimum is attained when ρ λ = 0 \rho_\lambda=0 ρ λ = 0 . When λ c r i t < λ < 1 n K ˉ \lambda_{\mathrm{crit}}<\lambda<\frac{1}{\sqrt{n\bar K}} λ crit < λ < n K ˉ 1 , solve (29) for Θ λ \Theta_\lambda Θ λ and recover ρ λ = K ˉ − 1 Θ λ \rho_\lambda=\sqrt{\bar K-1}\Theta_\lambda ρ λ = K ˉ − 1 Θ λ .
Asymptotic behaviour of ρ λ \rho_\lambda ρ λ . On one hand, as λ → 0 + \lambda\to0^+ λ → 0 + , one obviously has lim λ → 0 + ∥ B ∗ A ~ ∗ ∥ F = ∞ \lim_{\lambda\to0^+}\|B^*\widetilde{A}^*\|_F=\infty lim λ → 0 + ∥ B ∗ A ∗ ∥ F = ∞ , since by Lemma D.2 we have Z λ ∗ = B ∗ A ~ ∗ Z_\lambda^*=B^*\widetilde{A}^* Z λ ∗ = B ∗ A ∗ where Z λ ∗ Z_\lambda^* Z λ ∗ is the unique minimizer of (Problem three) and ∥ Z λ ∗ ∥ ∗ = ρ λ → ∞ \|Z_\lambda^*\|_*=\rho_\lambda\to\infty ∥ Z λ ∗ ∥ ∗ = ρ λ → ∞ . Then, we characterize its asymptotic direction, which is equivalent to study the limit of log g a ( x 0 ) log g b ( x 0 ) \frac{\log g_a(x_0)}{\log g_b(x_0)} l o g g b ( x 0 ) l o g g a ( x 0 ) and log g a ( x 0 ) log g c ( x 0 ) \frac{\log g_a(x_0)}{\log g_c(x_0)} l o g g c ( x 0 ) l o g g a ( x 0 ) as λ → 0 + \lambda\to0^+ λ → 0 + . We will use x 0 , λ x_{0,\lambda} x 0 , λ denote the choice of x 0 x_0 x 0 for fix regularization effect λ \lambda λ .
First, the equation to derive for x 0 x_0 x 0 is as follows
g b ( x ) g c ( x ) = 1 1 + K K − K ˉ \frac{g_b(x)}{g_c(x)}=\frac{1}{1+\sqrt{\frac{K}{K-\bar K}}} g c ( x ) g b ( x ) = 1 + K − K ˉ K 1
⟺ ρ λ K ˉ n ⋅ − x 2 K ˉ − 1 + ( K K − K ˉ ) 3 / 2 ( 1 − x ) 2 x 2 + K K − K ˉ ( 1 − x ) 2 = − log ( 1 + K K − K ˉ ) . \Longleftrightarrow
\frac{\rho_\lambda}{\bar K\sqrt{n}}\cdot
\frac{-\frac{x^2}{\bar K-1}+\left(\frac{K}{K-\bar K}\right)^{3/2}(1-x)^2}
{x^2+\frac{K}{K-\bar K}(1-x)^2}
=-\log\left(1+\sqrt{\frac{K}{K-\bar K}}\right). ⟺ K ˉ n ρ λ ⋅ x 2 + K − K ˉ K ( 1 − x ) 2 − K ˉ − 1 x 2 + ( K − K ˉ K ) 3/2 ( 1 − x ) 2 = − log ( 1 + K − K ˉ K ) .
Since lim λ → 0 + ρ λ = ∞ \lim_{\lambda\to0+}\rho_\lambda=\infty lim λ → 0 + ρ λ = ∞ , one must have x 0 , λ x_{0,\lambda} x 0 , λ approaches the solution of the following quadratic equation
x 0 , λ 2 K ‾ − 1 = ( K K − K ‾ ) 3 / 2 ( 1 − x 0 , λ ) 2 . \frac{x_{0,\lambda}^{2}}{\overline{K}-1}=\left(\frac{K}{K-\overline{K}}\right)^{3/2}(1-x_{0,\lambda})^{2}. K − 1 x 0 , λ 2 = ( K − K K ) 3/2 ( 1 − x 0 , λ ) 2 .
Moreover, one can see that the expression for log g a ( x 0 , λ ) log g b ( x 0 , λ ) \frac{\log g_a(x_{0,\lambda})}{\log g_b(x_{0,\lambda})} l o g g b ( x 0 , λ ) l o g g a ( x 0 , λ ) and log g a ( x 0 , λ ) log g c ( x 0 , λ ) \frac{\log g_a(x_{0,\lambda})}{\log g_c(x_{0,\lambda})} l o g g c ( x 0 , λ ) l o g g a ( x 0 , λ ) takes the following form
log g a ( x 0 , λ ) log g b ( x 0 , λ ) = x 0 , λ 2 + K K − K ‾ ( 1 − x 0 , λ ) 2 − x 0 , λ 2 K ‾ − 1 + K K − K ‾ ( 1 − x 0 , λ ) 2 \frac{\log g_a(x_{0,\lambda})}{\log g_b(x_{0,\lambda})}
=
\frac{x_{0,\lambda}^{2}+\sqrt{\frac{K}{K-\overline{K}}}(1-x_{0,\lambda})^{2}}
{-\frac{x_{0,\lambda}^{2}}{\overline{K}-1}+\sqrt{\frac{K}{K-\overline{K}}}(1-x_{0,\lambda})^{2}} log g b ( x 0 , λ ) log g a ( x 0 , λ ) = − K − 1 x 0 , λ 2 + K − K K ( 1 − x 0 , λ ) 2 x 0 , λ 2 + K − K K ( 1 − x 0 , λ ) 2
log g a ( x 0 , λ ) log g c ( x 0 , λ ) = x 0 , λ 2 + K K − K ‾ ( 1 − x 0 , λ ) 2 − K ‾ K − K ‾ ⋅ K K − K ‾ ( 1 − x 0 , λ ) 2 . \frac{\log g_a(x_{0,\lambda})}{\log g_c(x_{0,\lambda})}
=
\frac{x_{0,\lambda}^{2}+\sqrt{\frac{K}{K-\overline{K}}}(1-x_{0,\lambda})^{2}}
{-\frac{\overline{K}}{K-\overline{K}}\cdot\sqrt{\frac{K}{K-\overline{K}}}(1-x_{0,\lambda})^{2}}. log g c ( x 0 , λ ) log g a ( x 0 , λ ) = − K − K K ⋅ K − K K ( 1 − x 0 , λ ) 2 x 0 , λ 2 + K − K K ( 1 − x 0 , λ ) 2 .
Thus, combine the above two equations, we can compute that
lim λ → 0 + log g a ( x 0 , λ ) log g b ( x 0 , λ ) = lim λ → 0 + log g a ( x 0 , λ ) log g c ( x 0 , λ ) = − ( K − 1 ) . \lim_{\lambda\to0+}\frac{\log g_a(x_{0,\lambda})}{\log g_b(x_{0,\lambda})}
=
\lim_{\lambda\to0+}\frac{\log g_a(x_{0,\lambda})}{\log g_c(x_{0,\lambda})}
=-(K-1). λ → 0 + lim log g b ( x 0 , λ ) log g a ( x 0 , λ ) = λ → 0 + lim log g c ( x 0 , λ ) log g a ( x 0 , λ ) = − ( K − 1 ) .
Connection between ρ λ \rho_\lambda ρ λ and Θ λ \Theta_\lambda Θ λ . In previous analysis, we have shown that
ρ λ = ∥ B λ ∗ A λ ∗ ∥ ∗ , Θ λ = ∥ B λ ∗ A λ ∗ ∥ F (30) \rho_\lambda=\lVert B_\lambda^*A_\lambda^*\rVert_*,
\qquad
\Theta_\lambda=\lVert B_\lambda^*A_\lambda^*\rVert_F \tag*{(30)} ρ λ = ∥ B λ ∗ A λ ∗ ∥ ∗ , Θ λ = ∥ B λ ∗ A λ ∗ ∥ F (30)
It remains to relate ρ λ \rho_\lambda ρ λ and Θ λ \Theta_\lambda Θ λ across the regularization regimes. Write A ‾ λ ∗ : = A λ ∗ X f t \overline{A}_\lambda^*:=A_\lambda^*X_{\mathrm{ft}} A λ ∗ := A λ ∗ X ft for the data-coordinate factor denoted by A ~ ∗ \widetilde{A}^* A ∗ above. By Lemma D.1, B λ ∗ A λ ∗ = B λ ∗ A ‾ λ ∗ X f t ⊤ B_\lambda^*A_\lambda^*=B_\lambda^*\overline{A}_\lambda^*X_{\mathrm{ft}}^\top B λ ∗ A λ ∗ = B λ ∗ A λ ∗ X ft ⊤ . Since X f t ⊤ X f t = I K n ‾ X_{\mathrm{ft}}^\top X_{\mathrm{ft}}=I_{K\overline{n}} X ft ⊤ X ft = I K n , the weight-space and data-coordinate products have the same nonzero singular values, and hence the same nuclear and Frobenius norms.
First, it is obvious in the high-penalty regime, ρ λ = Θ λ = 0 \rho_\lambda=\Theta_\lambda=0 ρ λ = Θ λ = 0 .
Second, in the intermediate regime , the data-coordinate product is
B λ ∗ A ‾ λ ∗ = ρ λ K n ‾ ( K − 1 ) ( 0 ( K − K ‾ ) × K ‾ ) ⊗ 1 n ‾ ⊤ . B_\lambda^*\overline{A}_\lambda^*
=
\frac{\rho_\lambda}{\sqrt{K\overline{n}(K-1)}}
\left(\mathbf{0}_{(K-\overline{K})\times\overline{K}}\right)\otimes\mathbf{1}_{\overline{n}}^\top. B λ ∗ A λ ∗ = K n ( K − 1 ) ρ λ ( 0 ( K − K ) × K ) ⊗ 1 n ⊤ .
Consequently,
ρ λ = ∥ B λ ∗ A ‾ λ ∗ ∥ ∗ , Θ λ = ∥ B λ ∗ A ‾ λ ∗ ∥ F = ρ λ K n ‾ − 1 . \rho_\lambda=\lVert B_\lambda^*\overline{A}_\lambda^*\rVert_*,
\qquad
\Theta_\lambda=\lVert B_\lambda^*\overline{A}_\lambda^*\rVert_F
=\frac{\rho_\lambda}{\sqrt{K\overline{n}-1}}. ρ λ = ∥ B λ ∗ A λ ∗ ∥ ∗ , Θ λ = ∥ B λ ∗ A λ ∗ ∥ F = K n − 1 ρ λ .
Equivalently, the data-coordinate product in the intermediate regime is
B λ ∗ A ‾ λ ∗ = Θ λ n ‾ K ( 0 ( K − K ‾ ) × K ‾ ) ⊗ 1 n ‾ ⊤ . B_\lambda^*\overline{A}_\lambda^*
=
\frac{\Theta_\lambda}{\sqrt{\overline{n}K}}
\left(\mathbf{0}_{(K-\overline{K})\times\overline{K}}\right)\otimes\mathbf{1}_{\overline{n}}^\top. B λ ∗ A λ ∗ = n K Θ λ ( 0 ( K − K ) × K ) ⊗ 1 n ⊤ .
Multiplying by X f t ⊤ X_{\mathrm{ft}}^\top X ft ⊤ recovers the weight-space formula (8) . Third, in the low-penalty regime , the preceding asymptotic analysis gives lim λ → 0 + ∥ B λ ∗ A ‾ λ ∗ ∥ F = ∞ \lim_{\lambda\to0+}\lVert B_\lambda^*\overline{A}_\lambda^*\rVert_F=\infty lim λ → 0 + ∥ B λ ∗ A λ ∗ ∥ F = ∞ and characterizes the limiting direction.
The solution of (5). Finally, one can use Lemma D.1 to characterize the optimal solution of (5) based on the above results.
Proof of General Version
In this subsection we present the general version of Theorem 3.1: under the near-orthogonal regime in Assumption 2.1 (i.e., ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 small but possibly nonzero), the optimal LoRA update B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ remains close, in Frobenius norm, to its closed-form orthogonal-case description, with the deviation controlled linearly in ϵ 1 \epsilon_1 ϵ 1 and ϵ 2 \epsilon_2 ϵ 2 .
Notation. Write G : = X f t ⊤ X f t ∈ R K n ‾ × K n ‾ G:=X_{\mathrm{ft}}^\top X_{\mathrm{ft}}\in\mathbb{R}^{K\overline{n}\times K\overline{n}} G := X ft ⊤ X ft ∈ R K n × K n , R : = G 1 / 2 R:=G^{1/2} R := G 1/2 , Q : = X f t R − 1 ∈ R d × K n ‾ Q:=X_{\mathrm{ft}}R^{-1}\in\mathbb{R}^{d\times K\overline{n}} Q := X ft R − 1 ∈ R d × K n (so that Q ⊤ Q = I K n ‾ Q^\top Q=I_{K\overline{n}} Q ⊤ Q = I K n and X f t = Q R X_{\mathrm{ft}}=QR X ft = QR ), S : = W p r e X f t ∈ R K × K n ‾ S:=W_{\mathrm{pre}}X_{\mathrm{ft}}\in\mathbb{R}^{K\times K\overline{n}} S := W pre X ft ∈ R K × K n , and let N p r e : = ∣ D p r e ∣ N_{\mathrm{pre}}:=|\mathcal{D}_{\mathrm{pre}}| N pre := ∣ D pre ∣ be the number of pre-training samples. Set δ : = ( K ˉ n − 1 ) ϵ 1 \delta:= (\bar{K}n-1)\epsilon_1 δ := ( K ˉ n − 1 ) ϵ 1 . For any S ˉ ∈ R K × K ˉ n \bar{S} \in\mathbb{R}^{K \times\bar{K}n} S ˉ ∈ R K × K ˉ n and any positive-definite R ˉ ∈ R K ˉ n × K ˉ n \bar{R} \in\mathbb{R}^{\bar{K}n \times\bar{K}n} R ˉ ∈ R K ˉ n × K ˉ n , define
Φ ( Z ; S ˉ , R ˉ ) : = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , ( Z R ˉ + S ˉ ) e ( c − 1 ) n + j ) + λ ∥ Z ∥ ∗ , (31) \Phi(Z;\bar{S},\bar{R}) := \frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}(y_c,(Z\bar{R}+\bar{S})e_{(c-1)n+j})+\lambda\lVert Z\rVert_*, \tag*{(31)} Φ ( Z ; S ˉ , R ˉ ) := K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , ( Z R ˉ + S ˉ ) e ( c − 1 ) n + j ) + λ ∥ Z ∥ ∗ , (31)
Z λ ∗ ( S ˉ , R ˉ ) : = arg min Z ∈ R K × K ˉ n Φ ( Z ; S ˉ , R ˉ ) . (32) Z_\lambda^*(\bar{S},\bar{R}) := \underset{Z\in\mathbb{R}^{K\times\bar{K}n}}{\arg\min}\ \Phi(Z;\bar{S},\bar{R}). \tag*{(32)} Z λ ∗ ( S ˉ , R ˉ ) := Z ∈ R K × K ˉ n arg min Φ ( Z ; S ˉ , R ˉ ) . (32)
The minimizer Z λ ∗ Z_\lambda^* Z λ ∗ is unique by Lemma G.6 (for S ˉ = 0 \bar{S}=0 S ˉ = 0 , R ˉ = I \bar{R}=I R ˉ = I ; the same argument applies for general R ˉ ≻ 0 \bar{R}\succ0 R ˉ ≻ 0 ).
Theorem D.2 (General version of Theorem 3.1). Fix λ > 0 \lambda>0 λ > 0 and suppose Assumptions 2.1–2.4 hold with ϵ 1 , ϵ 2 ∈ [ 0 , 1 ) \epsilon_1,\epsilon_2\in[0,1) ϵ 1 , ϵ 2 ∈ [ 0 , 1 ) . There exist constants δ 0 ∈ ( 0 , 1 / 2 ] \delta_0\in(0,1/2] δ 0 ∈ ( 0 , 1/2 ] , s 0 ∈ ( 0 , 1 ] s_0\in(0,1] s 0 ∈ ( 0 , 1 ] , and μ 0 , C 0 > 0 \mu_0,C_0>0 μ 0 , C 0 > 0 , depending on λ , K , K ˉ , n , ∥ Z λ ∗ ( 0 , I ) ∥ ∗ \lambda,K,\bar{K},n,\lVert Z_\lambda^*(0,I)\rVert_* λ , K , K ˉ , n , ∥ Z λ ∗ ( 0 , I ) ∥ ∗ but not on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 , such that whenever δ : = ( K ˉ n − 1 ) ϵ 1 ≤ δ 0 \delta:=(\bar{K}n-1)\epsilon_1\le\delta_0 δ := ( K ˉ n − 1 ) ϵ 1 ≤ δ 0 and ∥ S ∥ F ≤ s 0 \lVert S\rVert_F\le s_0 ∥ S ∥ F ≤ s 0 ,
∥ B λ ∗ A λ ∗ − Z λ ∗ ( 0 , I ) X f t ⊤ ∥ F ≤ C 0 μ 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z λ ∗ ( 0 , I ) ∥ F ) ) . (33) \left\lVert B_\lambda^*A_\lambda^*-Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top\right\rVert_F\le\frac{C_0}{\mu_0}\left(\lVert S\rVert_F+\delta\left(1+\lVert Z_\lambda^*(0,I)\rVert_F\right)\right). \tag*{(33)} B λ ∗ A λ ∗ − Z λ ∗ ( 0 , I ) X ft ⊤ F ≤ μ 0 C 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z λ ∗ ( 0 , I ) ∥ F ) ) . (33)
In particular, as ϵ 1 , ϵ 2 → 0 \epsilon_1,\epsilon_2\to0 ϵ 1 , ϵ 2 → 0 , B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ converges in Frobenius norm to its orthogonal-case form Z λ ∗ ( 0 , I ) X f t ⊤ Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top Z λ ∗ ( 0 , I ) X ft ⊤ .
Remark D.2 (Bounding ∥ S ∥ F \lVert S\rVert_F ∥ S ∥ F in ϵ 2 \epsilon_2 ϵ 2 ). The bound (33) is stated directly in terms of ∥ S ∥ F = ∥ W p r e X f t ∥ F \lVert S\rVert_F=\lVert W_{\mathrm{pre}}X_{\mathrm{ft}}\rVert_F ∥ S ∥ F = ∥ W pre X ft ∥ F . To translate this into a bound in ϵ 2 \epsilon_2 ϵ 2 , one needs a lower bound on σ min ( X p r e ) \sigma_{\min}(X_{\mathrm{pre}}) σ m i n ( X pre ) . By Proposition G.1 and Assumption 2.4, each row w k w_k w k of W p r e W_{\mathrm{pre}} W pre admits the representation w k = X p r e β k w_k=X_{\mathrm{pre}}\beta_k w k = X pre β k with ∥ β k ∥ 2 ≤ ∥ w k ∥ 2 / σ min ( X p r e ) \lVert\beta_k\rVert_2\le\lVert w_k\rVert_2/\sigma_{\min}(X_{\mathrm{pre}}) ∥ β k ∥ 2 ≤ ∥ w k ∥ 2 / σ m i n ( X pre ) . Combined with Assumption 2.1(c),
∣ w k ⊤ x ˉ c , j ∣ = ∣ β k ⊤ X p r e ⊤ x ˉ c , j ∣ ≤ ∥ β k ∥ 2 N p r e ϵ 2 ≤ ∥ w k ∥ 2 N p r e ϵ 2 σ min ( X p r e ) , (34) \left|w_k^\top\bar{x}_{c,j}\right|=\left|\beta_k^\top X_{\mathrm{pre}}^\top\bar{x}_{c,j}\right|\le\lVert\beta_k\rVert_2\sqrt{N_{\mathrm{pre}}}\epsilon_2\le\frac{\lVert w_k\rVert_2\sqrt{N_{\mathrm{pre}}}\epsilon_2}{\sigma_{\min}(X_{\mathrm{pre}})}, \tag*{(34)} w k ⊤ x ˉ c , j = β k ⊤ X pre ⊤ x ˉ c , j ≤ ∥ β k ∥ 2 N pre ϵ 2 ≤ σ m i n ( X pre ) ∥ w k ∥ 2 N pre ϵ 2 , (34)
hence ∥ S ∥ F ≤ K n N p r e ∥ W p r e ∥ F ϵ 2 / σ min ( X p r e ) \lVert S\rVert_F\le\sqrt{K n N_{\mathrm{pre}}}\lVert W_{\mathrm{pre}}\rVert_F\epsilon_2/\sigma_{\min}(X_{\mathrm{pre}}) ∥ S ∥ F ≤ K n N pre ∥ W pre ∥ F ϵ 2 / σ m i n ( X pre ) . In particular, ∥ S ∥ F → 0 \lVert S\rVert_F\to0 ∥ S ∥ F → 0 as ϵ 2 → 0 \epsilon_2\to0 ϵ 2 → 0 whenever σ min ( X p r e ) \sigma_{\min}(X_{\mathrm{pre}}) σ m i n ( X pre ) is bounded below by a positive constant. The cleanest specialization is the orthonormal case X p r e ⊤ X p r e = I N p r e X_{\mathrm{pre}}^\top X_{\mathrm{pre}}=I_{N_{\mathrm{pre}}} X pre ⊤ X pre = I N pre (so σ min ( X p r e ) = 1 \sigma_{\min}(X_{\mathrm{pre}})=1 σ m i n ( X pre ) = 1 ), which yields
∥ S ∥ F ≤ K n N p r e ∥ W p r e ∥ F ϵ 2 . (35) \lVert S\rVert_F\le\sqrt{K n N_{\mathrm{pre}}}\lVert W_{\mathrm{pre}}\rVert_F\epsilon_2. \tag*{(35)} ∥ S ∥ F ≤ K n N pre ∥ W pre ∥ F ϵ 2 . (35)
The same conclusion holds, up to a multiplicative conditioning constant, whenever σ min ( X p r e ) \sigma_{\min}(X_{\mathrm{pre}}) σ m i n ( X pre ) can be lower-bounded (e.g., near-orthogonal pre-training features, or unit-norm pre-training features with bounded condition number). We highlight that we do not claim (35) is tight. The goal of Remark D.2 is to show that ∥ S ∥ F \lVert S\rVert_F ∥ S ∥ F can be controlled by ϵ 2 \epsilon_2 ϵ 2 under mild assumptions, so that the right-hand side of (33) goes to zero as $\epsilon_2\to 0.
Proof of Theorem D.2. We first reformulate the optimization problem in Z Z Z as it was done in the previous proof. By the same change of variable used in the proof of Lemma D.1, decompose A = A ~ Q ⊤ + A ⊥ A=\tilde{A}Q^\top+A_\perp A = A ~ Q ⊤ + A ⊥ where A ~ : = A Q ∈ R r × K ˉ n \tilde{A}:=AQ\in\mathbb{R}^{r\times\bar{K}n} A ~ := A Q ∈ R r × K ˉ n and A ⊥ : = A ( I d − Q Q ⊤ ) A_\perp:=A(I_d-QQ^\top) A ⊥ := A ( I d − Q Q ⊤ ) . Then A x c , j = A R ~ e ( c − 1 ) n + j Ax_{c,j}=A\tilde{R}e_{(c-1)n+j} A x c , j = A R ~ e ( c − 1 ) n + j and (5) can be rewritten as
L ( B , A ) = 1 K ˉ n ∑ c , j L C E ( y c , ( B A ~ R ~ + S ) e ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ~ ∥ F 2 + ∥ A ⊥ ∥ F 2 ) . (36) \mathcal{L}(B,A)=\frac{1}{\bar{K}n}\sum_{c,j}\mathcal{L}_{\mathrm{CE}}(y_c,(B\tilde{A}\tilde{R}+S)e_{(c-1)n+j})+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert\tilde{A}\rVert_F^2+\lVert A_\perp\rVert_F^2\right). \tag*{(36)} L ( B , A ) = K ˉ n 1 c , j ∑ L CE ( y c , ( B A ~ R ~ + S ) e ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ~ ∥ F 2 + ∥ A ⊥ ∥ F 2 ) . (36)
The component A ⊥ A_\perp A ⊥ contributes only to the regularizer and is therefore zero at any minimizer. Setting Z = B A ~ Z=B\tilde{A} Z = B A ~ and applying the variational form of the nuclear norm (Lemma G.2), the LoRA problem becomes the rank-constrained minimization min Z ∈ R K × K ˉ n , rank ( Z ) ≤ r Φ ( Z ; S , R ) \min_{Z\in\mathbb{R}^{K\times\bar{K}n},\operatorname{rank}(Z)\le r}\Phi(Z;S,R) min Z ∈ R K × K ˉ n , rank ( Z ) ≤ r Φ ( Z ; S , R ) , where Φ ( Z ; S ˉ , R ˉ ) : = L ( Z ; S ˉ , R ˉ ) + λ ∥ Z ∥ ∗ \Phi(Z;\bar{S},\bar{R}):=L(Z;\bar{S},\bar{R})+\lambda\lVert Z\rVert_* Φ ( Z ; S ˉ , R ˉ ) := L ( Z ; S ˉ , R ˉ ) + λ ∥ Z ∥ ∗ . Let Z λ ∗ ( S , R ) Z_\lambda^*(S,R) Z λ ∗ ( S , R ) denote the unique unconstrained minimizer of Φ ( ⋅ ; S , R ) \Phi(\cdot;S,R) Φ ( ⋅ ; S , R ) over R K × K ˉ n \mathbb{R}^{K\times\bar{K}n} R K × K ˉ n (Lemma G.6). We first establish a perturbation bound for this unconstrained problem, then verify that its minimizer satisfies the rank constraint.
Then, we provide a spectral bound on R R R . By Assumption 2.1, G i i = ∥ x ˉ i ∥ 2 = 1 G_{ii}=\lVert\bar{x}_i\rVert^2=1 G ii = ∥ x ˉ i ∥ 2 = 1 and ∣ G i j ∣ = ∣ x ˉ i ⊤ x ˉ j ∣ ≤ ϵ 1 \lvert G_{ij}\rvert=\lvert\bar{x}_i^\top\bar{x}_j\rvert\le\epsilon_1 ∣ G ij ∣ = ∣ x ˉ i ⊤ x ˉ j ∣ ≤ ϵ 1 for i ≠ j i\ne j i = j . By Gershgorin’s circle theorem,
∥ G − I ∥ o p ≤ max i ∑ j ≠ i ∣ G i j ∣ ≤ ( K ˉ n − 1 ) ϵ 1 = δ . (37) \lVert G-I\rVert_{\mathrm{op}}\le\max_i\sum_{j\ne i}\lvert G_{ij}\rvert\le(\bar{K}n-1)\epsilon_1=\delta. \tag*{(37)} ∥ G − I ∥ op ≤ i max j = i ∑ ∣ G ij ∣ ≤ ( K ˉ n − 1 ) ϵ 1 = δ . (37)
Hence every eigenvalue of G G G lies in [ 1 − δ , 1 + δ ] [1-\delta,1+\delta] [ 1 − δ , 1 + δ ] , so σ i ( R ) = σ i ( G ) ∈ [ 1 − δ , 1 + δ ] \sigma_i(R)=\sqrt{\sigma_i(G)}\in[\sqrt{1-\delta},\sqrt{1+\delta}] σ i ( R ) = σ i ( G ) ∈ [ 1 − δ , 1 + δ ] , which gives σ min ( R ) ≥ 1 − δ \sigma_{\min}(R)\ge\sqrt{1-\delta} σ m i n ( R ) ≥ 1 − δ . Using ∣ x − 1 ∣ ≤ ∣ x − 1 ∣ \lvert\sqrt{x}-1\rvert\le\lvert x-1\rvert ∣ x − 1 ∣ ≤ ∣ x − 1 ∣ for x ≥ 0 x\ge0 x ≥ 0 ,
∥ R − I ∥ o p = max i ∣ σ i ( G ) − 1 ∣ ≤ max i ∣ σ i ( G ) − 1 ∣ = ∥ G − I ∥ o p ≤ δ . (38) \lVert R-I\rVert_{\mathrm{op}}=\max_i\lvert\sqrt{\sigma_i(G)}-1\rvert\le\max_i\lvert\sigma_i(G)-1\rvert=\lVert G-I\rVert_{\mathrm{op}}\le\delta. \tag*{(38)} ∥ R − I ∥ op = i max ∣ σ i ( G ) − 1 ∣ ≤ i max ∣ σ i ( G ) − 1 ∣ = ∥ G − I ∥ op ≤ δ . (38)
Based on the previous spectral bound, we can now bound the deviation of Z λ ∗ ( S , R ) Z_\lambda^*(S,R) Z λ ∗ ( S , R ) from Z λ ∗ ( 0 , I ) Z_\lambda^*(0,I) Z λ ∗ ( 0 , I ) , which will lead to the desired bound on ∥ B λ ∗ A λ ∗ − Z λ ∗ ( 0 , I ) X f t ⊤ ∥ F \lVert B_\lambda^*A_\lambda^*-Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top\rVert_F ∥ B λ ∗ A λ ∗ − Z λ ∗ ( 0 , I ) X ft ⊤ ∥ F . Let L ( Z ; S ˉ , R ˉ ) : = Φ ( Z ; S ˉ , R ˉ ) − λ ∥ Z ∥ ∗ L(Z;\bar S,\bar R):=\Phi(Z;\bar S,\bar R)-\lambda\lVert Z\rVert_* L ( Z ; S ˉ , R ˉ ) := Φ ( Z ; S ˉ , R ˉ ) − λ ∥ Z ∥ ∗ be the smooth (cross-entropy) part of the objective, and write Z 1 : = Z λ ∗ ( S , R ) Z_1:=Z_\lambda^*(S,R) Z 1 := Z λ ∗ ( S , R ) , Z 0 : = Z λ ∗ ( 0 , I ) Z_0:=Z_\lambda^*(0,I) Z 0 := Z λ ∗ ( 0 , I ) . The first-order optimality conditions read
− ∇ Z L ( Z 1 ; S , R ) ∈ λ ∂ ∥ Z 1 ∥ ∗ , − ∇ Z L ( Z 0 ; 0 , I ) ∈ λ ∂ ∥ Z 0 ∥ ∗ . (39) -\nabla_Z L(Z_1;S,R)\in\lambda\partial\lVert Z_1\rVert_*,\qquad-\nabla_Z L(Z_0;0,I)\in\lambda\partial\lVert Z_0\rVert_*. \tag*{(39)} − ∇ Z L ( Z 1 ; S , R ) ∈ λ ∂ ∥ Z 1 ∥ ∗ , − ∇ Z L ( Z 0 ; 0 , I ) ∈ λ ∂ ∥ Z 0 ∥ ∗ . (39)
Subdifferentials of convex functions are monotone, so ⟨ G 1 − G 0 , Z 1 − Z 0 ⟩ ≥ 0 \langle G_1-G_0,Z_1-Z_0\rangle\ge0 ⟨ G 1 − G 0 , Z 1 − Z 0 ⟩ ≥ 0 for any G i ∈ ∂ ∥ Z i ∥ ∗ G_i\in\partial\lVert Z_i\rVert_* G i ∈ ∂ ∥ Z i ∥ ∗ . Substituting,
⟨ ∇ L ( Z 1 ; S , R ) − ∇ L ( Z 0 ; 0 , I ) , Z 1 − Z 0 ⟩ ≤ 0. (40) \langle\nabla L(Z_1;S,R)-\nabla L(Z_0;0,I),Z_1-Z_0\rangle\le0. \tag*{(40)} ⟨ ∇ L ( Z 1 ; S , R ) − ∇ L ( Z 0 ; 0 , I ) , Z 1 − Z 0 ⟩ ≤ 0. (40)
We work under the preliminary bound ∥ S ∥ F ≤ 1 \lVert S\rVert_F\le1 ∥ S ∥ F ≤ 1 and will choose s 0 ≤ 1 s_0\le1 s 0 ≤ 1 below. By Lemma G.6, both Z 0 Z_0 Z 0 and Z 1 Z_1 Z 1 lie in the subspace { Z : 1 K ⊤ Z = 0 } \{Z:\mathbf{1}_K^\top Z=0\} { Z : 1 K ⊤ Z = 0 } . We next establish an a priori F-norm bound to enable application of local strong convexity. The unperturbed bound ∥ Z 0 ∥ ∗ ≤ log K / λ \lVert Z_0\rVert_*\le\log K/\lambda ∥ Z 0 ∥ ∗ ≤ log K / λ is already given in the proof of Lemma D.1 (using Φ ( 0 ; 0 , I ) = log K \Phi(0;0,I)=\log K Φ ( 0 ; 0 , I ) = log K ). The same argument extends to the perturbed problem: by optimality of Z 1 Z_1 Z 1 and feasibility of Z = 0 Z=0 Z = 0 ,
λ ∥ Z 1 ∥ F ≤ λ ∥ Z 1 ∥ ∗ ≤ Φ ( Z 1 ; S , R ) ≤ Φ ( 0 ; S , R ) = L ( 0 ; S , R ) ≤ log K + 2 ∥ S ∥ F , (41) \lambda\lVert Z_1\rVert_F\le\lambda\lVert Z_1\rVert_*\le\Phi(Z_1;S,R)\le\Phi(0;S,R)=L(0;S,R)\le\log K+2\lVert S\rVert_F, \tag*{(41)} λ ∥ Z 1 ∥ F ≤ λ ∥ Z 1 ∥ ∗ ≤ Φ ( Z 1 ; S , R ) ≤ Φ ( 0 ; S , R ) = L ( 0 ; S , R ) ≤ log K + 2 ∥ S ∥ F , (41)
where the last inequality uses L C E ( y c , z ) = − z c + log ∑ i e z i ≤ log K + 2 ∥ z ∥ ∞ L_{\mathrm{CE}}(y_c,z)=-z_c+\log\sum_i e^{z_i}\le\log K+2\lVert z\rVert_\infty L CE ( y c , z ) = − z c + log ∑ i e z i ≤ log K + 2 ∥ z ∥ ∞ together with ∥ S e ( c − 1 ) n + j ∥ ∞ ≤ ∥ S ∥ F \lVert S e_{(c-1)n+j}\rVert_\infty\le\lVert S\rVert_F ∥ S e ( c − 1 ) n + j ∥ ∞ ≤ ∥ S ∥ F . Define
R 0 : = log K + 2 λ , B : = { Z ∈ R K × K n ˉ : ∥ Z ∥ F ≤ R 0 , 1 K ⊤ Z = 0 } , R_0:=\frac{\log K+2}{\lambda},\qquad\mathcal{B}:=\{Z\in\mathbb{R}^{K\times K\bar n}:\lVert Z\rVert_F\le R_0,\ \mathbf{1}_K^\top Z=0\}, R 0 := λ log K + 2 , B := { Z ∈ R K × K n ˉ : ∥ Z ∥ F ≤ R 0 , 1 K ⊤ Z = 0 } ,
so that { Z 0 , Z 1 } ⊂ B \{Z_0,Z_1\}\subset\mathcal{B} { Z 0 , Z 1 } ⊂ B . By Lemma G.7 applied with radius R 0 R_0 R 0 , L ( ⋅ ; 0 , I ) L(\cdot;0,I) L ( ⋅ ; 0 , I ) is μ 0 \mu_0 μ 0 -strongly convex on B \mathcal{B} B in directions Δ Z ∈ { Δ Z : 1 K ⊤ Δ Z = 0 } \Delta Z\in\{\Delta Z:\mathbf{1}_K^\top\Delta Z=0\} Δ Z ∈ { Δ Z : 1 K ⊤ Δ Z = 0 } , where μ 0 = μ 0 ( R 0 ) > 0 \mu_0=\mu_0(R_0)>0 μ 0 = μ 0 ( R 0 ) > 0 . Consequently
⟨ ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 0 ; 0 , I ) , Z 1 − Z 0 ⟩ ≥ μ 0 ∥ Z 1 − Z 0 ∥ F 2 . (42) \langle\nabla L(Z_1;0,I)-\nabla L(Z_0;0,I),Z_1-Z_0\rangle\ge\mu_0\lVert Z_1-Z_0\rVert_F^2. \tag*{(42)} ⟨ ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 0 ; 0 , I ) , Z 1 − Z 0 ⟩ ≥ μ 0 ∥ Z 1 − Z 0 ∥ F 2 . (42)
Subtracting from (40),
μ 0 ∥ Z 1 − Z 0 ∥ F 2 ≤ ⟨ ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 1 ; S , R ) , Z 1 − Z 0 ⟩ (43) \mu_0\lVert Z_1-Z_0\rVert_F^2\le\langle\nabla L(Z_1;0,I)-\nabla L(Z_1;S,R),Z_1-Z_0\rangle \tag*{(43)} μ 0 ∥ Z 1 − Z 0 ∥ F 2 ≤ ⟨ ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 1 ; S , R ) , Z 1 − Z 0 ⟩ (43)
≤ ∥ ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 1 ; S , R ) ∥ F ∥ Z 1 − Z 0 ∥ F , (44) \le\lVert\nabla L(Z_1;0,I)-\nabla L(Z_1;S,R)\rVert_F\lVert Z_1-Z_0\rVert_F, \tag*{(44)} ≤ ∥ ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 1 ; S , R ) ∥ F ∥ Z 1 − Z 0 ∥ F , (44)
hence
μ 0 ∥ Z 1 − Z 0 ∥ F ≤ ∥ ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 1 ; S , R ) ∥ F . (45) \mu_0\lVert Z_1-Z_0\rVert_F\le\lVert\nabla L(Z_1;0,I)-\nabla L(Z_1;S,R)\rVert_F. \tag*{(45)} μ 0 ∥ Z 1 − Z 0 ∥ F ≤ ∥ ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 1 ; S , R ) ∥ F . (45)
It remains to bound the right-hand side of (45). Direct calculation gives
∇ L ( Z ; S ˉ , R ˉ ) = 1 K n ˉ ∑ c , j σ c , j ( Z ; S ˉ , R ˉ ) ( R ˉ e ( c − 1 ) n + j ) ⊤ , (46) \nabla L(Z;\bar S,\bar R)=\frac{1}{K\bar n}\sum_{c,j}\sigma_{c,j}(Z;\bar S,\bar R)(\bar R e_{(c-1)n+j})^\top, \tag*{(46)} ∇ L ( Z ; S ˉ , R ˉ ) = K n ˉ 1 c , j ∑ σ c , j ( Z ; S ˉ , R ˉ ) ( R ˉ e ( c − 1 ) n + j ) ⊤ , (46)
where σ c , j ( Z ; S ˉ , R ˉ ) : = softmax ( ( Z R ˉ + S ˉ ) e ( c − 1 ) n + j ) − y c ∈ R K \sigma_{c,j}(Z;\bar S,\bar R):=\operatorname{softmax}((Z\bar R+\bar S)e_{(c-1)n+j})-y_c\in\mathbb{R}^K σ c , j ( Z ; S ˉ , R ˉ ) := softmax (( Z R ˉ + S ˉ ) e ( c − 1 ) n + j ) − y c ∈ R K . Splitting,
∇ L ( Z ; S , R ) − ∇ L ( Z ; 0 , I ) = 1 K n ˉ ∑ c , j [ ( σ c , j ( Z ; S , R ) − σ c , j ( Z ; 0 , I ) ) ( R e ( c − 1 ) n + j ) ⊤ + σ c , j ( Z ; 0 , I ) ( ( R − I ) e ( c − 1 ) n + j ) ⊤ ] . (47) \begin{aligned}
\nabla L(Z;S,R)-\nabla L(Z;0,I)=\frac{1}{K\bar n}\sum_{c,j}[(\sigma_{c,j}(Z;S,R)-\sigma_{c,j}(Z;0,I))(R e_{(c-1)n+j})^\top\\
&+\sigma_{c,j}(Z;0,I)((R-I)e_{(c-1)n+j})^\top]. \tag*{(47)}
\end{aligned} ∇ L ( Z ; S , R ) − ∇ L ( Z ; 0 , I ) = K n ˉ 1 c , j ∑ [( σ c , j ( Z ; S , R ) − σ c , j ( Z ; 0 , I )) ( R e ( c − 1 ) n + j ) ⊤ + σ c , j ( Z ; 0 , I ) (( R − I ) e ( c − 1 ) n + j ) ⊤ ] . (47)
The softmax map is 1-Lipschitz, so
∥ σ c , j ( Z ; S , R ) − σ c , j ( Z ; 0 , I ) ∥ 2 ≤ ∥ ( ( Z R + S ) − Z ) e ( c − 1 ) n + j ∥ 2 ≤ ∥ Z ∥ F ∥ R − I ∥ o p + ∥ S e ( c − 1 ) n + j ∥ 2 . (48) \begin{aligned}
\lVert\sigma_{c,j}(Z;S,R)-\sigma_{c,j}(Z;0,I)\rVert_2\le\lVert((ZR+S)-Z)e_{(c-1)n+j}\rVert_2 \\
&\le\lVert Z\rVert_F\lVert R-I\rVert_{\mathrm{op}}+\lVert S e_{(c-1)n+j}\rVert_2. \tag*{(48)}
\end{aligned} ∥ σ c , j ( Z ; S , R ) − σ c , j ( Z ; 0 , I ) ∥ 2 ≤ ∥(( Z R + S ) − Z ) e ( c − 1 ) n + j ∥ 2 ≤ ∥ Z ∥ F ∥ R − I ∥ op + ∥ S e ( c − 1 ) n + j ∥ 2 . (48)
and ∥ σ c , j ( Z ; 0 , I ) ∥ 2 ≤ 2 \|\sigma_{c,j}(Z;0,I)\|_{2}\le\sqrt{2} ∥ σ c , j ( Z ; 0 , I ) ∥ 2 ≤ 2 since σ c , j \sigma_{c,j} σ c , j is the difference of a probability vector and a one-hot label, both in [ 0 , 1 ] K [0,1]^{K} [ 0 , 1 ] K . Combining via the triangle inequality and Cauchy–Schwarz (using ∑ c , j ∥ S e ( c − 1 ) n + j ∥ 2 ≤ K ˉ n ∥ S ∥ F \sum_{c,j}\|Se_{(c-1)n+j}\|_{2}\le\sqrt{\bar{K}n}\|S\|_{F} ∑ c , j ∥ S e ( c − 1 ) n + j ∥ 2 ≤ K ˉ n ∥ S ∥ F and ∑ c , j ∥ ( R − I ) e ( c − 1 ) n + j ∥ 2 ≤ K ˉ n ∥ R − I ∥ F ≤ K ˉ n δ \sum_{c,j}\|(R-I)e_{(c-1)n+j}\|_{2}\le\sqrt{\bar{K}n}\|R-I\|_{F}\le\sqrt{\bar{K}n}\delta ∑ c , j ∥ ( R − I ) e ( c − 1 ) n + j ∥ 2 ≤ K ˉ n ∥ R − I ∥ F ≤ K ˉ n δ ),
∥ ∇ L ( Z ; S , R ) − ∇ L ( Z ; 0 , I ) ∥ F ≤ ( 1 + δ ) ( δ ∥ Z ∥ F + ∥ S ∥ F K ˉ n ) + 2 δ . (49) \|\nabla L(Z;S,R)-\nabla L(Z;0,I)\|_{F}\le(1+\delta)\left(\delta\|Z\|_{F}+\frac{\|S\|_{F}}{\sqrt{\bar{K}n}}\right)+\sqrt{2}\delta. \tag*{(49)} ∥∇ L ( Z ; S , R ) − ∇ L ( Z ; 0 , I ) ∥ F ≤ ( 1 + δ ) ( δ ∥ Z ∥ F + K ˉ n ∥ S ∥ F ) + 2 δ . (49)
Evaluating at Z = Z 1 Z=Z_{1} Z = Z 1 , using ∥ Z 1 ∥ F ≤ ∥ Z 0 ∥ F + ∥ Z 1 − Z 0 ∥ F \|Z_{1}\|_{F}\le\|Z_{0}\|_{F}+\|Z_{1}-Z_{0}\|_{F} ∥ Z 1 ∥ F ≤ ∥ Z 0 ∥ F + ∥ Z 1 − Z 0 ∥ F , and substituting into (45):
( μ 0 − ( 1 + δ ) δ ) ∥ Z 1 − Z 0 ∥ F ≤ ( 1 + δ ) ( δ ∥ Z 0 ∥ F + ∥ S ∥ F K ˉ n ) + 2 δ . (50) \left(\mu_{0}-(1+\delta)\delta\right)\|Z_{1}-Z_{0}\|_{F}\le(1+\delta)\left(\delta\|Z_{0}\|_{F}+\frac{\|S\|_{F}}{\sqrt{\bar{K}n}}\right)+\sqrt{2}\delta. \tag*{(50)} ( μ 0 − ( 1 + δ ) δ ) ∥ Z 1 − Z 0 ∥ F ≤ ( 1 + δ ) ( δ ∥ Z 0 ∥ F + K ˉ n ∥ S ∥ F ) + 2 δ . (50)
Set η 0 : = min ( 1 / 2 , μ 0 / 3 ) ∈ ( 0 , 1 / 2 ] \eta_{0}:=\min(1/2,\mu_{0}/3)\in(0,1/2] η 0 := min ( 1/2 , μ 0 /3 ) ∈ ( 0 , 1/2 ] . For any δ ≤ η 0 \delta\le\eta_{0} δ ≤ η 0 we have 1 + δ ≤ 3 / 2 1+\delta\le3/2 1 + δ ≤ 3/2 and
( 1 + δ ) δ ≤ 3 2 δ ≤ 3 2 μ 0 3 = μ 0 2 , (1+\delta)\delta\le\frac{3}{2}\delta\le\frac{3}{2}\frac{\mu_{0}}{3}=\frac{\mu_{0}}{2}, ( 1 + δ ) δ ≤ 2 3 δ ≤ 2 3 3 μ 0 = 2 μ 0 ,
so μ 0 − ( 1 + δ ) δ ≥ μ 0 / 2 > 0 \mu_{0}-(1+\delta)\delta\ge\mu_{0}/2>0 μ 0 − ( 1 + δ ) δ ≥ μ 0 /2 > 0 . Dividing through and using 1 + δ ≤ 3 / 2 1+\delta\le3/2 1 + δ ≤ 3/2 , 2 δ ≤ 2 δ ( 1 + ∥ Z 0 ∥ F ) \sqrt{2}\delta\le\sqrt{2}\delta(1+\|Z_{0}\|_{F}) 2 δ ≤ 2 δ ( 1 + ∥ Z 0 ∥ F ) ,
∥ Z 1 − Z 0 ∥ F ≤ C 1 μ 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z 0 ∥ F ) ) , (51) \|Z_{1}-Z_{0}\|_{F}\le\frac{C_{1}}{\mu_{0}}\left(\|S\|_{F}+\delta(1+\|Z_{0}\|_{F})\right), \tag*{(51)} ∥ Z 1 − Z 0 ∥ F ≤ μ 0 C 1 ( ∥ S ∥ F + δ ( 1 + ∥ Z 0 ∥ F ) ) , (51)
where C 1 C_{1} C 1 is an absolute constant absorbing the factors 2 2 2 , 3 / K ˉ n 3/\sqrt{\bar{K}n} 3/ K ˉ n , 2 \sqrt{2} 2 from the previous display.
By Lemma D.5 below, there exist thresholds δ 0 ′ ≤ η 0 \delta_{0}'\le\eta_{0} δ 0 ′ ≤ η 0 and s 0 ≤ 1 s_{0}\le1 s 0 ≤ 1 such that rank ( Z 1 ) ≤ K ˉ ≤ r \operatorname{rank}(Z_{1})\le\bar{K}\le r rank ( Z 1 ) ≤ K ˉ ≤ r whenever δ ≤ δ 0 ′ \delta\le\delta_{0}' δ ≤ δ 0 ′ and ∥ S ∥ F ≤ s 0 \|S\|_{F}\le s_{0} ∥ S ∥ F ≤ s 0 . Choose the theorem’s threshold δ 0 : = δ 0 ′ \delta_{0}:=\delta_{0}' δ 0 := δ 0 ′ . Under the joint conditions in the statement, the rank constraint is therefore non-binding, so B ∗ A ~ ∗ = Z 1 B^{*}\tilde{A}^{*}=Z_{1} B ∗ A ~ ∗ = Z 1 and B ∗ A ∗ = Z 1 Q T B^{*}A^{*}=Z_{1}Q^{\mathsf{T}} B ∗ A ∗ = Z 1 Q T .
Finally, we bound ∥ B λ ∗ A λ ∗ − Z 0 X f t T ∥ F \|B^{*}_{\lambda}A^{*}_{\lambda}-Z_{0}X_{\mathrm{ft}}^{\mathsf{T}}\|_{F} ∥ B λ ∗ A λ ∗ − Z 0 X ft T ∥ F . From the reformulation above, B λ ∗ A λ ∗ = Z 1 Q T B^{*}_{\lambda}A^{*}_{\lambda}=Z_{1}Q^{\mathsf{T}} B λ ∗ A λ ∗ = Z 1 Q T . Using X f t = Q R X_{\mathrm{ft}}=QR X ft = QR , hence X f t T = R Q T X_{\mathrm{ft}}^{\mathsf{T}}=RQ^{\mathsf{T}} X ft T = R Q T , we obtain Z 0 X f t T = Z 0 R Q T Z_{0}X_{\mathrm{ft}}^{\mathsf{T}}=Z_{0}RQ^{\mathsf{T}} Z 0 X ft T = Z 0 R Q T , so
∥ B λ ∗ A λ ∗ − Z 0 X f t T ∥ F = ∥ ( Z 1 − Z 0 R ) Q T ∥ F = ∥ Z 1 − Z 0 R ∥ F (since Q T Q = I ) ≤ ∥ Z 1 − Z 0 ∥ F + ∥ Z 0 ( I − R ) ∥ F ≤ ∥ Z 1 − Z 0 ∥ F + δ ∥ Z 0 ∥ F . (52) \begin{aligned}
\|B^{*}_{\lambda}A^{*}_{\lambda}-Z_{0}X_{\mathrm{ft}}^{\mathsf{T}}\|_{F}
&=\|(Z_{1}-Z_{0}R)Q^{\mathsf{T}}\|_{F}\\
&=\|Z_{1}-Z_{0}R\|_{F} && \text{(since } Q^{\mathsf{T}}Q=I\text{)}\\
&\le\|Z_{1}-Z_{0}\|_{F}+\|Z_{0}(I-R)\|_{F}\\
&\le\|Z_{1}-Z_{0}\|_{F}+\delta\|Z_{0}\|_{F}. \tag*{(52)}
\end{aligned} ∥ B λ ∗ A λ ∗ − Z 0 X ft T ∥ F = ∥ ( Z 1 − Z 0 R ) Q T ∥ F = ∥ Z 1 − Z 0 R ∥ F ≤ ∥ Z 1 − Z 0 ∥ F + ∥ Z 0 ( I − R ) ∥ F ≤ ∥ Z 1 − Z 0 ∥ F + δ ∥ Z 0 ∥ F . (since Q T Q = I ) (52)
Combining with (52) and using δ ∥ Z 0 ∥ F ≤ δ ( 1 + ∥ Z 0 ∥ F ) \delta\|Z_{0}\|_{F}\le\delta(1+\|Z_{0}\|_{F}) δ ∥ Z 0 ∥ F ≤ δ ( 1 + ∥ Z 0 ∥ F ) ,
∥ B λ ∗ A λ ∗ − Z 0 X f t T ∥ F ≤ C 1 μ 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z 0 ∥ F ) ) + δ ( 1 + ∥ Z 0 ∥ F ) (53) \|B^{*}_{\lambda}A^{*}_{\lambda}-Z_{0}X_{\mathrm{ft}}^{\mathsf{T}}\|_{F}\le\frac{C_{1}}{\mu_{0}}\left(\|S\|_{F}+\delta(1+\|Z_{0}\|_{F})\right)+\delta(1+\|Z_{0}\|_{F}) \tag*{(53)} ∥ B λ ∗ A λ ∗ − Z 0 X ft T ∥ F ≤ μ 0 C 1 ( ∥ S ∥ F + δ ( 1 + ∥ Z 0 ∥ F ) ) + δ ( 1 + ∥ Z 0 ∥ F ) (53)
≤ C 1 + μ 0 μ 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z 0 ∥ F ) ) . (54) \le\frac{C_{1}+\mu_{0}}{\mu_{0}}\left(\|S\|_{F}+\delta(1+\|Z_{0}\|_{F})\right). \tag*{(54)} ≤ μ 0 C 1 + μ 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z 0 ∥ F ) ) . (54)
Setting C 0 : = C 1 + μ 0 C_{0}:=C_{1}+\mu_{0} C 0 := C 1 + μ 0 (which depends only on λ \lambda λ , K K K , K ˉ \bar{K} K ˉ , n n n , ∥ Z 0 ∥ ∗ \|Z_{0}\|_{*} ∥ Z 0 ∥ ∗ , as required) gives (33). □ \square □
Lemma D.5 (Rank stability of the unconstrained minimizer). Adopt the notation from the proof of Theorem D.2. Let Z λ ∗ ( S ˉ , R ˉ ) Z_{\lambda}^{*}(\bar{S},\bar{R}) Z λ ∗ ( S ˉ , R ˉ ) denote the unique unconstrained minimizer of Φ ( ⋅ ; S ˉ , R ˉ ) = L ( ⋅ ; S ˉ , R ˉ ) + λ ∥ ⋅ ∥ ∗ \Phi(\cdot;\bar{S},\bar{R})=L(\cdot;\bar{S},\bar{R})+\lambda\|\cdot\|_{*} Φ ( ⋅ ; S ˉ , R ˉ ) = L ( ⋅ ; S ˉ , R ˉ ) + λ ∥ ⋅ ∥ ∗ over R K × K ˉ n \mathbb{R}^{K\times\bar{K}n} R K × K ˉ n (Lemma G.6), and write Z 0 : = Z λ ∗ ( 0 , I ) Z_{0}:=Z_{\lambda}^{*}(0,I) Z 0 := Z λ ∗ ( 0 , I ) . There exist constants δ 0 ′ ∈ ( 0 , η 0 ] \delta_{0}'\in(0,\eta_{0}] δ 0 ′ ∈ ( 0 , η 0 ] and s 0 ∈ ( 0 , 1 ] s_{0}\in(0,1] s 0 ∈ ( 0 , 1 ] , depending on λ \lambda λ , K K K , K ˉ \bar{K} K ˉ , n n n , μ 0 \mu_{0} μ 0 , C 1 C_{1} C 1 , ∥ Z 0 ∥ ∗ \|Z_{0}\|_{*} ∥ Z 0 ∥ ∗ but not on ϵ 1 , ϵ 2 \epsilon_{1},\epsilon_{2} ϵ 1 , ϵ 2 , such that whenever δ ≤ δ 0 ′ \delta\le\delta_{0}' δ ≤ δ 0 ′ and ∥ S ∥ F ≤ s 0 \|S\|_{F}\le s_{0} ∥ S ∥ F ≤ s 0 ,
rank ( Z λ ∗ ( S , R ) ) ≤ K ˉ . \operatorname{rank}(Z_{\lambda}^{*}(S,R))\le\bar{K}. rank ( Z λ ∗ ( S , R )) ≤ K ˉ .
Proof. Write Z 1 : = Z λ ∗ ( S , R ) Z_{1}:=Z_{\lambda}^{*}(S,R) Z 1 := Z λ ∗ ( S , R ) and G i : = − ∇ L ( Z i ; S ˉ i , R ˉ i ) / λ G_{i}:=-\nabla L(Z_{i};\bar{S}_{i},\bar{R}_{i})/\lambda G i := − ∇ L ( Z i ; S ˉ i , R ˉ i ) / λ for i ∈ { 0 , 1 } i\in\{0,1\} i ∈ { 0 , 1 } , with ( S ˉ 0 , R ˉ 0 ) = ( 0 , I ) (\bar{S}_{0},\bar{R}_{0})=(0,I) ( S ˉ 0 , R ˉ 0 ) = ( 0 , I ) and ( S ˉ 1 , R ˉ 1 ) = ( S , R ) (\bar{S}_{1},\bar{R}_{1})=(S,R) ( S ˉ 1 , R ˉ 1 ) = ( S , R ) .
By the within-class permutation argument in the proof of Lemma D.2, Z 0 = V ⊗ 1 n T Z_{0}=V\otimes\mathbf{1}_{n}^{\mathsf{T}} Z 0 = V ⊗ 1 n T for some V ∈ R K × K ˉ V\in\mathbb{R}^{K\times\bar{K}} V ∈ R K × K ˉ .
Consequently, σ ( Z 0 e ( c − 1 ) n + j ) = σ ( V : , c ) = : p c \sigma(Z_{0}e_{(c-1)n+j})=\sigma(V_{:,c})=:p_{c} σ ( Z 0 e ( c − 1 ) n + j ) = σ ( V : , c ) =: p c depends only on c c c . Letting D : = ∑ c = 1 K ˉ ( p c − y c ) e c T ∈ R K × K ˉ D:=\sum_{c=1}^{\bar{K}}(p_{c}-y_{c})e_{c}^{\mathsf{T}}\in\mathbb{R}^{K\times\bar{K}} D := ∑ c = 1 K ˉ ( p c − y c ) e c T ∈ R K × K ˉ ,
∇ L ( Z 0 ; 0 , I ) = 1 K ˉ n ∑ c = 1 K ˉ ( p c − y c ) ∑ j = 1 n e ( c − 1 ) n + j T = 1 K ˉ n D ⊗ 1 n T , (55) \nabla L(Z_{0};0,I)=\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}(p_{c}-y_{c})\sum_{j=1}^{n}e_{(c-1)n+j}^{\mathsf{T}}=\frac{1}{\bar{K}n}D\otimes\mathbf{1}_{n}^{\mathsf{T}}, \tag*{(55)} ∇ L ( Z 0 ; 0 , I ) = K ˉ n 1 c = 1 ∑ K ˉ ( p c − y c ) j = 1 ∑ n e ( c − 1 ) n + j T = K ˉ n 1 D ⊗ 1 n T , (55)
so rank ( ∇ L ( Z 0 ; 0 , I ) ) ≤ rank ( D ) ≤ K ˉ \operatorname{rank}(\nabla L(Z_0;0,I)) \le\operatorname{rank}(D) \le\bar K rank ( ∇ L ( Z 0 ; 0 , I )) ≤ rank ( D ) ≤ K ˉ , and
σ i ( G 0 ) = 0 for all i > K ˉ . (56) \sigma_i(G_0)=0 \quad\text{for all } i>\bar K . \tag*{(56)} σ i ( G 0 ) = 0 for all i > K ˉ . (56)
Then, we study the first-order optimality condition for the unconstrained convex problem reads G 1 ∈ ∂ ∥ Z 1 ∥ ∗ G_1 \in\partial\|Z_1\|_* G 1 ∈ ∂ ∥ Z 1 ∥ ∗ . Letting Z 1 = U 1 Σ 1 V 1 T Z_1=U_1\Sigma_1V_1^\mathsf{T} Z 1 = U 1 Σ 1 V 1 T be a compact SVD of rank r 1 : = rank ( Z 1 ) r_1:=\operatorname{rank}(Z_1) r 1 := rank ( Z 1 ) , the standard subdifferential characterization of the nuclear norm [50 ] gives G 1 = U 1 V 1 T + W G_1=U_1V_1^\mathsf{T}+W G 1 = U 1 V 1 T + W with U 1 T W = 0 U_1^\mathsf{T}W=0 U 1 T W = 0 , W V 1 = 0 WV_1=0 W V 1 = 0 , and ∥ W ∥ op ≤ 1 \|W\|_{\operatorname{op}}\le1 ∥ W ∥ op ≤ 1 . Since U 1 V 1 T U_1V_1^\mathsf{T} U 1 V 1 T and W W W have orthogonal column and row spaces, their singular values combine: σ i ( G 1 ) = 1 \sigma_i(G_1)=1 σ i ( G 1 ) = 1 for i ≤ r 1 i\le r_1 i ≤ r 1 , and σ i ( G 1 ) = σ i − r 1 ( W ) ≤ 1 \sigma_i(G_1)=\sigma_{i-r_1}(W)\le1 σ i ( G 1 ) = σ i − r 1 ( W ) ≤ 1 for i > r 1 i>r_1 i > r 1 . Applying Weyl’s inequality for singular values to G 1 G_1 G 1 and G 0 G_0 G 0 , together with (57),
σ K ˉ + 1 ( G 1 ) ≤ σ K ˉ + 1 ( G 0 ) + ∥ G 1 − G 0 ∥ op = ∥ G 1 − G 0 ∥ op ≤ ∥ G 1 − G 0 ∥ F . (57) \sigma_{\bar K+1}(G_1)\le\sigma_{\bar K+1}(G_0)+\|G_1-G_0\|_{\operatorname{op}}=\|G_1-G_0\|_{\operatorname{op}}\le\|G_1-G_0\|_F . \tag*{(57)} σ K ˉ + 1 ( G 1 ) ≤ σ K ˉ + 1 ( G 0 ) + ∥ G 1 − G 0 ∥ op = ∥ G 1 − G 0 ∥ op ≤ ∥ G 1 − G 0 ∥ F . (57)
Next, based on the previous analysis, we bound the difference between the gradients. Decompose
∇ L ( Z 1 ; S , R ) − ∇ L ( Z 0 ; 0 , I ) = ∇ L ( Z 1 ; S , R ) − ∇ L ( Z 1 ; 0 , I ) ⏟ = : ( I ) + ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 0 ; 0 , I ) ⏟ = : ( I I ) . \nabla L(Z_1;S,R)-\nabla L(Z_0;0,I)=\underbrace{\nabla L(Z_1;S,R)-\nabla L(Z_1;0,I)}_{=:(\mathrm{I})}+\underbrace{\nabla L(Z_1;0,I)-\nabla L(Z_0;0,I)}_{=:(\mathrm{II})}. ∇ L ( Z 1 ; S , R ) − ∇ L ( Z 0 ; 0 , I ) = =: ( I ) ∇ L ( Z 1 ; S , R ) − ∇ L ( Z 1 ; 0 , I ) + =: ( II ) ∇ L ( Z 1 ; 0 , I ) − ∇ L ( Z 0 ; 0 , I ) .
The bound (59) on (I) follows from the gradient calculation in the proof of Theorem D.2 (the display preceding (52)): for δ ≤ η 0 \delta\le\eta_0 δ ≤ η 0 ,
∥ ( I ) ∥ F ≤ ( 1 + δ ) ( δ ∥ Z 1 ∥ F + ∥ S ∥ F K ˉ n ) + 2 δ . (58) \|(\mathrm{I})\|_F\le(1+\delta)\left(\delta\|Z_1\|_F+\frac{\|S\|_F}{\sqrt{\bar K n}}\right)+\sqrt{2}\,\delta. \tag*{(58)} ∥ ( I ) ∥ F ≤ ( 1 + δ ) ( δ ∥ Z 1 ∥ F + K ˉ n ∥ S ∥ F ) + 2 δ . (58)
For (II), the softmax map σ : R K → Δ K − 1 \sigma:\mathbb{R}^K\to\Delta^{K-1} σ : R K → Δ K − 1 has Jacobian diag ( p ) − p p T \operatorname{diag}(p)-pp^\mathsf{T} diag ( p ) − p p T with operator norm at most 1 1 1 , and is therefore 1-Lipschitz in ℓ 2 \ell_2 ℓ 2 . Since { e ( c − 1 ) n + j } c , j \{e_{(c-1)n+j}\}_{c,j} { e ( c − 1 ) n + j } c , j is an orthonormal basis of R K ˉ n \mathbb{R}^{\bar K n} R K ˉ n ,
∥ ( I I ) ∥ F 2 = 1 ( K ˉ n ) 2 ∑ c , j ∥ σ ( Z 1 e ( c − 1 ) n + j ) − σ ( Z 0 e ( c − 1 ) n + j ) ∥ 2 2 ≤ ∥ Z 1 − Z 0 ∥ F 2 ( K ˉ n ) 2 . (59) \|(\mathrm{II})\|_F^2=\frac{1}{(\bar K n)^2}\sum_{c,j}\left\|\sigma\left(Z_1e_{(c-1)n+j}\right)-\sigma\left(Z_0e_{(c-1)n+j}\right)\right\|_2^2\le\frac{\|Z_1-Z_0\|_F^2}{(\bar K n)^2}. \tag*{(59)} ∥ ( II ) ∥ F 2 = ( K ˉ n ) 2 1 c , j ∑ σ ( Z 1 e ( c − 1 ) n + j ) − σ ( Z 0 e ( c − 1 ) n + j ) 2 2 ≤ ( K ˉ n ) 2 ∥ Z 1 − Z 0 ∥ F 2 . (59)
By (52), ∥ Z 1 − Z 0 ∥ F ≤ C 1 μ 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z 0 ∥ F ) ) \|Z_1-Z_0\|_F\le\frac{C_1}{\mu_0}\left(\|S\|_F+\delta(1+\|Z_0\|_F)\right) ∥ Z 1 − Z 0 ∥ F ≤ μ 0 C 1 ( ∥ S ∥ F + δ ( 1 + ∥ Z 0 ∥ F ) ) . Combined with ∥ Z 1 ∥ F ≤ ∥ Z 0 ∥ F + ∥ Z 1 − Z 0 ∥ F \|Z_1\|_F\le\|Z_0\|_F+\|Z_1-Z_0\|_F ∥ Z 1 ∥ F ≤ ∥ Z 0 ∥ F + ∥ Z 1 − Z 0 ∥ F and restricting to δ ≤ η 0 \delta\le\eta_0 δ ≤ η 0 and ∥ S ∥ F ≤ 1 \|S\|_F\le1 ∥ S ∥ F ≤ 1 , there exists a constant
C 2 : = ( 1 + η 0 ) ( ∥ Z 0 ∥ F + C 1 μ 0 ( 1 + ∥ Z 0 ∥ F ) + 1 K ˉ n ) + 2 + C 1 ( 1 + ∥ Z 0 ∥ F ) μ 0 K ˉ n , C_2:=(1+\eta_0)\left(\|Z_0\|_F+\frac{C_1}{\mu_0}(1+\|Z_0\|_F)+\frac{1}{\sqrt{\bar K n}}\right)+\sqrt{2}+\frac{C_1(1+\|Z_0\|_F)}{\mu_0\bar K n}, C 2 := ( 1 + η 0 ) ( ∥ Z 0 ∥ F + μ 0 C 1 ( 1 + ∥ Z 0 ∥ F ) + K ˉ n 1 ) + 2 + μ 0 K ˉ n C 1 ( 1 + ∥ Z 0 ∥ F ) ,
depending only on λ \lambda λ , K K K , K ˉ \bar K K ˉ , n n n , μ 0 \mu_0 μ 0 , C 1 C_1 C 1 , ∥ Z 0 ∥ ∗ \|Z_0\|_* ∥ Z 0 ∥ ∗ , such that
∥ ∇ L ( Z 1 ; S , R ) − ∇ L ( Z 0 ; 0 , I ) ∥ F ≤ C 2 ( δ + ∥ S ∥ F ) . (60) \|\nabla L(Z_1;S,R)-\nabla L(Z_0;0,I)\|_F\le C_2\left(\delta+\|S\|_F\right). \tag*{(60)} ∥∇ L ( Z 1 ; S , R ) − ∇ L ( Z 0 ; 0 , I ) ∥ F ≤ C 2 ( δ + ∥ S ∥ F ) . (60)
Finally, we use the above bounds to establish a condition under which the rank of Z 1 Z_1 Z 1 is bounded. Set
δ 0 ′ : = min ( η 0 , λ 4 C 2 ) , s 0 : = min ( 1 , λ 4 C 2 ) . \delta'_0:=\min\left(\eta_0,\frac{\lambda}{4C_2}\right),\qquad s_0:=\min\left(1,\frac{\lambda}{4C_2}\right). δ 0 ′ := min ( η 0 , 4 C 2 λ ) , s 0 := min ( 1 , 4 C 2 λ ) .
For δ ≤ δ 0 ′ \delta\le\delta'_0 δ ≤ δ 0 ′ and ∥ S ∥ F ≤ s 0 \|S\|_F\le s_0 ∥ S ∥ F ≤ s 0 ,
∥ G 1 − G 0 ∥ F = 1 λ ∥ ∇ L ( Z 1 ; S , R ) − ∇ L ( Z 0 ; 0 , I ) ∥ F ≤ C 2 ( δ 0 ′ + s 0 ) λ ≤ 1 2 < 1. \|G_1-G_0\|_F=\frac{1}{\lambda}\|\nabla L(Z_1;S,R)-\nabla L(Z_0;0,I)\|_F\le\frac{C_2(\delta'_0+s_0)}{\lambda}\le\frac{1}{2}<1. ∥ G 1 − G 0 ∥ F = λ 1 ∥∇ L ( Z 1 ; S , R ) − ∇ L ( Z 0 ; 0 , I ) ∥ F ≤ λ C 2 ( δ 0 ′ + s 0 ) ≤ 2 1 < 1.
Combined with (58), σ K ˉ + 1 ( G 1 ) < 1 \sigma_{\bar K+1}(G_1)<1 σ K ˉ + 1 ( G 1 ) < 1 . Since at most K ˉ \bar K K ˉ singular values of G 1 G_1 G 1 can equal 1 by Step 2, we conclude rank ( Z 1 ) ≤ K ˉ \operatorname{rank}(Z_1)\le\bar K rank ( Z 1 ) ≤ K ˉ .
Proof of Theorem 3.2 and Corollary 3.1
In this section, we first present the proofs of Theorem 3.2 and Corollary 3.1. Then, under additional smallness assumptions on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 , we show that the same conclusions still hold, with additional error terms that depend on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 .
Proof of Theorem 3.2
For convenience, we will first define the following notations. Let W L o R A λ = W p r e + B λ ∗ A λ ∗ W^{\lambda}_{\mathrm{LoRA}} = W_{\mathrm{pre}} + B^{*}_{\lambda}A^{*}_{\lambda} W LoRA λ = W pre + B λ ∗ A λ ∗ , and define the following shorthand for margins as: γ p r e = γ ( W p r e ; D p r e ) \gamma_{\mathrm{pre}} = \gamma(W_{\mathrm{pre}}; \mathcal{D}_{\mathrm{pre}}) γ pre = γ ( W pre ; D pre ) , γ f t , λ = γ ( B λ ∗ A λ ∗ ; D f t ) \gamma_{\mathrm{ft},\lambda} = \gamma(B^{*}_{\lambda}A^{*}_{\lambda}; \mathcal{D}_{\mathrm{ft}}) γ ft , λ = γ ( B λ ∗ A λ ∗ ; D ft ) , and γ λ = γ ( W L o R A λ ; D p r e ∪ D f t ) \gamma_{\lambda} = \gamma(W^{\lambda}_{\mathrm{LoRA}}; \mathcal{D}_{\mathrm{pre}} \cup\mathcal{D}_{\mathrm{ft}}) γ λ = γ ( W LoRA λ ; D pre ∪ D ft ) . Additionally, given an K ˉ \bar{K} K ˉ -class max-margin classifier W f t ∗ ∈ R K ˉ × d W^{*}_{\mathrm{ft}} \in\mathbb{R}^{\bar{K} \times d} W ft ∗ ∈ R K ˉ × d on the fine-tuning data, let γ f t p a d = γ ( ( [ W f t ∗ ; 0 ( K − K ˉ ) × d ] ) ; D f t ) \gamma^{\mathrm{pad}}_{\mathrm{ft}} = \gamma(([W^{*}_{\mathrm{ft}}; 0_{(K-\bar{K})\times d}]); \mathcal{D}_{\mathrm{ft}}) γ ft pad = γ (([ W ft ∗ ; 0 ( K − K ˉ ) × d ]) ; D ft ) . When K ˉ < K \bar{K} < K K ˉ < K this is in general a lower bound on max W ∈ R K × d γ ( W ; D f t ) \max_{W \in\mathbb{R}^{K \times d}} \gamma(W; \mathcal{D}_{\mathrm{ft}}) max W ∈ R K × d γ ( W ; D ft ) , with equality iff K ˉ = K \bar{K} = K K ˉ = K . In the balanced case K ˉ = K \bar{K} = K K ˉ = K , γ f t p a d = γ f t ∗ \gamma^{\mathrm{pad}}_{\mathrm{ft}} = \gamma^{*}_{\mathrm{ft}} γ ft pad = γ ft ∗ as defined in the main theorem.
With these definitions in place, we restate Theorem 3.2 in expanded form covering the general K ˉ ≤ K \bar{K} \le K K ˉ ≤ K setting. The expanded version specifies the closed-form values of Θ λ \Theta_{\lambda} Θ λ and γ f t , λ \gamma_{\mathrm{ft},\lambda} γ ft , λ in three regimes when K ˉ < K \bar{K} < K K ˉ < K . In the balanced case K ˉ = K \bar{K} = K K ˉ = K , the intermediate and low-penalty regimes coincide in their ETF geometry and margin formulas, recovering the two-regime statement of Theorem 3.2. More precisely, λ c r i t = 0 \lambda_{\mathrm{crit}} = 0 λ crit = 0 , so the intermediate formula holds for every 0 < λ < 1 / ( K n ) 0 < \lambda< 1/(K\sqrt{n}) 0 < λ < 1/ ( K n ) , and the limit λ → 0 + \lambda\to0^{+} λ → 0 + retains the same ETF geometry.
Theorem E.1 (Detailed version of Theorem 3.2). Adopt the setup of Theorem 3.1 and let Θ λ : = ∥ B λ ∗ A λ ∗ ∥ F \Theta_{\lambda} := \lVert B^{*}_{\lambda}A^{*}_{\lambda}\rVert_{F} Θ λ := ∥ B λ ∗ A λ ∗ ∥ F . The normalized margins of W L o R A λ W^{\lambda}_{\mathrm{LoRA}} W LoRA λ on the pre-training and fine-tuning data are
γ ( W L o R A λ ; D p r e ) = γ p r e ρ p r e Θ λ 2 + ρ p r e 2 , γ ( W L o R A λ ; D f t ) = γ f t , λ Θ λ Θ λ 2 + ρ p r e 2 , (61) \gamma(W^{\lambda}_{\mathrm{LoRA}}; \mathcal{D}_{\mathrm{pre}}) = \gamma_{\mathrm{pre}} \frac{\rho_{\mathrm{pre}}}{\sqrt{\Theta_{\lambda}^{2} + \rho_{\mathrm{pre}}^{2}}}, \qquad\gamma(W^{\lambda}_{\mathrm{LoRA}}; \mathcal{D}_{\mathrm{ft}}) = \gamma_{\mathrm{ft},\lambda} \frac{\Theta_{\lambda}}{\sqrt{\Theta_{\lambda}^{2} + \rho_{\mathrm{pre}}^{2}}}, \tag*{(61)} γ ( W LoRA λ ; D pre ) = γ pre Θ λ 2 + ρ pre 2 ρ pre , γ ( W LoRA λ ; D ft ) = γ ft , λ Θ λ 2 + ρ pre 2 Θ λ , (61) which yield the closed-form Adapt( λ ) (\lambda) ( λ ) and Forget( λ ) (\lambda) ( λ ) in (13) via Adapt( λ ) = γ ( W L o R A λ ; D f t ) (\lambda) = \gamma(W^{\lambda}_{\mathrm{LoRA}}; \mathcal{D}_{\mathrm{ft}}) ( λ ) = γ ( W LoRA λ ; D ft ) (since γ ( W p r e ; D f t ) = 0 \gamma(W_{\mathrm{pre}}; \mathcal{D}_{\mathrm{ft}}) = 0 γ ( W pre ; D ft ) = 0 in the strict-orthogonal regime) and Forget( λ ) = γ p r e − γ ( W L o R A λ ; D p r e ) (\lambda) = \gamma_{\mathrm{pre}} - \gamma(W^{\lambda}_{\mathrm{LoRA}}; \mathcal{D}_{\mathrm{pre}}) ( λ ) = γ pre − γ ( W LoRA λ ; D pre ) . Moreover, Θ λ \Theta_{\lambda} Θ λ and γ f t , λ \gamma_{\mathrm{ft},\lambda} γ ft , λ take the following values:
(i) High-penalty regime λ ≥ 1 K n \lambda\ge\frac{1}{K\sqrt{n}} λ ≥ K n 1 : Θ λ = γ f t , λ = 0 \Theta_{\lambda} = \gamma_{\mathrm{ft},\lambda} = 0 Θ λ = γ ft , λ = 0 , and hence Adapt( λ ) = (\lambda) = ( λ ) = Forget( λ ) = 0 (\lambda) = 0 ( λ ) = 0 .
(ii) Intermediate regime λ c r i t < λ < 1 K n \lambda_{\mathrm{crit}} < \lambda< \frac{1}{K\sqrt{n}} λ crit < λ < K n 1 : Θ λ = ρ λ K ˉ − 1 \Theta_{\lambda} = \frac{\rho_{\lambda}}{\sqrt{\bar{K}-1}} Θ λ = K ˉ − 1 ρ λ , γ f t , λ = γ f t p a d \gamma_{\mathrm{ft},\lambda} = \gamma^{\mathrm{pad}}_{\mathrm{ft}} γ ft , λ = γ ft pad , and Θ λ \Theta_{\lambda} Θ λ is a strictly decreasing function of λ \lambda λ . Both Adapt( λ ) (\lambda) ( λ ) and Forget( λ ) (\lambda) ( λ ) are therefore strictly decreasing in λ \lambda λ on this regime.
(iii) Low-penalty regime 0 < λ ≤ λ c r i t 0 < \lambda\le\lambda_{\mathrm{crit}} 0 < λ ≤ λ crit :
Θ λ = n K ˉ a λ 2 + n K ˉ ( K ˉ − 1 ) b λ 2 + n K ˉ ( K − K ˉ ) c λ 2 , γ f t , λ = a λ + c λ Θ λ . \Theta_{\lambda} = \sqrt{n\bar{K}a_{\lambda}^{2} + n\bar{K}(\bar{K}-1)b_{\lambda}^{2} + n\bar{K}(K-\bar{K})c_{\lambda}^{2}}, \qquad\gamma_{\mathrm{ft},\lambda} = \frac{a_{\lambda} + c_{\lambda}}{\Theta_{\lambda}}. Θ λ = n K ˉ a λ 2 + n K ˉ ( K ˉ − 1 ) b λ 2 + n K ˉ ( K − K ˉ ) c λ 2 , γ ft , λ = Θ λ a λ + c λ .
Remark E.1. Full fine-tuning as an alternative method. While our work focuses on LoRA, it is instructive to briefly contrast it with the more conventional strategy of full fine-tuning. Full fine-tuning updates all parameters of the pretrained model, which is considerably less efficient in both computation and storage, whereas LoRA achieves adaptation through a compact low-rank parameterization. In the linear classification setting, the two approaches correspond to the following regularized objectives:
min B , A L ( W p r e + B A ; D f t ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) , LoRA objective , (62) \min_{B,A} \; L(W_{\mathrm{pre}} + BA; \mathcal{D}_{\mathrm{ft}}) + \frac{\lambda}{2}(\lVert B\rVert_{F}^{2} + \lVert A\rVert_{F}^{2}),
\qquad\textit{LoRA objective}, \tag*{(62)} B , A min L ( W pre + B A ; D ft ) + 2 λ (∥ B ∥ F 2 + ∥ A ∥ F 2 ) , LoRA objective , (62)
min W L ( W ; D f t ) + λ 2 ∥ W − W p r e ∥ F 2 , Full fine-tuning objective . (63) \min_{W} \; L(W; \mathcal{D}_{\mathrm{ft}}) + \frac{\lambda}{2}\lVert W - W_{\mathrm{pre}}\rVert_{F}^{2},
\qquad\textit{Full fine-tuning objective}. \tag*{(63)} W min L ( W ; D ft ) + 2 λ ∥ W − W pre ∥ F 2 , Full fine-tuning objective . (63)
Beyond efficiency, the two formulations induce fundamentally different (implicit) biases. By exploiting the variational form of the nuclear norm [[42 ] ], one can show that for any solution ( B ∗ , A ∗ ) (B^{*}, A^{*}) ( B ∗ , A ∗ ) of (63), the induced update W L o R A λ W^{\lambda}_{\mathrm{LoRA}} W LoRA λ is equivalently the solution of
min W rank ( W − W p r e ) ≤ r L ( W ; D f t ) + λ ∥ W − W p r e ∥ ∗ , Implicit bias of LoRA fine-tuning , (64) \min_{\substack{W \\ \operatorname{rank}(W-W_{\mathrm{pre}})\le r}} \; L(W; \mathcal{D}_{\mathrm{ft}}) + \lambda\lVert W-W_{\mathrm{pre}}\rVert_{*},
\qquad\textit{Implicit bias of LoRA fine-tuning}, \tag*{(64)} W rank ( W − W pre ) ≤ r min L ( W ; D ft ) + λ ∥ W − W pre ∥ ∗ , Implicit bias of LoRA fine-tuning , (64)
which highlights LoRA’s connection to nuclear-norm regularization under rank constraints. We refer the readers to Jang et al. [ [23 ] for the proof of the argument. Understanding the consequences of this distinction between nuclear norm regularization and Frobenius norm regularization remains an open direction for future work. Importantly, our goal is not to compare LoRA and full fine-tuning, but to rigorously analyze why LoRA exhibits reduced forgetting and to provide principled guidance for selecting regularization parameters in adapter-based fine-tuning.
Proof. Our key observation is that when one computes the margin of W L o R A λ W_{\mathrm{LoRA}}^\lambda W LoRA λ on the pre-training and fine-tuning datasets, only W p r e W_{\mathrm{pre}} W pre or B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ is activated. This is because by Theorem 3.1, the product of the optimal LoRA adapters lies in the span of fine-tuning data, and by Proposition G.1 together with Assumption 2.1, W p r e W_{\mathrm{pre}} W pre lies in the span of pre-training data. Thus, under Assumption 2.1, B λ ∗ A λ ∗ x = 0 B_\lambda^*A_\lambda^*x=0 B λ ∗ A λ ∗ x = 0 if x ∈ D p r e x\in\mathcal{D}_{\mathrm{pre}} x ∈ D pre , and W p r e x = 0 W_{\mathrm{pre}}x=0 W pre x = 0 if x ∈ D f t x\in\mathcal{D}_{\mathrm{ft}} x ∈ D ft . In particular, γ ( W p r e ; D f t ) = 0 \gamma(W_{\mathrm{pre}};\mathcal{D}_{\mathrm{ft}})=0 γ ( W pre ; D ft ) = 0 , so Adapt ( λ ) = γ ( W L o R A λ ; D f t ) \operatorname{Adapt}(\lambda)=\gamma(W_{\mathrm{LoRA}}^\lambda;\mathcal{D}_{\mathrm{ft}}) Adapt ( λ ) = γ ( W LoRA λ ; D ft ) . Before proving that the margins take the structures shown in Theorem E.1 , we introduce the following notation. Let W p r e = [ w 1 , … , w K ] ⊤ W_{\mathrm{pre}}=[w_1,\ldots,w_K]^\top W pre = [ w 1 , … , w K ] ⊤ and B λ ∗ A λ ∗ = [ δ w λ , 1 , … , δ w λ , K ] ⊤ B_\lambda^*A_\lambda^*=[\delta w_{\lambda,1},\ldots,\delta w_{\lambda,K}]^\top B λ ∗ A λ ∗ = [ δ w λ , 1 , … , δ w λ , K ] ⊤ .
Now, when we compute the margin of W L o R A λ W_{\mathrm{LoRA}}^\lambda W LoRA λ on pre-training data, one has
γ ( W L o R A λ ; D p r e ) = min ( x i , c , y c ) ∈ D p r e ( w c + δ λ , c ) ⊤ x − max i ≠ y ( w i + δ λ , i ) ⊤ x ∥ W L o R A λ ∥ F = min ( x i , c , y c ) ∈ D p r e w c ⊤ x − max i ≠ y w i ⊤ x ∥ W p r e ∥ F ⋅ ∥ W p r e ∥ F ∥ W L o R A λ ∥ F = γ p r e ⋅ ρ p r e ρ p r e 2 + ∥ B λ ∗ A λ ∗ ∥ F 2 . \begin{aligned}
\gamma(W_{\mathrm{LoRA}}^\lambda;\mathcal{D}_{\mathrm{pre}})=\frac{\min_{(x_i,c,y_c)\in\mathcal{D}_{\mathrm{pre}}}(w_c+\delta_{\lambda,c})^\top x-\max_{i\ne y}(w_i+\delta_{\lambda,i})^\top x}{\lVert W_{\mathrm{LoRA}}^\lambda\rVert_F} \\
&=\frac{\min_{(x_i,c,y_c)\in\mathcal{D}_{\mathrm{pre}}}w_c^\top x-\max_{i\ne y}w_i^\top x}{\lVert W_{\mathrm{pre}}\rVert_F}\cdot\frac{\lVert W_{\mathrm{pre}}\rVert_F}{\lVert W_{\mathrm{LoRA}}^\lambda\rVert_F} \\
&=\gamma_{\mathrm{pre}}\cdot\frac{\rho_{\mathrm{pre}}}{\sqrt{\rho_{\mathrm{pre}}^2+\lVert B_\lambda^*A_\lambda^*\rVert_F^2}}.
\end{aligned} γ ( W LoRA λ ; D pre ) = ∥ W LoRA λ ∥ F min ( x i , c , y c ) ∈ D pre ( w c + δ λ , c ) ⊤ x − max i = y ( w i + δ λ , i ) ⊤ x = ∥ W pre ∥ F min ( x i , c , y c ) ∈ D pre w c ⊤ x − max i = y w i ⊤ x ⋅ ∥ W LoRA λ ∥ F ∥ W pre ∥ F = γ pre ⋅ ρ pre 2 + ∥ B λ ∗ A λ ∗ ∥ F 2 ρ pre .
Then, for the fine-tuning data, one has
γ ( W L o R A λ ; D f t ) = min ( x i , c , y c ) ∈ D f t ( w c + δ λ , c ) ⊤ x − max i ≠ y ( w i + δ λ , i ) ⊤ x ∥ W L o R A λ ∥ F = C ( B λ ∗ A λ ∗ ; D f t ) ρ p r e 2 + ∥ B λ ∗ A λ ∗ ∥ F 2 , = γ f t , λ ⋅ ∥ B λ ∗ A λ ∗ ∥ F ρ p r e 2 + ∥ B λ ∗ A λ ∗ ∥ F 2 . \begin{aligned}
\gamma(W_{\mathrm{LoRA}}^\lambda;\mathcal{D}_{\mathrm{ft}})=\frac{\min_{(x_i,c,y_c)\in\mathcal{D}_{\mathrm{ft}}}(w_c+\delta_{\lambda,c})^\top x-\max_{i\ne y}(w_i+\delta_{\lambda,i})^\top x}{\lVert W_{\mathrm{LoRA}}^\lambda\rVert_F} \\
&=\frac{C(B_\lambda^*A_\lambda^*;\mathcal{D}_{\mathrm{ft}})}{\sqrt{\rho_{\mathrm{pre}}^2+\lVert B_\lambda^*A_\lambda^*\rVert_F^2}}, \\
&=\gamma_{\mathrm{ft},\lambda}\cdot\frac{\lVert B_\lambda^*A_\lambda^*\rVert_F}{\sqrt{\rho_{\mathrm{pre}}^2+\lVert B_\lambda^*A_\lambda^*\rVert_F^2}}.
\end{aligned} γ ( W LoRA λ ; D ft ) = ∥ W LoRA λ ∥ F min ( x i , c , y c ) ∈ D ft ( w c + δ λ , c ) ⊤ x − max i = y ( w i + δ λ , i ) ⊤ x = ρ pre 2 + ∥ B λ ∗ A λ ∗ ∥ F 2 C ( B λ ∗ A λ ∗ ; D ft ) , = γ ft , λ ⋅ ρ pre 2 + ∥ B λ ∗ A λ ∗ ∥ F 2 ∥ B λ ∗ A λ ∗ ∥ F .
The last equality also holds when B λ ∗ A λ ∗ = 0 B_\lambda^*A_\lambda^*=0 B λ ∗ A λ ∗ = 0 by the convention γ ( 0 ; D ) = 0 \gamma(0;\mathcal{D})=0 γ ( 0 ; D ) = 0 . These are exactly the formulas in (62) with Θ λ = ∥ B λ ∗ A λ ∗ ∥ F \Theta_\lambda=\lVert B_\lambda^*A_\lambda^*\rVert_F Θ λ = ∥ B λ ∗ A λ ∗ ∥ F , and substituting them into the definitions of Adapt ( λ ) \operatorname{Adapt}(\lambda) Adapt ( λ ) and Forget ( λ ) \operatorname{Forget}(\lambda) Forget ( λ ) recovers (13). It remains to compute ∥ B λ ∗ A λ ∗ ∥ F \lVert B_\lambda^*A_\lambda^*\rVert_F ∥ B λ ∗ A λ ∗ ∥ F in each regime.
High-penalty regime. In this regime, B λ ∗ A λ ∗ = 0 B_\lambda^*A_\lambda^*=0 B λ ∗ A λ ∗ = 0 , so ∥ B λ ∗ A λ ∗ ∥ F = 0 \lVert B_\lambda^*A_\lambda^*\rVert_F=0 ∥ B λ ∗ A λ ∗ ∥ F = 0 .
Intermediate regime. In the intermediate regime, one has
B λ ∗ A λ ∗ = ρ λ K n ( K ˉ − 1 ) ( ( M K ˉ 0 ( K − K ˉ ) × K ˉ ) ⊗ I n ) X f t ⊤ , (65) B_\lambda^*A_\lambda^*=\frac{\rho_\lambda}{\sqrt{K n(\bar K-1)}}\left(\left(\begin{array}{c}M_{\bar K}\\0_{(K-\bar K)\times\bar K}\end{array}\right)\otimes I_n\right)X_{\mathrm{ft}}^\top, \tag*{(65)} B λ ∗ A λ ∗ = K n ( K ˉ − 1 ) ρ λ ( ( M K ˉ 0 ( K − K ˉ ) × K ˉ ) ⊗ I n ) X ft ⊤ , (65)
and one can compute its norm as follows
∥ B λ ∗ A λ ∗ ∥ F 2 = ρ λ 2 K n ( K ˉ − 1 ) ⋅ K ˉ K ˉ − 1 ∥ ( ( I K ˉ − 1 K ˉ 1 K ˉ 1 K ˉ ⊤ ) ⊗ I n ) X f t ⊤ ∥ F 2 = ρ λ 2 K n ( K ˉ − 1 ) ⋅ K ˉ K ˉ − 1 ⋅ K ˉ ( n ⋅ ( K ˉ − 1 K ˉ ) 2 + ( K ˉ − 1 ) n K ˉ 2 ) = ρ λ 2 K ˉ − 1 . \begin{aligned}
\lVert B_\lambda^*A_\lambda^*\rVert_F^2
=\frac{\rho_\lambda^2}{K n(\bar K-1)}\cdot\frac{\bar K}{\bar K-1}\left\lVert\left(\left(I_{\bar K}-\frac{1}{\bar K}1_{\bar K}1_{\bar K}^\top\right)\otimes I_n\right)X_{\mathrm{ft}}^\top\right\rVert_F^2 \\
&=\frac{\rho_\lambda^2}{K n(\bar K-1)}\cdot\frac{\bar K}{\bar K-1}\cdot\bar K\left(n\cdot\left(\frac{\bar K-1}{\bar K}\right)^2+\frac{(\bar K-1)n}{\bar K^2}\right) \\
&=\frac{\rho_\lambda^2}{\bar K-1}.
\end{aligned} ∥ B λ ∗ A λ ∗ ∥ F 2 = K n ( K ˉ − 1 ) ρ λ 2 ⋅ K ˉ − 1 K ˉ ( ( I K ˉ − K ˉ 1 1 K ˉ 1 K ˉ ⊤ ) ⊗ I n ) X ft ⊤ F 2 = K n ( K ˉ − 1 ) ρ λ 2 ⋅ K ˉ − 1 K ˉ ⋅ K ˉ ( n ⋅ ( K ˉ K ˉ − 1 ) 2 + K ˉ 2 ( K ˉ − 1 ) n ) = K ˉ − 1 ρ λ 2 .
Thus, Θ λ = ∥ B λ ∗ A λ ∗ ∥ F = ρ λ K ˉ − 1 \Theta_\lambda=\lVert B_\lambda^*A_\lambda^*\rVert_F=\frac{\rho_\lambda}{\sqrt{\bar K-1}} Θ λ = ∥ B λ ∗ A λ ∗ ∥ F = K ˉ − 1 ρ λ .
Low-penalty regime. In this regime, B λ ∗ A λ ∗ B_\lambda^*A_\lambda^* B λ ∗ A λ ∗ takes the following form
B λ ∗ A λ ∗ = ( ( ( a λ + b λ ) I K ˉ − b λ 1 K ˉ 1 K ˉ ⊤ − c λ 1 K − K ˉ 1 K ˉ ⊤ ) ⊗ I n ) X f t ⊤ , (66) B_\lambda^*A_\lambda^*=\left(\left(\begin{array}{cc}(a_\lambda+b_\lambda)I_{\bar K}-b_\lambda1_{\bar K}1_{\bar K}^\top& -c_\lambda1_{K-\bar K}1_{\bar K}^\top\end{array}\right)\otimes I_n\right)X_{\mathrm{ft}}^\top, \tag*{(66)} B λ ∗ A λ ∗ = ( ( ( a λ + b λ ) I K ˉ − b λ 1 K ˉ 1 K ˉ ⊤ − c λ 1 K − K ˉ 1 K ˉ ⊤ ) ⊗ I n ) X ft ⊤ , (66)
and its Frobenius norm is
∥ B λ ∗ A λ ∗ ∥ F = n ∥ ( ( a λ + b λ ) I K ˉ − b λ 1 K ˉ 1 K ˉ ⊤ − c λ 1 K − K ˉ 1 K ˉ ⊤ ) ∥ F = n ⋅ K ˉ a λ 2 + K ˉ ( K ˉ − 1 ) b λ 2 + ( K − K ˉ ) K c λ 2 . \begin{aligned}
\lVert B_\lambda^*A_\lambda^*\rVert_F=\sqrt{n}\left\lVert\left(\begin{array}{cc}(a_\lambda+b_\lambda)I_{\bar K}-b_\lambda1_{\bar K}1_{\bar K}^\top& -c_\lambda1_{K-\bar K}1_{\bar K}^\top\end{array}\right)\right\rVert_F \\
&=\sqrt{n}\cdot\sqrt{\bar K a_\lambda^2+\bar K(\bar K-1)b_\lambda^2+(K-\bar K)Kc_\lambda^2}.
\end{aligned} ∥ B λ ∗ A λ ∗ ∥ F = n ( ( a λ + b λ ) I K ˉ − b λ 1 K ˉ 1 K ˉ ⊤ − c λ 1 K − K ˉ 1 K ˉ ⊤ ) F = n ⋅ K ˉ a λ 2 + K ˉ ( K ˉ − 1 ) b λ 2 + ( K − K ˉ ) K c λ 2 .
For any sample x ˉ c , j ∈ D f t \bar{x}_{c,j} \in\mathcal{D}_{\mathrm{ft}} x ˉ c , j ∈ D ft in class c ≤ K ˉ c \le\bar{K} c ≤ K ˉ , the score on the correct class is a λ a_{\lambda} a λ , the score on any other class k ≤ K ˉ , k ≠ c k \le\bar{K}, k \ne c k ≤ K ˉ , k = c is − b λ -b_{\lambda} − b λ , and the score on any class k > K ˉ k > \bar{K} k > K ˉ is − c λ -c_{\lambda} − c λ . Hence the per-sample margin is a λ − max ( − b λ , − c λ ) = a λ + min ( b λ , c λ ) a_{\lambda} - \max(-b_{\lambda},-c_{\lambda}) = a_{\lambda} + \min(b_{\lambda},c_{\lambda}) a λ − max ( − b λ , − c λ ) = a λ + min ( b λ , c λ ) . By the defining equation of x 0 x_0 x 0 in Lemma D.4, g b ( x 0 ) / g c ( x 0 ) = 1 / ( K / ( K − K ˉ ) + 1 ) < 1 g_b(x_0)/g_c(x_0) = 1/(\sqrt{K}/(K-\bar{K})+1) < 1 g b ( x 0 ) / g c ( x 0 ) = 1/ ( K / ( K − K ˉ ) + 1 ) < 1 , so log g b ( x 0 ) < log g c ( x 0 ) \log g_b(x_0) < \log g_c(x_0) log g b ( x 0 ) < log g c ( x 0 ) . Combined with b λ = − log g b ( x 0 ) b_{\lambda} = -\log g_b(x_0) b λ = − log g b ( x 0 ) and c λ = − log g c ( x 0 ) c_{\lambda} = -\log g_c(x_0) c λ = − log g c ( x 0 ) , this gives b λ > c λ b_{\lambda} > c_{\lambda} b λ > c λ , so min ( b λ , c λ ) = c λ \min(b_{\lambda},c_{\lambda}) = c_{\lambda} min ( b λ , c λ ) = c λ and the unnormalized fine-tuning margin equals a λ + c λ a_{\lambda} + c_{\lambda} a λ + c λ . Dividing by Θ λ = ∥ B λ ∗ A λ ∗ ∥ F \Theta_{\lambda} = \lVert B_{\lambda}^{*}A_{\lambda}^{*}\rVert_F Θ λ = ∥ B λ ∗ A λ ∗ ∥ F yields γ f t , λ = ( a λ + c λ ) / Θ λ \gamma_{\mathrm{ft},\lambda} = (a_{\lambda} + c_{\lambda})/\Theta_{\lambda} γ ft , λ = ( a λ + c λ ) / Θ λ .
Proof of Corollary 3.1
Proof. The first identity in Corollary 3.1,
γ ( W L o R A λ ; D p r e ∪ D f t ) = min { γ p r e − Forget ( λ ) , Adapt ( λ ) } , \gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}})=\min\{\gamma_{\mathrm{pre}}-\operatorname{Forget}(\lambda),\operatorname{Adapt}(\lambda)\}, γ ( W LoRA λ ; D pre ∪ D ft ) = min { γ pre − Forget ( λ ) , Adapt ( λ )} ,
follows from the definition of the union margin together with γ ( W L o R A λ ; D p r e ) = γ p r e − Forget ( λ ) \gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}})=\gamma_{\mathrm{pre}}-\operatorname{Forget}(\lambda) γ ( W LoRA λ ; D pre ) = γ pre − Forget ( λ ) and γ ( W L o R A λ ; D f t ) = Adapt ( λ ) \gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{ft}})=\operatorname{Adapt}(\lambda) γ ( W LoRA λ ; D ft ) = Adapt ( λ ) from Theorem 3.2. It remains to maximize this minimum over λ \lambda λ .
We work in the balanced setting K ˉ = K \bar{K}=K K ˉ = K of Corollary 3.1. In the high-penalty regime ( λ ≥ 1 / ( K n ) ) (\lambda\ge1/(K\sqrt{n})) ( λ ≥ 1/ ( K n )) , Θ λ = 0 \Theta_{\lambda}=0 Θ λ = 0 and so Adapt ( λ ) = 0 \operatorname{Adapt}(\lambda)=0 Adapt ( λ ) = 0 , which gives a union margin of 0. Every 0 < λ < 1 / ( K n ) 0<\lambda<1/(K\sqrt{n}) 0 < λ < 1/ ( K n ) gives a positive union margin, so the optimum lies in this interval.
On this interval, γ f t , λ = γ f t ∗ \gamma_{\mathrm{ft},\lambda}=\gamma_{\mathrm{ft}}^{*} γ ft , λ = γ ft ∗ is constant. By the balanced-case solution in Appendix D, Θ λ \Theta_{\lambda} Θ λ decreases continuously and strictly from + ∞ +\infty + ∞ to 0 as λ \lambda λ increases from 0 to 1 / ( K n ) 1/(K\sqrt{n}) 1/ ( K n ) . From (62), the pre-training margin is strictly decreasing in Θ λ \Theta_{\lambda} Θ λ and the fine-tuning margin is strictly increasing in Θ λ \Theta_{\lambda} Θ λ . Their minimum is therefore uniquely maximized where they are equal:
γ p r e ρ p r e Θ λ 2 + ρ p r e 2 = γ f t ∗ Θ λ Θ λ 2 + ρ f t 2 ⟺ Θ λ = Θ ∗ : = ρ p r e γ p r e γ f t ∗ . \gamma_{\mathrm{pre}}\frac{\rho_{\mathrm{pre}}}{\sqrt{\Theta_{\lambda}^{2}+\rho_{\mathrm{pre}}^{2}}}=\gamma_{\mathrm{ft}}^{*}\frac{\Theta_{\lambda}}{\sqrt{\Theta_{\lambda}^{2}+\rho_{\mathrm{ft}}^{2}}}
\Longleftrightarrow\Theta_{\lambda}=\Theta^{*}:=\frac{\rho_{\mathrm{pre}}\gamma_{\mathrm{pre}}}{\gamma_{\mathrm{ft}}^{*}}. γ pre Θ λ 2 + ρ pre 2 ρ pre = γ ft ∗ Θ λ 2 + ρ ft 2 Θ λ ⟺ Θ λ = Θ ∗ := γ ft ∗ ρ pre γ pre .
Since 0 < Θ ∗ < ∞ 0<\Theta^{*}<\infty 0 < Θ ∗ < ∞ , there is a unique λ ∗ ∈ ( 0 , 1 / ( K n ) ) \lambda^{*}\in(0,1/(K\sqrt{n})) λ ∗ ∈ ( 0 , 1/ ( K n )) with Θ λ ∗ = Θ ∗ \Theta_{\lambda^{*}}=\Theta^{*} Θ λ ∗ = Θ ∗ . No additional balancing condition is needed. Substituting Θ ∗ \Theta^{*} Θ ∗ into (62) gives
γ ( W L o R A λ ∗ ; D p r e ∪ D f t ) = γ p r e γ f t ∗ γ p r e 2 + ( γ f t ∗ ) 2 = ( 1 ( γ f t ∗ ) 2 + 1 γ p r e 2 ) − 1 / 2 . \gamma(W_{\mathrm{LoRA}}^{\lambda^{*}};\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}})=\frac{\gamma_{\mathrm{pre}}\gamma_{\mathrm{ft}}^{*}}{\sqrt{\gamma_{\mathrm{pre}}^{2}+(\gamma_{\mathrm{ft}}^{*})^{2}}}=\left(\frac{1}{(\gamma_{\mathrm{ft}}^{*})^{2}}+\frac{1}{\gamma_{\mathrm{pre}}^{2}}\right)^{-1/2}. γ ( W LoRA λ ∗ ; D pre ∪ D ft ) = γ pre 2 + ( γ ft ∗ ) 2 γ pre γ ft ∗ = ( ( γ ft ∗ ) 2 1 + γ pre 2 1 ) − 1/2 .
Proof of General Version
In this subsection we present the general version of Theorem 3.2: under the near-orthogonal regime in Assumption 2.1, the normalized margins of W L o R A λ W_{\mathrm{LoRA}}^{\lambda} W LoRA λ on pre-training, fine-tuning, and combined data remain close to their strict-orthogonal closed-form values from Theorem E.1, with deviations controlled by the perturbation error E λ + ϵ 2 E_{\lambda}+\epsilon_{2} E λ + ϵ 2 defined below.
Notation. We adopt the notation of Theorem D.2: G G G , R R R , Q Q Q , S S S , δ \delta δ , δ 0 \delta_{0} δ 0 , s 0 s_{0} s 0 , μ 0 \mu_{0} μ 0 , C 0 C_{0} C 0 , and Z λ ∗ ( S ˉ , R ˉ ) Z_{\lambda}^{*}(\bar{S},\bar{R}) Z λ ∗ ( S ˉ , R ˉ ) , with the key bound (33). Additionally, write
Θ λ ( 0 ) : = ∥ Z λ ∗ ( 0 , I ) ∥ F , γ f t , λ ( 0 ) : = γ f t , λ ∣ ϵ 1 = ϵ 2 = 0 , (67) \Theta_{\lambda}^{(0)}:=\lVert Z_{\lambda}^{*}(0,I)\rVert_F,\qquad\gamma_{\mathrm{ft},\lambda}^{(0)}:=\left.\gamma_{\mathrm{ft},\lambda}\right|_{\epsilon_{1}=\epsilon_{2}=0}, \tag*{(67)} Θ λ ( 0 ) := ∥ Z λ ∗ ( 0 , I ) ∥ F , γ ft , λ ( 0 ) := γ ft , λ ∣ ϵ 1 = ϵ 2 = 0 , (67)
for the strict-orthogonal adapter Frobenius norm and the strict-orthogonal fine-tuning margin from Theorem E.1. (When X f t ⊤ X f t = I X_{\mathrm{ft}}^{\top}X_{\mathrm{ft}}=I X ft ⊤ X ft = I , Θ λ ( 0 ) = ∥ B λ ∗ A λ ∗ ∥ F \Theta_{\lambda}^{(0)}=\lVert B_{\lambda}^{*}A_{\lambda}^{*}\rVert_F Θ λ ( 0 ) = ∥ B λ ∗ A λ ∗ ∥ F and γ f t , λ ( 0 ) = γ f t , λ \gamma_{\mathrm{ft},\lambda}^{(0)}=\gamma_{\mathrm{ft},\lambda} γ ft , λ ( 0 ) = γ ft , λ .) Set
E λ : = C 0 μ 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z λ ∗ ( 0 , I ) ∥ F ) ) (the bound in (33)). (68) E_{\lambda}:=\frac{C_{0}}{\mu_{0}}\left(\lVert S\rVert_F+\delta\left(1+\lVert Z_{\lambda}^{*}(0,I)\rVert_F\right)\right)\qquad\text{(the bound in (33)).} \tag*{(68)} E λ := μ 0 C 0 ( ∥ S ∥ F + δ ( 1 + ∥ Z λ ∗ ( 0 , I ) ∥ F ) ) (the bound in (33)). (68)
The construction C 0 = C 1 + μ 0 C_{0}=C_{1}+\mu_{0} C 0 = C 1 + μ 0 in the proof of Theorem D.2 gives C 0 / μ 0 > 1 C_{0}/\mu_{0}>1 C 0 / μ 0 > 1 , so ∥ S ∥ F ≤ E λ \lVert S\rVert_F\le E_{\lambda} ∥ S ∥ F ≤ E λ and δ ≤ E λ \delta\le E_{\lambda} δ ≤ E λ . Therefore, ∥ S ∥ F = O ( ϵ 2 ) \lVert S\rVert_F=O(\epsilon_{2}) ∥ S ∥ F = O ( ϵ 2 ) , and E λ → 0 E_{\lambda}\to0 E λ → 0 as ϵ 1 , ϵ 2 → 0 \epsilon_{1},\epsilon_{2}\to0 ϵ 1 , ϵ 2 → 0 .
Theorem E.2 (General version of Theorem 3.2). *Adopt the setup of Theorem D.2 (so δ ≤ δ 0 \delta\le\delta_{0} δ ≤ δ 0 and ∥ S ∥ F ≤ s 0 \lVert S\rVert_F\le s_{0} ∥ S ∥ F ≤ s 0 ). Assume additionally that pre-training samples are unit-norm. There exists a constant
C > 0 C>0 C > 0 , depending on λ \lambda λ , K K K , K ˉ \bar{K} K ˉ , n n n , ρ p r e \rho_{\mathrm{pre}} ρ pre , μ 0 \mu_0 μ 0 , C 0 C_0 C 0 , ∥ Z λ ∗ ( 0 , I ) ∥ ∗ \lVert Z_{\lambda}^{\ast}(0,I)\rVert_{\ast} ∥ Z λ ∗ ( 0 , I ) ∥ ∗ but not on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 , such that
∣ γ ( W L o R A λ ; D p r e ) − γ p r e ρ p r e ( Θ λ ( 0 ) ) 2 + ρ p r e 2 ∣ ≤ C ( E λ + ϵ 2 ) , (69) \left|\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}})-\gamma_{\mathrm{pre}}\frac{\rho_{\mathrm{pre}}}{\sqrt{(\Theta_{\lambda}^{(0)})^2+\rho_{\mathrm{pre}}^2}}\right|\leq C(E_{\lambda}+\epsilon_2), \tag*{(69)} γ ( W LoRA λ ; D pre ) − γ pre ( Θ λ ( 0 ) ) 2 + ρ pre 2 ρ pre ≤ C ( E λ + ϵ 2 ) , (69)
∣ γ ( W L o R A λ ; D f t ) − γ f t , λ ( 0 ) Θ λ ( 0 ) ( Θ λ ( 0 ) ) 2 + ρ p r e 2 ∣ ≤ C ( E λ + ϵ 2 ) . (70) \left|\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{ft}})-\gamma_{\mathrm{ft},\lambda}^{(0)}\frac{\Theta_{\lambda}^{(0)}}{\sqrt{(\Theta_{\lambda}^{(0)})^2+\rho_{\mathrm{pre}}^2}}\right|\leq C(E_{\lambda}+\epsilon_2). \tag*{(70)} γ ( W LoRA λ ; D ft ) − γ ft , λ ( 0 ) ( Θ λ ( 0 ) ) 2 + ρ pre 2 Θ λ ( 0 ) ≤ C ( E λ + ϵ 2 ) . (70)
The corresponding bound on γ ( W L o R A λ ; D p r e ∪ D f t ) = min { ⋅ , ⋅ } \gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}})=\min\{\cdot,\cdot\} γ ( W LoRA λ ; D pre ∪ D ft ) = min { ⋅ , ⋅ } inherits the same right-hand side. In particular, as E λ + ϵ 2 → 0 E_{\lambda}+\epsilon_2\to0 E λ + ϵ 2 → 0 , the three margins converge to their strict-orthogonal closed-form values from Theorem E.1 .
Proof. Throughout the proof, C C C denotes a positive constant whose value may change line to line, but always depends only on the quantities listed in the theorem statement.
We first establish leakage bounds quantifying how much pre-training and fine-tuning components mix in the near-orthogonal regime. Since S = W p r e X f t S=W_{\mathrm{pre}}X_{\mathrm{ft}} S = W pre X ft , for any fine-tuning sample x ˉ j \bar{x}_j x ˉ j and any row w k w_k w k of W p r e W_{\mathrm{pre}} W pre ,
∣ w k T x ˉ j ∣ = ∣ S k j ∣ ≤ ∥ S ∥ F ≤ E λ . (71) \lvert w_k^{\mathsf T}\bar{x}_j\rvert=\lvert S_{kj}\rvert\leq\lVert S\rVert_F\leq E_{\lambda}. \tag*{(71)} ∣ w k T x ˉ j ∣ = ∣ S k j ∣ ≤ ∥ S ∥ F ≤ E λ . (71)
For the symmetric direction, write δ λ , k : = ( B λ ∗ A λ ∗ ) k T \delta_{\lambda,k}:=(B_{\lambda}^{\ast}A_{\lambda}^{\ast})_k^{\mathsf T} δ λ , k := ( B λ ∗ A λ ∗ ) k T for the k k k -th row of B λ ∗ A λ ∗ B_{\lambda}^{\ast}A_{\lambda}^{\ast} B λ ∗ A λ ∗ . By the reformulation in the proof of Theorem D.2, B λ ∗ A λ ∗ = Z λ ∗ ( S , R ) Q T B_{\lambda}^{\ast}A_{\lambda}^{\ast}=Z_{\lambda}^{\ast}(S,R)Q^{\mathsf T} B λ ∗ A λ ∗ = Z λ ∗ ( S , R ) Q T , so for any unit-norm x ∈ D p r e x\in\mathcal{D}_{\mathrm{pre}} x ∈ D pre ,
∣ δ λ , k T x ∣ ≤ ∥ ( Z λ ∗ ( S , R ) ) k ∥ 2 ∥ Q T x ∥ 2 . (72) \lvert\delta_{\lambda,k}^{\mathsf T}x\rvert\leq\lVert(Z_{\lambda}^{\ast}(S,R))_k\rVert_2\lVert Q^{\mathsf T}x\rVert_2. \tag*{(72)} ∣ δ λ , k T x ∣ ≤ ∥( Z λ ∗ ( S , R ) ) k ∥ 2 ∥ Q T x ∥ 2 . (72)
Since ∥ Q T x ∥ 2 = ∥ R − 1 X f t T x ∥ 2 ≤ ∥ R − 1 ∥ o p ∥ X f t T x ∥ 2 \lVert Q^{\mathsf T}x\rVert_2=\lVert R^{-1}X_{\mathrm{ft}}^{\mathsf T}x\rVert_2\leq\lVert R^{-1}\rVert_{\mathrm{op}}\lVert X_{\mathrm{ft}}^{\mathsf T}x\rVert_2 ∥ Q T x ∥ 2 = ∥ R − 1 X ft T x ∥ 2 ≤ ∥ R − 1 ∥ op ∥ X ft T x ∥ 2 , ∥ R − 1 ∥ o p ≤ 1 / 1 − δ \lVert R^{-1}\rVert_{\mathrm{op}}\leq1/\sqrt{1-\delta} ∥ R − 1 ∥ op ≤ 1/ 1 − δ , and ∥ X f t T x ∥ 2 2 = ∑ l ∣ x ˉ l T x ∣ 2 ≤ K n ϵ ˉ 2 2 \lVert X_{\mathrm{ft}}^{\mathsf T}x\rVert_2^2=\sum_l\lvert\bar{x}_l^{\mathsf T}x\rvert^2\leq K n\bar{\epsilon}_2^2 ∥ X ft T x ∥ 2 2 = ∑ l ∣ x ˉ l T x ∣ 2 ≤ K n ϵ ˉ 2 2 by Assumption 2.1(c), we have ∣ δ λ , k T x ∣ ≤ K n ϵ 2 1 − δ ∥ ( Z λ ∗ ( S , R ) ) k ∥ 2 \lvert\delta_{\lambda,k}^{\mathsf T}x\rvert\leq\frac{\sqrt{K n}\epsilon_2}{\sqrt{1-\delta}}\lVert(Z_{\lambda}^{\ast}(S,R))_k\rVert_2 ∣ δ λ , k T x ∣ ≤ 1 − δ K n ϵ 2 ∥( Z λ ∗ ( S , R ) ) k ∥ 2 . Using ∥ Z λ ∗ ( S , R ) ∥ F ≤ ∥ Z λ ∗ ( 0 , I ) ∥ F + E λ \lVert Z_{\lambda}^{\ast}(S,R)\rVert_F\leq\lVert Z_{\lambda}^{\ast}(0,I)\rVert_F+E_{\lambda} ∥ Z λ ∗ ( S , R ) ∥ F ≤ ∥ Z λ ∗ ( 0 , I ) ∥ F + E λ from (52),
∣ δ λ , k T x ∣ ≤ C ( ∥ Z λ ∗ ( 0 , I ) ∥ F + E λ ) ϵ 2 ≤ C ϵ 2 . (73) \lvert\delta_{\lambda,k}^{\mathsf T}x\rvert\leq C\bigl(\lVert Z_{\lambda}^{\ast}(0,I)\rVert_F+E_{\lambda}\bigr)\epsilon_2\leq C\epsilon_2. \tag*{(73)} ∣ δ λ , k T x ∣ ≤ C ( ∥ Z λ ∗ ( 0 , I ) ∥ F + E λ ) ϵ 2 ≤ C ϵ 2 . (73)
Then, we bound the perturbation of the Frobenius norm ∥ W L o R A λ ∥ F \lVert W_{\mathrm{LoRA}}^{\lambda}\rVert_F ∥ W LoRA λ ∥ F from ρ p r e 2 + ( Θ λ ( 0 ) ) 2 \sqrt{\rho_{\mathrm{pre}}^2+(\Theta_{\lambda}^{(0)})^2} ρ pre 2 + ( Θ λ ( 0 ) ) 2 . Expanding,
∥ W L o R A λ ∥ F 2 = ∥ W p r e ∥ F 2 + 2 ⟨ W p r e , B λ ∗ A λ ∗ ⟩ + ∥ B λ ∗ A λ ∗ ∥ F 2 . (74) \lVert W_{\mathrm{LoRA}}^{\lambda}\rVert_F^2=\lVert W_{\mathrm{pre}}\rVert_F^2+2\langle W_{\mathrm{pre}},B_{\lambda}^{\ast}A_{\lambda}^{\ast}\rangle+\lVert B_{\lambda}^{\ast}A_{\lambda}^{\ast}\rVert_F^2. \tag*{(74)} ∥ W LoRA λ ∥ F 2 = ∥ W pre ∥ F 2 + 2 ⟨ W pre , B λ ∗ A λ ∗ ⟩ + ∥ B λ ∗ A λ ∗ ∥ F 2 . (74)
Each row of B λ ∗ A λ ∗ B_{\lambda}^{\ast}A_{\lambda}^{\ast} B λ ∗ A λ ∗ lies in span ( D f t ) \operatorname{span}(\mathcal{D}_{\mathrm{ft}}) span ( D ft ) (the column space of X f t X_{\mathrm{ft}} X ft ), so
∣ ⟨ W p r e , B λ ∗ A λ ∗ ⟩ ∣ = ∣ ∑ k w k T δ λ , k ∣ ≤ ∑ k ∥ P X f t w k ∥ 2 ∥ ( B λ ∗ A λ ∗ ) k ∥ 2 , (75) \left|\langle W_{\mathrm{pre}},B_{\lambda}^{\ast}A_{\lambda}^{\ast}\rangle\right|=\left|\sum_k w_k^{\mathsf T}\delta_{\lambda,k}\right|\leq\sum_k\lVert P_{X_{\mathrm{ft}}}w_k\rVert_2\lVert(B_{\lambda}^{\ast}A_{\lambda}^{\ast})_k\rVert_2, \tag*{(75)} ∣ ⟨ W pre , B λ ∗ A λ ∗ ⟩ ∣ = k ∑ w k T δ λ , k ≤ k ∑ ∥ P X ft w k ∥ 2 ∥( B λ ∗ A λ ∗ ) k ∥ 2 , (75)
where P X f t P_{X_{\mathrm{ft}}} P X ft is the orthogonal projection onto span ( D f t ) \operatorname{span}(\mathcal{D}_{\mathrm{ft}}) span ( D ft ) . The identity W p r e Q = S R − 1 W_{\mathrm{pre}}Q=SR^{-1} W pre Q = S R − 1 gives ∥ W p r e Q ∥ F ≤ ∥ S ∥ F / 1 − δ \lVert W_{\mathrm{pre}}Q\rVert_F\leq\lVert S\rVert_F/\sqrt{1-\delta} ∥ W pre Q ∥ F ≤ ∥ S ∥ F / 1 − δ . Thus, Cauchy–Schwarz and B λ ∗ A λ ∗ = Z λ ∗ ( S , R ) Q T B_{\lambda}^{\ast}A_{\lambda}^{\ast}=Z_{\lambda}^{\ast}(S,R)Q^{\mathsf T} B λ ∗ A λ ∗ = Z λ ∗ ( S , R ) Q T yield
∣ ⟨ W p r e , B λ ∗ A λ ∗ ⟩ ∣ ≤ ∥ S ∥ F ∥ B λ ∗ A λ ∗ ∥ F 1 − δ ≤ C ( Θ λ ( 0 ) + E λ ) ∥ S ∥ F . (76) \left|\langle W_{\mathrm{pre}},B_{\lambda}^{\ast}A_{\lambda}^{\ast}\rangle\right|\leq\frac{\lVert S\rVert_F\lVert B_{\lambda}^{\ast}A_{\lambda}^{\ast}\rVert_F}{\sqrt{1-\delta}}\leq C\bigl(\Theta_{\lambda}^{(0)}+E_{\lambda}\bigr)\lVert S\rVert_F. \tag*{(76)} ∣ ⟨ W pre , B λ ∗ A λ ∗ ⟩ ∣ ≤ 1 − δ ∥ S ∥ F ∥ B λ ∗ A λ ∗ ∥ F ≤ C ( Θ λ ( 0 ) + E λ ) ∥ S ∥ F . (76)
Write Δ λ : = B λ ∗ A λ ∗ \Delta_{\lambda}:=B_{\lambda}^{\ast}A_{\lambda}^{\ast} Δ λ := B λ ∗ A λ ∗ . Equation (51) and Q T Q = I Q^{\mathsf T}Q=I Q T Q = I give ∣ ∥ Δ λ ∥ F − Θ λ ( 0 ) ∣ ≤ E λ \left|\lVert\Delta_{\lambda}\rVert_F-\Theta_{\lambda}^{(0)}\right|\leq E_{\lambda} ∥ Δ λ ∥ F − Θ λ ( 0 ) ≤ E λ . Comparing first with ρ p r e 2 + ∥ Δ λ ∥ F 2 \sqrt{\rho_{\mathrm{pre}}^2+\lVert\Delta_{\lambda}\rVert_F^2} ρ pre 2 + ∥ Δ λ ∥ F 2 and then using this norm bound yields
∣ ∥ W L o R A λ ∥ F − ρ p r e 2 + ( Θ λ ( 0 ) ) 2 ∣ ≤ 2 ∣ ⟨ W p r e , Δ λ ⟩ ∣ ∥ W L o R A λ ∥ F + ρ p r e 2 + ∥ Δ λ ∥ F 2 + E λ ≤ 2 ∥ S ∥ F 1 − δ + E λ ≤ C ( E λ + ϵ 2 ) . (77) \begin{aligned}
\left|\lVert W_{\mathrm{LoRA}}^{\lambda}\rVert_F-\sqrt{\rho_{\mathrm{pre}}^2+(\Theta_{\lambda}^{(0)})^2}\right|
\leq\frac{2\left|\langle W_{\mathrm{pre}},\Delta_{\lambda}\rangle\right|}{\lVert W_{\mathrm{LoRA}}^{\lambda}\rVert_F+\sqrt{\rho_{\mathrm{pre}}^2+\lVert\Delta_{\lambda}\rVert_F^2}}+E_{\lambda} \\
&\leq\frac{2\lVert S\rVert_F}{\sqrt{1-\delta}}+E_{\lambda}\leq C(E_{\lambda}+\epsilon_2). \tag*{(77)}
\end{aligned} ∥ W LoRA λ ∥ F − ρ pre 2 + ( Θ λ ( 0 ) ) 2 ≤ ∥ W LoRA λ ∥ F + ρ pre 2 + ∥ Δ λ ∥ F 2 2 ∣ ⟨ W pre , Δ λ ⟩ ∣ + E λ ≤ 1 − δ 2 ∥ S ∥ F + E λ ≤ C ( E λ + ϵ 2 ) . (77)
Next, we bound the perturbation of the unnormalized margins C ( W L o R A λ ; ⋅ ) \mathcal{C}(W_{\mathrm{LoRA}}^{\lambda};\cdot) C ( W LoRA λ ; ⋅ ) on each dataset. For pre-training data ( x , y c ) ∈ D p r e (x,y_c)\in\mathcal{D}_{\mathrm{pre}} ( x , y c ) ∈ D pre and k ≠ c k\ne c k = c ,
[ ( w c + δ λ , c ) − ( w k + δ λ , k ) ] T x = ( w c − w k ) T x + ( δ λ , c − δ λ , k ) T x . (78) \bigl[(w_c+\delta_{\lambda,c})-(w_k+\delta_{\lambda,k})\bigr]^{\mathsf T}x=(w_c-w_k)^{\mathsf T}x+(\delta_{\lambda,c}-\delta_{\lambda,k})^{\mathsf T}x. \tag*{(78)} [ ( w c + δ λ , c ) − ( w k + δ λ , k ) ] T x = ( w c − w k ) T x + ( δ λ , c − δ λ , k ) T x . (78)
Taking the minimum over ( x , c , k ) (x,c,k) ( x , c , k ) and combining min ( w c − w k ) ⊤ x = C ( W p r e ; D p r e ) = ρ p r e γ p r e \min(w_c-w_k)^\top x=C(W_{\mathrm{pre}};D_{\mathrm{pre}})=\rho_{\mathrm{pre}}\gamma_{\mathrm{pre}} min ( w c − w k ) ⊤ x = C ( W pre ; D pre ) = ρ pre γ pre (definition of γ p r e \gamma_{\mathrm{pre}} γ pre ) with ∣ δ λ , c ⊤ x − δ λ , k ⊤ x ∣ ≤ 2 C ϵ 2 |\delta_{\lambda,c}^\top x-\delta_{\lambda,k}^\top x|\le2C\epsilon_2 ∣ δ λ , c ⊤ x − δ λ , k ⊤ x ∣ ≤ 2 C ϵ 2 from (74),
∣ C ( W L o R A λ ; D p r e ) − ρ p r e γ p r e ∣ ≤ C ϵ 2 . (79) \left|C(W_{\mathrm{LoRA}}^\lambda;D_{\mathrm{pre}})-\rho_{\mathrm{pre}}\gamma_{\mathrm{pre}}\right|\le C\epsilon_2 . \tag*{(79)} C ( W LoRA λ ; D pre ) − ρ pre γ pre ≤ C ϵ 2 . (79)
For fine-tuning data ( x ˉ , y c ) ∈ D f t (\bar{x},y_c)\in D_{\mathrm{ft}} ( x ˉ , y c ) ∈ D ft and k ≠ c k\ne c k = c ,
[ ( w c + δ λ , c ) − ( w k + δ λ , k ) ] ⊤ x ˉ = ( δ λ , c − δ λ , k ) ⊤ x ˉ + ( w c − w k ) ⊤ x ˉ . (80) \left[(w_c+\delta_{\lambda,c})-(w_k+\delta_{\lambda,k})\right]^\top\bar{x}=(\delta_{\lambda,c}-\delta_{\lambda,k})^\top\bar{x}+(w_c-w_k)^\top\bar{x}. \tag*{(80)} [ ( w c + δ λ , c ) − ( w k + δ λ , k ) ] ⊤ x ˉ = ( δ λ , c − δ λ , k ) ⊤ x ˉ + ( w c − w k ) ⊤ x ˉ . (80)
The unnormalized fine-tuning margin is 2-Lipschitz in the F-norm of the classifier matrix (since for any W , W ′ ∈ R K × d W,W'\in\mathbb{R}^{K\times d} W , W ′ ∈ R K × d and any unit-norm x ˉ \bar{x} x ˉ , ∣ ( ( W − W ′ ) c − ( W − W ′ ) k ) ⊤ x ˉ ∣ ≤ 2 ∥ W − W ′ ∥ F |((W-W')_c-(W-W')_k)^\top\bar{x}|\le2\|W-W'\|_F ∣ (( W − W ′ ) c − ( W − W ′ ) k ) ⊤ x ˉ ∣ ≤ 2∥ W − W ′ ∥ F ), so by Theorem D.2,
∣ C ( B λ ∗ A λ ∗ ; D f t ) − C ( Z λ ∗ ( 0 , I ) X f t ⊤ ; D f t ) ∣ ≤ 2 E λ . (81) \left|C(B_\lambda^*A_\lambda^*;D_{\mathrm{ft}})-C(Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top;D_{\mathrm{ft}})\right|\le2E_\lambda. \tag*{(81)} C ( B λ ∗ A λ ∗ ; D ft ) − C ( Z λ ∗ ( 0 , I ) X ft ⊤ ; D ft ) ≤ 2 E λ . (81)
We further compare C ( Z λ ∗ ( 0 , I ) X f t ⊤ ; D f t ) C(Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top;D_{\mathrm{ft}}) C ( Z λ ∗ ( 0 , I ) X ft ⊤ ; D ft ) — the unnormalized margin of the strict-orthogonal classifier evaluated on the actual (near-orthogonal) data — with its strict-orthogonal value Θ λ ( 0 ) γ f t , λ ( 0 ) \Theta_\lambda^{(0)}\gamma_{\mathrm{ft},\lambda}^{(0)} Θ λ ( 0 ) γ ft , λ ( 0 ) . For any sample x ˉ c ′ , j ′ ∈ D f t \bar{x}_{c',j'}\in D_{\mathrm{ft}} x ˉ c ′ , j ′ ∈ D ft ,
X f t ⊤ x ˉ c ′ , j ′ = e ( c ′ − 1 ) n + j ′ + r c ′ , j ′ , r c ′ , j ′ : = X f t ⊤ x ˉ c ′ , j ′ − e ( c ′ − 1 ) n + j ′ , (82) X_{\mathrm{ft}}^\top\bar{x}_{c',j'}=e_{(c'-1)n+j'}+r_{c',j'},\qquad r_{c',j'}:=X_{\mathrm{ft}}^\top\bar{x}_{c',j'}-e_{(c'-1)n+j'} , \tag*{(82)} X ft ⊤ x ˉ c ′ , j ′ = e ( c ′ − 1 ) n + j ′ + r c ′ , j ′ , r c ′ , j ′ := X ft ⊤ x ˉ c ′ , j ′ − e ( c ′ − 1 ) n + j ′ , (82)
where the ( ( c ′ ′ − 1 ) n + j ′ ′ ) ((c''-1)n+j'') (( c ′′ − 1 ) n + j ′′ ) -th entry of r c ′ , j ′ r_{c',j'} r c ′ , j ′ is x ˉ c ′ ′ , j ′ ′ ⊤ x ˉ c ′ , j ′ \bar{x}_{c'',j''}^\top\bar{x}_{c',j'} x ˉ c ′′ , j ′′ ⊤ x ˉ c ′ , j ′ for ( c ′ ′ , j ′ ′ ) ≠ ( c ′ , j ′ ) (c'',j'')\ne(c',j') ( c ′′ , j ′′ ) = ( c ′ , j ′ ) and 0 otherwise. By Assumption 2.1(b),
∥ r c ′ , j ′ ∥ 2 2 ≤ ∑ ( c ′ ′ , j ′ ′ ) ≠ ( c ′ , j ′ ) ϵ 1 2 = ( K ˉ n − 1 ) ϵ 1 2 , (83) \|r_{c',j'}\|_2^2\le\sum_{(c'',j'')\ne(c',j')}\epsilon_1^2=(\bar{K}n-1)\epsilon_1^2 , \tag*{(83)} ∥ r c ′ , j ′ ∥ 2 2 ≤ ( c ′′ , j ′′ ) = ( c ′ , j ′ ) ∑ ϵ 1 2 = ( K ˉ n − 1 ) ϵ 1 2 , (83)
hence ∥ r c ′ , j ′ ∥ 2 ≤ K ˉ n − 1 ϵ 1 = δ / K ˉ n − 1 \|r_{c',j'}\|_2\le\sqrt{\bar{K}n-1}\epsilon_1=\delta/\sqrt{\bar{K}n-1} ∥ r c ′ , j ′ ∥ 2 ≤ K ˉ n − 1 ϵ 1 = δ / K ˉ n − 1 . For any class k ∈ [ K ] k\in[K] k ∈ [ K ] ,
∣ ( Z λ ∗ ( 0 , I ) X f t ⊤ x ˉ c ′ , j ′ ) k − ( Z λ ∗ ( 0 , I ) ) k , ( c ′ − 1 ) n + j ′ ∣ = ∣ ( Z λ ∗ ( 0 , I ) ) k r c ′ , j ′ ∣ ≤ ∥ ( Z λ ∗ ( 0 , I ) ) k ∥ 2 ∥ r c ′ , j ′ ∥ 2 , (84) \left|(Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top\bar{x}_{c',j'})_k-(Z_\lambda^*(0,I))_{k,(c'-1)n+j'}\right|=\left|(Z_\lambda^*(0,I))_kr_{c',j'}\right|\le\|(Z_\lambda^*(0,I))_k\|_2\|r_{c',j'}\|_2 , \tag*{(84)} ( Z λ ∗ ( 0 , I ) X ft ⊤ x ˉ c ′ , j ′ ) k − ( Z λ ∗ ( 0 , I ) ) k , ( c ′ − 1 ) n + j ′ = ∣ ( Z λ ∗ ( 0 , I ) ) k r c ′ , j ′ ∣ ≤ ∥ ( Z λ ∗ ( 0 , I ) ) k ∥ 2 ∥ r c ′ , j ′ ∥ 2 , (84)
where ( Z λ ∗ ( 0 , I ) ) k (Z_\lambda^*(0,I))_k ( Z λ ∗ ( 0 , I ) ) k denotes the k k k -th row. Since max k ∥ ( Z λ ∗ ( 0 , I ) ) k ∥ 2 ≤ ∥ Z λ ∗ ( 0 , I ) ∥ F = Θ λ ( 0 ) \max_k\|(Z_\lambda^*(0,I))_k\|_2\le\|Z_\lambda^*(0,I)\|_F=\Theta_\lambda^{(0)} max k ∥ ( Z λ ∗ ( 0 , I ) ) k ∥ 2 ≤ ∥ Z λ ∗ ( 0 , I ) ∥ F = Θ λ ( 0 ) , the difference of two-class scores satisfies
∣ [ ( Z λ ∗ ( 0 , I ) X f t ⊤ x ˉ c ′ , j ′ ) c − ( Z λ ∗ ( 0 , I ) X f t ⊤ x ˉ c ′ , j ′ ) k ] − [ ( Z λ ∗ ( 0 , I ) ) c , ( c ′ − 1 ) n + j ′ − ( Z λ ∗ ( 0 , I ) ) k , ( c ′ − 1 ) n + j ′ ] ∣ ≤ 2 Θ λ ( 0 ) δ K ˉ n − 1 (85) \left|\left[(Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top\bar{x}_{c',j'})_c-(Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top\bar{x}_{c',j'})_k\right]-\left[(Z_\lambda^*(0,I))_{c,(c'-1)n+j'}-(Z_\lambda^*(0,I))_{k,(c'-1)n+j'}\right]\right|\le\frac{2\Theta_\lambda^{(0)}\delta}{\sqrt{\bar{K}n-1}} \tag*{(85)} [ ( Z λ ∗ ( 0 , I ) X ft ⊤ x ˉ c ′ , j ′ ) c − ( Z λ ∗ ( 0 , I ) X ft ⊤ x ˉ c ′ , j ′ ) k ] − [ ( Z λ ∗ ( 0 , I ) ) c , ( c ′ − 1 ) n + j ′ − ( Z λ ∗ ( 0 , I ) ) k , ( c ′ − 1 ) n + j ′ ] ≤ K ˉ n − 1 2 Θ λ ( 0 ) δ (85)
for every x ˉ ∈ D f t \bar{x}\in D_{\mathrm{ft}} x ˉ ∈ D ft and every pair c ≠ k c\ne k c = k . Setting c = c ′ c=c' c = c ′ and taking the minimum over c ′ ∈ [ K ˉ ] c'\in[\bar{K}] c ′ ∈ [ K ˉ ] , j ′ ∈ [ n ] j'\in[n] j ′ ∈ [ n ] , and k ∈ [ K ] ∖ { c ′ } k\in[K]\setminus\{c'\} k ∈ [ K ] ∖ { c ′ } , the 1-Lipschitz property of min \min min gives
∣ C ( Z λ ∗ ( 0 , I ) X f t ⊤ ; D f t ) − min c ′ , j ′ , k ≠ c ′ [ ( Z λ ∗ ( 0 , I ) ) c ′ , ( c ′ − 1 ) n + j ′ − ( Z λ ∗ ( 0 , I ) ) k , ( c ′ − 1 ) n + j ′ ] ∣ ≤ 2 Θ λ ( 0 ) δ K ˉ n − 1 . (86) \left|C(Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top;D_{\mathrm{ft}})-\min_{c',j',k\ne c'}\left[(Z_\lambda^*(0,I))_{c',(c'-1)n+j'}-(Z_\lambda^*(0,I))_{k,(c'-1)n+j'}\right]\right|\le\frac{2\Theta_\lambda^{(0)}\delta}{\sqrt{\bar{K}n-1}} . \tag*{(86)} C ( Z λ ∗ ( 0 , I ) X ft ⊤ ; D ft ) − c ′ , j ′ , k = c ′ min [ ( Z λ ∗ ( 0 , I ) ) c ′ , ( c ′ − 1 ) n + j ′ − ( Z λ ∗ ( 0 , I ) ) k , ( c ′ − 1 ) n + j ′ ] ≤ K ˉ n − 1 2 Θ λ ( 0 ) δ . (86)
The displayed minimum is precisely the strict-orthogonal unnormalized margin Θ λ ( 0 ) γ f t , λ ( 0 ) \Theta_\lambda^{(0)}\gamma_{\mathrm{ft},\lambda}^{(0)} Θ λ ( 0 ) γ ft , λ ( 0 ) (it is the minimum-margin definition applied to Z λ ∗ ( 0 , I ) Z_\lambda^*(0,I) Z λ ∗ ( 0 , I ) on a one-hot K ˉ n \bar{K}n K ˉ n -dimensional input set, which is what Theorem E.1 characterizes). Hence
∣ C ( Z λ ∗ ( 0 , I ) X f t ⊤ ; D f t ) − Θ λ ( 0 ) γ f t , λ ( 0 ) ∣ ≤ 2 Θ λ ( 0 ) K ˉ n − 1 δ ≤ C δ . (87) \left|C(Z_\lambda^*(0,I)X_{\mathrm{ft}}^\top;D_{\mathrm{ft}})-\Theta_\lambda^{(0)}\gamma_{\mathrm{ft},\lambda}^{(0)}\right|\le\frac{2\Theta_\lambda^{(0)}}{\sqrt{\bar{K}n-1}}\delta\le C\delta. \tag*{(87)} C ( Z λ ∗ ( 0 , I ) X ft ⊤ ; D ft ) − Θ λ ( 0 ) γ ft , λ ( 0 ) ≤ K ˉ n − 1 2 Θ λ ( 0 ) δ ≤ C δ . (87)
The leakage from the pre-training rows on D f t D_{\mathrm{ft}} D ft contributes at most 2 ∥ S ∥ F 2\|S\|_F 2∥ S ∥ F to the per-pair score difference via (72), so taking the min,
∣ C ( W L o R A λ ; D f t ) − C ( B λ ∗ A λ ∗ ; D f t ) ∣ ≤ 2 ∥ S ∥ F ≤ 2 E λ . (88) \left|C(W_{\mathrm{LoRA}}^\lambda;D_{\mathrm{ft}})-C(B_\lambda^*A_\lambda^*;D_{\mathrm{ft}})\right|\le2\|S\|_F\le2E_\lambda. \tag*{(88)} C ( W LoRA λ ; D ft ) − C ( B λ ∗ A λ ∗ ; D ft ) ≤ 2∥ S ∥ F ≤ 2 E λ . (88)
Combining with (82) and (88) via the triangle inequality (and noting δ ≤ ( μ 0 / C 0 ) E λ \delta\le(\mu_0/C_0)E_\lambda δ ≤ ( μ 0 / C 0 ) E λ by definition of E λ E_\lambda E λ , so the δ \delta δ from (88) is absorbed into C E λ CE_\lambda C E λ ),
∣ C ( W L o R A λ ; D f t ) − Θ λ ( 0 ) γ f t , λ ( 0 ) ∣ ≤ C ( E λ + ϵ 2 ) . (89) \left|C(W_{\mathrm{LoRA}}^\lambda;D_{\mathrm{ft}})-\Theta_\lambda^{(0)}\gamma_{\mathrm{ft},\lambda}^{(0)}\right|\le C(E_\lambda+\epsilon_2). \tag*{(89)} C ( W LoRA λ ; D ft ) − Θ λ ( 0 ) γ ft , λ ( 0 ) ≤ C ( E λ + ϵ 2 ) . (89)
Finally, we combine the bounds. For a dataset of unit-norm inputs, ∣ γ ( W ; D ) ∣ ≤ 2 |\gamma(W;\mathcal{D})| \le\sqrt{2} ∣ γ ( W ; D ) ∣ ≤ 2 whenever W ≠ 0 W \ne0 W = 0 .
Let a : = ∥ W L o R A λ ∥ F a := \lVert W_{\mathrm{LoRA}}^{\lambda}\rVert_F a := ∥ W LoRA λ ∥ F , b : = ρ p r e 2 + ( Θ λ ( 0 ) ) 2 b := \sqrt{\rho_{\mathrm{pre}}^2 + (\Theta_{\lambda}^{(0)})^2} b := ρ pre 2 + ( Θ λ ( 0 ) ) 2 , u : = C ( W L o R A λ ; D ) u := C(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}) u := C ( W LoRA λ ; D ) , and let u 0 u_0 u 0 be the corresponding strict-orthogonal unnormalized margin. Then
∣ u a − u 0 b ∣ = ∣ u − u 0 b + u a b − a b ∣ ≤ ∣ u − u 0 ∣ + 2 ∣ a − b ∣ b . (90) \left|\frac{u}{a}-\frac{u_0}{b}\right|=\left|\frac{u-u_0}{b}+\frac{u}{a}\frac{b-a}{b}\right|\leq\frac{|u-u_0|+\sqrt{2}|a-b|}{b}. \tag*{(90)} a u − b u 0 = b u − u 0 + a u b b − a ≤ b ∣ u − u 0 ∣ + 2 ∣ a − b ∣ . (90)
Since b ≥ ρ p r e > 0 b \ge\rho_{\mathrm{pre}} > 0 b ≥ ρ pre > 0 , (78) combined with (80) gives (70) and combined with (90) gives (71). Taking the minimum of the pre-training and fine-tuning margins gives the combined-data bound by the 1-Lipschitz property of min.
We now translate Theorem E.2 into perturbation bounds on Adapt( λ ) (\lambda) ( λ ) and Forget( λ ) (\lambda) ( λ ) , and on the optimal-trade-off value of Corollary 3.1.
Corollary E.1 (Near-orthogonal bounds on Adapt and Forget). Under the hypotheses of Theorem E.2, with C C C as there,
∣ Adapt ( λ ) − γ f t , λ ( 0 ) Θ λ ( 0 ) ( Θ λ ( 0 ) ) 2 + ρ p r e 2 ∣ ≤ C ( E λ + ϵ 2 ) , (91) \left|\operatorname{Adapt}(\lambda)-\gamma_{\mathrm{ft},\lambda}^{(0)}\frac{\Theta_{\lambda}^{(0)}}{\sqrt{(\Theta_{\lambda}^{(0)})^2+\rho_{\mathrm{pre}}^2}}\right|\leq C(E_{\lambda}+\epsilon_2), \tag*{(91)} Adapt ( λ ) − γ ft , λ ( 0 ) ( Θ λ ( 0 ) ) 2 + ρ pre 2 Θ λ ( 0 ) ≤ C ( E λ + ϵ 2 ) , (91) ∣ Forget ( λ ) − γ p r e ( 1 − ρ p r e ( Θ λ ( 0 ) ) 2 + ρ p r e 2 ) ∣ ≤ C ( E λ + ϵ 2 ) . (92) \left|\operatorname{Forget}(\lambda)-\gamma_{\mathrm{pre}}\left(1-\frac{\rho_{\mathrm{pre}}}{\sqrt{(\Theta_{\lambda}^{(0)})^2+\rho_{\mathrm{pre}}^2}}\right)\right|\leq C(E_{\lambda}+\epsilon_2). \tag*{(92)} Forget ( λ ) − γ pre 1 − ( Θ λ ( 0 ) ) 2 + ρ pre 2 ρ pre ≤ C ( E λ + ϵ 2 ) . (92) In particular, as E λ + ϵ 2 → 0 E_{\lambda}+\epsilon_2 \to0 E λ + ϵ 2 → 0 , both Adapt( λ ) (\lambda) ( λ ) and Forget( λ ) (\lambda) ( λ ) converge to their strict-orthogonal closed-form values from Theorem E.1.
Proof. For the Forget bound, note that γ p r e = γ ( W p r e ; D p r e ) \gamma_{\mathrm{pre}}=\gamma(W_{\mathrm{pre}};\mathcal{D}_{\mathrm{pre}}) γ pre = γ ( W pre ; D pre ) does not depend on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 and that Forget( λ ) = γ p r e − γ ( W L o R A λ ; D p r e ) (\lambda)=\gamma_{\mathrm{pre}}-\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}) ( λ ) = γ pre − γ ( W LoRA λ ; D pre ) , so (93) is just (70) with the γ p r e \gamma_{\mathrm{pre}} γ pre term absorbed. For the Adapt bound, use Adapt( λ ) = γ ( W L o R A λ ; D f t ) − γ ( W p r e ; D f t ) (\lambda)=\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{ft}})-\gamma(W_{\mathrm{pre}};\mathcal{D}_{\mathrm{ft}}) ( λ ) = γ ( W LoRA λ ; D ft ) − γ ( W pre ; D ft ) . The leakage bound (72) controls the second term: for any x ˉ ∈ D f t \bar{x}\in\mathcal{D}_{\mathrm{ft}} x ˉ ∈ D ft and any pair c ≠ k c\ne k c = k , ∣ ( w c − w k ) T x ˉ ∣ ≤ 2 ∥ S ∥ F |(w_c-w_k)^{\mathsf{T}}\bar{x}|\leq2\lVert S\rVert_F ∣ ( w c − w k ) T x ˉ ∣ ≤ 2 ∥ S ∥ F , so taking the min over ( x ˉ , c , k ) (\bar{x},c,k) ( x ˉ , c , k ) and dividing by ∥ W p r e ∥ F = ρ p r e \lVert W_{\mathrm{pre}}\rVert_F=\rho_{\mathrm{pre}} ∥ W pre ∥ F = ρ pre gives ∣ γ ( W p r e ; D f t ) ∣ ≤ 2 ∥ S ∥ F / ρ p r e ≤ 2 E λ / ρ p r e |\gamma(W_{\mathrm{pre}};\mathcal{D}_{\mathrm{ft}})|\leq2\lVert S\rVert_F/\rho_{\mathrm{pre}}\leq2E_{\lambda}/\rho_{\mathrm{pre}} ∣ γ ( W pre ; D ft ) ∣ ≤ 2 ∥ S ∥ F / ρ pre ≤ 2 E λ / ρ pre . Combining with (71) via the triangle inequality yields (92).
Corollary E.2 (Near-orthogonal version of Corollary 3.1). Let γ f t p a d , ( 0 ) \gamma_{\mathrm{ft}}^{\mathrm{pad},(0)} γ ft pad , ( 0 ) denote the strict-orthogonal reference value of γ f t p a d \gamma_{\mathrm{ft}}^{\mathrm{pad}} γ ft pad . Adopt the hypotheses of Theorem E.2, and set Θ ∗ : = ρ p r e γ p r e / γ f t p a d , ( 0 ) \Theta^*:=\rho_{\mathrm{pre}}\gamma_{\mathrm{pre}}/\gamma_{\mathrm{ft}}^{\mathrm{pad},(0)} Θ ∗ := ρ pre γ pre / γ ft pad , ( 0 ) . If K ˉ < K \bar{K}<K K ˉ < K , assume Θ ∗ < Θ λ c r i t ( 0 ) \Theta^*<\Theta_{\lambda_{\mathrm{crit}}}^{(0)} Θ ∗ < Θ λ crit ( 0 ) ; when K ˉ = K \bar{K}=K K ˉ = K , no additional balancing condition is needed. Let λ ∗ \lambda^* λ ∗ be the unique solution of Θ λ ∗ ( 0 ) = Θ ∗ \Theta_{\lambda^*}^{(0)}=\Theta^* Θ λ ∗ ( 0 ) = Θ ∗ in ( λ c r i t , 1 / ( K ˉ n ) ) (\lambda_{\mathrm{crit}},1/(\bar{K}\sqrt{n})) ( λ crit , 1/ ( K ˉ n )) , and choose a compact subinterval Λ \Lambda Λ of this open interval containing λ ∗ \lambda^* λ ∗ . Writing δ 0 ( λ ) , s 0 ( λ ) \delta_0(\lambda),s_0(\lambda) δ 0 ( λ ) , s 0 ( λ ) for the local thresholds in Theorem D.2, require
δ ≤ inf λ ∈ Λ δ 0 ( λ ) , ∥ S ∥ F ≤ inf λ ∈ Λ s 0 ( λ ) . \delta\leq\inf_{\lambda\in\Lambda}\delta_0(\lambda),\qquad\lVert S\rVert_F\leq\inf_{\lambda\in\Lambda}s_0(\lambda). δ ≤ λ ∈ Λ inf δ 0 ( λ ) , ∥ S ∥ F ≤ λ ∈ Λ inf s 0 ( λ ) . Suppose E λ ≤ E ˉ E_{\lambda}\leq\bar{E} E λ ≤ E ˉ for all λ ∈ Λ \lambda\in\Lambda λ ∈ Λ , and the constant C C C in Theorem E.2 can be taken uniform on Λ \Lambda Λ with value C ˉ \bar{C} C ˉ . Then
∣ max λ ∈ Λ γ ( W L o R A λ ; D p r e ∪ D f t ) − ( 1 ( γ f t p a d , ( 0 ) ) 2 + 1 ρ p r e 2 ) − 1 / 2 ∣ ≤ C ˉ ( E ˉ + ϵ 2 ) . (93) \left|\max_{\lambda\in\Lambda}\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}})-\left(\frac{1}{(\gamma_{\mathrm{ft}}^{\mathrm{pad},(0)})^2}+\frac{1}{\rho_{\mathrm{pre}}^2}\right)^{-1/2}\right|\leq\bar{C}(\bar{E}+\epsilon_2). \tag*{(93)} λ ∈ Λ max γ ( W LoRA λ ; D pre ∪ D ft ) − ( ( γ ft pad , ( 0 ) ) 2 1 + ρ pre 2 1 ) − 1/2 ≤ C ˉ ( E ˉ + ϵ 2 ) . (93) In particular, as E ˉ + ϵ 2 → 0 \bar{E}+\epsilon_2\to0 E ˉ + ϵ 2 → 0 (with Λ \Lambda Λ fixed), the maximum converges to the corresponding strict-orthogonal intermediate-regime balancing value, which reduces to the value in Corollary 3.1 when K ˉ = K \bar{K}=K K ˉ = K .
Proof. By Theorem E.2 applied uniformly on Λ \Lambda Λ , together with the 1-Lipschitz property of min,
∣ γ ( W L o R A λ ; D p r e ∪ D f t ) − γ ( 0 ) ( λ ) ∣ ≤ C ˉ ( E ˉ + ϵ 2 ) for every λ ∈ Λ , \left|\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}})-\gamma^{(0)}(\lambda)\right|\leq\bar{C}(\bar{E}+\epsilon_2)\quad\text{for every }\lambda\in\Lambda, γ ( W LoRA λ ; D pre ∪ D ft ) − γ ( 0 ) ( λ ) ≤ C ˉ ( E ˉ + ϵ 2 ) for every λ ∈ Λ ,
where γ ( 0 ) ( λ ) : = min { γ p r e ρ p r e ( Θ λ ( 0 ) ) 2 + ρ p r e 2 , γ f t , λ ( 0 ) Θ λ ( 0 ) ( Θ λ ( 0 ) ) 2 + ρ p r e 2 } \gamma^{(0)}(\lambda):=\min\left\{\frac{\gamma_{\mathrm{pre}}\rho_{\mathrm{pre}}}{\sqrt{(\Theta_{\lambda}^{(0)})^2+\rho_{\mathrm{pre}}^2}},\frac{\gamma_{\mathrm{ft},\lambda}^{(0)}\Theta_{\lambda}^{(0)}}{\sqrt{(\Theta_{\lambda}^{(0)})^2+\rho_{\mathrm{pre}}^2}}\right\} γ ( 0 ) ( λ ) := min { ( Θ λ ( 0 ) ) 2 + ρ pre 2 γ pre ρ pre , ( Θ λ ( 0 ) ) 2 + ρ pre 2 γ ft , λ ( 0 ) Θ λ ( 0 ) } is the strict-orthogonal union margin. Taking sup λ ∈ Λ \sup_{\lambda\in\Lambda} sup λ ∈ Λ on both sides and using ∣ sup Λ f − sup Λ g ∣ ≤ sup Λ ∣ f − g ∣ |\sup_{\Lambda}f-\sup_{\Lambda}g|\leq\sup_{\Lambda}|f-g| ∣ sup Λ f − sup Λ g ∣ ≤ sup Λ ∣ f − g ∣ ,
∣ max λ ∈ Λ γ ( W L o R A λ ; D p r e ∪ D f t ) − max λ ∈ Λ γ ( 0 ) ( λ ) ∣ ≤ C ˉ ( E ˉ + ϵ 2 ) . \left|\max_{\lambda\in\Lambda}\gamma(W_{\mathrm{LoRA}}^{\lambda};\mathcal{D}_{\mathrm{pre}}\cup\mathcal{D}_{\mathrm{ft}})-\max_{\lambda\in\Lambda}\gamma^{(0)}(\lambda)\right|\leq\bar{C}(\bar{E}+\epsilon_2). λ ∈ Λ max γ ( W LoRA λ ; D pre ∪ D ft ) − λ ∈ Λ max γ ( 0 ) ( λ ) ≤ C ˉ ( E ˉ + ϵ 2 ) .
On Λ \Lambda Λ , γ f t , λ ( 0 ) = γ f t p a d , ( 0 ) \gamma_{\mathrm{ft},\lambda}^{(0)}=\gamma_{\mathrm{ft}}^{\mathrm{pad},(0)} γ ft , λ ( 0 ) = γ ft pad , ( 0 ) and Θ λ ( 0 ) \Theta_{\lambda}^{(0)} Θ λ ( 0 ) is strictly decreasing in λ \lambda λ . The pre-training term in γ ( 0 ) \gamma^{(0)} γ ( 0 ) is strictly decreasing in Θ λ ( 0 ) \Theta_{\lambda}^{(0)} Θ λ ( 0 ) , while the fine-tuning term is strictly increasing. Their minimum is therefore uniquely maximized when Θ λ ( 0 ) = Θ ∗ \Theta_{\lambda}^{(0)}=\Theta^* Θ λ ( 0 ) = Θ ∗ , which occurs at λ ∗ ∈ Λ \lambda^*\in\Lambda λ ∗ ∈ Λ . Hence max λ ∈ Λ γ ( 0 ) ( λ ) = γ ( 0 ) ( λ ∗ ) = ( 1 / ( γ f t p a d , ( 0 ) ) 2 + 1 / γ p r e 2 ) − 1 / 2 \max_{\lambda\in\Lambda}\gamma^{(0)}(\lambda)=\gamma^{(0)}(\lambda^*)=(1/(\gamma_{\mathrm{ft}}^{\mathrm{pad},(0)})^2+1/\gamma_{\mathrm{pre}}^2)^{-1/2} max λ ∈ Λ γ ( 0 ) ( λ ) = γ ( 0 ) ( λ ∗ ) = ( 1/ ( γ ft pad , ( 0 ) ) 2 + 1/ γ pre 2 ) − 1/2 , proving (93) for both K ˉ = K \bar K=K K ˉ = K and K ˉ < K \bar K<K K ˉ < K .
Remark E.2 (Explicit bounds on E ˉ \bar E E ˉ and C ˉ \bar C C ˉ ). The constants in Corollary E.2 can be chosen with the following explicit upper bounds. Set E ˉ : = sup λ ∈ Λ E λ \bar E:=\sup_{\lambda\in\Lambda}E_\lambda E ˉ := sup λ ∈ Λ E λ and Θ max ( 0 ) : = max λ ∈ Λ Θ ( 0 ) \Theta_{\max}^{(0)}:=\max_{\lambda\in\Lambda}\Theta^{(0)} Θ m a x ( 0 ) := max λ ∈ Λ Θ ( 0 ) . Since a : = min Λ > 0 a:=\min\Lambda>0 a := min Λ > 0 , the proof of Theorem D.2 uses radii bounded by ( log K + 2 ) / a (\log K+2)/a ( log K + 2 ) / a . Thus μ 0 ( λ ) \mu_0(\lambda) μ 0 ( λ ) can be chosen uniformly bounded below by a positive constant and C 0 ( λ ) C_0(\lambda) C 0 ( λ ) uniformly bounded above. Write κ Λ : = sup λ ∈ Λ C 0 ( λ ) / μ 0 ( λ ) < ∞ \kappa_\Lambda:=\sup_{\lambda\in\Lambda}C_0(\lambda)/\mu_0(\lambda)<\infty κ Λ := sup λ ∈ Λ C 0 ( λ ) / μ 0 ( λ ) < ∞ .
Bound on E ˉ \bar E E ˉ . For the conversion to an epsilon-only bound, assume the uniform positive singular-value lower bound discussed in Remark D.2. Combining the definition of E λ E_\lambda E λ in (68) with δ = ( K ˉ n − 1 ) ϵ 1 \delta=(\bar K n-1)\epsilon_1 δ = ( K ˉ n − 1 ) ϵ 1 from Theorem D.2 and the explicit bound ∥ S ∥ F ≤ K ˉ n N p r e ρ p r e ϵ 2 / σ min ( X p r e ) \lVert S\rVert_F\le\sqrt{\bar K n}\,N_{\mathrm{pre}}\rho_{\mathrm{pre}}\epsilon_2/\sigma_{\min}(X_{\mathrm{pre}}) ∥ S ∥ F ≤ K ˉ n N pre ρ pre ϵ 2 / σ m i n ( X pre ) from that remark,
E ˉ ≤ κ Λ ( K ˉ n N p r e ρ p r e σ min ( X p r e ) ϵ 2 + ( K ˉ n − 1 ) ( 1 + Θ max ( 0 ) ) ϵ 1 ) , (94) \bar E\le\kappa_\Lambda\left(\frac{\sqrt{\bar K n}\,N_{\mathrm{pre}}\rho_{\mathrm{pre}}}{\sigma_{\min}(X_{\mathrm{pre}})}\epsilon_2+(\bar K n-1)(1+\Theta_{\max}^{(0)})\epsilon_1\right), \tag*{(94)} E ˉ ≤ κ Λ ( σ m i n ( X pre ) K ˉ n N pre ρ pre ϵ 2 + ( K ˉ n − 1 ) ( 1 + Θ m a x ( 0 ) ) ϵ 1 ) , (94)
which is linear in ϵ 1 \epsilon_1 ϵ 1 and ϵ 2 \epsilon_2 ϵ 2 .
Bound on C ˉ \bar C C ˉ . The constant C C C in the proof of Theorem E.2 is built from three sources:
the direct leakage bound (71) has coefficient 1 1 1 in ∥ S ∥ F \lVert S\rVert_F ∥ S ∥ F , the projection bound (76) contributes L p r o j : = 2 L_{\mathrm{proj}}:=\sqrt{2} L proj := 2 , and (73) contributes L f t : = 2 K ˉ n L_{\mathrm{ft}}:=\sqrt{2\bar K n} L ft := 2 K ˉ n , uniformly since δ ≤ 1 / 2 \delta\le1/2 δ ≤ 1/2 ;
Cauchy–Schwarz over K K K classes contributes a factor K \sqrt{K} K ;
the unit-norm input bound ∣ γ ( W ; D ) ∣ ≤ 2 |\gamma(W;\mathcal D)|\le\sqrt{2} ∣ γ ( W ; D ) ∣ ≤ 2 and (90) contribute a factor at most ( 1 + 2 ) / ρ p r e (1+\sqrt{2})/\rho_{\mathrm{pre}} ( 1 + 2 ) / ρ pre when converting numerator and norm errors to normalized-margin errors.
Using E λ ϵ 2 ≤ E λ E_\lambda\epsilon_2\le E_\lambda E λ ϵ 2 ≤ E λ in (73) and absorbing universal numerical factors into C ∗ > 0 C_*>0 C ∗ > 0 ,
C ˉ ≤ C ∗ K ( 1 + L p r o j + L f t ) ρ p r e ( 1 + ρ p r e + Θ max ( 0 ) ) . (95) \bar C\le\frac{C_*\sqrt{K}(1+L_{\mathrm{proj}}+L_{\mathrm{ft}})}{\rho_{\mathrm{pre}}}(1+\rho_{\mathrm{pre}}+\Theta_{\max}^{(0)}). \tag*{(95)} C ˉ ≤ ρ pre C ∗ K ( 1 + L proj + L ft ) ( 1 + ρ pre + Θ m a x ( 0 ) ) . (95)
Under the stated conditioning requirement, substituting (94) and (95) into the right-hand side of (93) yields a fully explicit linear-in-( ϵ 1 , ϵ 2 ) (\epsilon_1,\epsilon_2) ( ϵ 1 , ϵ 2 ) bound on the gap between the near-orthogonal optimum and the strict-orthogonal value.
Proof of Theorem 3.3
In this section, we first present the proof of Theorem 3.3. Then, under additional smallness assumptions on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 , we show that the same conclusions still hold, with additional error terms that depend on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 .
Proof of Theorem 3.3
Proof. For α ∈ R ≥ 0 T \alpha\in\mathbb{R}_{\ge0}^T α ∈ R ≥ 0 T , following the same argument as in Appendix § 11 , one can show that the margin of linear classifier W L o R A W_{\mathrm{LoRA}} W LoRA for each task is
γ ( W L o R A ( α ) ; D p r e ) = γ p r e ρ p r e ρ p r e 2 + ∑ j = 1 T α j 2 Θ λ , j 2 , (96) \gamma(W_{\mathrm{LoRA}}(\alpha);\mathcal D_{\mathrm{pre}})=\gamma_{\mathrm{pre}}\frac{\rho_{\mathrm{pre}}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}\alpha_j^2\Theta_{\lambda,j}^2}}, \tag*{(96)} γ ( W LoRA ( α ) ; D pre ) = γ pre ρ pre 2 + ∑ j = 1 T α j 2 Θ λ , j 2 ρ pre , (96)
γ ( W L o R A ( α ) ; D i ) = γ i α i Θ λ , i ρ p r e 2 + ∑ j = 1 T α j 2 Θ λ , j 2 , i = 1 , … , T . (97) \gamma(W_{\mathrm{LoRA}}(\alpha);\mathcal D_i)=\gamma_i\frac{\alpha_i\Theta_{\lambda,i}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}\alpha_j^2\Theta_{\lambda,j}^2}},\qquad i=1,\ldots,T. \tag*{(97)} γ ( W LoRA ( α ) ; D i ) = γ i ρ pre 2 + ∑ j = 1 T α j 2 Θ λ , j 2 α i Θ λ , i , i = 1 , … , T . (97)
Then, our goal is to solve the following optimization problem
max α 1 , … , α T ≥ 0 min ( γ p r e ρ p r e ρ p r e 2 + ∑ j = 1 T α j 2 Θ λ , j 2 , min i ≤ T γ i α i Θ λ , i ρ p r e 2 + ∑ j = 1 T α j 2 Θ λ , j 2 ) \max_{\alpha_1,\ldots,\alpha_T\ge0}\min\left(\gamma_{\mathrm{pre}}\frac{\rho_{\mathrm{pre}}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}\alpha_j^2\Theta_{\lambda,j}^2}},\min_{i\le T}\gamma_i\frac{\alpha_i\Theta_{\lambda,i}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}\alpha_j^2\Theta_{\lambda,j}^2}}\right) α 1 , … , α T ≥ 0 max min γ pre ρ pre 2 + ∑ j = 1 T α j 2 Θ λ , j 2 ρ pre , i ≤ T min γ i ρ pre 2 + ∑ j = 1 T α j 2 Θ λ , j 2 α i Θ λ , i
For convenience, we introduce the following notation x i = α i Θ λ , i ≥ 0 x_i=\alpha_i\Theta_{\lambda,i}\ge0 x i = α i Θ λ , i ≥ 0 , i ∈ [ T ] i\in[T] i ∈ [ T ] , then the above optimization problem can be rewritten as
max x 1 , … , x T ≥ 0 min ( ρ p r e γ p r e ρ p r e 2 + ∑ j = 1 T x j 2 , min i ∈ [ T ] γ i x i ρ p r e 2 + ∑ j = 1 T x j 2 ) . (98) \max_{x_1,\ldots,x_T\ge0}\min\left(\frac{\rho_{\mathrm{pre}}\gamma_{\mathrm{pre}}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}x_j^2}},\min_{i\in[T]}\frac{\gamma_i x_i}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}x_j^2}}\right). \tag*{(98)} x 1 , … , x T ≥ 0 max min ρ pre 2 + ∑ j = 1 T x j 2 ρ pre γ pre , i ∈ [ T ] min ρ pre 2 + ∑ j = 1 T x j 2 γ i x i . (98)
Let S : = ρ p r e 2 + ∑ j = 1 T x j 2 S:=\sqrt{\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}x_j^2} S := ρ pre 2 + ∑ j = 1 T x j 2 and t : = min { γ i x i S , γ p r e ρ p r e S } t:=\min\left\{\frac{\gamma_i x_i}{S},\frac{\gamma_{\mathrm{pre}}\rho_{\mathrm{pre}}}{S}\right\} t := min { S γ i x i , S γ pre ρ pre } , then Problem 99 is equivalent to
max t , S , x 1 , … , x T t s . t . x i ≥ 0 , γ i x i ≥ t S , i ∈ [ T ] γ p r e ρ p r e ≥ t S , ∑ i = 1 T x i 2 + ρ p r e 2 = S 2 , S > 0. (99) \begin{aligned}
\max_{t,S,x_1,\ldots,x_T}\quad&t\\
\mathrm{s.t.}\quad&x_i\ge0,\quad\gamma_i x_i\ge tS,\quad i\in[T]\\
&\gamma_{\mathrm{pre}}\rho_{\mathrm{pre}}\ge tS,\\
&\sum_{i=1}^{T}x_i^2+\rho_{\mathrm{pre}}^2=S^2,\quad S>0. \tag*{(99)}
\end{aligned} t , S , x 1 , … , x T max s.t. t x i ≥ 0 , γ i x i ≥ tS , i ∈ [ T ] γ pre ρ pre ≥ tS , i = 1 ∑ T x i 2 + ρ pre 2 = S 2 , S > 0. (99)
We first claim at optimum, the inequalities w.r.t. x i x_i x i will be tight, i.e., x i = t S γ i x_i=\frac{tS}{\gamma_i} x i = γ i tS , ∀ i ∈ [ T ] \forall i\in[T] ∀ i ∈ [ T ] . In this case, one can see that
S 2 = ρ p r e 2 + ∑ j = 1 T x j 2 ≥ S 2 t 2 ⋅ ( ∑ j = 1 T 1 γ j 2 + 1 γ p r e 2 ) . S^2=\rho_{\mathrm{pre}}^2+\sum_{j=1}^{T}x_j^2\ge S^2t^2\cdot\left(\sum_{j=1}^{T}\frac{1}{\gamma_j^2}+\frac{1}{\gamma_{\mathrm{pre}}^2}\right). S 2 = ρ pre 2 + j = 1 ∑ T x j 2 ≥ S 2 t 2 ⋅ ( j = 1 ∑ T γ j 2 1 + γ pre 2 1 ) .
Thus, the optimal value of the objective is t ≤ 1 1 γ p r e 2 + ∑ j = 1 T 1 γ j 2 t\le\frac{1}{\sqrt{\frac{1}{\gamma_{\mathrm{pre}}^2}+\sum_{j=1}^{T}\frac{1}{\gamma_j^2}}} t ≤ γ pre 2 1 + ∑ j = 1 T γ j 2 1 1 , and the equality is achieved under the condition γ i x i = γ p r e ρ p r e \gamma_i x_i=\gamma_{\mathrm{pre}}\rho_{\mathrm{pre}} γ i x i = γ pre ρ pre , which is equivalent to α i = ρ p r e γ p r e γ i Θ λ , i \alpha_i=\frac{\rho_{\mathrm{pre}}\gamma_{\mathrm{pre}}}{\gamma_i\Theta_{\lambda,i}} α i = γ i Θ λ , i ρ pre γ pre , i = 1 , … , T i=1,\ldots,T i = 1 , … , T .
Finally, we show that at the optimum of Problem 100, both the adapter and pre-training inequalities are tight.
Case 1: some adapter inequality is slack. Assume for the optimal solution ( t ∗ , S ∗ , x ∗ ) (t^*,S^*,x^*) ( t ∗ , S ∗ , x ∗ ) there exists j ∈ [ T ] j\in[T] j ∈ [ T ] such that γ j x j ∗ > t ∗ S ∗ \gamma_jx_j^*>t^*S^* γ j x j ∗ > t ∗ S ∗ . Decrease x j ∗ x_j^* x j ∗ slightly to x j ∗ − ϵ x_j^*-\epsilon x j ∗ − ϵ with ϵ > 0 \epsilon>0 ϵ > 0 . The new S S S is
S ~ = ρ p r e 2 + ∑ i ≠ j ( x i ∗ ) 2 + ( x j ∗ − ϵ ) 2 < S ∗ . \widetilde{S}=\sqrt{\rho_{\mathrm{pre}}^2+\sum_{i\ne j}(x_i^*)^2+(x_j^*-\epsilon)^2}<S^*. S = ρ pre 2 + i = j ∑ ( x i ∗ ) 2 + ( x j ∗ − ϵ ) 2 < S ∗ .
For ϵ \epsilon ϵ sufficiently small, every other adapter inequality remains satisfied (since S ~ < S ∗ \widetilde{S}<S^* S < S ∗ ), the pre-training inequality remains satisfied (γ p r e ρ p r e ≥ t ∗ S ∗ > t ∗ S ~ \gamma_{\mathrm{pre}}\rho_{\mathrm{pre}}\ge t^*S^*>t^*\widetilde{S} γ pre ρ pre ≥ t ∗ S ∗ > t ∗ S ), and γ j ( x j ∗ − ϵ ) > t ∗ S ~ \gamma_j(x_j^*-\epsilon)>t^*\widetilde{S} γ j ( x j ∗ − ϵ ) > t ∗ S by continuity. Therefore one can increase t ∗ t^* t ∗ until one inequality becomes tight, raising the objective and contradicting optimality.
Case 2: all adapter inequalities are tight but the pre-training inequality is slack. Assume γ i x i ∗ = t ∗ S ∗ \gamma_i x_i^*=t^*S^* γ i x i ∗ = t ∗ S ∗ for every i ∈ [ T ] i\in[T] i ∈ [ T ] but γ p r e ρ p r e > t ∗ S ∗ \gamma_{\mathrm{pre}}\rho_{\mathrm{pre}}>t^*S^* γ pre ρ pre > t ∗ S ∗ . Take x ~ i : = c x i ∗ \widetilde{x}_i:=cx_i^* x i := c x i ∗ for some c > 1 c>1 c > 1 , leaving ρ p r e \rho_{\mathrm{pre}} ρ pre unchanged. Then
S ~ = ρ p r e 2 + c 2 ∑ i ( x i ∗ ) 2 = c 2 ( S ∗ ) 2 − ρ p r e 2 ( c 2 − 1 ) < c S ∗ . \widetilde{S}=\sqrt{\rho_{\mathrm{pre}}^2+c^2\sum_i(x_i^*)^2}=\sqrt{c^2(S^*)^2-\rho_{\mathrm{pre}}^2(c^2-1)}<cS^*. S = ρ pre 2 + c 2 i ∑ ( x i ∗ ) 2 = c 2 ( S ∗ ) 2 − ρ pre 2 ( c 2 − 1 ) < c S ∗ .
Setting t ~ : = c t ∗ S ∗ / S ~ \widetilde{t}:=ct^*S^*/\widetilde{S} t := c t ∗ S ∗ / S keeps all adapter inequalities tight (since γ i x ~ i = c γ i x i ∗ = c t ∗ S ∗ = t ~ S ~ \gamma_i\widetilde{x}_i=c\gamma_i x_i^*=ct^*S^*=\widetilde{t}\widetilde{S} γ i x i = c γ i x i ∗ = c t ∗ S ∗ = t S ), and t ~ > t ∗ \widetilde{t}>t^* t > t ∗ from S ~ < c S ∗ \widetilde{S}<cS^* S < c S ∗ . For c c c close enough to 1, the pre-training inequality γ p r e ρ p r e ≥ t ~ S ~ = c t ∗ S ∗ \gamma_{\mathrm{pre}}\rho_{\mathrm{pre}}\ge\widetilde{t}\widetilde{S}=ct^*S^* γ pre ρ pre ≥ t S = c t ∗ S ∗ remains satisfied by continuity. Hence ( t ~ , S ~ , x ~ ) (\widetilde{t},\widetilde{S},\widetilde{x}) ( t , S , x ) is feasible with t ~ > t ∗ \widetilde{t}>t^* t > t ∗ , again contradicting optimality.
Combining the two cases, all inequalities must be tight at any optimum.
Proof of General Version
In this subsection we present the general version of Theorem 3.3 : under a near-orthogonal multi-task assumption (defined below), the margins of the merged model W L o R A ( α ) W_{\mathrm{LoRA}}(\alpha) W LoRA ( α ) on the pre-training data and on each fine-tuning task remain close to the strict-orthogonal closed-form values from Theorem 3.3 , with deviations controlled by the perturbation error E E E defined below. Consequently, the optimal mixing coefficients of Theorem 3.3 continue to (approximately) maximize the merged margin.
Near-orthogonal multi-task assumption. Let X i ∈ R d × K ˉ i n i X_i \in\mathbb{R}^{d \times\bar{K}_i n_i} X i ∈ R d × K ˉ i n i denote the data matrix of task i ∈ [ T ] i \in[T] i ∈ [ T ] . We replace Assumption 3.1 by the following near-orthogonal version: there exist ϵ 1 , ϵ 2 ∈ [ 0 , 1 ) \epsilon_1, \epsilon_2 \in[0,1) ϵ 1 , ϵ 2 ∈ [ 0 , 1 ) such that
(a) Every fine-tuning sample has unit norm: ∥ x ˉ ∥ 2 = 1 \lVert\bar{x}\rVert_2 = 1 ∥ x ˉ ∥ 2 = 1 for every x ˉ ∈ ⋃ i = 1 T D i \bar{x} \in\bigcup_{i=1}^{T} D_i x ˉ ∈ ⋃ i = 1 T D i .
(b) Within-task: for each i ∈ [ T ] i \in[T] i ∈ [ T ] and each pair of distinct x ˉ , x ˉ ′ ∈ D i \bar{x}, \bar{x}' \in D_i x ˉ , x ˉ ′ ∈ D i , ∣ x ˉ T x ˉ ′ ∣ ≤ ϵ 1 \lvert\bar{x}^{\mathsf{T}}\bar{x}'\rvert\le\epsilon_1 ∣ x ˉ T x ˉ ′ ∣ ≤ ϵ 1 .
(c) Cross-task and pre-training: for each i ≠ j i \ne j i = j in [ T ] [T] [ T ] , every x ˉ ∈ D i \bar{x} \in D_i x ˉ ∈ D i and x ˉ ′ ∈ D j \bar{x}' \in D_j x ˉ ′ ∈ D j satisfy ∣ x ˉ T x ˉ ′ ∣ ≤ ϵ 2 \lvert\bar{x}^{\mathsf{T}}\bar{x}'\rvert\le\epsilon_2 ∣ x ˉ T x ˉ ′ ∣ ≤ ϵ 2 ; and for every x ˉ ∈ ⋃ i D i \bar{x} \in\bigcup_i D_i x ˉ ∈ ⋃ i D i and every x ∈ D p r e x \in D_{\mathrm{pre}} x ∈ D pre , ∣ x ˉ T x ∣ ≤ ϵ 2 \lvert\bar{x}^{\mathsf{T}}x\rvert\le\epsilon_2 ∣ x ˉ T x ∣ ≤ ϵ 2 .
The strict-orthogonal Assumption 3.1 is recovered at ϵ 1 = ϵ 2 = 0 \epsilon_1 = \epsilon_2 = 0 ϵ 1 = ϵ 2 = 0 .
Notation. For each task i ∈ [ T ] i \in[T] i ∈ [ T ] we adopt the per-task analogues of the notation of Theorem D.2: G i : = X i T X i G_i := X_i^{\mathsf{T}}X_i G i := X i T X i , R i : = G i 1 / 2 R_i := G_i^{1/2} R i := G i 1/2 , Q i : = X i R i − 1 Q_i := X_iR_i^{-1} Q i := X i R i − 1 , S i : = W p r e X i S_i := W_{\mathrm{pre}}X_i S i := W pre X i , δ i : = ( K ˉ i n i − 1 ) ϵ 1 \delta_i := (\bar{K}_i n_i - 1)\epsilon_1 δ i := ( K ˉ i n i − 1 ) ϵ 1 , and the unique optimum Z λ , i ∗ ( S ˉ , R ˉ ) Z_{\lambda,i}^{*}(\bar{S},\bar{R}) Z λ , i ∗ ( S ˉ , R ˉ ) of Φ i ( Z ; S ˉ , R ˉ ) \Phi_i(Z;\bar{S},\bar{R}) Φ i ( Z ; S ˉ , R ˉ ) defined as in Theorem D.2 but with X i X_i X i , K ˉ i \bar{K}_i K ˉ i , n i n_i n i in place of X f t X_{\mathrm{ft}} X ft , K ˉ \bar{K} K ˉ , n n n . We take δ 0 \delta_0 δ 0 , s 0 s_0 s 0 , μ 0 \mu_0 μ 0 as the minima of their per-task counterparts and C 0 C_0 C 0 as the maximum. By the per-task reformulation, B i ∗ A i ∗ = Z λ , i ∗ ( S i , R i ) Q i T B_i^{*}A_i^{*} = Z_{\lambda,i}^{*}(S_i,R_i)Q_i^{\mathsf{T}} B i ∗ A i ∗ = Z λ , i ∗ ( S i , R i ) Q i T , where the rank constraint rank ( Z ) ≤ r \operatorname{rank}(Z) \le r rank ( Z ) ≤ r is non-binding for δ i \delta_i δ i , ∥ S i ∥ F \lVert S_i\rVert_F ∥ S i ∥ F sufficiently small by Lemma D.5 applied per task. Write
Θ λ , i ( 0 ) : = ∥ Z λ , i ∗ ( 0 , I ) ∥ F , γ i ( 0 ) : = γ i ∣ ϵ 1 = ϵ 2 = 0 , (100) \Theta_{\lambda,i}^{(0)} := \lVert Z_{\lambda,i}^{*}(0,I)\rVert_F,\qquad\gamma_i^{(0)} := \left.\gamma_i\right|_{\epsilon_1=\epsilon_2=0}, \tag*{(100)} Θ λ , i ( 0 ) := ∥ Z λ , i ∗ ( 0 , I ) ∥ F , γ i ( 0 ) := γ i ∣ ϵ 1 = ϵ 2 = 0 , (100)
for the strict-orthogonal per-task adapter Frobenius norm and per-task margin (so Θ λ , i ( 0 ) = Θ λ , i \Theta_{\lambda,i}^{(0)} = \Theta_{\lambda,i} Θ λ , i ( 0 ) = Θ λ , i and γ i ( 0 ) = γ i \gamma_i^{(0)} = \gamma_i γ i ( 0 ) = γ i from Theorem 3.3 when $\epsilon_1 = \epsilon_2 = 0). Set
E λ , i : = C 0 μ 0 ( ∥ S i ∥ F + δ i ( 1 + Θ λ , i ( 0 ) ) ) , E : = max i ∈ [ T ] E λ , i + ϵ 2 . (101) E_{\lambda,i} := \frac{C_0}{\mu_0}\left(\lVert S_i\rVert_F + \delta_i(1+\Theta_{\lambda,i}^{(0)})\right),\qquad E := \max_{i\in[T]} E_{\lambda,i}+\epsilon_2. \tag*{(101)} E λ , i := μ 0 C 0 ( ∥ S i ∥ F + δ i ( 1 + Θ λ , i ( 0 ) ) ) , E := i ∈ [ T ] max E λ , i + ϵ 2 . (101)
The common constants satisfy C 0 / μ 0 ≥ 1 C_0/\mu_0 \ge1 C 0 / μ 0 ≥ 1 , so ∥ S i ∥ F ≤ E λ , i ≤ E \lVert S_i\rVert_F \le E_{\lambda,i} \le E ∥ S i ∥ F ≤ E λ , i ≤ E . Under a uniform positive lower bound on σ min ( X p r e ) \sigma_{\min}(X_{\mathrm{pre}}) σ m i n ( X pre ) , ∥ S i ∥ F = O ( ϵ 2 ) \lVert S_i\rVert_F = O(\epsilon_2) ∥ S i ∥ F = O ( ϵ 2 ) uniformly in i i i (Remark D.2), so E → 0 E \to0 E → 0 as ϵ 1 , ϵ 2 → 0 \epsilon_1,\epsilon_2 \to0 ϵ 1 , ϵ 2 → 0 .
Theorem F.1 (General version of Theorem 3.3 ). Adopt the near-orthogonal multi-task assumption above with δ i ≤ δ 0 \delta_i \le\delta_0 δ i ≤ δ 0 and ∥ S i ∥ F ≤ s 0 \lVert S_i\rVert_F \le s_0 ∥ S i ∥ F ≤ s 0 for every i ∈ [ T ] i \in[T] i ∈ [ T ] , using the common thresholds chosen above. Suppose each task is trained in the intermediate or low-penalty regime, and the pre-training samples have unit norm. There exist constants E 0 ∈ ( 0 , 1 ] E_0 \in(0,1] E 0 ∈ ( 0 , 1 ] and C > 0 C > 0 C > 0 , depending on λ \lambda λ , K K K , { K ˉ i , n i } i = 1 T \{\bar{K}_i,n_i\}_{i=1}^{T} { K ˉ i , n i } i = 1 T , ρ p r e \rho_{\mathrm{pre}} ρ pre , μ 0 \mu_0 μ 0 , C 0 C_0 C 0 , { ∥ Z λ , i ∗ ( 0 , I ) ∥ ∗ } i = 1 T \{\lVert Z_{\lambda,i}^{*}(0,I)\rVert_{*}\}_{i=1}^{T} {∥ Z λ , i ∗ ( 0 , I ) ∥ ∗ } i = 1 T but not on ϵ 1 , ϵ 2 \epsilon_1,\epsilon_2 ϵ 1 , ϵ 2 or α \alpha α , such that whenever E ≤ E 0 E \le E_0 E ≤ E 0 , the following bounds hold uniformly for all nonnegative coefficient vectors α ∈ R ≥ 0 T \alpha\in\mathbb{R}_{\geq0}^{T} α ∈ R ≥ 0 T :
∣ γ ( W L o R A ( α ) ; D p r e ) − γ p r e ρ p r e ρ p r e 2 + ∑ i = 1 T α i 2 ( Θ λ , i ( 0 ) ) 2 ∣ ≤ C E , (102) \left|\gamma(W_{\mathrm{LoRA}}(\alpha);D_{\mathrm{pre}})-\frac{\gamma_{\mathrm{pre}}\rho_{\mathrm{pre}}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{i=1}^{T}\alpha_i^2\left(\Theta_{\lambda,i}^{(0)}\right)^2}}\right|\le CE, \tag*{(102)} γ ( W LoRA ( α ) ; D pre ) − ρ pre 2 + ∑ i = 1 T α i 2 ( Θ λ , i ( 0 ) ) 2 γ pre ρ pre ≤ C E , (102) ∣ γ ( W L o R A ( α ) ; D j ) − γ j ( 0 ) α j Θ λ , j ( 0 ) ρ p r e 2 + ∑ i = 1 T α i 2 ( Θ λ , i ( 0 ) ) 2 ∣ ≤ C E , j ∈ [ T ] . (103) \left|\gamma(W_{\mathrm{LoRA}}(\alpha);D_j)-\frac{\gamma_j^{(0)}\alpha_j\Theta_{\lambda,j}^{(0)}}{\sqrt{\rho_{\mathrm{pre}}^2+\sum_{i=1}^{T}\alpha_i^2\left(\Theta_{\lambda,i}^{(0)}\right)^2}}\right|\le CE,\qquad j\in[T]. \tag*{(103)} γ ( W LoRA ( α ) ; D j ) − ρ pre 2 + ∑ i = 1 T α i 2 ( Θ λ , i ( 0 ) ) 2 γ j ( 0 ) α j Θ λ , j ( 0 ) ≤ C E , j ∈ [ T ] . (103)
Write F ( α ) : = γ ( W L o R A ( α ) ; D p r e ∪ ⋃ i D i ) F(\alpha):=\gamma(W_{\mathrm{LoRA}}(\alpha);D_{\mathrm{pre}}\cup\bigcup_iD_i) F ( α ) := γ ( W LoRA ( α ) ; D pre ∪ ⋃ i D i ) and let F ( 0 ) ( α ) F^{(0)}(\alpha) F ( 0 ) ( α ) be the minimum of the reference margins displayed above. The 1-Lipschitz property of min \min min gives ∣ F ( α ) − F ( 0 ) ( α ) ∣ ≤ C E \lvert F(\alpha)-F^{(0)}(\alpha)\rvert\le CE ∣ F ( α ) − F ( 0 ) ( α )∣ ≤ C E uniformly over α ≥ 0 \alpha\ge0 α ≥ 0 . In particular, the strict-orthogonal optimum α i ∗ = ρ p r e γ p r e / ( γ i ( 0 ) Θ λ , i ( 0 ) ) \alpha_i^{*}=\rho_{\mathrm{pre}}\gamma_{\mathrm{pre}}/(\gamma_i^{(0)}\Theta_{\lambda,i}^{(0)}) α i ∗ = ρ pre γ pre / ( γ i ( 0 ) Θ λ , i ( 0 ) ) satisfies
0 ≤ sup α ∈ R ≥ 0 T F ( α ) − F ( α ∗ ) ≤ 2 C E . 0\le\sup_{\alpha\in\mathbb{R}_{\geq0}^{T}}F(\alpha)-F(\alpha^{*})\le2CE. 0 ≤ α ∈ R ≥ 0 T sup F ( α ) − F ( α ∗ ) ≤ 2 C E .
Thus α ∗ \alpha^{*} α ∗ remains optimal up to an additive O ( E ) O(E) O ( E ) error in achieved margin, and all margins converge uniformly to their strict-orthogonal reference values as E → 0 E\to0 E → 0 .
Proof. Throughout, C C C denotes a positive constant whose value may change line to line, but always depends only on the quantities listed in the theorem statement. We work with E ≤ 1 E\le1 E ≤ 1 and choose E 0 ≤ 1 E_0\le1 E 0 ≤ 1 below. We first establish leakage bounds. Since S i = W p r e X i S_i = W_{\mathrm{pre}}X_i S i = W pre X i , for every task i ∈ [ T ] i \in[T] i ∈ [ T ] , sample x ˉ j ( i ) ∈ D i \bar{x}^{(i)}_j \in\mathcal{D}_i x ˉ j ( i ) ∈ D i , and row w k w_k w k of W p r e W_{\mathrm{pre}} W pre ,
∣ w k ⊤ x ˉ j ( i ) ∣ = ∣ ( S i ) k j ∣ ≤ ∥ S i ∥ F ≤ E λ , i . (104) \left|w_k^{\top}\bar{x}^{(i)}_j\right| = \left|(S_i)_{kj}\right| \le\left\|S_i\right\|_F \le E_{\lambda,i}. \tag*{(104)} w k ⊤ x ˉ j ( i ) = ∣ ( S i ) k j ∣ ≤ ∥ S i ∥ F ≤ E λ , i . (104)
For each i i i , write δ λ , i , k : = ( B i ∗ A i ∗ ) k ⊤ \delta_{\lambda,i,k} := \left(B_i^*A_i^*\right)^\top_k δ λ , i , k := ( B i ∗ A i ∗ ) k ⊤ for the k k k -th row of B i ∗ A i ∗ B_i^*A_i^* B i ∗ A i ∗ . Using B i ∗ A i ∗ = Z λ , i ∗ ( S i , R i ) Q i ⊤ B_i^*A_i^* = Z_{\lambda,i}^*(S_i,R_i)Q_i^\top B i ∗ A i ∗ = Z λ , i ∗ ( S i , R i ) Q i ⊤ , we obtain (i) for any unit-norm x ∈ D p r e x \in\mathcal{D}_{\mathrm{pre}} x ∈ D pre ,
∣ δ λ , i , k ⊤ x ∣ ≤ ∥ ( Z λ , i ∗ ( S i , R i ) ) k ∥ 2 ∥ Q i ⊤ x ∥ 2 ≤ C ϵ 2 , (105) \left|\delta_{\lambda,i,k}^{\top}x\right| \le\left\|\left(Z_{\lambda,i}^*(S_i,R_i)\right)_k\right\|_2\left\|Q_i^\top x\right\|_2 \le C\epsilon_2, \tag*{(105)} δ λ , i , k ⊤ x ≤ ( Z λ , i ∗ ( S i , R i ) ) k 2 Q i ⊤ x 2 ≤ C ϵ 2 , (105)
where the last step uses ∥ Q i ⊤ x ∥ 2 ≤ K ˉ i n i ϵ 2 / 1 − δ i \left\|Q_i^\top x\right\|_2 \le\sqrt{\bar{K}_i n_i}\epsilon_2/\sqrt{1-\delta_i} Q i ⊤ x 2 ≤ K ˉ i n i ϵ 2 / 1 − δ i (from ∣ x ˉ ( i ) , ⊤ x ∣ ≤ ϵ 2 \left|\bar{x}^{(i),\top}x\right| \le\epsilon_2 x ˉ ( i ) , ⊤ x ≤ ϵ 2 ) and ∥ Z λ , i ∗ ( S i , R i ) ∥ F ≤ Θ λ , i ( 0 ) + E λ , i ≤ C \left\|Z_{\lambda,i}^*(S_i,R_i)\right\|_F \le\Theta_{\lambda,i}^{(0)}+E_{\lambda,i}\le C Z λ , i ∗ ( S i , R i ) F ≤ Θ λ , i ( 0 ) + E λ , i ≤ C . (ii) For any x ˉ ∈ D j \bar{x}\in\mathcal{D}_j x ˉ ∈ D j with j ≠ i j\ne i j = i , the same argument with X j X_j X j in place of { x } \{x\} { x } gives
∣ δ λ , i , k ⊤ x ˉ ∣ ≤ C ϵ 2 , (106) \left|\delta_{\lambda,i,k}^{\top}\bar{x}\right| \le C\epsilon_2, \tag*{(106)} δ λ , i , k ⊤ x ˉ ≤ C ϵ 2 , (106)
since each entry of X i ⊤ x ˉ X_i^\top\bar{x} X i ⊤ x ˉ is bounded by ϵ 2 \epsilon_2 ϵ 2 (cross-task assumption). The adapter leakage bounds are O ( ϵ 2 ) O(\epsilon_2) O ( ϵ 2 ) uniformly in i , k i,k i , k , and the pre-trained classifier leakage is controlled by ∥ S i ∥ F \left\|S_i\right\|_F ∥ S i ∥ F .
Then, we bound the perturbation of the merged-model Frobenius norm. Expanding,
∥ W L o R A ( α ) ∥ F 2 = ρ p r e 2 + ∑ i = 1 T α i 2 ∥ B i ∗ A i ∗ ∥ F 2 + 2 ∑ i = 1 T α i ⟨ W p r e , B i ∗ A i ∗ ⟩ + ∑ i ≠ j α i α j ⟨ B i ∗ A i ∗ , B j ∗ A j ∗ ⟩ . (107) \left\|W_{\mathrm{LoRA}}(\alpha)\right\|_F^2 = \rho_{\mathrm{pre}}^2 + \sum_{i=1}^{T}\alpha_i^2\left\|B_i^*A_i^*\right\|_F^2 + 2\sum_{i=1}^{T}\alpha_i\left\langle W_{\mathrm{pre}},B_i^*A_i^*\right\rangle+ \sum_{i\ne j}\alpha_i\alpha_j\left\langle B_i^*A_i^*,B_j^*A_j^*\right\rangle. \tag*{(107)} ∥ W LoRA ( α ) ∥ F 2 = ρ pre 2 + i = 1 ∑ T α i 2 ∥ B i ∗ A i ∗ ∥ F 2 + 2 i = 1 ∑ T α i ⟨ W pre , B i ∗ A i ∗ ⟩ + i = j ∑ α i α j ⟨ B i ∗ A i ∗ , B j ∗ A j ∗ ⟩ . (107)
The identity W p r e Q i = S i R i − 1 W_{\mathrm{pre}}Q_i=S_iR_i^{-1} W pre Q i = S i R i − 1 and the same projection argument as in the proof of Theorem E.2 give
∣ ⟨ W p r e , B i ∗ A i ∗ ⟩ ∣ ≤ ∥ S i ∥ F ∥ B i ∗ A i ∗ ∥ F 1 − δ i ≤ C ( Θ λ , i ( 0 ) + E λ , i ) ∥ S i ∥ F . (108) \left|\left\langle W_{\mathrm{pre}},B_i^*A_i^*\right\rangle\right| \le\frac{\left\|S_i\right\|_F\left\|B_i^*A_i^*\right\|_F}{\sqrt{1-\delta_i}} \le C\left(\Theta_{\lambda,i}^{(0)}+E_{\lambda,i}\right)\left\|S_i\right\|_F. \tag*{(108)} ∣ ⟨ W pre , B i ∗ A i ∗ ⟩ ∣ ≤ 1 − δ i ∥ S i ∥ F ∥ B i ∗ A i ∗ ∥ F ≤ C ( Θ λ , i ( 0 ) + E λ , i ) ∥ S i ∥ F . (108)
For the cross-task term with i ≠ j i\ne j i = j , each row ( B i ∗ A i ∗ ) k (B_i^*A_i^*)_k ( B i ∗ A i ∗ ) k lies in span ( X i ) \operatorname{span}(X_i) span ( X i ) and ( B j ∗ A j ∗ ) k ∈ span ( X j ) (B_j^*A_j^*)_k\in\operatorname{span}(X_j) ( B j ∗ A j ∗ ) k ∈ span ( X j ) . Writing u ∈ span ( X i ) u\in\operatorname{span}(X_i) u ∈ span ( X i ) as u = X i α u=X_i\alpha u = X i α with ∥ α ∥ 2 ≤ ∥ u ∥ 2 / σ min ( R i ) ≤ ∥ u ∥ 2 / 1 − δ i ≤ C ∥ u ∥ 2 \|\alpha\|_2\le\|u\|_2/\sigma_{\min}(R_i)\le\|u\|_2/\sqrt{1-\delta_i}\le C\|u\|_2 ∥ α ∥ 2 ≤ ∥ u ∥ 2 / σ m i n ( R i ) ≤ ∥ u ∥ 2 / 1 − δ i ≤ C ∥ u ∥ 2 (and similarly for v = X j β v=X_j\beta v = X j β ), we have u ⊤ v = α ⊤ X i ⊤ X j β u^\top v=\alpha^\top X_i^\top X_j\beta u ⊤ v = α ⊤ X i ⊤ X j β with each entry of X i ⊤ X j X_i^\top X_j X i ⊤ X j bounded by ϵ 2 \epsilon_2 ϵ 2 in absolute value, hence ∣ u ⊤ v ∣ ≤ ∥ X i ⊤ X j ∥ o p ∥ α ∥ 2 ∥ β ∥ 2 ≤ C ϵ 2 ∥ u ∥ 2 ∥ v ∥ 2 \left|u^\top v\right|\le\left\|X_i^\top X_j\right\|_{\mathrm{op}}\|\alpha\|_2\|\beta\|_2\le C\epsilon_2\|u\|_2\|v\|_2 u ⊤ v ≤ X i ⊤ X j op ∥ α ∥ 2 ∥ β ∥ 2 ≤ C ϵ 2 ∥ u ∥ 2 ∥ v ∥ 2 . Summing this bound over rows k k k and applying Cauchy–Schwarz,
∣ ⟨ B i ∗ A i ∗ , B j ∗ A j ∗ ⟩ ∣ ≤ C ϵ 2 ∥ B i ∗ A i ∗ ∥ F ∥ B j ∗ A j ∗ ∥ F ≤ C ϵ 2 ( Θ λ , i ( 0 ) + E λ , i ) ( Θ λ , j ( 0 ) + E λ , j ) . (109) \left|\left\langle B_i^*A_i^*,B_j^*A_j^*\right\rangle\right| \le C\epsilon_2\left\|B_i^*A_i^*\right\|_F\left\|B_j^*A_j^*\right\|_F \le C\epsilon_2\left(\Theta_{\lambda,i}^{(0)}+E_{\lambda,i}\right)\left(\Theta_{\lambda,j}^{(0)}+E_{\lambda,j}\right). \tag*{(109)} ⟨ B i ∗ A i ∗ , B j ∗ A j ∗ ⟩ ≤ C ϵ 2 ∥ B i ∗ A i ∗ ∥ F B j ∗ A j ∗ F ≤ C ϵ 2 ( Θ λ , i ( 0 ) + E λ , i ) ( Θ λ , j ( 0 ) + E λ , j ) . (109)
Set a α : = ∥ W L o R A ( α ) ∥ F a_\alpha:= \left\|W_{\mathrm{LoRA}}(\alpha)\right\|_F a α := ∥ W LoRA ( α ) ∥ F and b α 2 : = ρ p r e 2 + ∑ i α i 2 ( Θ λ , i ( 0 ) ) 2 b_\alpha^2 := \rho_{\mathrm{pre}}^2+\sum_i\alpha_i^2\left(\Theta_{\lambda,i}^{(0)}\right)^2 b α 2 := ρ pre 2 + ∑ i α i 2 ( Θ λ , i ( 0 ) ) 2 . Combining (109), (110) with the per-task bound ∣ ∥ B i ∗ A i ∗ ∥ F − Θ λ , i ( 0 ) ∣ ≤ E λ , i + O ( δ i Θ λ , i ( 0 ) ) \left|\left\|B_i^*A_i^*\right\|_F-\Theta_{\lambda,i}^{(0)}\right|\le E_{\lambda,i}+O(\delta_i\Theta_{\lambda,i}^{(0)}) ∥ B i ∗ A i ∗ ∥ F − Θ λ , i ( 0 ) ≤ E λ , i + O ( δ i Θ λ , i ( 0 ) ) from Theorem D.2 and using E ≤ 1 E\le1 E ≤ 1 yields, for some D > 0 D>0 D > 0 independent of α , ϵ 1 , ϵ 2 \alpha,\epsilon_1,\epsilon_2 α , ϵ 1 , ϵ 2 ,
∣ a α 2 − b α 2 ∣ ≤ D E ( 1 + ∥ α ∥ 2 2 ) . \left|a_\alpha^2-b_\alpha^2\right|\le DE(1+\|\alpha\|_2^2). a α 2 − b α 2 ≤ D E ( 1 + ∥ α ∥ 2 2 ) .
Here the linear and mixed coefficient terms are controlled using ∥ α ∥ 1 ≤ T ∥ α ∥ 2 \|\alpha\|_1\le\sqrt{T}\|\alpha\|_2 ∥ α ∥ 1 ≤ T ∥ α ∥ 2 and ∥ α ∥ 1 2 ≤ T ∥ α ∥ 2 2 \|\alpha\|_1^2\le T\|\alpha\|_2^2 ∥ α ∥ 1 2 ≤ T ∥ α ∥ 2 2 . The stated regimes ensure Θ λ , i ( 0 ) > 0 \Theta_{\lambda,i}^{(0)}>0 Θ λ , i ( 0 ) > 0 for every i i i , so
c : = min { ρ p r e , min i Θ λ , i ( 0 ) } > 0 , b α 2 ≥ c 2 ( 1 + ∥ α ∥ 2 2 ) . c:=\min\left\{\rho_{\mathrm{pre}},\min_i\Theta_{\lambda,i}^{(0)}\right\}>0,\qquad b_\alpha^2\ge c^2(1+\|\alpha\|_2^2). c := min { ρ pre , i min Θ λ , i ( 0 ) } > 0 , b α 2 ≥ c 2 ( 1 + ∥ α ∥ 2 2 ) .
Thus, setting κ : = D / c 2 \kappa:=D/c^2 κ := D / c 2 , we obtain the uniform relative bound
∣ a α 2 − b α 2 ∣ ≤ κ E b α 2 . (110) \left|a_\alpha^2-b_\alpha^2\right|\le\kappa E b_\alpha^2. \tag*{(110)} a α 2 − b α 2 ≤ κ E b α 2 . (110)
Choose E 0 : = min { 1 , ( 2 κ ) − 1 } E_0:=\min\{1,(2\kappa)^{-1}\} E 0 := min { 1 , ( 2 κ ) − 1 } . For E ≤ E 0 E\le E_0 E ≤ E 0 , this gives, uniformly over α ≥ 0 \alpha\ge0 α ≥ 0 ,
a α ≥ b α 2 > 0 , ∣ a α − b α ∣ b α = ∣ a α 2 − b α 2 ∣ b α ( a α + b α ) ≤ κ E . a_\alpha\ge\frac{b_\alpha}{\sqrt{2}}>0,\qquad\frac{\left|a_\alpha-b_\alpha\right|}{b_\alpha}=\frac{\left|a_\alpha^2-b_\alpha^2\right|}{b_\alpha(a_\alpha+b_\alpha)}\le\kappa E. a α ≥ 2 b α > 0 , b α ∣ a α − b α ∣ = b α ( a α + b α ) a α 2 − b α 2 ≤ κ E .
Next, we bound the perturbation of the unnormalized margins on each dataset. For ( x , y c ) ∈ D p r e (x,y_c)\in\mathcal{D}_{\mathrm{pre}} ( x , y c ) ∈ D pre and k ≠ c k\ne c k = c , decomposing
[ ( w c + ∑ i α i δ λ , i , c ) − ( w k + ∑ i α i δ λ , i , k ) ] ⊤ x = ( w c − w k ) ⊤ x + ∑ i = 1 T α i ( δ λ , i , c − δ λ , i , k ) ⊤ x , (111) \left[\left(w_c+\sum_i\alpha_i\delta_{\lambda,i,c}\right)-\left(w_k+\sum_i\alpha_i\delta_{\lambda,i,k}\right)\right]^\top x=(w_c-w_k)^\top x+\sum_{i=1}^{T}\alpha_i\left(\delta_{\lambda,i,c}-\delta_{\lambda,i,k}\right)^\top x, \tag*{(111)} [ ( w c + i ∑ α i δ λ , i , c ) − ( w k + i ∑ α i δ λ , i , k ) ] ⊤ x = ( w c − w k ) ⊤ x + i = 1 ∑ T α i ( δ λ , i , c − δ λ , i , k ) ⊤ x , (111)
and applying (105) to each adapter term,
∣ C ( W L o R A ( α ) ; D p r e ) − ρ p r e γ p r e ∣ ≤ C ∥ α ∥ 1 ϵ 2 . (112) \left|C(W_{\mathrm{LoRA}}(\alpha);D_{\mathrm{pre}})-\rho_{\mathrm{pre}}\gamma_{\mathrm{pre}}\right|\le C\|\alpha\|_1\epsilon_2. \tag*{(112)} ∣ C ( W LoRA ( α ) ; D pre ) − ρ pre γ pre ∣ ≤ C ∥ α ∥ 1 ϵ 2 . (112)
For ( x ˉ , y c ) ∈ D j (\bar{x},y_c)\in D_j ( x ˉ , y c ) ∈ D j and k ≠ c k\ne c k = c ,
[ ⋯ ] ⊤ x ˉ = α j ( δ λ , j , c − δ λ , j , k ) ⊤ x ˉ + ∑ i ≠ j α i ( δ λ , i , c − δ λ , i , k ) ⊤ x ˉ + ( w c − w k ) ⊤ x ˉ . (113) [\cdots]^{\top}\bar{x}=\alpha_j(\delta_{\lambda,j,c}-\delta_{\lambda,j,k})^{\top}\bar{x}+\sum_{i\ne j}\alpha_i(\delta_{\lambda,i,c}-\delta_{\lambda,i,k})^{\top}\bar{x}+(w_c-w_k)^{\top}\bar{x}. \tag*{(113)} [ ⋯ ] ⊤ x ˉ = α j ( δ λ , j , c − δ λ , j , k ) ⊤ x ˉ + i = j ∑ α i ( δ λ , i , c − δ λ , i , k ) ⊤ x ˉ + ( w c − w k ) ⊤ x ˉ . (113)
The first term in (113) is the within-task contribution. Since α j ≥ 0 \alpha_j\ge0 α j ≥ 0 , it can be taken outside the minimum. By the same argument as in the proof of Theorem E.2 (Step 3 there), ∣ min ( w . r . t . ( x ˉ , c , k ) ) α j ( δ λ , j , c − δ λ , j , k ) ⊤ x ˉ − α j Θ λ , j ( 0 ) γ j ( 0 ) ∣ ≤ C ∣ α j ∣ ( E λ , j + δ j ) \left|\min_{(w.r.t.\ (\bar{x},c,k))}\alpha_j(\delta_{\lambda,j,c}-\delta_{\lambda,j,k})^{\top}\bar{x}-\alpha_j\Theta_{\lambda,j}^{(0)}\gamma_j^{(0)}\right|\le C|\alpha_j|(E_{\lambda,j}+\delta_j) min ( w . r . t . ( x ˉ , c , k )) α j ( δ λ , j , c − δ λ , j , k ) ⊤ x ˉ − α j Θ λ , j ( 0 ) γ j ( 0 ) ≤ C ∣ α j ∣ ( E λ , j + δ j ) . The cross-task adapter terms are bounded via (106) : ∣ α i ( δ λ , i , c − δ λ , i , k ) ⊤ x ˉ ∣ ≤ 2 C ∣ α i ∣ ϵ 2 |\alpha_i(\delta_{\lambda,i,c}-\delta_{\lambda,i,k})^{\top}\bar{x}|\le2C|\alpha_i|\epsilon_2 ∣ α i ( δ λ , i , c − δ λ , i , k ) ⊤ x ˉ ∣ ≤ 2 C ∣ α i ∣ ϵ 2 , and the pre-training leakage on D j D_j D j contributes at most 2 ∥ S j ∥ F ≤ 2 E λ , j 2\|S_j\|_F\le2E_{\lambda,j} 2∥ S j ∥ F ≤ 2 E λ , j via (104) . Combining via the triangle inequality,
∣ C ( W L o R A ( α ) ; D j ) − α j Θ λ , j ( 0 ) γ j ( 0 ) ∣ ≤ C ( 1 + ∥ α ∥ 1 ) E , (114) \left|C(W_{\mathrm{LoRA}}(\alpha);D_j)-\alpha_j\Theta_{\lambda,j}^{(0)}\gamma_j^{(0)}\right|\le C(1+\|\alpha\|_1)E, \tag*{(114)} C ( W LoRA ( α ) ; D j ) − α j Θ λ , j ( 0 ) γ j ( 0 ) ≤ C ( 1 + ∥ α ∥ 1 ) E , (114)
where we have absorbed δ j \delta_j δ j into E E E via δ j ≤ ( μ 0 / C 0 ) E λ , j ≤ ( μ 0 / C 0 ) E \delta_j\le(\mu_0/C_0)E_{\lambda,j}\le(\mu_0/C_0)E δ j ≤ ( μ 0 / C 0 ) E λ , j ≤ ( μ 0 / C 0 ) E .
Finally, fix any dataset D ∈ { D p r e , D 1 , … , D T } D\in\{D_{\mathrm{pre}},D_1,\ldots,D_T\} D ∈ { D pre , D 1 , … , D T } and set u α : = C ( W L o R A ( α ) ; D ) u_\alpha:=C(W_{\mathrm{LoRA}}(\alpha);D) u α := C ( W LoRA ( α ) ; D ) . Its reference numerator u α ( 0 ) u_\alpha^{(0)} u α ( 0 ) is ρ p r e γ p r e \rho_{\mathrm{pre}}\gamma_{\mathrm{pre}} ρ pre γ pre for D p r e D_{\mathrm{pre}} D pre , and α j Θ λ , j ( 0 ) γ j ( 0 ) \alpha_j\Theta_{\lambda,j}^{(0)}\gamma_j^{(0)} α j Θ λ , j ( 0 ) γ j ( 0 ) for D j D_j D j . Equations (112) –(114) give ∣ u α − u α ( 0 ) ∣ ≤ C ( 1 + ∥ α ∥ 1 ) E |u_\alpha-u_\alpha^{(0)}|\le C(1+\|\alpha\|_1)E ∣ u α − u α ( 0 ) ∣ ≤ C ( 1 + ∥ α ∥ 1 ) E . Since
1 + ∥ α ∥ 1 b α ≤ T + 1 1 + ∥ α ∥ 2 2 b α ≤ T + 1 c , \frac{1+\|\alpha\|_1}{b_\alpha}\le\frac{\sqrt{T+1}\sqrt{1+\|\alpha\|_2^2}}{b_\alpha}\le\frac{\sqrt{T+1}}{c}, b α 1 + ∥ α ∥ 1 ≤ b α T + 1 1 + ∥ α ∥ 2 2 ≤ c T + 1 ,
the numerator error divided by b α b_\alpha b α is uniformly O ( E ) O(E) O ( E ) . All inputs have unit norm and a α > 0 a_\alpha>0 a α > 0 , so ∣ u α / a α ∣ ≤ 2 |u_\alpha/a_\alpha|\le\sqrt{2} ∣ u α / a α ∣ ≤ 2 . Applying the ratio estimate (90) and the relative norm bound above,
∣ u α a α − u α ( 0 ) b α ∣ ≤ ∣ u α − u α ( 0 ) ∣ + 2 ∣ a α − b α ∣ b α ≤ C E . \left|\frac{u_\alpha}{a_\alpha}-\frac{u_\alpha^{(0)}}{b_\alpha}\right|\le\frac{|u_\alpha-u_\alpha^{(0)}|+\sqrt{2}|a_\alpha-b_\alpha|}{b_\alpha}\le CE. a α u α − b α u α ( 0 ) ≤ b α ∣ u α − u α ( 0 ) ∣ + 2 ∣ a α − b α ∣ ≤ C E .
This proves (102) –(103) uniformly in α \alpha α . Taking the minimum across datasets gives ∣ F ( α ) − F ( 0 ) ( α ) ∣ ≤ C E |F(\alpha)-F^{(0)}(\alpha)|\le CE ∣ F ( α ) − F ( 0 ) ( α ) ∣ ≤ C E uniformly.
For the optimality claim, Theorem 3.3 gives F ( 0 ) ( α ) ≤ F ( 0 ) ( α ∗ ) F^{(0)}(\alpha)\le F^{(0)}(\alpha^*) F ( 0 ) ( α ) ≤ F ( 0 ) ( α ∗ ) for every α ≥ 0 \alpha\ge0 α ≥ 0 . Therefore,
F ( α ) ≤ F ( 0 ) ( α ) + C E ≤ F ( 0 ) ( α ∗ ) + C E ≤ F ( α ∗ ) + 2 C E . F(\alpha)\le F^{(0)}(\alpha)+CE\le F^{(0)}(\alpha^*)+CE\le F(\alpha^*)+2CE. F ( α ) ≤ F ( 0 ) ( α ) + C E ≤ F ( 0 ) ( α ∗ ) + C E ≤ F ( α ∗ ) + 2 C E .
Taking the supremum over all nonnegative coefficient vectors proves the stated 2 C E 2CE 2 C E near-optimality bound.
Lemmas
This section first lists the preliminary lemmas (cited from external sources) used throughout the proofs, and then gives full proofs of the paper-specific supporting lemmas.
Preliminary Lemmas
In this subsection, we provide several preliminary lemmas that will be used in the proof.
Lemma G.1 (Theorem 3 in Riedel [ [43 ] ). Let A ∈ R l × l A\in\mathbb{R}^{l\times l} A ∈ R l × l with rank ( A ) < l \operatorname{rank}(A)<l rank ( A ) < l , and v i , w i v_i,w_i v i , w i , i = 1 , 2 i=1,2 i = 1 , 2 be vectors in R l \mathbb{R}^l R l . Let v 1 ∈ M ( A ) v_1\in M(A) v 1 ∈ M ( A ) , w 1 ⊥ M ( A ) w_1\perp M(A) w 1 ⊥ M ( A ) , and v 2 ∈ M ( A ∗ ) v_2\in M(A^*) v 2 ∈ M ( A ∗ ) , w 2 ⊥ M ( A ∗ ) w_2\perp M(A^*) w 2 ⊥ M ( A ∗ ) , where M ( A ) M(A) M ( A ) denotes the range of A A A . Assume w 2 ∥ w 1 w_2\parallel w_1 w 2 ∥ w 1 and w i ≠ 0 w_i\ne0 w i = 0 , i = 1 , 2 i=1,2 i = 1 , 2 . Let Ω = A + ( v 1 + w 1 ) ( v 2 + w 2 ) ∗ \Omega=A+(v_1+w_1)(v_2+w_2)^* Ω = A + ( v 1 + w 1 ) ( v 2 + w 2 ) ∗ , Then
Ω † = A † − w 2 v 2 ∗ A † ∥ w 2 ∥ 2 − A † v 1 w 1 ∗ ∥ w 1 ∥ 2 + ( 1 + v 2 ∗ A † v 1 ) ⋅ w 2 w 1 ∗ ∥ w 1 ∥ 2 ∥ w 2 ∥ 2 . \Omega^\dagger=A^\dagger-\frac{w_2v_2^*A^\dagger}{\|w_2\|^2}-\frac{A^\dagger v_1w_1^*}{\|w_1\|^2}+(1+v_2^*A^\dagger v_1)\cdot\frac{w_2w_1^*}{\|w_1\|^2\|w_2\|^2}. Ω † = A † − ∥ w 2 ∥ 2 w 2 v 2 ∗ A † − ∥ w 1 ∥ 2 A † v 1 w 1 ∗ + ( 1 + v 2 ∗ A † v 1 ) ⋅ ∥ w 1 ∥ 2 ∥ w 2 ∥ 2 w 2 w 1 ∗ . We refer the readers to Riedel [ [43 ] for the proof.
Lemma G.2 (Variational form of the nuclear norm). For any fixed Z ∈ R K × m Z\in\mathbb{R}^{K\times m} Z ∈ R K × m , we have
∥ Z ∥ ∗ = min Z = B A 1 2 ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) . \|Z\|_*=\min_{Z=BA}\frac{1}{2}\left(\|B\|_F^2+\|A\|_F^2\right). ∥ Z ∥ ∗ = Z = B A min 2 1 ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) .
We refer the readers to Recht et al. [ [42 ] for the proof.
Lemma G.3. For any Z ∈ R K × m Z \in\mathbb{R}^{K \times m} Z ∈ R K × m ,
∥ Z ∥ ∗ = max V ∈ R m × K , ∥ V ∥ o p ≤ 1 Tr ( Z V ) . \lVert Z \rVert_* = \max_{V \in\mathbb{R}^{m \times K},\ \lVert V \rVert_{\mathrm{op}} \le1} \operatorname{Tr}(ZV). ∥ Z ∥ ∗ = V ∈ R m × K , ∥ V ∥ op ≤ 1 max Tr ( Z V ) . For any orthogonal projection P ∈ R K × K P \in\mathbb{R}^{K \times K} P ∈ R K × K and any Z ∈ R K × m Z \in\mathbb{R}^{K \times m} Z ∈ R K × m ,
∥ P Z ∥ ∗ ≤ ∥ P ∥ o p ∥ Z ∥ ∗ ≤ ∥ Z ∥ ∗ , \lVert PZ \rVert_* \le\lVert P \rVert_{\mathrm{op}} \lVert Z \rVert_* \le\lVert Z \rVert_*, ∥ P Z ∥ ∗ ≤ ∥ P ∥ op ∥ Z ∥ ∗ ≤ ∥ Z ∥ ∗ , Equality ∥ P Z ∥ ∗ = ∥ Z ∥ ∗ \lVert PZ \rVert_* = \lVert Z \rVert_* ∥ P Z ∥ ∗ = ∥ Z ∥ ∗ holds if and only if P Z = Z PZ = Z P Z = Z . Moreover, for any two matrices A , B ∈ R K × m A, B \in\mathbb{R}^{K \times m} A , B ∈ R K × m , one has
∑ i = 1 min ( K , m ) σ i ( A B ⊤ ) ≤ ∑ i = 1 min ( K , m ) σ i ( A ) σ i ( B ) . \sum_{i=1}^{\min(K,m)} \sigma_i(AB^{\top}) \le\sum_{i=1}^{\min(K,m)} \sigma_i(A)\sigma_i(B). i = 1 ∑ m i n ( K , m ) σ i ( A B ⊤ ) ≤ i = 1 ∑ m i n ( K , m ) σ i ( A ) σ i ( B ) .
We refer the readers to Horn and Johnson [ [19 ] for the proof.
Lemma G.4 (Berge’s Maximum Theorem). Let X X X , Θ \Theta Θ be topological spaces, f : X × Θ → R f : X \times\Theta\to\mathbb{R} f : X × Θ → R be a continuous function on the product X × Θ X \times\Theta X × Θ , and C : Θ → X C : \Theta\to X C : Θ → X be a compact-valued correspondence such that C ( θ ) ≠ ∅ C(\theta) \ne\varnothing C ( θ ) = ∅ for all θ ∈ Θ \theta\in\Theta θ ∈ Θ . Define the marginal function f ∗ : Θ → R f^* : \Theta\to\mathbb{R} f ∗ : Θ → R and the set of minimizers C ∗ : Θ → X C^* : \Theta\to X C ∗ : Θ → X by
f ∗ ( θ ) = sup { f ( x , θ ) : x ∈ C ( θ ) } f^*(\theta) = \sup\{f(x,\theta) : x \in C(\theta)\} f ∗ ( θ ) = sup { f ( x , θ ) : x ∈ C ( θ )} C ∗ ( θ ) = arg max { f ( x , θ ) : x ∈ C ( θ ) } = { x ∈ C ( θ ) : f ( x ; θ ) = f ∗ ( θ ) } . C^*(\theta) = \arg\max\{f(x,\theta) : x \in C(\theta)\} = \{x \in C(\theta) : f(x;\theta) = f^*(\theta)\}. C ∗ ( θ ) = arg max { f ( x , θ ) : x ∈ C ( θ )} = { x ∈ C ( θ ) : f ( x ; θ ) = f ∗ ( θ )} . If C C C is continuous at θ \theta θ , then the value function f ∗ ( θ ) f^*(\theta) f ∗ ( θ ) is continuous, and the set of maximizers C ∗ ( θ ) C^*(\theta) C ∗ ( θ ) is upper-hemicontinuous with nonempty and compact values. Moreover, if C ∗ ( θ ) C^*(\theta) C ∗ ( θ ) is single-valued, and thus is a continuous function rather than a correspondence.
We refer the readers to Sundaram [ [47 ] for the proof.
Lemma G.5 (Invariance under Permutation). Suppose X X X satisfies Π X Π ⊤ = X \Pi X\Pi^{\top} = X Π X Π ⊤ = X for any permutation Π \Pi Π , then X = a I + c 11 ⊤ X = aI + c\mathbf{1}\mathbf{1}^{\top} X = a I + c 1 1 ⊤ for any constants a , c ∈ R a, c \in\mathbb{R} a , c ∈ R .
We refer the readers to Lemma 10 in Hong and Ling [ [18 ] for the proof.
Proposition G.1 (Representer Theorem for Hard-margin SVM Problem). Let { ( x c , i , y c ) } i = 1 , c = 1 n c , K \{(x_{c,i}, y_c)\}_{i=1,c=1}^{n_c,K} {( x c , i , y c ) } i = 1 , c = 1 n c , K be the dataset with features x c , i ∈ R d x_{c,i} \in\mathbb{R}^{d} x c , i ∈ R d and class labels y c ∈ R K y_c \in\mathbb{R}^{K} y c ∈ R K . Then, let W m m W^{\mathrm{mm}} W mm be the solution of the following hard-margin SVM problem, i.e.,
min W ∈ R K × d 1 2 ∥ W ∥ F 2 s.t. ∀ i ∈ [ K ] , ∀ j ∈ [ n i ] , ∀ k ≠ i : w i ⊤ x i , j ≥ w k ⊤ x i , j + 1 , \min_{W \in\mathbb{R}^{K \times d}} \frac{1}{2}\lVert W \rVert_F^2 \quad\text{s.t.} \quad\forall i \in[K],\ \forall j \in[n_i],\ \forall k \ne i : \ w_i^{\top}x_{i,j} \ge w_k^{\top}x_{i,j} + 1, W ∈ R K × d min 2 1 ∥ W ∥ F 2 s.t. ∀ i ∈ [ K ] , ∀ j ∈ [ n i ] , ∀ k = i : w i ⊤ x i , j ≥ w k ⊤ x i , j + 1 , where W = [ w 1 , ⋯ , w K ] ⊤ W = [w_1,\cdots,w_K]^{\top} W = [ w 1 , ⋯ , w K ] ⊤ . Then, there exists a set of scalars { α i , c , j } i = 1 , c = 1 , j = 1 K , K , n i \{\alpha_{i,c,j}\}_{i=1,c=1,j=1}^{K,K,n_i} { α i , c , j } i = 1 , c = 1 , j = 1 K , K , n i such that each row of W m m W^{\mathrm{mm}} W mm has the following structure
w i m m = ∑ c = 1 K ∑ j = 1 n c α i , c , j x c , j . w_i^{\mathrm{mm}} = \sum_{c=1}^{K}\sum_{j=1}^{n_c}\alpha_{i,c,j}x_{c,j}. w i mm = c = 1 ∑ K j = 1 ∑ n c α i , c , j x c , j .
We refer the readers to Scholkopf and Smola [2018] for the proof.
Proof of Lemma D.1
Proof. By Proposition G.1 and Assumption 2.4, each row of W p r e W_{\mathrm{pre}} W pre lies in the span of the pre-training features. Under Assumption 2.1(c) with ϵ 2 = 0 \epsilon_2 = 0 ϵ 2 = 0 , every fine-tuning feature is orthogonal to this span, so
W p r e X f t = 0. (115) W_{\mathrm{pre}}X_{\mathrm{ft}} = 0. \tag*{(115)} W pre X ft = 0. (115)
Moreover, Assumption 2.1(a)–(b) with ϵ 1 = 0 \epsilon_1 = 0 ϵ 1 = 0 gives X f t ⊤ X f t = I K ˉ n X_{\mathrm{ft}}^{\top}X_{\mathrm{ft}} = I_{\bar{K}n} X ft ⊤ X ft = I K ˉ n . We apply change of variable as follows: A ~ = A X f t ∈ R r × K ˉ n \widetilde{A} = AX_{\mathrm{ft}} \in\mathbb{R}^{r \times\bar{K}n} A = A X ft ∈ R r × K ˉ n .
1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , ( W p r e + B A ) x c , j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , W p r e X f t e ( c − 1 ) n + j + B A X f t e ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , W p r e X f t e ( c − 1 ) n + j + B A Q R e ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A Q ∥ F 2 + ∥ A Q ⊥ ∥ F 2 ) = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , W p r e X f t e ( c − 1 ) n + j + B A ~ R e ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ~ ∥ F 2 + ∥ A Q ⊥ ∥ F 2 ) . \begin{aligned}
&\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,(W_{\mathrm{pre}}+BA)x_{c,j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert A\rVert_F^2\right)\\
={}&\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,W_{\mathrm{pre}}X_{\mathrm{ft}}e_{(c-1)n+j}+BAX_{\mathrm{ft}}e_{(c-1)n+j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert A\rVert_F^2\right)\\
={}&\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,W_{\mathrm{pre}}X_{\mathrm{ft}}e_{(c-1)n+j}+BAQRe_{(c-1)n+j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert AQ\rVert_F^2+\lVert AQ_{\perp}\rVert_F^2\right)\\
={}&\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,W_{\mathrm{pre}}X_{\mathrm{ft}}e_{(c-1)n+j}+B\widetilde{A}Re_{(c-1)n+j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert\widetilde{A}\rVert_F^2+\lVert AQ_{\perp}\rVert_F^2\right).
\end{aligned} = = = K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , ( W pre + B A ) x c , j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , W pre X ft e ( c − 1 ) n + j + B A X ft e ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , W pre X ft e ( c − 1 ) n + j + B A QR e ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A Q ∥ F 2 + ∥ A Q ⊥ ∥ F 2 ) K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , W pre X ft e ( c − 1 ) n + j + B A R e ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 + ∥ A Q ⊥ ∥ F 2 ) .
Under this transformation, the objective in (5) can be rewritten as follows
1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , ( W p r e + B A ) x c , j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , B A X f t e ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , B A ~ e ~ ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A X f t ∥ F 2 + ∥ A X f t , ⊥ ∥ F 2 ) = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , B A ~ e ~ ( c − 1 ) n + j ) + λ 2 ( ∥ B ∥ F 2 + ∥ A ~ ∥ F 2 + ∥ A X f t , ⊥ ∥ F 2 ) . \begin{aligned}
&\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,(W_{\mathrm{pre}}+BA)x_{c,j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert A\rVert_F^2\right)\\
={}&\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,BAX_{\mathrm{ft}}e_{(c-1)n+j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert A\rVert_F^2\right)\\
={}&\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,B\widetilde{A}\widetilde{e}_{(c-1)n+j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert AX_{\mathrm{ft}}\rVert_F^2+\lVert AX_{\mathrm{ft},\perp}\rVert_F^2\right)\\
={}&\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,B\widetilde{A}\widetilde{e}_{(c-1)n+j}\right)+\frac{\lambda}{2}\left(\lVert B\rVert_F^2+\lVert\widetilde{A}\rVert_F^2+\lVert AX_{\mathrm{ft},\perp}\rVert_F^2\right).
\end{aligned} = = = K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , ( W pre + B A ) x c , j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , B A X ft e ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 ) K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , B A e ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A X ft ∥ F 2 + ∥ A X ft , ⊥ ∥ F 2 ) K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , B A e ( c − 1 ) n + j ) + 2 λ ( ∥ B ∥ F 2 + ∥ A ∥ F 2 + ∥ A X ft , ⊥ ∥ F 2 ) .
Notice that the component A X f t , ⊥ AX_{\mathrm{ft},\perp} A X ft , ⊥ contributes only to the regularization term and does not affect the cross-entropy loss. Hence at any global minimizer ( B ∗ , A ∗ ) (B^*,A^*) ( B ∗ , A ∗ ) of (5) we must have
A X f t , ⊥ = 0. AX_{\mathrm{ft},\perp}=0. A X ft , ⊥ = 0.
Moreover, if ( B ∗ , A ∗ ) (B^*,A^*) ( B ∗ , A ∗ ) is a global minimizer of the original problem, then
( B ∗ , A ∗ X f t ) (B^*,A^*X_{\mathrm{ft}}) ( B ∗ , A ∗ X ft )
is a global minimizer of the objective in (25) . Conversely, let ( B ∗ , A ~ ∗ ) (B^*,\widetilde{A}^*) ( B ∗ , A ∗ ) be any global minimizer of (25), then one can solve the following equation to obtain ( B ∗ , A ∗ ) (B^*,A^*) ( B ∗ , A ∗ ) which is a global minimizer of the original objective in (5).
A ∗ X f t = A ~ ∗ , A ∗ X f t , ⊥ = 0. (116) A^*X_{\mathrm{ft}}=\widetilde{A}^*,\qquad A^*X_{\mathrm{ft},\perp}=0. \tag*{(116)} A ∗ X ft = A ∗ , A ∗ X ft , ⊥ = 0. (116)
□ \square □
Proof of Lemma D.2
Proof. To show that there exists a unique value ρ λ \rho_{\lambda} ρ λ such that the (Problem one) and (Problem two) enjoys the same set of solutions, we first introduce two additional optimization problems.
min Z 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , Z e ( c − 1 ) n + j ) + λ ∥ Z ∥ ∗ (Problem three) \min_Z\ \frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,Ze_{(c-1)n+j}\right)+\lambda\lVert Z\rVert_*
\qquad\text{(Problem three)} Z min K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , Z e ( c − 1 ) n + j ) + λ ∥ Z ∥ ∗ (Problem three)
min Z 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , Z e ( c − 1 ) n + j ) s.t. ∥ Z ∥ ∗ ≤ ρ λ (Problem four) \min_Z\ \frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}\left(y_c,Ze_{(c-1)n+j}\right)
\quad\text{s.t.}\ \lVert Z\rVert_*\leq\rho_{\lambda}
\qquad\text{(Problem four)} Z min K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , Z e ( c − 1 ) n + j ) s.t. ∥ Z ∥ ∗ ≤ ρ λ (Problem four)
where Z ∈ R K × K ˉ n Z\in\mathbb{R}^{K\times\bar{K}n} Z ∈ R K × K ˉ n . Let S i ( λ ) S_i(\lambda) S i ( λ ) , i = 1 , 2 , 3 , 4 i=1,2,3,4 i = 1 , 2 , 3 , 4 be the solution sets of the above four optimization problems for the same fixed λ \lambda λ . We will show that
S 1 ( λ ) = S 2 ( λ ) , S 3 ( λ ) = S 4 ( λ ) S_1(\lambda)=S_2(\lambda), S_3(\lambda)=S_4(\lambda) S 1 ( λ ) = S 2 ( λ ) , S 3 ( λ ) = S 4 ( λ ) .
S 3 ( λ ) , S 4 ( λ ) S_3(\lambda), S_4(\lambda) S 3 ( λ ) , S 4 ( λ ) contains only one element for any fixed λ > 0 \lambda>0 λ > 0 .
∀ B ∗ , A ~ ∗ ∈ S 1 ( λ ) \forall B^*,\widetilde{A}^*\in S_1(\lambda) ∀ B ∗ , A ∗ ∈ S 1 ( λ ) , one has B ∗ A ~ ∗ ∈ S 3 ( λ ) B^*\widetilde{A}^*\in S_3(\lambda) B ∗ A ∗ ∈ S 3 ( λ ) .
Let S 3 ( λ ) = S 4 ( λ ) = { Z λ ∗ } S_3(\lambda)=S_4(\lambda)=\{Z_\lambda^*\} S 3 ( λ ) = S 4 ( λ ) = { Z λ ∗ } , then there exists B ∗ A ~ ∗ = Z λ ∗ B^*\widetilde{A}^*=Z_\lambda^* B ∗ A ∗ = Z λ ∗ such that ( B ∗ , A ~ ∗ ) ∈ S 1 ( λ ) (B^*,\widetilde{A}^*)\in S_1(\lambda) ( B ∗ , A ∗ ) ∈ S 1 ( λ ) .
We first establish that (Problem three) has a unique solution. Our analysis is based on the following lemma.
Lemma G.6 (Cross entropy loss with nuclear norm has a unique minimizer). For any λ > 0 \lambda>0 λ > 0 , define
ϕ ( Z ; λ ) = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n L C E ( y c , Z e ( c − 1 ) n + j ) + λ ∥ Z ∥ ∗ , \phi(Z;\lambda)=\frac{1}{\bar{K}n}\sum_{c=1}^{\bar{K}}\sum_{j=1}^{n}\mathcal{L}_{\mathrm{CE}}(y_c,Z e_{(c-1)n+j})+\lambda\lVert Z\rVert_*, ϕ ( Z ; λ ) = K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n L CE ( y c , Z e ( c − 1 ) n + j ) + λ ∥ Z ∥ ∗ , where y c y_c y c is a one-hot vector with c c c -th index equals one. Then ϕ ( Z ; λ ) \phi(Z;\lambda) ϕ ( Z ; λ ) has a unique solution.
We refer the readers to later sections in Appendix G.4 for the proof.
Based on Lemma G.6, (Problem three) has a unique solution, denoted by Z λ ∗ Z_\lambda^* Z λ ∗ .
Before relating this solution to the fixed-width factorization in (Problem one), we verify its rank feasibility. Let Π = diag ( Π 1 , … , Π K ˉ ) \Pi=\operatorname{diag}(\Pi_1,\ldots,\Pi_{\bar{K}}) Π = diag ( Π 1 , … , Π K ˉ ) , where each Π c ∈ R n × n \Pi_c\in\mathbb{R}^{n\times n} Π c ∈ R n × n is a permutation matrix. Permuting columns within a class leaves the mean cross entropy unchanged, and ∥ Z Π ∥ ∗ = ∥ Z ∥ ∗ \lVert Z\Pi\rVert_*=\lVert Z\rVert_* ∥ Z Π ∥ ∗ = ∥ Z ∥ ∗ by unitary invariance. Thus ϕ ( Z Π ; λ ) = ϕ ( Z ; λ ) \phi(Z\Pi;\lambda)=\phi(Z;\lambda) ϕ ( Z Π ; λ ) = ϕ ( Z ; λ ) , and uniqueness implies Z λ ∗ Π = Z λ ∗ Z_\lambda^*\Pi=Z_\lambda^* Z λ ∗ Π = Z λ ∗ for every such Π \Pi Π . Consequently, columns within each class coincide, so
Z λ ∗ = V λ ⊗ 1 n T , V λ ∈ R K × K ˉ , rank ( Z λ ∗ ) ≤ K ˉ ≤ r , Z_\lambda^*=V_\lambda\otimes\mathbf{1}_n^{\mathsf{T}},\qquad V_\lambda\in\mathbb{R}^{K\times\bar{K}},\qquad\operatorname{rank}(Z_\lambda^*)\leq\bar{K}\leq r, Z λ ∗ = V λ ⊗ 1 n T , V λ ∈ R K × K ˉ , rank ( Z λ ∗ ) ≤ K ˉ ≤ r ,
where the final inequality follows from Assumption 2.3.
Set q = rank ( Z λ ∗ ) q=\operatorname{rank}(Z_\lambda^*) q = rank ( Z λ ∗ ) . If q > 0 q>0 q > 0 , take the compact SVD Z λ ∗ = U q Σ q V q T Z_\lambda^*=U_q\Sigma_qV_q^{\mathsf{T}} Z λ ∗ = U q Σ q V q T , with U q ∈ R K n ˉ × q U_q\in\mathbb{R}^{K\bar{n}\times q} U q ∈ R K n ˉ × q and V q ∈ R K ˉ n × q V_q\in\mathbb{R}^{\bar{K}n\times q} V q ∈ R K ˉ n × q , and define
B 0 = [ U q Σ q 1 / 2 0 K × ( r − q ) ] ∈ R K × r , B_0=\left[U_q\Sigma_q^{1/2}\ \ 0_{K\times(r-q)}\right]\in\mathbb{R}^{K\times r}, B 0 = [ U q Σ q 1/2 0 K × ( r − q ) ] ∈ R K × r ,
A ~ 0 = [ Σ q 1 / 2 V q T 0 ( r − q ) × K ˉ n ] ∈ R r × K ˉ n . \widetilde{A}_0=\begin{bmatrix}\Sigma_q^{1/2}V_q^{\mathsf{T}}\\0_{(r-q)\times\bar{K}n}\end{bmatrix}\in\mathbb{R}^{r\times\bar{K}n}. A 0 = [ Σ q 1/2 V q T 0 ( r − q ) × K ˉ n ] ∈ R r × K ˉ n .
If q = 0 q=0 q = 0 , take B 0 = 0 B_0=0 B 0 = 0 and A ~ 0 = 0 \widetilde{A}_0=0 A 0 = 0 with the same dimensions. In either case,
B 0 A ~ 0 = Z λ ∗ , ∥ B 0 ∥ F 2 = ∥ A ~ 0 ∥ F 2 = ∥ Z λ ∗ ∥ ∗ . B_0\widetilde{A}_0=Z_\lambda^*,\qquad\lVert B_0\rVert_F^2=\lVert\widetilde{A}_0\rVert_F^2=\lVert Z_\lambda^*\rVert_*. B 0 A 0 = Z λ ∗ , ∥ B 0 ∥ F 2 = ∥ A 0 ∥ F 2 = ∥ Z λ ∗ ∥ ∗ .
By Lemma G.2, the objective of (Problem one) at any ( B , A ~ ) (B,\widetilde{A}) ( B , A ) is at least ϕ ( B A ~ ; λ ) ≥ ϕ ( Z λ ∗ ; λ ) \phi(B\widetilde{A};\lambda)\geq\phi(Z_\lambda^*;\lambda) ϕ ( B A ; λ ) ≥ ϕ ( Z λ ∗ ; λ ) , and ( B 0 , A ~ 0 ) (B_0,\widetilde{A}_0) ( B 0 , A 0 ) attains this lower bound. Hence the minimum values of (Problem one) and (Problem three) agree. Every ( B ∗ , A ~ ∗ ) ∈ S 1 ( λ ) (B^*,\widetilde{A}^*)\in S_1(\lambda) ( B ∗ , A ∗ ) ∈ S 1 ( λ ) must therefore satisfy
B ∗ A ~ ∗ = Z λ ∗ , 2 ∥ Z λ ∗ ∥ ∗ = ∥ B ∗ ∥ F 2 + ∥ A ~ ∗ ∥ F 2 . B^*\widetilde{A}^*=Z_\lambda^*,\qquad2\lVert Z_\lambda^*\rVert_*=\lVert B^*\rVert_F^2+\lVert\widetilde{A}^*\rVert_F^2. B ∗ A ∗ = Z λ ∗ , 2 ∥ Z λ ∗ ∥ ∗ = ∥ B ∗ ∥ F 2 + ∥ A ∗ ∥ F 2 .
Equality in ∥ B ∗ A ~ ∗ ∥ ∗ ≤ ∥ B ∗ ∥ F ∥ A ~ ∗ ∥ F ≤ ( ∥ B ∗ ∥ F 2 + ∥ A ~ ∗ ∥ F 2 ) / 2 \lVert B^*\widetilde{A}^*\rVert_*\leq\lVert B^*\rVert_F\lVert\widetilde{A}^*\rVert_F\leq(\lVert B^*\rVert_F^2+\lVert\widetilde{A}^*\rVert_F^2)/2 ∥ B ∗ A ∗ ∥ ∗ ≤ ∥ B ∗ ∥ F ∥ A ∗ ∥ F ≤ (∥ B ∗ ∥ F 2 + ∥ A ∗ ∥ F 2 ) /2 also gives ∥ B ∗ ∥ F 2 = ∥ A ~ ∗ ∥ F 2 = ∥ Z λ ∗ ∥ ∗ \lVert B^*\rVert_F^2=\lVert\widetilde{A}^*\rVert_F^2=\lVert Z_\lambda^*\rVert_* ∥ B ∗ ∥ F 2 = ∥ A ∗ ∥ F 2 = ∥ Z λ ∗ ∥ ∗ .
Then, one can choose ρ λ = ∥ Z λ ∗ ∥ ∗ \rho_\lambda=\lVert Z_\lambda^*\rVert_* ρ λ = ∥ Z λ ∗ ∥ ∗ , and based on strong duality, (Problem three) and (Problem four) admits the same set of solutions, i.e., S 3 ( λ ) = S 4 ( λ ) S_3(\lambda)=S_4(\lambda) S 3 ( λ ) = S 4 ( λ ) .
Next, we show that ∀ ( B ∗ , A ~ ∗ ) ∈ S 2 ( λ ) \forall(B^*,\widetilde{A}^*)\in S_2(\lambda) ∀ ( B ∗ , A ∗ ) ∈ S 2 ( λ ) , it must satisfy
B ∗ A ~ ∗ ∈ S 4 ( λ ) , 2 ∥ B ∗ A ~ ∗ ∥ ∗ = ∥ B ∗ ∥ F 2 + ∥ A ~ ∗ ∥ F 2 . B^*\widetilde{A}^*\in S_4(\lambda),\qquad2\lVert B^*\widetilde{A}^*\rVert_*=\lVert B^*\rVert_F^2+\lVert\widetilde{A}^*\rVert_F^2. B ∗ A ∗ ∈ S 4 ( λ ) , 2 ∥ B ∗ A ∗ ∥ ∗ = ∥ B ∗ ∥ F 2 + ∥ A ∗ ∥ F 2 .
This is because first, the minimum objective value of (Problem two) and (Problem four) must be equal. Given Z λ ∗ ∈ S 4 ( λ ) Z_\lambda^*\in S_4(\lambda) Z λ ∗ ∈ S 4 ( λ ) , one can do balanced factorization, and obtain the corresponding B ∗ , A ~ ∗ B^*,\widetilde{A}^* B ∗ , A ∗ . This implies that the minimum objective value of (Problem two) is less than or equal to that of (Problem four). On the other hand, ∀ ( B ∗ , A ~ ∗ ) ∈ S 2 ( λ ) \forall(B^*,\widetilde{A}^*)\in S_2(\lambda) ∀ ( B ∗ , A ∗ ) ∈ S 2 ( λ ) , one has $\lVert B^\widetilde{A}^ \rVert_\leq\frac{\lVert B^ \rVert_F^2+\lVert\widetilde{A}^*\rVert_F^2}{2}\leq ρ λ \rho_{\lambda} ρ λ which implies B ∗ A ~ ∗ B^{*}\widetilde{A}^{*} B ∗ A ∗ is a feasible solution of (Problem four). Thus, the minimum objective value of (Problem four) is less than or equal to that of (Problem two). Combine these together, we conclude that the minimum objective value of (Problem two) and (Problem four) must be equal, and ∀ ( B ∗ , A ~ ∗ ) ∈ S 2 ( λ ) \forall(B^{*},\widetilde{A}^{*})\in S_{2}(\lambda) ∀ ( B ∗ , A ∗ ) ∈ S 2 ( λ ) , B ∗ A ~ ∗ ∈ S 4 ( λ ) B^{*}\widetilde{A}^{*}\in S_{4}(\lambda) B ∗ A ∗ ∈ S 4 ( λ ) must hold. Since S 4 ( λ ) = { Z λ ∗ } S_{4}(\lambda)=\{Z_{\lambda}^{*}\} S 4 ( λ ) = { Z λ ∗ } , the preceding norm inequalities are equalities, and ∥ B ∗ ∥ F 2 = ∥ A ~ ∗ ∥ F 2 = ρ λ \lVert B^{*}\rVert_{F}^{2}=\lVert\widetilde{A}^{*}\rVert_{F}^{2}=\rho_{\lambda} ∥ B ∗ ∥ F 2 = ∥ A ∗ ∥ F 2 = ρ λ .
Finally, we show S 1 ( λ ) = S 2 ( λ ) S_{1}(\lambda)=S_{2}(\lambda) S 1 ( λ ) = S 2 ( λ ) .
This is because, on one hand, ∀ B 1 ∗ , A ~ 1 ∗ ∈ S 1 ( λ ) \forall B_{1}^{*},\widetilde{A}_{1}^{*}\in S_{1}(\lambda) ∀ B 1 ∗ , A 1 ∗ ∈ S 1 ( λ ) , from previous reasoning, they must satisfy
B 1 ∗ A ~ 1 ∗ ∈ S 3 ( λ ) = S 4 ( λ ) , ∥ B 1 ∗ ∥ F 2 = ∥ A ~ 1 ∗ ∥ F 2 = ρ λ , B_{1}^{*}\widetilde{A}_{1}^{*}\in S_{3}(\lambda)=S_{4}(\lambda),\qquad\lVert B_{1}^{*}\rVert_{F}^{2}=\lVert\widetilde{A}_{1}^{*}\rVert_{F}^{2}=\rho_{\lambda}, B 1 ∗ A 1 ∗ ∈ S 3 ( λ ) = S 4 ( λ ) , ∥ B 1 ∗ ∥ F 2 = ∥ A 1 ∗ ∥ F 2 = ρ λ ,
and this indicates B 1 ∗ , A ~ 1 ∗ ∈ S 2 ( λ ) B_{1}^{*},\widetilde{A}_{1}^{*}\in S_{2}(\lambda) B 1 ∗ , A 1 ∗ ∈ S 2 ( λ ) .
On the other hand, ∀ B 2 ∗ , A ~ 2 ∗ ∈ S 2 ( λ ) \forall B_{2}^{*},\widetilde{A}_{2}^{*}\in S_{2}(\lambda) ∀ B 2 ∗ , A 2 ∗ ∈ S 2 ( λ ) , they must satisfy
B 2 ∗ A ~ 2 ∗ ∈ S 3 ( λ ) = S 4 ( λ ) , ∥ B 2 ∗ ∥ F 2 = ∥ A ~ 2 ∗ ∥ F 2 = ρ λ , B_{2}^{*}\widetilde{A}_{2}^{*}\in S_{3}(\lambda)=S_{4}(\lambda),\qquad\lVert B_{2}^{*}\rVert_{F}^{2}=\lVert\widetilde{A}_{2}^{*}\rVert_{F}^{2}=\rho_{\lambda}, B 2 ∗ A 2 ∗ ∈ S 3 ( λ ) = S 4 ( λ ) , ∥ B 2 ∗ ∥ F 2 = ∥ A 2 ∗ ∥ F 2 = ρ λ ,
and this indicates ( B 2 ∗ , A ~ 2 ∗ ) ∈ S 1 ( λ ) (B_{2}^{*},\widetilde{A}_{2}^{*})\in S_{1}(\lambda) ( B 2 ∗ , A 2 ∗ ) ∈ S 1 ( λ ) . Thus, we finish the proof for S 1 ( λ ) = S 2 ( λ ) S_{1}(\lambda)=S_{2}(\lambda) S 1 ( λ ) = S 2 ( λ ) .
The last step is to show ρ λ \rho_{\lambda} ρ λ is a continuous and non-increasing function of λ \lambda λ that satisfies
lim ρ → 0 + ρ λ = ∞ , lim ρ → ∞ ρ λ = 0. \lim_{\rho\to0^{+}}\rho_{\lambda}=\infty,\qquad\lim_{\rho\to\infty}\rho_{\lambda}=0. ρ → 0 + lim ρ λ = ∞ , ρ → ∞ lim ρ λ = 0.
Continuity of ρ λ \rho_{\lambda} ρ λ . In this part, we will use Lemma G.4 to prove the solution to (Problem three) is continuous, which implies that ρ λ \rho_{\lambda} ρ λ is a continuous function w.r.t. λ \lambda λ . Notice Lemma G.4 is presented for maximization problem, one can replace the objective f f f to − f -f − f to extend it to minimization problem. Moreover, it is obvious that the objective function is continuous, convex, and the set of minimizers is single-valued for any fixed λ > 0 \lambda>0 λ > 0 . Thus, in order to apply Lemma G.4 to show the continuity of ρ λ \rho_{\lambda} ρ λ , we only need to check C ( λ ) C(\lambda) C ( λ ) , which is the range of Z Z Z , is a compact-valued correspondence. At first glance, this seems wrong since Z Z Z can take any values in R K × K n ˉ \mathbb{R}^{K\times K\bar{n}} R K × K n ˉ in (Problem three). However, we will show that one can constrain the domain of Z Z Z which leads to the same minimizer. Notice that when Z = 0 Z=0 Z = 0 , the objective in (Problem three) takes the value log K \log K log K . Thus, we can choose C ( λ ) = { Z | ∥ Z ∥ ∗ ≤ log K λ } C(\lambda)=\left\{Z\middle|\lVert Z\rVert_{*}\leq\frac{\log K}{\lambda}\right\} C ( λ ) = { Z ∥ Z ∥ ∗ ≤ λ l o g K } , which is a compact set. Therefore, for any fix λ \lambda λ , one can apply Lemma G.4 to show Z λ ∗ Z_{\lambda}^{*} Z λ ∗ is continuous w.r.t. λ \lambda λ , which implies ρ λ : = ∥ Z λ ∗ ∥ ∗ \rho_{\lambda}:=\lVert Z_{\lambda}^{*}\rVert_{*} ρ λ := ∥ Z λ ∗ ∥ ∗ is also continuous w.r.t. λ \lambda λ .
ρ λ \rho_{\lambda} ρ λ is a non-increasing function. For any 0 < λ 1 < λ 2 0<\lambda_{1}<\lambda_{2} 0 < λ 1 < λ 2 , let Z λ 1 , Z λ 2 Z_{\lambda_{1}},Z_{\lambda_{2}} Z λ 1 , Z λ 2 be the solutions for (Problem three) with the corresponding regularization penalty. Then, our first observation is
ϕ ( Z λ 2 ; λ 2 ) ≤ ϕ ( Z λ 1 ; λ 2 ) \phi(Z_{\lambda_{2}};\lambda_{2})\leq\phi(Z_{\lambda_{1}};\lambda_{2}) ϕ ( Z λ 2 ; λ 2 ) ≤ ϕ ( Z λ 1 ; λ 2 )
⟺ ϕ ( Z λ 2 ; λ 1 ) + ( λ 2 − λ 1 ) ∥ Z λ 2 ∥ ∗ ≤ ϕ ( Z λ 1 ; λ 1 ) + ( λ 2 − λ 1 ) ∥ Z λ 1 ∥ ∗ \Longleftrightarrow\phi(Z_{\lambda_{2}};\lambda_{1})+(\lambda_{2}-\lambda_{1})\lVert Z_{\lambda_{2}}\rVert_{*}\leq\phi(Z_{\lambda_{1}};\lambda_{1})+(\lambda_{2}-\lambda_{1})\lVert Z_{\lambda_{1}}\rVert_{*} ⟺ ϕ ( Z λ 2 ; λ 1 ) + ( λ 2 − λ 1 ) ∥ Z λ 2 ∥ ∗ ≤ ϕ ( Z λ 1 ; λ 1 ) + ( λ 2 − λ 1 ) ∥ Z λ 1 ∥ ∗
⟺ ϕ ( Z λ 2 ; λ 1 ) − ϕ ( Z λ 1 ; λ 1 ) ≤ ( λ 2 − λ 1 ) ( ∥ Z λ 1 ∥ ∗ − ∥ Z λ 2 ∥ ∗ ) \Longleftrightarrow\phi(Z_{\lambda_{2}};\lambda_{1})-\phi(Z_{\lambda_{1}};\lambda_{1})\leq(\lambda_{2}-\lambda_{1})(\lVert Z_{\lambda_{1}}\rVert_{*}-\lVert Z_{\lambda_{2}}\rVert_{*}) ⟺ ϕ ( Z λ 2 ; λ 1 ) − ϕ ( Z λ 1 ; λ 1 ) ≤ ( λ 2 − λ 1 ) (∥ Z λ 1 ∥ ∗ − ∥ Z λ 2 ∥ ∗ )
⟺ ϕ ( Z λ 2 ; λ 1 ) − ϕ ( Z λ 1 ; λ 1 ) ≤ ( λ 2 − λ 1 ) ( ρ ( λ 1 ) − ρ ( λ 2 ) ) . \Longleftrightarrow\phi(Z_{\lambda_{2}};\lambda_{1})-\phi(Z_{\lambda_{1}};\lambda_{1})\leq(\lambda_{2}-\lambda_{1})(\rho(\lambda_{1})-\rho(\lambda_{2})). ⟺ ϕ ( Z λ 2 ; λ 1 ) − ϕ ( Z λ 1 ; λ 1 ) ≤ ( λ 2 − λ 1 ) ( ρ ( λ 1 ) − ρ ( λ 2 )) .
Notice Z λ 1 ∈ arg min ϕ ( Z ; λ 1 ) Z_{\lambda_{1}}\in\arg\min\phi(Z;\lambda_{1}) Z λ 1 ∈ arg min ϕ ( Z ; λ 1 ) , thus, we have ϕ ( Z λ 2 ; λ 1 ) − ϕ ( Z λ 1 ; λ 1 ) ≥ 0 \phi(Z_{\lambda_{2}};\lambda_{1})-\phi(Z_{\lambda_{1}};\lambda_{1})\geq0 ϕ ( Z λ 2 ; λ 1 ) − ϕ ( Z λ 1 ; λ 1 ) ≥ 0 , and it leads to ρ ( λ 1 ) ≥ ρ ( λ 2 ) \rho(\lambda_{1})\geq\rho(\lambda_{2}) ρ ( λ 1 ) ≥ ρ ( λ 2 ) . Therefore, ρ λ \rho_{\lambda} ρ λ is non-increasing in λ \lambda λ . If ρ ( λ 1 ) = ρ ( λ 2 ) \rho(\lambda_{1})=\rho(\lambda_{2}) ρ ( λ 1 ) = ρ ( λ 2 ) , the preceding comparison and uniqueness imply Z λ 1 = Z λ 2 = : Z Z_{\lambda_{1}}=Z_{\lambda_{2}}=:Z Z λ 1 = Z λ 2 =: Z . Write ℓ ( Z ) : = ϕ ( Z ; 0 ) \ell(Z):=\phi(Z;0) ℓ ( Z ) := ϕ ( Z ; 0 ) . By first-order optimality, choose G i ∈ ∂ ∥ Z ∥ ∗ G_{i}\in\partial\lVert Z\rVert_{*} G i ∈ ∂ ∥ Z ∥ ∗ such that
∇ ℓ ( Z ) + λ i G i = 0 , i = 1 , 2. \nabla\ell(Z)+\lambda_{i}G_{i}=0,\qquad i=1,2. ∇ ℓ ( Z ) + λ i G i = 0 , i = 1 , 2.
Subtracting these optimality conditions gives
λ 1 G 1 = λ 2 G 2 . (117) \lambda_{1}G_{1}=\lambda_{2}G_{2}. \tag*{(117)} λ 1 G 1 = λ 2 G 2 . (117)
Every nuclear-norm subgradient satisfies ⟨ G i , Z ⟩ = ∥ Z ∥ ∗ \langle G_{i},Z\rangle=\lVert Z\rVert_{*} ⟨ G i , Z ⟩ = ∥ Z ∥ ∗ . Taking Frobenius inner products of (118) with Z Z Z therefore yields
( λ 1 − λ 2 ) ∥ Z ∥ ∗ = 0. (\lambda_{1}-\lambda_{2})\lVert Z\rVert_{*}=0. ( λ 1 − λ 2 ) ∥ Z ∥ ∗ = 0.
Since 0 < λ 1 < λ 2 0<\lambda_{1}<\lambda_{2} 0 < λ 1 < λ 2 , this forces ∥ Z ∥ ∗ = 0 \lVert Z\rVert_{*}=0 ∥ Z ∥ ∗ = 0 , hence Z = 0 Z=0 Z = 0 . Thus, when ρ ( λ 1 ) = ρ ( λ 2 ) \rho(\lambda_{1})=\rho(\lambda_{2}) ρ ( λ 1 ) = ρ ( λ 2 ) holds, one must have Z λ 1 = Z λ 2 = 0 Z_{\lambda_{1}}=Z_{\lambda_{2}}=0 Z λ 1 = Z λ 2 = 0 .
Asymptotic behaviour of ρ λ \rho_\lambda ρ λ . In this part, we aim to show
lim λ → 0 ρ λ = ∞ , lim λ → ∞ ρ λ = 0. \lim_{\lambda\to0}\rho_\lambda=\infty,\qquad\lim_{\lambda\to\infty}\rho_\lambda=0. λ → 0 lim ρ λ = ∞ , λ → ∞ lim ρ λ = 0.
We first use proof by contradiction to show that lim λ → ∞ ρ λ = 0 \lim_{\lambda\to\infty}\rho_\lambda=0 lim λ → ∞ ρ λ = 0 . Assume there exists M > 0 M>0 M > 0 such that ∀ λ > 0 \forall\lambda>0 ∀ λ > 0 , one has ρ λ ≥ M \rho_\lambda\ge M ρ λ ≥ M . Then, we take a series λ i \lambda_i λ i such that lim i → ∞ λ i = ∞ \lim_{i\to\infty}\lambda_i=\infty lim i → ∞ λ i = ∞ , and let S = { ρ ( λ i ) } i = 1 ∞ S=\{\rho(\lambda_i)\}_{i=1}^{\infty} S = { ρ ( λ i ) } i = 1 ∞ . Based on the assumption, we know all the elements in S S S has a lower bound M M M . Moreover, we know ∀ λ i \forall\lambda_i ∀ λ i , ρ ( λ i ) ≤ log K λ i \rho(\lambda_i)\le\frac{\log K}{\lambda_i} ρ ( λ i ) ≤ λ i l o g K . This is because ϕ ( 0 ; λ ) = log K \phi(0;\lambda)=\log K ϕ ( 0 ; λ ) = log K which is independent of the choices of λ i \lambda_i λ i . Thus, there exists M ′ M' M ′ such that ∀ ρ λ ∈ S \forall\rho_\lambda\in S ∀ ρ λ ∈ S , we have ρ λ ≤ M ′ \rho_\lambda\le M' ρ λ ≤ M ′ . Based on Bolzano–Weierstrass theorem, there exists a subsequence λ i k \lambda_{i_k} λ i k such that lim k → ∞ ρ ( λ i k ) = M ∗ \lim_{k\to\infty}\rho(\lambda_{i_k})=M^* lim k → ∞ ρ ( λ i k ) = M ∗ where M ≤ M ∗ ≤ M ′ M\le M^*\le M' M ≤ M ∗ ≤ M ′ . However, in this case, lim k → ∞ ϕ ( Z λ i k ∗ ; λ i k ) ≥ lim k → ∞ λ i k ρ ( λ i k ) = ∞ \lim_{k\to\infty}\phi(Z^*_{\lambda_{i_k}};\lambda_{i_k})\ge\lim_{k\to\infty}\lambda_{i_k}\rho(\lambda_{i_k})=\infty lim k → ∞ ϕ ( Z λ i k ∗ ; λ i k ) ≥ lim k → ∞ λ i k ρ ( λ i k ) = ∞ . This is in contradictory to the assumption that Z λ i k ∗ Z^*_{\lambda_{i_k}} Z λ i k ∗ minimizes ϕ ( Z ; λ i k ) \phi(Z;\lambda_{i_k}) ϕ ( Z ; λ i k ) and ϕ ( 0 ; λ ) = log K \phi(0;\lambda)=\log K ϕ ( 0 ; λ ) = log K . Thus, one must have lim λ → ∞ ρ λ = 0 \lim_{\lambda\to\infty}\rho_\lambda=0 lim λ → ∞ ρ λ = 0 .
Next, we first use proof by contradiction to show that lim λ → 0 ρ λ = ∞ \lim_{\lambda\to0}\rho_\lambda=\infty lim λ → 0 ρ λ = ∞ . Assume there exists N > 0 N>0 N > 0 such that ∀ λ > 0 \forall\lambda>0 ∀ λ > 0 , one has ρ λ ≤ N \rho_\lambda\le N ρ λ ≤ N . For every finite R > 0 R>0 R > 0 , ϕ ( Z ; 0 ) \phi(Z;0) ϕ ( Z ; 0 ) is continuous and strictly positive on the compact nuclear-norm ball { Z : ∥ Z ∥ ∗ ≤ R } \{Z:\|Z\|_*\le R\} { Z : ∥ Z ∥ ∗ ≤ R } . Consequently,
ϵ R : = min ∥ Z ∥ ∗ ≤ R ϕ ( Z ; 0 ) > 0. \epsilon_R:=\min_{\|Z\|_*\le R}\phi(Z;0)>0. ϵ R := ∥ Z ∥ ∗ ≤ R min ϕ ( Z ; 0 ) > 0.
Then, we pick a series λ k = 1 k 2 \lambda_k=\frac{1}{k^2} λ k = k 2 1 , R k = k R_k=k R k = k , and choose
Z ~ k = k ( 1 n T 0 n T 0 n T ⋯ 0 n T 0 n T 1 n T 0 n T ⋯ 0 n T 0 n T 0 n T ⋯ 0 n T 1 n T 0 ( K − K ˉ ) × K ˉ n ) . \widetilde{Z}_k=k\begin{pmatrix}
\mathbf{1}_n^\mathsf{T} & \mathbf{0}_n^\mathsf{T} & \mathbf{0}_n^\mathsf{T} & \cdots& \mathbf{0}_n^\mathsf{T}\\
\mathbf{0}_n^\mathsf{T} & \mathbf{1}_n^\mathsf{T} & \mathbf{0}_n^\mathsf{T} & \cdots& \mathbf{0}_n^\mathsf{T}\\
\mathbf{0}_n^\mathsf{T} & \mathbf{0}_n^\mathsf{T} & \cdots& \mathbf{0}_n^\mathsf{T} & \mathbf{1}_n^\mathsf{T}\\
& & \mathbf{0}_{(K-\bar K)\times\bar K n} & &
\end{pmatrix}. Z k = k 1 n T 0 n T 0 n T 0 n T 1 n T 0 n T 0 n T 0 n T ⋯ 0 ( K − K ˉ ) × K ˉ n ⋯ ⋯ 0 n T 0 n T 0 n T 1 n T .
Based on the optimality of Z λ k ∗ Z^*_{\lambda_k} Z λ k ∗ , one must have
ϕ ( Z λ k ∗ ; λ k ) ≤ ϕ ( Z ~ k ; λ k ) = log ( 1 + ( K − 1 ) exp ( − k ) ) + K ˉ n k . \phi(Z^*_{\lambda_k};\lambda_k)\le\phi(\widetilde{Z}_k;\lambda_k)=\log\left(1+(K-1)\exp(-k)\right)+\frac{\bar K\sqrt{n}}{k}. ϕ ( Z λ k ∗ ; λ k ) ≤ ϕ ( Z k ; λ k ) = log ( 1 + ( K − 1 ) exp ( − k ) ) + k K ˉ n .
On the other hand,
ϕ ( Z λ k ∗ ; λ k ) ≥ ϕ ( Z λ k ∗ ; 0 ) ≥ ϵ N > 0. \phi(Z^*_{\lambda_k};\lambda_k)\ge\phi(Z^*_{\lambda_k};0)\ge\epsilon_N>0. ϕ ( Z λ k ∗ ; λ k ) ≥ ϕ ( Z λ k ∗ ; 0 ) ≥ ϵ N > 0.
Combine these two inequalities together, one has
ϵ N ≤ log ( 1 + ( K − 1 ) exp ( − k ) ) + K ˉ n k . \epsilon_N\le\log\left(1+(K-1)\exp(-k)\right)+\frac{\bar K\sqrt{n}}{k}. ϵ N ≤ log ( 1 + ( K − 1 ) exp ( − k ) ) + k K ˉ n .
However, one can choose k k k sufficiently large that the above inequality breaks. Thus, there cannot exist a N > 0 N>0 N > 0 such that ρ λ ≤ N \rho_\lambda\le N ρ λ ≤ N holds ∀ λ > 0 \forall\lambda>0 ∀ λ > 0 . Therefore, one must have lim λ → 0 ρ λ = ∞ \lim_{\lambda\to0}\rho_\lambda=\infty lim λ → 0 ρ λ = ∞ . □ \square □
Proof of Lemma G.6
Proof. Our starting point is the following lemma which is developed in Hong and Ling [ [18 ] .
Lemma G.7 (Cross entropy loss is locally strongly convex in restricted direction). Define
ϕ ( Z ) = 1 K ˉ n ∑ c = 1 K ˉ ∑ j = 1 n ℓ C E ( y c , Z e ( c − 1 ) n + j ) , \phi(Z)=\frac{1}{\bar K n}\sum_{c=1}^{\bar K}\sum_{j=1}^{n}\ell_{\mathrm{CE}}(y_c,Z e_{(c-1)n+j}), ϕ ( Z ) = K ˉ n 1 c = 1 ∑ K ˉ j = 1 ∑ n ℓ CE ( y c , Z e ( c − 1 ) n + j ) , where y c y_c y c is a one-hot vector with c c c -th entry equal to one. For any R > 0 R>0 R > 0 , there exists μ ( R ) > 0 \mu(R)>0 μ ( R ) > 0 such that on the compact set { Z ∈ R K × K n : ∥ Z ∥ F ≤ R } \{Z\in\mathbb{R}^{K\times K n}: \|Z\|_F\le R\} { Z ∈ R K × K n : ∥ Z ∥ F ≤ R } , ϕ ( Z ) \phi(Z) ϕ ( Z ) is μ ( R ) \mu(R) μ ( R ) -strongly convex in any direction Δ Z ∈ { Δ Z : 1 K T Δ Z = 0 } \Delta_Z\in\{\Delta_Z:\mathbf{1}_K^\mathsf{T}\Delta_Z=0\} Δ Z ∈ { Δ Z : 1 K T Δ Z = 0 } .
This bounded-set statement follows from the pointwise restricted Hessian bound in the proof of Lemma 5.1 in Hong and Ling [ [18 ] .
Fix λ > 0 \lambda>0 λ > 0 and write ϕ ( Z ; λ ) : = ϕ ( Z ) + λ ∥ Z ∥ ∗ \phi(Z;\lambda):=\phi(Z)+\lambda\|Z\|_* ϕ ( Z ; λ ) := ϕ ( Z ) + λ ∥ Z ∥ ∗ . Since cross entropy is nonnegative, ϕ ( Z ; λ ) ≥ λ ∥ Z ∥ ∗ ≥ λ ∥ Z ∥ F \phi(Z;\lambda)\ge\lambda\|Z\|_*\ge\lambda\|Z\|_F ϕ ( Z ; λ ) ≥ λ ∥ Z ∥ ∗ ≥ λ ∥ Z ∥ F .
Hence ϕ ( ⋅ ; λ ) \phi(\cdot;\lambda) ϕ ( ⋅ ; λ ) is continuous and coercive, so it attains a global minimum. Every global minimizer Z ∗ Z^* Z ∗ satisfies
λ ∥ Z ∗ ∥ F ≤ ϕ ( Z ∗ ; λ ) ≤ ϕ ( 0 ; λ ) = log K , \lambda\lVert Z^*\rVert_F \le\phi(Z^*;\lambda) \le\phi(0;\lambda)=\log K, λ ∥ Z ∗ ∥ F ≤ ϕ ( Z ∗ ; λ ) ≤ ϕ ( 0 ; λ ) = log K ,
and therefore ∥ Z ∗ ∥ F ≤ log K / λ \lVert Z^*\rVert_F \le\log K/\lambda ∥ Z ∗ ∥ F ≤ log K / λ .
We next show that every global minimizer is centered. Let Z ∗ Z^* Z ∗ be any global minimizer and decompose it as
Z ∗ = 1 K 1 K 1 K T Z ∗ + ( I K − 1 K 1 K 1 K T ) Z ∗ . Z^*=\frac{1}{K}\mathbf{1}_K\mathbf{1}_K^{\mathsf T}Z^*+\left(I_K-\frac{1}{K}\mathbf{1}_K\mathbf{1}_K^{\mathsf T}\right)Z^*. Z ∗ = K 1 1 K 1 K T Z ∗ + ( I K − K 1 1 K 1 K T ) Z ∗ .
For simplicity, let P = I K − 1 K 1 K 1 K T P=I_K-\frac{1}{K}\mathbf{1}_K\mathbf{1}_K^{\mathsf T} P = I K − K 1 1 K 1 K T be the projection matrix onto the space orthogonal to 1 K \mathbf{1}_K 1 K . The removed component 1 K 1 K 1 K T Z ∗ \frac{1}{K}\mathbf{1}_K\mathbf{1}_K^{\mathsf T}Z^* K 1 1 K 1 K T Z ∗ has constant entries within each column. Since cross entropy is invariant under such logit shifts, ϕ ( Z ∗ ) = ϕ ( P Z ∗ ) \phi(Z^*)=\phi(PZ^*) ϕ ( Z ∗ ) = ϕ ( P Z ∗ ) . By the contraction property in Lemma G.3 applied to the orthogonal projection P P P ,
∥ P Z ∗ ∥ ∗ ≤ ∥ Z ∗ ∥ ∗ , \lVert PZ^*\rVert_* \le\lVert Z^*\rVert_*, ∥ P Z ∗ ∥ ∗ ≤ ∥ Z ∗ ∥ ∗ ,
with equality if and only if P Z ∗ = Z ∗ PZ^*=Z^* P Z ∗ = Z ∗ . Combined with ϕ ( Z ∗ ) = ϕ ( P Z ∗ ) \phi(Z^*)=\phi(PZ^*) ϕ ( Z ∗ ) = ϕ ( P Z ∗ ) , we have ϕ ( P Z ∗ ; λ ) ≤ ϕ ( Z ∗ ; λ ) \phi(PZ^*;\lambda)\le\phi(Z^*;\lambda) ϕ ( P Z ∗ ; λ ) ≤ ϕ ( Z ∗ ; λ ) , and optimality of Z ∗ Z^* Z ∗ forces equality, hence P Z ∗ = Z ∗ PZ^*=Z^* P Z ∗ = Z ∗ and
1 K T Z ∗ = 0. \mathbf{1}_K^{\mathsf T}Z^*=0. 1 K T Z ∗ = 0.
Finally, choose R > log K / λ R>\log K/\lambda R > log K / λ . All global minimizers belong to the convex set
B R : = { Z ∈ R K × K ˉ n : 1 K T Z = 0 , ∥ Z ∥ F ≤ R } . B_R:=\{Z\in\mathbb{R}^{K\times\bar{K}n}:\mathbf{1}_K^{\mathsf T}Z=0,\ \lVert Z\rVert_F\le R\}. B R := { Z ∈ R K × K ˉ n : 1 K T Z = 0 , ∥ Z ∥ F ≤ R } .
By Lemma G.7 , ϕ ( Z ) \phi(Z) ϕ ( Z ) is μ ( R ) \mu(R) μ ( R ) -strongly convex on B R B_R B R . Since the nuclear norm is convex, the regularized objective ϕ ( Z ; λ ) \phi(Z;\lambda) ϕ ( Z ; λ ) is also μ ( R ) \mu(R) μ ( R ) -strongly convex on this set. Thus, for any two global minimizers Z 1 , Z 2 Z_1,Z_2 Z 1 , Z 2 ,
ϕ ( Z 1 + Z 2 2 ; λ ) ≤ ϕ ( Z 1 ; λ ) + ϕ ( Z 2 ; λ ) 2 − μ ( R ) 8 ∥ Z 1 − Z 2 ∥ F 2 . \phi\left(\frac{Z_1+Z_2}{2};\lambda\right)\le\frac{\phi(Z_1;\lambda)+\phi(Z_2;\lambda)}{2}-\frac{\mu(R)}{8}\lVert Z_1-Z_2\rVert_F^2. ϕ ( 2 Z 1 + Z 2 ; λ ) ≤ 2 ϕ ( Z 1 ; λ ) + ϕ ( Z 2 ; λ ) − 8 μ ( R ) ∥ Z 1 − Z 2 ∥ F 2 .
If Z 1 ≠ Z 2 Z_1\ne Z_2 Z 1 = Z 2 , their midpoint would have a strictly smaller objective value, contradicting global optimality. Therefore the global minimizer is unique.
Experiment
In this section, we present the detailed setup of the experiments in §4.
Code availability. Code is available at https://github.com/simonxu97/MaLoRA.
Hardware. All experiments are run on a single NVIDIA RTXA5000 GPU (24GB) per (benchmark, seed), allocated through an academic SLURM cluster. The backbones—ViT-Base/16 pre-trained on ImageNet-21k and XLM-RoBERTa Base—are loaded directly from public checkpoints; we do not pretrain any model from scratch. Per-domain LoRA finetuning runs for 20 fixed epochs with AdamW and takes 5–30 minutes per domain depending on the dataset, totalling roughly 0.5–3GPU-hours per (benchmark, seed) across all per-domain adapters and the pretrain-task adapter. Computational overhead and merge-selection runtime are reported in Appendix § 14 .§ 2.2 , using a dedicated matched-budget timing protocol.
Details for experiments in §H.1.2 and §4.1
Definition of superclasses of CIFAR-100. CIFAR-100 groups its 100 fine categories into 20 coarse superclasses as follows
Construction of pre-training and fine-tuning tasks. For every superclass S S S , let [ c 1 S , … , c 5 S ] [c^S_1,\ldots,c^S_5] [ c 1 S , … , c 5 S ] be a fixed random permutation of its five fine labels. We build three disjoint labelled sets
D p r e = ⋃ S { c 1 S , c 2 S , c 3 S } , D 1 = ⋃ S { c 4 S } , D 2 = ⋃ S { c 5 S } . \mathcal{D}_{\mathrm{pre}}=\bigcup_S\{c^S_1,c^S_2,c^S_3\},\qquad\mathcal{D}_1=\bigcup_S\{c^S_4\},\qquad\mathcal{D}_2=\bigcup_S\{c^S_5\}. D pre = S ⋃ { c 1 S , c 2 S , c 3 S } , D 1 = S ⋃ { c 4 S } , D 2 = S ⋃ { c 5 S } .
Hence D p r e \mathcal{D}_{\mathrm{pre}} D pre contains 60 fine classes (three per super-class) while each fine-tuning task D i \mathcal{D}_i D i contains exactly one new class per superclass, preserving maximal diversity yet zero overlap with D p r e \mathcal{D}_{\mathrm{pre}} D pre .
Table 2. CIFAR-100 coarse superclasses and their five fine labels.
Frozen feature extractors. We evaluate four widely used backbones, noting their different pre-training dataset:
ResNet-50 [17 ] (torchvision, supervised ImageNet-1K ).
ViT-Base/16 [11 ] (timm: supervised pre-training on ImageNet-21K , then fine-tuned on ImageNet-1K).
ConvNeXt-Tiny [32 ] pre-trained on ImageNet-12K, then fine-tuned on ImageNet-1K.
CLIP ViT-B/32 [39 ] (OpenAI’s contrastive pre-training on web-scale image–text pairs; no ImageNet supervision).
All models are kept frozen; we extract the CLS token (ViT/CLIP) or the global-average-pooled penultimate tensor (CNNs) at 224 × 224 224 \times224 224 × 224 resolution for every CIFAR-100 image.
Pre-training stage. A linear classifier W p r e ∈ R 20 × d W_{\mathrm{pre}} \in\mathbb{R}^{20 \times d} W pre ∈ R 20 × d (d d d = embedding dimension) is trained from scratch on D p r e \mathcal{D}_{\mathrm{pre}} D pre for 600 epochs with Adam (η = 0.2 \eta= 0.2 η = 0.2 ). The 20 rows correspond to the super-classes, not the 60 fine labels; this matches our theoretical model where each task’s labels share a common output.
Empirical Motivation for the Task Geometry Assumption
In this section, we conduct a comprehensive feature correlation analysis on CIFAR-100. This analysis quantifies both intra-class and inter-class correlations in the feature space to provide empirical motivation for the near-orthogonal task geometry in Assumption 2.1.
Let F θ ( x ) \mathcal{F}_{\theta}(x) F θ ( x ) denote the feature extraction function of a neural network with parameters θ \theta θ , where x ∈ R 3 × 32 × 32 x \in\mathbb{R}^{3 \times32 \times32} x ∈ R 3 × 32 × 32 represents an input image from CIFAR-100. For each architecture (ResNet-50, ViT-Base/16, CLIP ViT-B/32, and ConvNeXt-Tiny), we extract features from the penultimate layer, obtaining feature vectors f i = F θ ( x i ) ∈ R d f_i = \mathcal{F}_{\theta}(x_i) \in\mathbb{R}^{d} f i = F θ ( x i ) ∈ R d for each training sample x i x_i x i .
We organize the extracted features by class, creating sets S c = { f i : y i = c } \mathcal{S}_c = \{f_i : y_i = c\} S c = { f i : y i = c } for each class c ∈ { 0 , 1 , … , 99 } c \in\{0, 1, \ldots, 99\} c ∈ { 0 , 1 , … , 99 } , where y i y_i y i is the ground truth label of sample x i x_i x i .
Cosine-Similarity Computation. For any two feature vectors f i f_i f i , f j ∈ R d f_j \in\mathbb{R}^d f j ∈ R d , we compute the cosine similarity:
s ( f i , f j ) = ⟨ f i , f j ⟩ ∥ f i ∥ 2 ∥ f j ∥ 2 = ∑ k = 1 d f i ( k ) f j ( k ) ∑ k = 1 d ( f i ( k ) ) 2 ∑ k = 1 d ( f j ( k ) ) 2 , (118) s(f_i, f_j) = \frac{\langle f_i, f_j \rangle}{\lVert f_i \rVert_2 \lVert f_j \rVert_2} = \frac{\sum_{k=1}^{d} f_i^{(k)} f_j^{(k)}}{\sqrt{\sum_{k=1}^{d} \left(f_i^{(k)}\right)^2} \sqrt{\sum_{k=1}^{d} \left(f_j^{(k)}\right)^2}}, \tag*{(118)} s ( f i , f j ) = ∥ f i ∥ 2 ∥ f j ∥ 2 ⟨ f i , f j ⟩ = ∑ k = 1 d ( f i ( k ) ) 2 ∑ k = 1 d ( f j ( k ) ) 2 ∑ k = 1 d f i ( k ) f j ( k ) , (118)
where f i ( k ) f_i^{(k)} f i ( k ) denotes the k k k -th coordinate of f i f_i f i .
Intra-class Similarity. For each class c c c , we compute the average intra-class cosine similarity:
s i n t r a ( c ) = 1 ∣ S c ∣ ( ∣ S c ∣ − 1 ) / 2 ∑ f i , f j ∈ S c , i < j s ( f i , f j ) . (119) s_{\mathrm{intra}}(c) = \frac{1}{\lvert S_c \rvert(\lvert S_c \rvert- 1)/2} \sum_{\substack{f_i,f_j \in S_c,\\ i<j}} s(f_i, f_j). \tag*{(119)} s intra ( c ) = ∣ S c ∣ (∣ S c ∣ − 1 ) /2 1 f i , f j ∈ S c , i < j ∑ s ( f i , f j ) . (119)
This measures how aligned features are within the same class, indicating the consistency of learned representations for semantically similar samples.
Inter-class Similarity. For any two distinct classes c 1 ≠ c 2 c_1 \ne c_2 c 1 = c 2 , we compute the average inter-class cosine similarity:
s i n t e r ( c 1 , c 2 ) = 1 ∣ S c 1 ∣ ⋅ ∣ S c 2 ∣ ∑ f i ∈ S c 1 ∑ f j ∈ S c 2 s ( f i , f j ) . (120) s_{\mathrm{inter}}(c_1, c_2) = \frac{1}{\lvert S_{c_1} \rvert\cdot\lvert S_{c_2} \rvert} \sum_{f_i \in S_{c_1}} \sum_{f_j \in S_{c_2}} s(f_i, f_j). \tag*{(120)} s inter ( c 1 , c 2 ) = ∣ S c 1 ∣ ⋅ ∣ S c 2 ∣ 1 f i ∈ S c 1 ∑ f j ∈ S c 2 ∑ s ( f i , f j ) . (120)
This quantifies the alignment between features from different classes, with values closer to zero indicating more orthogonal (i.e., better separated) classes.
Similarity Matrix Construction. We construct a 100 × 100 100 \times100 100 × 100 similarity matrix S S S where:
S i j = { s i n t r a ( i ) if i = j s i n t e r ( i , j ) if i ≠ j . (121) S_{ij} =
\begin{cases}
s_{\mathrm{intra}}(i) & \text{if } i=j \\
s_{\mathrm{inter}}(i,j) & \text{if } i\ne j.
\end{cases} \tag*{(121)} S ij = { s intra ( i ) s inter ( i , j ) if i = j if i = j . (121)
Implementation Details. To ensure computational efficiency while maintaining statistical reliability, we subsample feature pairs for similarity computation. Specifically, for each class we randomly sample 100 points (10,000 in total). We first report the distribution of pairwise cosine similarities across all sampled points in Figure 2 . To further distinguish intra-class and inter-class relationships, we visualize the corresponding similarities as a heatmap in Figure 3 . For intra-class similarities, we randomly sample up to 1,000 distinct sample pairs per class (drawn without replacement to exclude trivial self-pairs whose cosine would be 1); for inter-class similarities, we sample up to 1,000 pairs for each class pair. This subsampling strategy yields robust similarity estimates while keeping the computation tractable.
Figure 2. Distributions of pairwise feature correlation across four pre-trained models. The red dashed line represent the mean value of correlations, and the dashed lines represent mean plus and minus one std.
Figure 3. Feature correlation analysis across different pretrained models.
Figure 2 highlights two key observations: (1) most pairwise cosine similarities are concentrated around zero; and (2) the mean similarity ± \pm ± one standard deviation remains below 0.15 for every architecture. These descriptive statistics provide qualitative motivation for studying a near-orthogonal reference geometry.
Figure 3 highlights three observations: (1) higher diagonal values indicate stronger within-class alignment; (2) near-zero off-diagonal values indicate weak average inter-class alignment; and (3) Table 3 quantifies this contrast, with inter-class means close to zero and intra-class means around 0.23–0.27. The weak average inter-class and within-class alignment provides qualitative motivation for studying a near-orthogonal reference geometry. Extending the analysis to richer within-class structure and stronger correlations remains an important direction for future work.
Table 3. Mean and standard deviation of intra-class and inter-class feature cosine similarities. Values closer to zero indicate greater orthogonality between features.
Effect of regularization
In this section, we numerically verify Theorem 3.2 .
We first pre-train a linear classifier on the pretraining task, and then fine-tune it using LoRA across 50 logarithmically spaced regularization strengths, with λ ∈ [ 10 − 5 , 10 − 1 ] \lambda\in[10^{-5}, 10^{-1}] λ ∈ [ 1 0 − 5 , 1 0 − 1 ] . Figure 4 shows that LoRA’s performance exhibits a non-monotonic relationship with λ \lambda λ , peaking at a moderate value. For ViT and ResNet, the theoretically predicted optimal λ \lambda λ is within a factor of 1.8 of the best empirical choice for every K ˉ \bar{K} K ˉ , and for ConvNeXt within a factor of 1.2 when K ˉ ∈ { 10 , 20 } \bar{K} \in\{10,20\} K ˉ ∈ { 10 , 20 } ; for CLIP, the empirical optimum is 1.8–4.4 times larger than predicted. Detailed comparisons are reported in Appendix H.
Figure 4. Accuracy of the fine-tuned model across varying regularization levels for four different pre-trained models. Each line represents a fine-tuning task, indicated by color. Stars highlight the best-performing regularization value for each task. Vertical dashed lines indicate the theoretically predicted optimal regularization parameter for each setting. Alignment between the star and the dashed line of the same color reflects how well our theory predicts the empirically optimal regularization level.
For each fine-tuning task, we freeze the pre-trained weights W p r e W_{\mathrm{pre}} W pre and attach a rank-r r r LoRA block, where r r r is set to the number of classes in the fine-tuning task. This satisfies Assumption 2.3 . We evaluate performance across three fine-tuning tasks with varying class counts: K ˉ = 5 , 10 , 20 \bar{K}=5,10,20 K ˉ = 5 , 10 , 20 . Each task is constructed by selecting the top K ˉ \bar{K} K ˉ superclasses from D 2 D_2 D 2 .
We fine-tune the model on each task using SGD for 3000 steps with a learning rate of 0.5, sweeping over regularization values λ ∈ [ 10 − 5 , 10 − 1 ] \lambda\in[10^{-5}, 10^{-1}] λ ∈ [ 1 0 − 5 , 1 0 − 1 ] with 50 logarithmically spaced regularization strengths. For each K ˉ \bar{K} K ˉ and each frozen feature extractor, we record both the empirically optimal regularization parameter and the theoretically predicted one from Theorem 3.2 . The detailed figures are given in Table 4 .
Table 4. Empirical (Emp) and theoretical (Thm) optimal regularization parameters λ \lambda λ for different pre-trained models and fine-tuning task sizes (K ˉ \bar{K} K ˉ ).
Details for experiments in §4.1
For the experiments in §4.1, we fine-tune D 1 , D 2 D_1,D_2 D 1 , D 2 using different LoRA adapters ( B i , A i ) , i = 1 , 2 (B_i,A_i), i=1,2 ( B i , A i ) , i = 1 , 2 with LoRA rank 20. We fine-tune the LoRA adapters using Adam ( η = 0.1 ) (\eta=0.1) ( η = 0.1 ) with regularization parameter λ = 5 × 10 − 7 \lambda=5\times10^{-7} λ = 5 × 1 0 − 7 . Let ( B i ∗ , A i ∗ ) , i = 1 , 2 (B_i^*,A_i^*), i=1,2 ( B i ∗ , A i ∗ ) , i = 1 , 2 be the optimal LoRA adapters we achieve at the end of fine-tuning, we merge them together as follows
W L o R A ( α 1 , α 2 ) = W p r e + α 1 B 1 ∗ A 1 ∗ + α 2 B 2 ∗ A 2 ∗ W_{\mathrm{LoRA}}(\alpha_1,\alpha_2) = W_{\mathrm{pre}} + \alpha_1 B_1^* A_1^* + \alpha_2 B_2^* A_2^* W LoRA ( α 1 , α 2 ) = W pre + α 1 B 1 ∗ A 1 ∗ + α 2 B 2 ∗ A 2 ∗
To seek for the optimal mixing coefficients ( α 1 , α 2 ) (\alpha_1,\alpha_2) ( α 1 , α 2 ) , we run grid search over a 50 × 50 50 \times50 50 × 50 lattice on ( 0 , 1 ) 2 (0,1)^2 ( 0 , 1 ) 2 , and compare it with the optimal theoretical mixing coefficients presented in Theorem 3.3 . We test different number of classes in the fine-tuning datasets, i.e., K ˉ = 5 , 10 , 20 \bar{K} = 5, 10, 20 K ˉ = 5 , 10 , 20 , and reports their results here.
Figure 5. Merged-model accuracy across architectures and different number of classes in the fine-tuning tasks. Each panel shows the accuracy of the merged model evaluated on the combined dataset, across a 50 × 50 50 \times50 50 × 50 grid of mixing coefficients ( α 1 , α 2 ) ∈ ( 0 , 1 ) 2 (\alpha_1,\alpha_2) \in(0,1)^2 ( α 1 , α 2 ) ∈ ( 0 , 1 ) 2 . For each architecture, the red star indicates the theoretically predicted optimal coefficients from Theorem 3.3 , while the red triangle marks the empirically optimal coefficients.
Details for experiments in §4.2
In §4.2, the datasets tested are all multi-domain benchmarks (See Table 7 for details).
Table 7: Benchmarks used in the main paper. “Domains” lists every domain in the dataset; the pretrain column marks the one folded into base_with_pretrain. “# merged” is the number of LoRA adapters that participate in the dense task-arithmetic merge (i.e. all non-pretrain adapters).
Table 7.
Table 5: Average normalized accuracy (%) following Ilharco et al. [ [22 ] : each task’s merged-model accuracy is divided by the per-domain finetuned model’s accuracy on that task, then averaged across domains within each benchmark. Mean ± std over 10 seeds. Bold = best per column. Values >100 indicate the merged model outperforms the per-task finetuned model on that benchmark.
Table 5.
Table 6: Average relative accuracy drop (%), the complement of normalized accuracy (1 − merged / oracle 1 - \text{merged}/\text{oracle} 1 − merged / oracle , averaged across domains within each benchmark and across benchmarks). Mean ± std over 10 seeds. Bold = smallest per column (least forgetting). Negative values indicate the merged model outperforms the per-task finetuned model on average. Avg is the equally weighted mean across benchmarks; its std is ∑ b σ b 2 / 6 \sqrt{\sum_b \sigma_b^2/6} ∑ b σ b 2 /6 , where σ b \sigma_b σ b is the per-benchmark std, assuming independent benchmarks.
Table 6.
Validation/test splits. For every domain, we hold out a stratified 10% validation split and a stratified 10% test split from the per-domain training set, with the remaining 80% used for fine-tuning. The split indices are deterministic in the configured random seed and are reused across all methods so that Soup, Task Arithmetic, and MaLoRA are evaluated on the same per-domain val/test sets within a seed. Validation accuracy averaged over all merged domains is the criterion for selecting the merge coefficient λ ∗ \lambda^* λ ∗ .
LoRA configuration. We use LoRA Hu et al. [ [20 ] with the following layout, applied identically to every adapter (per-domain, per-seed):
Layers transformed: the last four transformer layers, indexed { 8 , 9 , 10 , 11 } \{8, 9, 10, 11\} { 8 , 9 , 10 , 11 } (out of 12). This was identified as the best layer placement in our preliminary layer-sweep ablation.
Scaling factor: α L o R A = 16 \alpha_{\mathrm{LoRA}} = 16 α LoRA = 16 for all adapters.
LoRA dropout: 0.05.
Classifier head: a fresh classification head is initialized for each adapter and trained jointly with the LoRA parameters. At merge time, each merged adapter contributes both its classifier head and its LoRA delta to the weighted sum, so that the merged model has a coherent classifier without an external head-aggregation step.
Training and per-dataset hyperparameters. All adapters are trained with AdamW for 20 epochs (no early stopping; lora_patience = 0). Before enabling the LoRA branch, we warm up the classifier head for 5 epochs at learning rate 3 × 10 − 4 3 \times10^{-4} 3 × 1 0 − 4 (head_epochs = 5, head_lr = 3e-4); each domain trains its own head (train_shared_head = false). We use default PyTorch gradient clipping. The four hyperparameters that vary across datasets, i.e., LoRA rank r r r , learning rate, batch size, and weight decay, are summarized in Table 8 .
Table 8. Per-dataset training hyperparameters. All adapters use AdamW with LoRA α = 16 \alpha= 16 α = 16 , dropout 0.05, query and value projections of the final four transformer layers, and 20 training epochs. The pretrain domain is the one folded into the base_with_pretrain model, with the remaining domains fine-tuned on top.
Baseline methods. We compare the following methods for merging the LoRA adapters,
Uniform Soup Wortsman et al. [ [53 ] : α i = 1 / N \alpha_i = 1/N α i = 1/ N (equal weight across all merged adapters). Equivalent to Task Arithmetic at λ = 1 / N \lambda= 1/N λ = 1/ N .
Task Arithmetic (TA) Ilharco et al. [ [22 ] : α i = λ \alpha_i = \lambda α i = λ for every i i i , with λ \lambda λ tuned on validation.
DARE-Linear Yu et al. [ [56 ] : each task vector Δ W i \Delta W_i Δ W i is randomly sparsified by a Bernoulli mask with keep rate d d d and rescaled by 1 / d 1/d 1/ d , then linearly averaged with global scale λ \lambda λ , i.e. Δ W m e r g e d = ( λ / N ) ∑ i Δ W ~ i \Delta W_{\mathrm{merged}} = (\lambda/N) \sum_i \Delta\widetilde{W}_i Δ W merged = ( λ / N ) ∑ i Δ W i where Δ W ~ i \Delta\widetilde{W}_i Δ W i is the dropped-and-rescaled task vector. Both d d d and λ \lambda λ are tuned on validation.
DARE-TIES Yu et al. [ [56 ] : applies the DARE drop-and-rescale step above to each task vector, then aggregates via the TIES sign-election and disjoint-merge procedure with global scale λ \lambda λ . Both d d d and λ \lambda λ are tuned on validation.
TIES Yadav et al. [ [55 ] : each task vector is first trimmed to its top-d d d fraction of parameters by magnitude; per parameter, the dominant sign is elected across adapters by total signed magnitude; the merged delta averages only the sign-aligned values and is scaled by λ \lambda λ . Both d d d and λ \lambda λ are tuned on validation.
All baselines that involve density and λ \lambda λ are tuned over d ∈ { 0.2 , 0.3 , … , 0.9 } d \in\{0.2, 0.3, \ldots, 0.9\} d ∈ { 0.2 , 0.3 , … , 0.9 } and λ ∈ { 0.1 , 0.2 , … , 1.0 } \lambda\in\{0.1, 0.2, \ldots, 1.0\} λ ∈ { 0.1 , 0.2 , … , 1.0 } (80 configurations); Task Arithmetic uses the same λ \lambda λ grid (10 configurations).
Feature correlations across domains
We extend the CIFAR-100 analysis in Appendix § 14 .1.1 to the six multi-domain benchmarks in §4.2. For this analysis, we extract features using the stock pre-trained checkpoints google/vit-base-patch16-224-in21k for PACS, VLCS, and Terra Incognita, and xlm-roberta-base for MNLI, Amazon Reviews, and CivilComments. The backbones are kept fixed and used for inference only. Each domain is treated as a task, and the domain counts include both the pre-training domain and all fine-tuning domains.
In Table 9 , cross-domain mean correlations range from − 0.060 -0.060 − 0.060 to 0.060 0.060 0.060 , providing qualitative motivation for weak average cross-task alignment. Within domains, small mean correlations for different-class pairs on VLCS (0.008) and for both same-class and different-class pairs on Amazon Reviews and CivilComments (0.039–0.073) also motivate a near-orthogonal reference geometry. Stronger same-class alignment on PACS (0.376) and Terra Incognita (0.312) highlights structured departures from this geometry, showing that agreement with the reference geometry varies across benchmarks and pair categories.
Table 9. Feature correlations within and across classes and domains. Entries report mean correlations, with ± \pm ± denoting spread across class–domain cell pairs in each group. A cell is a class–domain combination.
Computational overhead
We measure end-to-end merge-selection runtime on all six benchmarks over ten random seeds, using one NVIDIA RTX A5000 GPU and a fixed budget of four CPU threads. For the six methods compared in Table 10 , we use a matched budget of 49 configurations per method, separately from the accuracy-selection grids described above. The timings measure merging and coefficient selection from trained adapters; adapter-training costs are reported separately in the hardware description.
Table 10. Mean end-to-end merge-selection runtime (seconds) over ten seeds per benchmark under the matched-budget timing protocol. Avg. is the arithmetic mean across the six benchmarks; the final column divides each method’s Avg. by MaLoRA’s.
MaLoRA estimates margins with one forward pass over each domain’s validation split, costing 1.05–1.48 times a single validation pass. The resulting margins are computed once and reused throughout the global-scale search. For fixed adapters and validation sets, this additional margin-computation cost is independent of the number of candidate scales.
Table 10 reports the mean runtime for each benchmark and the arithmetic mean across the six benchmarks. MaLoRA’s aggregate mean runtime is 972.6 s, compared with 950.3 s for TA, an increase of 22.3 s or approximately 2.35%. Its aggregate mean runtime is lower than those of DARE-Linear, DARE-TIES, and TIES (980.2–991.0 s).
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The claim made in the abstract can be found in Theorem 3.1, Theorem 3.2 , Theorem 3.3 , and Algorithm 1.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we discuss the limitations of the work in §5.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes] Justification: we list all the assumptions in §2.1, and provide proofs in Appendix D–Appendix F.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: we list all the model architectures, datasets, and training setups in Appendix § 14 .
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: all the experiments conducted in this paper use open source datasets, and we provide citations to them.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: in Appendix H, we provide details on the experiments.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: in Appendix H, we report the mean normalized accuracy and drop with std computed based on 10 repetitions.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: we report the usage of computational resources in Appendix H.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: we have reviewed the NeurIPS Code of Ethics and followed it.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [N/A]
Justification: there is no societal impact of the work performed.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: the paper poses no such risks.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: in §4, we cite the original paper that produces the datasets used in the experiment.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification: the paper does not release new assets.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: we document how we use LLM in Appendix A.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.