betaFeedback

WorksComputer Science

A Margin Perspective on LoRA: Robustness to Catastrophic Forgetting and Adapter Merging (MaLoRA)

Abstract

Low-rank adaptation (LoRA) is the de facto method for parameter-efficient fine-tuning of large neural networks, achieving strong task adaptation with minimal trainable parameters and low optimization cost. Beyond this efficiency, practitioners have observed two striking properties: robustness to catastrophic forgetting and the ability to merge independently trained adapters into a single model that performs competitively across multiple tasks. These properties are central to continual and multi-task learning, yet remain poorly understood. In this work, we provide a theoretical explanation through a unified margin-based perspective. We analyze LoRA in multiclass linear classification under a near-orthogonal task regime with ℓ2\ell_2-regularization, and characterize the optimal LoRA adapter across regularization regimes. Our analysis shows that, in an intermediate regime of regularization parameters, the optimal adapter aligns with the max-margin solution on the fine-tuning data. Building on this characterization, we derive two key consequences. First, we obtain closed-form expressions for the margins on pre-training and fine-tuning data, revealing a precise margin trade-off: the regularization parameter controls the balance between retention and adaptation. Second, we analyze adapter merging, proving that merged models achieve positive margin on each task and deriving optimal mixing coefficients that balance margins across tasks and maximize the margin over their union. These results lead to a simple, training-free merging rule, which we term margin-based LoRA merging (MaLoRA). Experiments on modern architectures and real datasets validate our theoretical predictions, showing that MaLoRA matches or outperforms several adapter-merging baselines across a range of vision and language classification tasks.

Introduction

Foundation models have transformed artificial intelligence across numerous domains. However, deploying these models in real-world applications typically requires fine-tuning on specialized datasets to meet accuracy, safety, and alignment requirements. Traditional full fine-tuning, which updates all parameters of a pre-trained model, incurs substantial computational cost, including large memory footprints, long training times, and significant storage demands. Low-Rank Adaptation (LoRA) [20] addresses this challenge by augmenting pre-trained weight matrices WW with low-rank adapters BB, AA while keeping WW frozen. Empirically, LoRA matches the performance of full fine-tuning while training only a small fraction of the parameters.1

40th Conference on Neural Information Processing Systems (NeurIPS 2026).

While LoRA was originally introduced for its computational efficiency, practitioners have since identified two striking empirical properties.

  1. Robustness to catastrophic forgetting. In fine-tuning and continual learning, the degradation of performance on previously learned tasks after adapting to new ones (a.k.a. catastrophic forgetting) is a fundamental challenge [37, 24, 40]. Surprisingly, recent studies show that LoRA exhibits strong robustness to catastrophic forgetting, even after substantial adaptation [52, 3, 38]. This robustness makes LoRA particularly appealing for continual learning.

  1. Effective adapter merging. In multi-task settings, maintaining separate fine-tuned models for each task is storage-intensive. Remarkably, LoRA adapters trained independently on different tasks can be merged (e.g., via simple weighted averaging) into a single adapter that performs well across all tasks [21, 55, 49, 56, 61]. This property enables flexible and efficient deployment.

Despite these empirical benefits, the theoretical mechanisms underlying LoRA’s robustness to catastrophic forgetting and its adapter merging capabilities remain poorly understood. In this work, we provide a unified theoretical explanation for both phenomena through a margin-based perspective.

Paper contributions. We study LoRA in multiclass linear classification under a near-orthogonal-task regime, in which fine-tuning samples are nearly orthogonal to one another and to the pre-training samples. In this setting, we characterize the global minimizer of the LoRA objective and use this characterization to analyze forgetting and adapter merging. Compared with prior theoretical work on LoRA (see Appendix § 8), our work makes the following contributions:

  • Characterization of the optimal solution. We characterize the global minimizer of LoRA across different regularization regimes (Theorem 3.1). In the high-penalty regime, the optimal adapter is zero. In the intermediate regime, the adapter aligns with the max-margin direction on the fine-tuning data. In the low-penalty regime, the adapter aligns with a non-max-margin classifier whose direction generally does not admit a simple closed form. Nevertheless, we show that this direction converges to an interpretable limit as the regularization parameter tends to zero.

  • Theory for reduced forgetting. We define forgetting as the reduction in margin on the pre-training data after fine-tuning, and derive closed-form expressions for the normalized margins on the pre-training data, the fine-tuning data, and their union (Theorem 3.2). Our analysis shows that these margins are governed by the regularization parameter and satisfy a precise trade-off: decreasing the regularization parameter increases the fine-tuning margin but also increases the Frobenius-norm ratio of the adapter to the pre-trained weights, leading to greater forgetting. We further identify the regularization parameter that maximizes the margin on the union of pre-training and fine-tuning tasks, thereby optimally balancing retention and adaptation.

  • Theory for adapter merging. Under the intermediate-regime assumption for each task, we derive closed-form expressions for the margin of the merged model on every task (Theorem 3.3). We show that the merged model attains strictly positive margin on each task, providing a theoretical explanation for the effectiveness of adapter merging. We also characterize, in closed form, the mixing coefficients that maximize the margin of the merged model on the union of all tasks, which naturally leads to margin-based LoRA merging (MaLoRA), a theoretically motivated adapter-merging method. This contrasts with many existing model-merging approaches, which rely primarily on heuristic prescriptions for choosing how to combine task-specific models.

  • Numerical validation. Experiments on real datasets and modern architectures support the theory in three ways: (i) in the linear probing setting, the empirically optimal regularization parameter agrees with the value predicted by our analysis; (ii) in the same setting, the empirically optimal adapter mixing weights agree with our closed-form coefficients; and (iii) in LoRA adapter merging, MaLoRA achieves performance comparable to, and in several cases better than, strong heuristic adapter-merging baselines across vision and language classification tasks.

Preliminaries

In this section, we first introduce the multiclass linear classification problem and its corresponding hard-margin SVM formulation. We then describe the specific learning problem considered in this work, which consists of a pre-training and fine-tuning phase. Finally, we state the key assumptions that underpin our analysis, and discuss their motivation and implications.

Background on multiclass linear classification

We first review the standard KK-class linear classification problem. Given a dataset D\mathcal{D}, the goal is to learn a weight matrix W∈RK×dW \in\mathbb{R}^{K \times d} that minimizes the mean empirical cross-entropy loss:

min⁡W∈RK×dL(W;D):=1∣D∣∑(x,y)∈DLCE(y,Wx),(1)\min_{W \in\mathbb{R}^{K \times d}} L(W;\mathcal{D}) := \frac{1}{|\mathcal{D}|} \sum_{(x,y)\in\mathcal{D}} \mathcal{L}_{\mathrm{CE}}(y,Wx), \tag*{(1)}

where x∈Rdx \in\mathbb{R}^{d} is the input and y∈RKy \in\mathbb{R}^{K} is a one-hot label vector. The cross-entropy loss is given by:

LCE(y,y^):=−∑c=1Kyclog⁡(exp⁡(y^c)∑i=1Kexp⁡(y^i)),y^=Wx.(2)\mathcal{L}_{\mathrm{CE}}(y,\hat{y}) := -\sum_{c=1}^{K} y_c \log\left(\frac{\exp(\hat{y}_c)}{\sum_{i=1}^{K}\exp(\hat{y}_i)}\right), \qquad\hat{y}=Wx. \tag*{(2)}

Recent studies have shown that when the data are linearly separable, gradient descent (GD) implicitly biases the solution of (1) toward the max-margin classifier [46, 34, 41]. Specifically, the iterates of GD converge in direction to the max-margin solution of the following hard-margin SVM problem2:

WSVM(D):=arg⁡min⁡W∈RK×d∥W∥F2subject toC(W;D)≥1,(3)W^{\mathrm{SVM}}(\mathcal{D}) := \mathop{\arg\min}_{W\in\mathbb{R}^{K\times d}} \lVert W\rVert_{F}^{2}\quad\text{subject to}\quad C(W;\mathcal{D})\ge1, \tag*{(3)}

where C(W;D)C(W;\mathcal{D}) is the minimum margin over the dataset:

C(W;D):=min⁡(x,y)∈Dmin⁡k≠c,yc=1(wcTx−wkTx),with W=[w1;…;wK]T.(4)C(W;\mathcal{D}) := \min_{(x,y)\in\mathcal{D}}\min_{\substack{k\ne c,y_c=1}}(w_c^{\mathsf{T}}x-w_k^{\mathsf{T}}x),\qquad\text{with } W=[w_1;\ldots;w_K]^{\mathsf{T}}. \tag*{(4)}

The hard-margin SVM problem in (3) seeks a weight matrix of minimum norm that separates all examples with at least unit margin between the correct class and the nearest competing class.

Intuitively, classification decisions depend primarily on the direction of the weight matrix rather than its scale. To enable comparisons across different scales of WW, we adopt the normalized margin used in [9], i.e., γ(W;D):=C(W;D)∥W∥F\gamma(W;\mathcal{D}):=\frac{C(W;\mathcal{D})}{\lVert W\rVert_F} for W≠0W\ne0. This definition has been used in recent theoretical studies [34, 41]. We additionally set γ(0;D):=0\gamma(0;\mathcal{D}):=0 by convention.

Pre-training and fine-tuning setup

We now describe the specific pre-training and fine-tuning framework used in our theoretical analysis.

Pre-training stage. Let Dpre\mathcal{D}_{\mathrm{pre}} be a labeled dataset with KK classes, and let Wpre∈RK×dW_{\mathrm{pre}}\in\mathbb{R}^{K\times d} denote a pre-trained linear classifier obtained by GD minimizing the cross-entropy loss L(W;Dpre)L(W;\mathcal{D}_{\mathrm{pre}}). When Dpre\mathcal{D}_{\mathrm{pre}} is linearly separable, this objective admits no finite minimizer. The GD iterates instead diverge in norm while converging in direction to the max-margin solution [34, 46]. We formalize this directional structure in Assumption 2.4.

Fine-tuning stage. We adapt the pre-trained model to a downstream dataset Dft\mathcal{D}_{\mathrm{ft}} consisting of nn additional samples from each of 2≤Kˉ≤K2\le\bar{K}\le K classes selected from the original KK pre-training classes. This restriction applies throughout the paper and appendix, including to every task in adapter merging. This setup captures practical scenarios such as domain-incremental learning [12], domain adaptation [Zohri­sadeh et al., 2019, Zhang et al., 2022], and bias-rebalancing fine-tuning [30, 48].

When LoRA is applied to adapt the pre-trained model to the fine-tuning task, we introduce a low-rank update to the weights, parameterized by matrices B∈RK×rB\in\mathbb{R}^{K\times r} and A∈Rr×dA\in\mathbb{R}^{r\times d}:

min⁡B,A L(Wpre+BA;Dft)+λ2(∥B∥F2+∥A∥F2).(5)\min_{B,A}\ L(W_{\mathrm{pre}}+BA;\mathcal{D}_{\mathrm{ft}})+\frac{\lambda}{2}\left(\lVert B\rVert_{F}^{2}+\lVert A\rVert_{F}^{2}\right). \tag*{(5)}

Here, the Frobenius-norm penalties explicitly constrain the adapters in weight space, limiting the deviation of the fine-tuned model from the pretrained initialization.

Although some empirical studies of LoRA do not include this penalty explicitly in the loss, they train the adapters using the AdamW optimizer [33], whose decoupled weight decay induces an effect analogous to L2L_2 regularization on BB and AA. Thus, the objective in (5) captures the regularization that is, explicitly or implicitly, used in standard LoRA fine-tuning practice [20, 52, 3].

Our goal is to understand the structure of the optimal solution to (5), and how it explains both LoRA’s robustness to catastrophic forgetting and its effectiveness in enabling adapter merging.

Assumptions and their implications

Our theoretical analysis uses the following assumptions. Let Nft=KˉnN_{\mathrm{ft}}=\bar{K}n be the number of fine-tuning samples, write Xft=[xˉ1,…,xˉNft]X_{\mathrm{ft}}=[\bar{x}_1,\ldots,\bar{x}_{N_{\mathrm{ft}}}], and let XpreX_{\mathrm{pre}} collect the pre-training feature vectors as columns.

Assumption 2.1 (Task geometry). The combined pre-training and fine-tuning datasets are linearly separable. There exist ϵ1,ϵ2∈[0,1)\epsilon_1,\epsilon_2\in[0,1) such that, for all i,j∈[Nft]i,j\in[N_{\mathrm{ft}}] and x∈Dprex\in\mathcal{D}_{\mathrm{pre}},

(a)∥xˉi∥2=∥x∥2=1,(b)∣xˉiTxˉj∣≤ϵ1 (i≠j),(c)∣xˉiTx∣≤ϵ2.(6)\begin{aligned} \text{(a)}\quad&\lVert\bar{x}_i\rVert_2=\lVert x\rVert_2=1,\qquad \text{(b)}\quad\lvert\bar{x}_i^{\mathsf{T}}\bar{x}_j\rvert\le\epsilon_1\ (i\ne j),\qquad \text{(c)}\quad\lvert\bar{x}_i^{\mathsf{T}}x\rvert\le\epsilon_2. \tag*{(6)} \end{aligned}

Assumption 2.2 (Dimension). In the strict-orthogonal case ϵ1=ϵ2=0\epsilon_1=\epsilon_2=0, the input dimension satisfies d≥Nft+rank⁡(Xpre)d\ge N_{\mathrm{ft}}+\operatorname{rank}(X_{\mathrm{pre}}). For near-orthogonal data, we assume (Nft−1)ϵ1<1(N_{\mathrm{ft}}-1)\epsilon_1<1.

Assumption 2.3 (LoRA rank). The LoRA rank parameter satisfies r≥Kˉr\ge\bar{K}, where Kˉ\bar{K} is the number of fine-tuning classes.

Assumption 2.4 (Pre-trained classifier). The pre-trained classifier WpreW_{\mathrm{pre}} is a scaled version of the hard-margin SVM solution on Dpre\mathcal{D}_{\mathrm{pre}}:

Wpre=ρpre⋅WpreSVM∥WpreSVM∥F,W_{\mathrm{pre}}=\rho_{\mathrm{pre}}\cdot\frac{W_{\mathrm{pre}}^{\mathrm{SVM}}}{\lVert W_{\mathrm{pre}}^{\mathrm{SVM}}\rVert_F},

for some scalar ρpre>0\rho_{\mathrm{pre}}>0, where WpreSVM=WSVM(Dpre)W_{\mathrm{pre}}^{\mathrm{SVM}}=W^{\mathrm{SVM}}(\mathcal{D}_{\mathrm{pre}}).

Assumption justification. The near-orthogonal task Assumption 2.1 is arguably a strong assumption in practice, but reflects the realistic geometric structure of fine-tuning data, and similar orthogonality-type assumptions are commonly used in theoretical analyses to enable closed-form characterizations [6, 5, 16, 26]. The feature-correlation analyses on CIFAR-100 and six multi-domain benchmarks in Appendices § 14.1.1 and § 14.§ 2.1 provide empirical motivation for studying this reference geometry. Assumption 2.2 makes the geometric feasibility and conditioning requirements explicit. The strict-case bound accounts for both the fine-tuning sample count and the pre-training feature span, specifying a high-dimensional, sample-limited setting. Assumption 2.3 requires only r≥Kˉr\ge\bar{K}, meaning the LoRA rank needs only to be at least the number of fine-tuning classes. Finally, Assumption 2.4 is motivated by recent theoretical results on the implicit bias of GD in multiclass classification [35, 41], as discussed in §2.1. ρpre\rho_{\mathrm{pre}} captures the (unspecified) scale at which pre-training was stopped, which arises naturally in implicit-bias analyses where the GD iterate diverges along the SVM direction.

For clarity of exposition, all theorems in the main paper are stated under the strict-orthogonal special case (ϵ1=ϵ2=0\epsilon_1=\epsilon_2=0) of Assumption 2.1. The general versions of these results for ϵ1,ϵ2>0\epsilon_1,\epsilon_2>0, together with their proofs, are deferred to the Appendix D–F.

Implications. The assumptions introduced above lead to a clean closed-form expression for the Kˉ\bar{K}-class hard-margin SVM solution for the fine-tuning task, and also imply near-orthogonality between the pre-trained weights WpreW_{\mathrm{pre}} and the fine-tuning dataset Dft\mathcal{D}_{\mathrm{ft}}, which underpins our subsequent theoretical analysis in §3. To formalize these implications, we begin by introducing the concept of a simplex equiangular tight frame (simplex ETF), which characterizes the geometry of the hard-margin SVM solution in our setting.

Definition 2.1 (Simplex ETF). For m≥2m\ge2, an mm-simplex ETF is a collection of mm vectors in Rp\mathbb{R}^{p} for some p≥mp\ge m, given by the columns of

Mm:=mm−1P(Im−1m1m1mT),M_m:=\sqrt{\frac{m}{m-1}}P\left(I_m-\frac{1}{m}\mathbf{1}_m\mathbf{1}_m^{\mathsf{T}}\right),

where P∈Rp×mP\in\mathbb{R}^{p\times m} satisfies PTP=ImP^{\mathsf{T}}P=I_m.

Throughout, we use the canonical choice p=mp=m and P=ImP=I_m, so Mm∈Rm×mM_m\in\mathbb{R}^{m\times m}. Using this definition, we now present a proposition that, under the strict-orthogonal specialization, characterizes the structure of the hard-margin SVM solution for the fine-tuning data (Dft\mathcal{D}_{\mathrm{ft}}) and the orthogonality between the pre-trained weights (WpreW_{\mathrm{pre}}) and the fine-tuning data.

References

  1. [1]Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–154, 2018.arxiv.org/abs/1711.09601
  2. [2]Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456–473, 2018.
  3. [3]Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673, 2024.
  4. [4]Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. CoRR, abs/1903.04561, 2019. URL http://arxiv.org/abs/1903.04561.
  5. [5]Etienne Boursier, Loucas Pullaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. In Advances in Neural Information Processing Systems, volume 35, pages 20105–20118, 2022.arxiv.org/abs/2206.00939
  6. [6]Phuong Bui Thi Mai and Christoph Lampert. The inductive bias of relu networks on orthogonally separable data. In 9th International Conference on Learning Representations, 2021.
  7. [7]Christopher J Burges and David Crisp. Uniqueness of the svm solution. Advances in neural information processing systems, 12, 1999.
  8. [8]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 8440–8451, 2020.arxiv.org/abs/1911.02116
  9. [9]Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research, 2(Dec):265–292, 2001.
  10. [10]Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabsreau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics, pages 1072–1080. PMLR, 2021.arxiv.org/abs/2010.04003
  11. [11]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2021.arxiv.org/abs/2010.11929
  12. [12]Yasushi Esaki, Satoshi Koide, and Takuro Kutsuna. One-shot domain incremental learning. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024.DOI
  13. [13]Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pages 4028–4079. PMLR, 2022.arxiv.org/abs/2205.09588
  14. [14]Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE international conference on computer vision, pages 1657–1664, 2013.DOI
  15. [15]Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43):e2103091118, 2021.DOI
  16. [16]Spencer Frei, Gal Vardi, Peter L Bartlett, Nathan Srebro, and Wei Hu. Implicit bias in leaky relu networks trained on high-dimensional data. arXiv preprint arXiv:2210.07082, 2022.arxiv.org/abs/2210.07082
  17. [17]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.arxiv.org/abs/1512.03385
  18. [18]Wanli Hong and Shuyang Ling. Neural collapse for unconstrained feature model under cross-entropy loss with imbalanced data. arXiv preprint arXiv:2309.09725, 2023.arxiv.org/abs/2309.09725
  19. [19]Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
  20. [20]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022.arxiv.org/abs/2106.09685
  21. [21]Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023.arxiv.org/abs/2307.13269
  22. [22]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannah Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.arxiv.org/abs/2212.04089
  23. [23]Uijeong Jang, Jason D Lee, and Ernest K Ryu. Lora training in the ntk regime has no spurious local minima. arXiv preprint arXiv:2402.11867, 2024.
  24. [24]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabsk a-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521–3526, 2017.DOI
  25. [25]Simon Kornblith, Jonathan Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2661–2671, 2019.arxiv.org/abs/1805.08974
  26. [26]Yiwen Kou, Zixiang Chen, and Quanquan Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks on nearly-orthogonal data. Advances in Neural Information Processing Systems, 36:30167–30221, 2023.
  27. [27]Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.(2009), 2009.
  28. [29]Xuhong Li, Yves Grandvalet, and Francois Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning (ICML), pages 2825–2834, 2018.
  29. [30]Zhiheng Li and Chenliang Xu. Discover the unknown biased attribute of an image classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14970–14979, 2021.
  30. [31]Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision (ECCV), pages 614–629, 2016.
  31. [32]Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1976–11986, 2022.
  32. [33]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.arxiv.org/abs/1711.05101
  33. [34]Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890, 2019.
  34. [35]Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems, 34:12978–12991, 2021.
  35. [36]Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610–23641. PMLR, 2023.
  36. [37]Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.DOI
  37. [38]Fuli Qiao and Mehrdad Mahdavi. Learn more, but bother less: parameter efficient continual learning. Advances in Neural Information Processing Systems, 37:97476–97498, 2024.DOI
  38. [39]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  39. [40]Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International conference on learning representations, 2021.
  40. [41]Hrithik Ravi, Clay Scott, Daniel Soudry, and Yutong Wang. The implicit bias of gradient descent on separable multiclass data. Advances in Neural Information Processing Systems, 37:81324–81359, 2024.arxiv.org/abs/2411.01350
  41. [42]Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):471–501, 2010.
  42. [43]Kurt S Riedel. A sherman–morrison–woodbury identity for rank augmenting matrices with application to centering. SIAM Journal on Matrix Analysis and Applications, 13(2):659–662, 1992.arxiv.org/abs/1803.10405
  43. [45]Jonathan Schwarz, Wojciech M Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning (ICML), pages 4528–4537, 2018.arxiv.org/abs/1805.06370
  44. [46]Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1–57, 2018.arxiv.org/abs/1710.10345
  45. [47]Rangarajan K Sundaram. A first course in optimization theory. Cambridge university press, 1996.
  46. [48]Taha ValizadehAslani, Yiwen Shi, Jing Wang, Ping Ren, Yi Zhang, Meng Hu, Liang Zhao, and Hualou Liang. Two-stage fine-tuning with chatgpt data augmentation for learning class-imbalanced data. Neurocomputing, 592:127801, 2024.DOI
  47. [49]Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. Lora-flow: Dynamic lora fusion for large language models in generative tasks. arXiv preprint arXiv:2402.11455, 2024.arxiv.org/abs/2402.11455
  48. [50]G Alistair Watson. Characterization of the subdifferential of some matrix norms. Linear Algebra Appl, 170(1):33–45, 1992.DOI
  49. [51]Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/N18-1101.arxiv.org/abs/1704.05426
  50. [52]Martin Wistuba, Prabhu Teja Sivaprasad, Lukas Balles, and Giovanni Zappella. Continual learning with low rank adaptation. arXiv preprint arXiv:2311.17601, 2023.arxiv.org/abs/2311.17601
  51. [53]Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965–23998. PMLR, 2022.arxiv.org/abs/2203.05482
  52. [54]Ziqing Xu, Hancheng Min, Lachlan Ewen MacDonald, Jinqi Luo, Salma Tarmoun, Enrique Mallada, and Rene Vidal. Understanding the learning dynamics of lora: A gradient flow perspective on low-rank adaptation. arXiv preprint arXiv:2503.06892, 2025.
  53. [55]Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models. arXiv preprint arXiv:2306.01708, 1, 2023.arxiv.org/abs/2306.01708
  54. [56]Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2311.03099
  55. [57]Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513, 2023.arxiv.org/abs/2310.17513
  56. [58]Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR, 2017.arxiv.org/abs/1703.04200
  57. [60]Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.arxiv.org/abs/1509.01626
  58. [61]Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. arXiv preprint arXiv:2402.09997, 2024.arxiv.org/abs/2402.09997

Paper details

vlnc:W_pce656u5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLM

View statement
Looking for related papers…
Contents