betaFeedback

WorksComputer Science

SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection

Abstract

The success of large language models in real-world applications relies on parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which inserts compact, trainable adapters into selected layers. In this paper, we propose SimplexLoRA, a LoRA variant that adaptively adjusts adapter ranks during fine-tuning based on their learned importance. Unlike standard LoRA, which employs a fixed rank across all layers, SimplexLoRA optimizes per-adapter importance weights on the nn-scaled simplex during a short warm-up and then freezes them, dynamically allocating capacity where it is most beneficial without increasing the overall parameter budget or inference cost. We evaluate SimplexLoRA across 12 language understanding, summarization, and reasoning tasks using DeBERTaV3, BART-large, Llama 3.1 8B, and Qwen3-8B. Our results show that SimplexLoRA consistently matches or outperforms standard LoRA and recent adaptive baselines, with the largest gains at low ranks. Our code is available at https://github.com/brain-lab-research/SimplexLoRA.

Introduction

LoRA [16] is a widely used parameter-efficient fine-tuning method [11] that parameterizes weight updates as W′=W0+ABW' = W^0 + AB, with W0∈Rm×dW^0 \in\mathbb{R}^{m \times d} frozen and trainable A∈Rm×rA \in\mathbb{R}^{m \times r}, B∈Rr×dB \in\mathbb{R}^{r \times d}, r≪m,dr \ll m,d. This approach achieves low inference overhead and substantially reduces memory during fine-tuning.

Despite its efficiency, standard LoRA uses a fixed rank across all layers, ignoring layer-wise differences in convergence and representation learning. Existing adaptive rank allocation methods attempt to address this issue but suffer from the critical drawbacks: (i) substantially more trainable parameters [Sheng et al., 2025], (ii) non-negligible inference-time overhead caused by dynamic pruning or gating mechanisms [Zhang et al., 2025], and (iii) deviations from the standard LoRA pipeline that hinder compatibility [Zhang et al., 2023b]. See § 2 for further discussion.

To address this limitation, we propose SimplexLoRA, which assigns an importance weight to each of the nn LoRA adapters, constrained to the nn-scaled simplex to enforce

Figure 1. SimplexLoRA framework. Per-adapter importance weights are constrained to the nn-scaled simplex, enabling dynamic allocation of a fixed rank budget across layers.

a fixed parameter budget. Key properties: (i) ranks learned automatically during fine-tuning; (ii) simplex constraint preserves the LoRA parameter budget; (iii) full LoRA compatibility; (iv) negligible training and inference overhead.

Related Work

Classical low-rank approaches

While LoRA has popularized parameter-efficient fine-tuning, it is not the only method to explore low-rank or structured adaptations. Earlier approaches include LoHA [26], which utilizes element-wise (Hadamard) interactions between low-rank matrices to reduce the parameter count without sacrificing expressivity, and Krona [Edalati et al., 2022], which employs the Kronecker product in place of the conventional low-rank matrix product to improve parameter efficiency while maintaining model capacity.

Subsequent methods extended or refined the LoRA framework. VERA [20] reduces adapter storage by sharing frozen random projection matrices across layers and learning small scaling vectors. rsLoRA [17] replaces the original LoRA scaling factor α/r\alpha/r [16] with the rank-stabilized factor α/r\alpha/\sqrt{r}. A complementary direction is taken by DoRA [23], which decomposes the adapted weight into magnitude and direction components and learns the magnitude together with a low-rank correction of the direction. Another line of work improves LoRA optimization and numerical efficiency. LoRA+ [12] uses different learning rates for the two adapter matrices, while QLoRA [8] enables memory-efficient fine-tuning through 4-bit quantization of frozen base-model weights. Both are orthogonal to our adaptive rank allocation method and can be combined with it.

Dynamic LoRA rank allocation

This topic is of particular relevance in the context of our work, as we focus on adaptive rank learning, enabling the model to automatically adjust the capacity of each adapter during fine-tuning. It has attracted significant attention in recent research [Zhang et al., 2023a,b; Sheng et al., 2025], as exploiting dynamically changing ranks allows models to utilize newly allocated parameters more efficiently.

The SVD-like representation of LoRA adapters proposed in AdaLoRA [Zhang et al., 2023b] replaces the standard two-matrix adapter with the parameterization PΛQP\Lambda Q and enables the construction of importance scores for individual singular components. Components outside the global importance budget are masked by setting the corresponding entries of Λ\Lambda to zero. To encourage the SVD-like structure, the method applies the orthogonality regularizer

R(P,Q)=∥PTP−I∥F2+∥QTQ−I∥F2.R(P,Q) = \lVert P^{\mathsf{T}}P-I\rVert_{F}^{2}+\lVert Q^{\mathsf{T}}Q-I\rVert_{F}^{2}.

During training, all three matrices PP, Λ\Lambda, and QQ are updated simultaneously. Although this parameterization avoids explicitly computing an SVD at every fine-tuning step, maintaining an initial rank above the target budget, regularly evaluating importance scores, and applying orthogonality regularization increase the training complexity compared with standard LoRA.

Several methods manage the rank budget more flexibly. ElaLoRA [4] uses SVD-based adaptation and gradient-derived importance scores to prune and expand ranks at scheduled intervals. FlexLoRA [22] uses normalized spectral entropy to guide rank pruning and expansion under a global budget. The dynamic-rank DoRA of Mao et al. [25] decomposes adapters into structured rank-one components and prunes them based on per-component importance. NormAL LoRA [1] uses rank-norm regularization to determine an appropriate rank for each module.

In a similar way, Sheng et al. (2025) proposes a rank-one LoRA series representation of the increment matrix:

ΔW=b1a1+⋯+bpap=[Bp−1bp][Ap−1ap],\Delta W=b_1a_1+\cdots+b_pa_p=\begin{bmatrix}B_{p-1}&b_p\end{bmatrix}\begin{bmatrix}A_{p-1}\\a_p\end{bmatrix},

where pp denotes the number of outer-loop iterations. This representation is applied to several layers, and after each outer-loop step a convergence criterion is evaluated: if the relative change between the (p−1)(p-1)-th and pp-th solutions is sufficiently small, the training of ΔW\Delta W is terminated. Unlike IncreLoRA, which expands modules according to gradient-based importance scores, AROMA uses the relative Frobenius-norm change between consecutive solutions as a stopping criterion.

Compared with these works, SimplexLoRA does not force adapter ranks to strictly increase or rely on expensive SVD computations. Instead, we allow the ranks to be learned as continuous parameters, enabling the model to respond dynamically to the task’s complexity. By leveraging simplex projection and weight freezing, we obtain a direct form of rank adaptation that avoids explicit importance scoring and iterative pruning.

SimplexLoRA Framework

For the LoRA method, we introduce trainable layer-wise scaling factors: Wi0+AiBi→Wi0+ωiAiBiW_i^0+A_iB_i\to W_i^0+\omega_iA_iB_i, where ωi≥0\omega_i\geq0 represents the importance coefficient assigned to the ii-th adapter. The weights ω=(ω1,…,ωn)\boldsymbol{\omega}=(\omega_1,\ldots,\omega_n) are constrained to the nn-scaled simplex, where nn is the number of LoRA adapters. This preserves LoRA magnitude, with ω=(1,…,1)=1\omega= (1,\ldots,1) = \mathbf{1} recovering standard LoRA initialization [16].

Motivated by the theoretical analysis in Appendix § 6, SimplexLoRA formulates the following alternating optimization problem:

min⁡ω∈n⋅Δn−1min⁡A,BL(ω,A,B)+λ2∥ω−1∥22(1)\min_{\omega\in n\cdot\Delta_{n-1}}\min_{A,B}\mathcal{L}(\omega,A,B)+\frac{\lambda}{2}\lVert\omega-\mathbf{1}\rVert_2^2 \tag*{(1)}

where A,B={Ai}i=1n,{Bi}i=1nA,B=\{A_i\}_{i=1}^{n},\{B_i\}_{i=1}^{n} denote the sets of all LoRA adapters, L\mathcal{L} is the loss, and the term (λ/2)∥ω−1∥22(\lambda/2)\lVert\omega-\mathbf{1}\rVert_2^2 prevents overfitting of the importance weights ω\omega.

SimplexLoRA Training Process

The SimplexLoRA algorithm employs an alternating optimization scheme with three phases, repeated for KK iterations.

Phase 1. Warm-up. For TT steps, we perform standard LoRA fine-tuning by jointly optimizing the adapter parameters A,BA,B and the importance weights ω\omega using gradient-based methods.

Phase 2. Simplex projection. The weight vector ω\omega is projected onto the scaled simplex n⋅Δn−1n\cdot\Delta_{n-1} via Euclidean projection [3]:

min⁡ω^∈n⋅Δn−1∥ω−ω^∥22\min_{\widehat{\omega}\in n\cdot\Delta_{n-1}}\lVert\omega-\widehat{\omega}\rVert_2^2

Phase 3. Rank update. Adapter ranks are dynamically resized while preserving the total parameter budget:

rinew=⌊r0⋅ωi⌋,r_i^{\mathrm{new}}=\lfloor r_0\cdot\omega_i\rfloor,

where r0r_0 is the initial (uniform) LoRA rank. When ranks change, the corresponding adapter matrices are resized and reinitialized following the standard LoRA scheme (see Appendix § 7).

Repeat Phases 1–3 for KK iterations.

Phase 4. Standard LoRA training with new ranks. After KK repetitions, we fix ω\omega to 1\mathbf{1} and reinitialize the adapter matrices AA and BB (Appendix § 7). In this sense Phases 1–3 act as a short rank-search procedure: their role is to determine the per-layer rank allocation, while the warm-up adapter weights are intentionally discarded so that the final stage is functionally identical to standard LoRA with a non-uniform rank profile. Standard training then resumes with these fixed ranks, and any LoRA-based optimization algorithm can be applied.

Scheduler. To ensure that the model resumes training after freezing the weights, we employ a linear scheduler with two training phases (see Figure 2). The first phase includes rank training with reduced learning rate. Once rank training is complete, the learning rate is reset to zero, and the warm-up begins to train the resulting LoRA layers. Note, however, that the learning rate for ω\omega is fixed during training, the scheduler exclusively adjusts the LoRA weights. As a result, the subsequent fine-tuning phase is functionally identical to the standard LoRA training process.

Figure 2. Scheduler for SimplexLoRA parameters.

Discussion. The total number of rank updates, K⋅TK\cdot T, is kept small to avoid rank collapse and to ensure that the majority of training focuses on optimizing the LoRA adapters. In practice, TT is negligible compared to the overall training duration. In the ablation study (§ 4.3), we analyze the sensitivity of SimplexLoRA to both hyperparameters KK and TT, evaluating their impact on performance.

Motivating Example of Rank Scaling

Figure 3 shows the final rank distribution from fine-tuning DeBERTaV3base_{\mathrm{base}} [14] on the MRPC dataset from GLUE [Wang et al., 2018], tuning only self-attention layers. The learned distribution reveals a clear pattern: value projection matrices in self-attention layers consistently receive the highest ranks, indicating their critical role. Rather than eliminating ranks of adapters with low weights, the method reduces their magnitude, which helps mitigate underfitting.

Figure 3. Illustration of the rank distribution of SimplexLoRA.

Experiments

We evaluate SimplexLoRA on a diverse set of benchmarks, including GLUE [Wang et al., 2018], XSum [27], CNN/Daily Mail [5], GSM8K [7], BoolQ [6], and HellaSwag [Zellers et al., 2019]. We exclude GLUE STS-B, whose regression metric requires custom evaluation and which overlaps with the QQP/MRPC similarity tasks. Experiments are conducted using DeBERTaV3base_{\mathrm{base}} [14], Llama 3.1 8B [2], Qwen3-8B [Yang et al., 2025], and BART-large [21] models. Hardware and hyperparameters are provided in Appendix D.

Out-of-the-box Comparison

We benchmark against baselines in default configurations under a number of training iterations. These span foundational PEFT methods (a performance floor) and recent dynamic-rank methods comparable to SimplexLoRA:

Standard PEFT methods: LoRA [16], LoHA [26], Krona [Edalati et al., 2022], rsLoRA [17], DoRA [23], and VERA [20].

Dynamic rank methods: AdaLoRA [Zhang et al., 2023b], NormAL LoRA [1], ElaLoRA [4], and FlexLoRA [22].

In this setting (Table 1), where adapters are applied only to self-attention layers, full fine-tuning, standard LoRA, and SimplexLoRA emerge as the most robust methods, with SimplexLoRA achieving the best average score at the same 0.24% parameter budget as standard LoRA. Other approaches require extensive hyperparameter tuning to achieve competitive performance. Consequently, we select standard LoRA as our primary baseline for further detailed comparison due to its widespread adoption and stability.

Method# ParamsMNLI AccSST-2 AccCoLA MccQQP Acc/F1QNLI AccRTE AccMRPC Acc/F1ALL Avg
Full Fine-Tuning184M (100%)0.89100.95410.68060.8962/0.86440.93830.83760.8848/0.91650.8689
LoRAr=8_{r=8} [16]442K (0.24%)0.87970.94500.69130.8802/0.84370.93010.84480.8897/0.92200.8696
LoHAr=8_{r=8} [26]884K (0.48%)0.85600.93920.62950.8674/0.82690.90850.80510.8628/0.90070.8440
Kronar=32_{r=32} [Edalati et al., 2022]57.6K (0.03%)0.75870.92890.59070.8275/0.78160.86050.75090.7500/0.84160.7878
VERAr=1024_{r=1024} [20]1.64M (0.88%)0.83550.94040.62520.8592/0.82030.89750.75810.8529/0.89620.8317
rsLoRAr=8_{r=8} [17]442K (0.24%)0.8814<u>0.9564</u>0.64140.8853/0.85060.93250.82670.8750/0.91220.8624
DoRAr=8_{r=8} [23]442K (0.24%)0.88110.94730.67480.8809/0.84510.92950.79780.8799/0.91390.8611
AdaLoRAr=8_{r=8} [Zhang et al., 2023b]664K (0.36%)0.83900.93920.62220.8534/0.80830.89990.78340.8859/0.91780.8388
NormAL LoRAr=8_{r=8} [1]442K (0.24%)0.88140.93280.68570.8768/0.84030.92820.82920.8870/0.91950.8645
ElaLoRAr=8_{r=8} [4]442K (0.24%)0.88530.95010.6638<u>0.8979</u>/0.86280.91460.80310.8782/0.91130.8630
FlexLoRAr=8_{r=8} [22]664K (0.36%)0.87110.95570.63310.8684/<u>0.8658</u>0.9343<u>0.8392</u>0.9014/0.92170.8656
SimplexLoRAr=8_{r=8}442K (0.24%)0.89840.95870.65720.9124/0.8844<u>0.9362</u>0.8339<u>0.8987</u>/0.92890.8788

Table 1. Comparative performance of "out-of-the-box" algorithms on DeBERTaV3base_{\mathrm{base}}. The best results are shown in bold, and the second-best are <u>underlined</u>. All experiment details are provided in Appendix D.

Comparison with LoRA

Table 2 compares SimplexLoRA with standard LoRA at multiple ranks on DeBERTaV3base_{\mathrm{base}} and Llama 3.1 8B (adapters in all linear layers, hyperparameters tuned per method). Across most GLUE tasks, SimplexLoRA consistently outperforms LoRA while maintaining the same computational cost. For generality, we also evaluate Qwen3-8B (GSM8K, BoolQ, HellaSwag) and BART-large (XSum, CNN/DailyMail) in Table 3. As on GLUE, SimplexLoRA matches or outperforms LoRA on most of these tasks, confirming the trend across architectures and task types. The improvement is most pronounced at low ranks (r∈{1,2}r \in\{1,2\}), where the tight capacity budget of standard LoRA gives adaptive allocation the most room to help.

ModelRankMethodMNLI AccSST-2 AccCoLA MccQQP Acc/F1QNLI AccRTE AccMRPC Acc/F1ALL Avg
DeBERTaV3base_{\text{base}}r=1r = 1L0.89120.94880.65580.8905/0.85440.92500.79430.8905/0.92130.8562
DeBERTaV3base_{\text{base}}r=1r = 1S0.89130.95760.67750.9018/0.87090.93170.81590.9093/0.93590.8690
DeBERTaV3base_{\text{base}}r=2r = 2L0.90040.94130.66000.9077/0.86310.93690.80680.9003/0.91930.8629
DeBERTaV3base_{\text{base}}r=2r = 2S0.89460.96080.66580.9178/0.88420.93380.82520.9103/0.93140.8717
DeBERTaV3base_{\text{base}}r=4r = 4L0.87800.95620.68090.9153/0.88660.95820.78160.9207/0.94710.8700
DeBERTaV3base_{\text{base}}r=4r = 4S0.90020.96100.68070.9244/0.89030.93580.82590.9044/0.93220.8756
DeBERTaV3base_{\text{base}}r=8r = 8L0.90300.96410.69650.9092/0.87230.94490.85370.9064/0.91270.8804
DeBERTaV3base_{\text{base}}r=8r = 8S0.90470.96190.69780.9129/0.89010.94470.83050.9267/0.94750.8826
Llama 3.1 8Br=1r = 1L0.89610.97510.70150.8891/0.83700.94110.90160.9154/0.94080.8886
Llama 3.1 8Br=1r = 1S0.88610.98010.68290.8891/0.83790.94310.92040.9278/0.94040.8898
Llama 3.1 8Br=2r = 2L0.89960.97510.69670.8851/0.83110.94210.91540.9204/0.94410.8900
Llama 3.1 8Br=2r = 2S0.89980.98010.69810.8851/0.83160.93810.93530.9254/0.94660.8933
Llama 3.1 8Br=4r = 4L0.89930.98000.69020.8851/0.82860.93910.91540.9254/0.94620.8899
Llama 3.1 8Br=4r = 4S0.90040.98230.70690.8891/0.83250.94210.93040.9204/0.94290.8941
Llama 3.1 8Br=8r = 8L0.89950.98510.69280.8851/0.83060.94210.91040.9154/0.94190.8892
Llama 3.1 8Br=8r = 8S0.90160.98510.70580.8881/0.83430.94310.93530.9204/0.94290.8952

Table 2. Comparative results of top-performing algorithms: LoRA (L) and SimplexLoRA (S) on DeBERTaV3base_{\text{base}} and Llama 3.1 8B. Results are averaged over 3 runs with different seeds. The best results per rank are shown in bold. All experiment details are provided in Appendix § 9.

RankMethodQwen3-8B GSM8K AccBoolQ AccHellaSwag AccALL AvgBART-large XSum ROUGE-1CNN/DailyMail ROUGE-1
r=1r = 1L0.85220.6550.7720.7600.3720.245
r=1r = 1S0.86660.6800.7750.7740.3730.249
r=2r = 2L0.86810.6620.7760.7690.3740.250
r=2r = 2S0.86730.6820.7910.7800.3760.254
r=4r = 4L0.85370.6910.7770.7740.3750.255
r=4r = 4S0.87190.7000.7800.7840.3780.256
r=8r = 8L0.86810.6910.7820.7800.3800.257
r=8r = 8S0.87410.6990.7900.7880.3790.259

Table 3. Comparative results: LoRA (L) and SimplexLoRA (S) on Qwen3-8B and BART-large. The best results per rank and model are shown in bold. All details are provided in Appendix § 9.

Ablation Study

As supplementary analysis, we grid-search hyperparameters on MNLI with DeBERTa to assess the robustness (Figure 4). Both methods are trained for 2048 steps using the best configuration from the main experiments; each cell reports the validation-accuracy difference.

Accuracy Difference from LoRA heatmap showing the influence of hyperparameters on SimplexLoRA performance

Figure 4. The influence of hyperparameters on SimplexLoRA performance.

The results show that SimplexLoRA is stable under a limited optimization budget (K⋅T<100K \cdot T < 100, i.e., less than 5% of total training). Very small values (e.g., K=3K = 3, T=5T = 5) lead to underfitting due to insufficient rank adaptation, while larger values (e.g., K≥10K \ge10 or T≥15T \ge15) tend to cause rank overfitting. Beyond these thresholds, we find that increasing the number of projection steps and inner iterations results in a performance plateau, with values showing minimal deviation from the average. This confirms the importance of keeping K⋅TK \cdot T low to maintain efficiency without performance loss. In all experiments, we use K=5K = 5 and T=10T = 10, which consistently yields strong performance.

Conclusion

We presented SimplexLoRA, a LoRA variant that allocates a fixed rank budget non-uniformly across layers via short alternating optimization on the nn-scaled simplex. Across 12 tasks and four backbones up to 8B parameters, SimplexLoRA matches or outperforms standard LoRA and recent adaptive baselines at the same parameter count and inference cost, with the largest gains in the low-rank regime.

Limitations

The limitations of the proposed strategy are as follows. First, SimplexLoRA introduces three additional hyperparameters: the number of iterations between simplex projection steps TT, the total number of such projections KK, and the weight decay coefficient used for regularization; the ablation in § 4.3 indicates that the method is robust within a broad range but does not eliminate the need to set these values. Second, the lack of experiments on models substantially larger than 8B parameters limits the generalizability of our findings. Third, while the per-task averages over three seeds consistently favor SimplexLoRA, on a few individual tasks (e.g., CoLA and RTE in the out-of-the-box DeBERTaV3 comparison) standard LoRA remains competitive or slightly better, indicating that the benefit of adaptive rank allocation is task-dependent. Fourth, a fundamental ambiguity remains during training: a small gradient norm in a layer may indicate either that the weights are well-optimized or that the gradients have vanished by the time they reach that layer, which can in principle bias the learned importance weights. On the ethical side, we see no specific risks beyond those already inherent to fine-tuning large language models: this work studies a parameter-efficient adaptation procedure, introduces no new data sources, uses only publicly available benchmarks with no proprietary or personal data and no human subjects, and reduces rather than increases the compute and memory footprint of fine-tuning.

Acknowledgments

The work of A. Beznosikov was supported by the Russian Science Foundation (project no. 21-71-30005-P).

Appendix

Supplementary Materials for SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection

SimplexLoRA: theoretical motivation

This section presents a heuristic argument that motivates the simplex-constrained reparameterization of § 3. We do not claim a formal optimization guarantee in the non-convex setting; the goal is to clarify why importance weights on a simplex are a natural device for redistributing adapter capacity.

Consider a loss function L(W1,…,Wn)\mathcal{L}(W^{1},\ldots,W^{n}) for a machine learning model with nn layers. In the context of fine-tuning an LLM using the LoRA algorithm, the optimization problem can be formulated as

min⁡A,B{L(A,B):=L(A1B1,…,AnBn)}.\min_{\mathcal{A},\mathcal{B}}\left\{\mathcal{L}(\mathcal{A},\mathcal{B}):=\mathcal{L}(A^{1}B^{1},\ldots,A^{n}B^{n})\right\}.

where A\mathcal{A} and B\mathcal{B} denote the sets of all LoRA adapters. To generalize this formulation, we introduce a non-negative scaling factor ωi≥0\omega_i \ge0 for each layer ii, which can be interpreted as rescaling the adapter matrix AiA^{i} by ωi\omega_i. For the purposes of the present derivation, we constrain the vector ω:=(ω1,…,ωn)\boldsymbol{\omega}:=(\omega_1,\ldots,\omega_n) to lie on the probability simplex Δn−1\Delta_{n-1}, i.e., ∑i=1nωi=1\sum_{i=1}^{n}\omega_i=1; the nn-scaled simplex used in the algorithm (§ 3) is the same object up to a factor of nn, chosen so that the uniform vector ω=1\boldsymbol{\omega}=1 recovers standard LoRA. The resulting optimization problem is then

min⁡A,B{L(ω1A1B1,…,ωnAnBn)}.\min_{\mathcal{A},\mathcal{B}}\left\{\mathcal{L}(\omega_1A^{1}B^{1},\ldots,\omega_nA^{n}B^{n})\right\}.

Note that this reparameterization does not alter the expressive power of the model, as the scaling factors can be absorbed into the adapter matrices.

A classical simplifying assumption in the optimization literature is that L\mathcal{L} is convex [18] with respect to all parameters {W1,…,Wn}\{W^{1},\ldots,W^{n}\}. For deep networks this is generally false, but convex analysis is still commonly used as a guiding lens: empirical studies report local convexity along training trajectories [19] (Zhou et al. 2019), and many non-convex results build on convex foundations [15, 10]. We adopt this assumption only to derive the upper bound below; the resulting prescription on ω\boldsymbol{\omega} is then transferred back to the non-convex setting as a design heuristic, not as a guarantee.

Therefore, if we take Ai⋅Bi:={0,…,AiBi,…,0}\mathcal{A}^{i}\cdot\mathcal{B}^{i}:=\{0,\ldots,A^{i}B^{i},\ldots,0\}, that is, we fine-tune only the ii-th layer, then, by the convexity of L\mathcal{L} (Jensen’s inequality), we obtain:

min⁡A,B{L(A,B)}=min⁡A,B{L(ω1A1B1,…,ωnAnBn)}=min⁡A,B{L(∑i=1nωi⋅{0,…,AiBi,…,0})}≤min⁡A,B{∑i=1nωiL(Ai⋅Bi)}=∑i=1nωimin⁡Ai,Bi{L(0,…,AiBi,…,0)}.\begin{aligned} \min_{\mathcal{A},\mathcal{B}}\{\mathcal{L}(\mathcal{A},\mathcal{B})\} &=\min_{\mathcal{A},\mathcal{B}}\{\mathcal{L}(\omega_1A^{1}B^{1},\ldots,\omega_nA^{n}B^{n})\}\\ &=\min_{\mathcal{A},\mathcal{B}}\left\{\mathcal{L}\left(\sum_{i=1}^{n}\omega_i\cdot\{0,\ldots,A^{i}B^{i},\ldots,0\}\right)\right\}\\ &\le\min_{\mathcal{A},\mathcal{B}}\left\{\sum_{i=1}^{n}\omega_i\mathcal{L}(\mathcal{A}^{i}\cdot\mathcal{B}^{i})\right\}\\ &=\sum_{i=1}^{n}\omega_i\min_{A^{i},B^{i}}\{\mathcal{L}(0,\ldots,A^{i}B^{i},\ldots,0)\}. \end{aligned}

Thus, the original fine-tuning problem is upper-bounded by a weighted sum of nn single-layer fine-tuning problems. Minimizing this upper bound over ω∈Δn−1\boldsymbol{\omega}\in\Delta_{n-1} allocates weight to the layer with the smallest single-layer minimum: the ii-th coordinate of the resulting ω\boldsymbol{\omega} exceeds the jj-th if and only if

min⁡Ai,BiL(Ai⋅Bi)≤min⁡Aj,BjL(Aj⋅Bj),\min_{A^{i},B^{i}}\mathcal{L}(\mathcal{A}^{i}\cdot\mathcal{B}^{i})\le\min_{A^{j},B^{j}}\mathcal{L}(\mathcal{A}^{j}\cdot\mathcal{B}^{j}),

since otherwise swapping their positions would yield a smaller upper bound. Reading this in reverse, components of ω\boldsymbol{\omega} rank the layers by their single-layer "fitting potential". SimplexLoRA uses this ranking as a proxy for layer importance and reallocates adapter rank accordingly: well-adapted layers receive higher ranks, weaker ones lower. We emphasize that this argument bounds the loss, not the rank distribution itself, so the connection to optimal ranks remains a heuristic; its empirical effectiveness is reported in § 3 and the ablation in § 4.3.

Note that to prevent ω\boldsymbol{\omega} from degenerating into a one-hot vector, which is a common issue in simplex-constrained optimization where a single component dominates, one can apply regularization. In our implementation, we use a standard Euclidean regularization term to encourage ω\boldsymbol{\omega} to remain closer to a uniform distribution across all components.

Adapter Reinitialization

To ensure that the reinitialized adapters do not introduce immediate perturbations to the model’s output, we follow the standard LoRA initialization scheme.

  • AiA^{i} initialized using the Kaiming uniform distribution [13] to preserve the variance of activations: Ai∼U(−din−0.5,din−0.5)A^{i} \sim U\left(-d_{in}^{-0.5},d_{in}^{-0.5}\right),

  • BiB^{i} initialized to zero.

This combination ensures that the initial product AiBi=0A^{i}B^{i} = 0, maintaining the model’s performance at the start of the final training stage while providing a stable gradient flow for the new parameters.

Simplex Projections

While softmax and weighted softmax transformations are commonly used for projecting onto the simplex, we observed severe performance degradation during training when applying these methods. Their behavior is highly sensitive to temperature scaling: low temperatures tend to produce nearly one-hot distributions, whereas high temperatures result in almost uniform outputs. Such degeneracies are particularly problematic in our setting, where the goal is to efficiently allocate the rank budget across multiple adapters without inducing excessive sparsity or uniformity. For these reasons, we utilize the standard Euclidean regularization in the optimization problem formulation equation (1), rather than alternatives such as KL divergence.

Experimental Details

The core algorithm is implemented in PyTorch using the Huggingface library. All experiments are conducted on NVIDIA A100-SXM4-80GB GPUs. Following standard LoRA practice, we use the scaling factor α/r\alpha/r with α=32\alpha= 32 and dynamically updated ranks rinewr_i^{\mathrm{new}} [16]. To support simplex projection and dynamic rank reallocation, we employ a modified AdamW optimizer [24] with specialized update steps for ω\omega, rank resizing, and a custom learning rate schedule.

Training and optimization hyperparameters for each experimental setup are summarized in Table 4, Table 5, Table 6, Table 7, and Table 8. The LR warm-up values reported in these tables refer to the learning-rate scheduler and are not SimplexLoRA rank-search hyperparameters. For the first table, the learning rate was either adopted from the original publications or selected via a small grid search. For the DeBERTa, Llama, and Qwen setups, the optimal configuration for each task was determined via an independent grid search over the specified hyperparameter ranges. For GSM8K, we use few-shot chain-of-thought prompting with flexible answer extraction during evaluation. All the results were averaged across 3 random seeds.

HyperparameterValue
OptimizerAdamW
β1,β2\beta_1, \beta_20.9,0.9990.9, 0.999
Model learning rateTuned (PEFT)
Learning rate (ω\omega)55
LR warm-up3030
Reg. weight decay00
Dropout0.050.05
Grad. accumulation66
Training budget512 iterations
Batch size3232
SchedulerLinear
Target layersQuery, Key, Value

Table 4. Hyperparameters for out-of-the-box comparison (Table 1).

HyperparameterValue / Grid
OptimizerAdamW
β1,β2\beta_1, \beta_20.9,0.9990.9, 0.999
Model learning rate{3,5,8,10,30,50,80,100}×10−5\{3, 5, 8, 10, 30, 50, 80, 100\}\times10^{-5}
Learning rate (ω\omega)55
LR warm-up{0,20,30,50,70,100,200}\{0, 20, 30, 50, 70, 100, 200\}
Reg. weight decay{0,100,10−1,…,10−6}\{0, 10^{0}, 10^{-1}, \ldots, 10^{-6}\}
Dropout0.050.05
Grad. accumulation66
Training budgettask-specific
Batch sizetask-specific
SchedulerTwo-phase (Linear)
Target layersAll linear layers
Task-specific (# epochs / batch size\#\ \text{epochs}\ /\ \text{batch size})
MNLI / SST-2 / QNLI / QQP(7/32),(10/32),(5/32),(5/32)(7 / 32), (10 / 32), (5 / 32), (5 / 32)
CoLA / RTE / MRPC(15/64),(50/64),(30/64)(15 / 64), (50 / 64), (30 / 64)

Table 5. Hyperparameters for DeBERTaV3base_{\mathrm{base}} experiments (Table 2).

HyperparameterValue / Grid
OptimizerAdamW
β1,β2\beta_1, \beta_20.9, 0.999
Model learning rate{3,5,8,10,30,50,80,100}×10−5\{3, 5, 8, 10, 30, 50, 80, 100\} \times10^{-5}
Learning rate (ω\omega)5
LR warm-up50
Reg. weight decay{0,100,10−1,…,10−6}\{0, 10^{0}, 10^{-1}, \ldots, 10^{-6}\}
Dropout0.05
Grad. accumulation1
Training budget1024 iterations
Batch size16
SchedulerTwo-phase (Cosine)
Target layersAll linear layers

Table 6. Hyperparameters for Llama 3.1 8B experiments (Table 2).

HyperparameterValue / Grid
OptimizerAdamW
β1,β2\beta_1, \beta_20.9, 0.999
Model learning rate{3,5,8,10,30,50,80,100}×10−5\{3, 5, 8, 10, 30, 50, 80, 100\} \times10^{-5}
Learning rate (ω\omega)5
LR warm-up350
Reg. weight decay{0,100,10−1,…,10−6}\{0, 10^{0}, 10^{-1}, \ldots, 10^{-6}\}
Dropout0.05
Grad. accumulation1
Training budget3500 iterations
Batch size4
SchedulerTwo-phase (Linear)
Target layersAll linear layers

Table 7. Hyperparameters for Qwen3-8B experiments (Table 3).

HyperparameterValue / Grid
OptimizerAdamW
β1,β2\beta_1, \beta_20.9, 0.999
Model learning rateTuned (PEFT)
Learning rate (ω\omega)5
LR warm-up50
Reg. weight decay10−410^{-4}
Model weight decay10−110^{-1}
Dropout0.05
Grad. accumulation16
Training budget15 epochs
Batch size8
SchedulerTwo-phase (Linear)
Beam size8
Max source / target length256/160
Target layersAll linear layers

Table 8. Hyperparameters for BART-large experiments (Table 3).

Use of AI Assistants

During the preparation of this work, the authors used a LLM to assist with several tasks. Specifically, the LLM was employed for code generation to create data visualization scripts, debugging existing code, and for proofreading the paper to improve grammatical accuracy and stylistic consistency. After using this tool, the authors reviewed and edited the content as needed for the final publication.

References

  1. [1]Aastik, Meghana Topu, Chinmay Kulkarni, and Pragya Paramita Sahu. NormAL LoRA: What is the perfect size? In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 19716–19731, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.1074. URL https://aclanthology.org/2025.findings-emnlp.1074/.DOI
  2. [2]AI@Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
  3. [3]Mathieu Blondel, Akinori Fujino, and Naonori Ueda. Large-scale multiclass support vector machine training via euclidean projection onto the simplex. In 2014 22nd International Conference on Pattern Recognition, pages 1289–1294, 2014. doi: 10.1109/ICPR.2014.231.DOI
  4. [4]Huandong Chang, Zicheng Ma, Mingyuan Ma, Zhenting Qi, Andrew Sabot, Hong Jiang, and H. T. Kung. ElaLoRA: Elastic and learnable low-rank adaptation for efficient model fine-tuning. arXiv preprint arXiv:2504.00254, 2025.arxiv.org/abs/2504.00254
  5. [5]Daqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358–2367, 2016. doi: 10.18653/v1/P16-1223.DOI
  6. [6]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.arxiv.org/abs/1905.10044
  7. [7]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  8. [8]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.arxiv.org/abs/2305.14314
  9. [10]Ilyas Fatkhullin, Jalal Etesami, Niao He, and Negar Kiyavash. Sharp analysis of stochastic optimization under global kurdyka-lojasiewicz inequality. Advances in Neural Information Processing Systems, 35:15836–15848, 2022.
  10. [11]Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024.arxiv.org/abs/2403.14608
  11. [12]Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024.arxiv.org/abs/2402.12354
  12. [13]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.arxiv.org/abs/1502.01852
  13. [14]Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTa: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021.
  14. [15]Oliver Hinder, Aaron Sidford, and Nimit Sohoni. Near-optimal methods for minimizing star-convex functions and beyond. In Conference on learning theory, pages 1894–1938. PMLR, 2020.arxiv.org/abs/1906.11985
  15. [16]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shen Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.arxiv.org/abs/2106.09685
  16. [17]Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732, 2023.arxiv.org/abs/2312.03732
  17. [18]Ahmed Khaled, Othmane Sebbouh, Nicolas Loizou, Robert M Gower, and Peter Richtárik. Unified analysis of stochastic gradient methods for composite convex and smooth optimization. Journal of Optimization Theory and Applications, 199(2):499–540, 2023.arxiv.org/abs/2006.11573
  18. [19]Bobby Kleinberg, Yuanzhi Li, and Yang Yuan. An alternative view: When does sgd escape local minima? In International conference on machine learning, pages 2698–2707. PMLR, 2018.arxiv.org/abs/1802.06175
  19. [20]Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023.arxiv.org/abs/2310.11454
  20. [21]Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdulrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.703.DOI
  21. [22]Muqing Liu, Chongjie Si, and Yuheng Jia. FlexLoRA: Entropy-guided flexible low-rank adaptation. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=tqnkbdYWm.arxiv.org/abs/2601.22905
  22. [23]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024.arxiv.org/abs/2402.09353
  23. [24]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  24. [25]Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. DoRA: Enhancing parameter-efficient fine-tuning with dynamic rank distribution. arXiv preprint arXiv:2405.17357, 2024.arxiv.org/abs/2405.17357
  25. [26]Hyeon-Woo Nam, Ye-Bin Moon, and Tae-Hyun Oh. Fed-Para: Low-rank hadamard product for communication-efficient federated learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=d71n4ftoCBy.
  26. [27]Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, 2018. doi: 10.18653/v1/D18-1206.

Paper details

vlnc:W_mprvzdt8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

a LLM

View statement
Looking for related papers…
Contents