betaFeedback

WorksComputer Science

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

Abstract

Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA applies adaptation throughout the transformer, despite heterogeneous contributions from its components. In this work, we conduct a systematic mechanistic interpretability analysis to understand how LoRA affects model internals. We observe that LoRA's adaptation effect is highly non-uniform, with significant component-wise and layer-wise sparsity. Most critically, we find that important layers are highly task-dependent, with limited overlap across tasks. Motivated by these insights, we propose Adaptive-LoRA, a calibration-driven method that automatically identifies task-specific layers via frozen-base zero-ablation, enabling efficient layer selection without exhaustive search. Extensive experiments on Qwen2.5-1.5B across 11 diverse reasoning tasks demonstrate that Adaptive-LoRA outperforms full-layer LoRA while using only 30% of layers, achieving a 3.5×3.5\times reduction in adapter parameters. This work bridges mechanistic interpretability and practical efficiency, providing both fundamental insights and a principled approach for adaptive fine-tuning.

Introduction

Large language models (LLMs) have achieved remarkable capabilities across diverse tasks, driving widespread adoption in both research and deployment [9, 28]. Adapting these models to specific downstream domains, however, remains computationally expensive, motivating parameter-efficient fine-tuning (PEFT) methods. Among these, Low-Rank Adaptation (LoRA) [13] has become the de facto standard, enabling adaptation through low-rank weight updates while keeping original parameters frozen.

Standard all-layer LoRA uses the same adaptation rank across transformer layers, despite their potentially unequal contributions. Layer-wise structured pruning studies [22] identify substantial variation in layer importance. Meanwhile, mechanistic work on induction heads [27] links specific attention circuits to in-context learning. These observations raise a fundamental question: if layers are not equally important for inference, why should they be equally adapted during fine-tuning?

LoRA variants improve adaptation through rank allocation [39], learning-rate optimization [11], and dynamic rank search [35]. We study a complementary question: can interventions on a frozen base model identify task-specific layers worth adapting before training begins? This connects the internal importance of model components to the allocation of adaptation capacity.

We address this question through mechanistic interpretability, building on studies of internal representations [8] and behavior-specific circuits [36]. Specifically, we employ path patching [36] to

40th Conference on Neural Information Processing Systems (NeurIPS 2026). quantify each layer’s causal contribution to task performance, providing a principled alternative to heuristic selection. Our investigation is guided by two research questions:

RQ1: How do different components and layers contribute to LoRA’s adaptation effect, and is this contribution uniform across the model?

RQ2: Are the important layers consistent across different tasks, or are they highly task-dependent?

Our mechanistic analysis on Qwen2.5-0.5B across four diverse datasets yields two key insights: (i) LoRA’s adaptation is highly non-uniform, with MLP components dominating over attention (58.4% vs. 41.6%) and contributions concentrated in early layers; (ii) important layers exhibit significant task-dependence, with only 39.5% average Jaccard similarity across tasks, challenging the uniform adaptation paradigm.

Building on these insights, we propose Adaptive-LoRA, a calibration-driven method that automatically identifies task-specific layers via frozen-base zero-ablation with a default calibration size of NN=30. Adaptive-LoRA operates in three phases (calibration, selection, and training) and can be adapted to different efficiency-performance trade-offs. Extensive experiments on Qwen2.5-1.5B [31] across 11 reasoning tasks demonstrate that Adaptive-LoRA outperforms full-layer LoRA baselines while using only 30% of layers, achieving a 3.5× reduction in adapter parameters.

Our contributions are:

  • We provide the first systematic mechanistic analysis of LoRA’s internal mechanisms, revealing significant heterogeneity in component-wise and layer-wise contributions.

  • We demonstrate that important layers are highly task-dependent (39.5% overlap), challenging the uniform adaptation paradigm.

  • We propose Adaptive-LoRA, a principled calibration-driven approach for task-specific layer selection via frozen-base zero-ablation, informed by path-patching analysis.

  • We empirically validate that Adaptive-LoRA outperforms full-layer baselines with 70% parameter reduction across 11 diverse reasoning tasks.

Related Work

Parameter-Efficient Fine-Tuning. Parameter-efficient fine-tuning (PEFT) aims to adapt large pre-trained models with minimal computational overhead. LoRA [13] approximates weight updates with low-rank matrices, significantly reducing trainable parameters. Subsequent work has focused on rank allocation [39], learning rate optimization [11], dynamic rank search [35], and weight decomposition [21]. Other PEFT paradigms include prefix tuning [17], adapters [12], and prompt tuning [16]. Continual-learning methods further structure low-rank updates through subspace constraints (InfLoRA [18]), hierarchical layer-wise adapters (TreeLoRA [30]), and gating (GainLoRA [19]). Our complementary focus is task-specific layer selection before training, guided by frozen-base interventions.

Mechanistic Interpretability. Mechanistic interpretability seeks to reverse-engineer model behaviors into human-understandable algorithms [26]. Activation patching (causal tracing) [23] and circuit analysis [36] enable researchers to isolate the contribution of specific components to model outputs. Automated circuit discovery [6] further identifies compact computational subgraphs associated with model behaviors. Our work applies these tools to dissect LoRA’s adaptation mechanism, moving beyond black-box performance metrics to understand how and where adaptation occurs within the network.

Layer Selection and Pruning. Layer selection in transfer learning has been explored in various contexts. Analysis of BERT attention [15] reveals repeated attention patterns across heads and task-dependent differences in their effects on performance. In the context of PEFT, heuristic strategies such as fine-tuning only the last few layers or specific middle layers have been proposed [10]. However, these static strategies fail to account for task diversity. Our work introduces a calibration-driven, task-specific selection mechanism based on causal importance, providing a principled alternative to static heuristics or magnitude-based pruning.

Mechanistic Analysis of LoRA

To understand the internal mechanisms of LoRA, we conduct a systematic analysis using activation ablation and path patching. We quantify the contribution of different model components across four diverse datasets: Alpaca-ZH (Chinese instruction), Alpaca-EN [34] (English instruction), Code-Alpaca [2] (code generation), and Dolly [7] (dialogue).

Experimental Setup

Task Formulation. We decompose LoRA’s effect along three dimensions: component type (MLP vs. Attention), projection type (Q/K/V), and layer depth (early/middle/late). This multi-granularity analysis enables us to pinpoint where adaptation occurs within the transformer architecture.

Model and Data. We use Qwen2.5-0.5B [31] (24 layers) as the base model for mechanistic analysis. Full LoRA adapters (rank=8, α=16\alpha=16, applied to Q/K/V projections) are trained on four datasets: Alpaca-ZH (52K Chinese instructions), Alpaca-EN (52K English instructions), Code-Alpaca (20K code generation), and Dolly (15K dialogue).

Contribution Measurement. We employ activation patching [23] to measure causal importance. For input xx and component cc, the contribution score measures the prediction change when component cc is intervened on:

Sc(x)=∣pLoRA(t∗∣x)−pablate(c)(t∗∣x)∣(1)S_c(x) = \left|p_{\mathrm{LoRA}}(t^* \mid x) - p_{\mathrm{ablate}(c)}(t^* \mid x)\right| \tag*{(1)}

where t∗t^* is the top predicted token under the full LoRA model. The component importance aggregated over calibration set DcalD_{\mathrm{cal}} is:

Ic=1∣Dcal∣∑x∈DcalSc(x)(2)I_c = \frac{1}{\lvert D_{\mathrm{cal}}\rvert}\sum_{x\in D_{\mathrm{cal}}} S_c(x) \tag*{(2)}

Component Contribution Analysis

A fundamental question is whether all model components contribute equally to adaptation. Prior work hypothesizes that MLPs store factual knowledge [23], while attention handles information routing. We investigate whether this functional separation holds during LoRA adaptation. Specifically, for each layer we intervene on each layer’s MLP or attention block output independently and measure the resulting prediction shift using Equation (1). LoRA adapters reside on Q/K/V; the MLP measurement captures mediation through the block.

As shown in Figure 1, MLP layers consistently contribute more than attention across all four datasets (58.4% vs. 41.6%), with remarkably stable ratios (57.3%–59.5%). This stability is noteworthy: despite the datasets spanning different languages (Chinese, English), modalities (code), and interaction styles (dialogue), the MLP-attention contribution ratio remains nearly constant. Q/K/V updates change the attention output, which enters the residual stream and is transformed by downstream MLP blocks. A separate output-replacement experiment isolates this pathway: we record block outputs in an adapter-disabled run, replace one block output in the adapted run with its reference, and retain the other adapted computations. MLP mediation is reproduced across two adapter configurations (Appendix § 10.4).

Figure 1. MLP vs. Attention. MLP dominates at 58.4% across all four datasets.

Finding 1: MLP Dominance. MLP blocks mediate 58.4% of the measured component effect vs. 41.6% for attention, with task-specific shares of 57.3%–59.5%.

Projection Analysis

Within attention’s 41.6% contribution, we further decompose the effect across Query (Q), Key (K), and Value (V) projections. For each attention layer, we selectively ablate the LoRA adapter applied to one projection while preserving the other two, then measure the resulting prediction shift. decomposition reveals which aspects of attention, namely routing (Q/K) versus content (V), drive the adaptation effect.

Figure 5 (Appendix § 12) shows that Value (V) accounts for approximately 46% of the normalized attention effect, compared with approximately 27% each for Q and K. V remains the largest individual projection share across the four datasets (44.3%–47.7%). Because V determines the content aggregated by attention, this pattern motivates studying content-pathway adaptation. The projection decomposition complements the downstream layer-selection experiments by identifying where the measured attention effect is concentrated.

Figure 5. Contribution of Q, K, V projections within attention across four datasets. Value (V) dominates at 46%, while Q and K contribute ∼27% each.

Finding 2: V Projection Dominance. Value (V) contributes 46% to attention adaptation, exceeding the individual Q (27%) and K (27%) shares, stable across all tasks.

This three-way decomposition also has implications for multi-adapter deployment. V adapters carry nearly half of the measured attention adaptation signal, making this pathway a natural target for more compact adapters. Their parameter cost is determined by the selected projections’ dimensions and ranks.

Layer-wise Contribution Patterns

We examine how adaptation contributions distribute across model depth by computing per-layer importance scores aggregated over all calibration samples. Each layer’s score reflects the total prediction shift under the component interventions, aggregating MLP and attention contributions at that depth.

Figure 2 presents the layer-wise contribution profiles. The task-specific rankings in Figure 3 further show non-uniform importance across depth. They combine a shared early-layer core with task-specific middle and late layers. Early changes can influence a longer downstream computation path, while later contributions depend on the target task. This motivates using measured layer importance rather than depth alone to allocate adaptation capacity; the fixed-budget downstream comparison in Section 5 tests the practical value of that choice.

Figure 2. Layer-wise contribution profiles across 24 layers and four datasets.

Task-specific layer importance heatmap

Figure 3. Task-specific layer importance heatmap. White stars mark the top-7 layers per task. Only L0–L2 are universally important; remaining selections are task-specific. Average Jaccard similarity: 39.5%.

Finding 3: Unequal Layer Contributions. Layer importance is non-uniform across depth, with both shared early-layer importance and task-specific contributions at later depths.

Task-Specific Layer Importance

Finding 3 identifies non-uniform importance across depth. We next examine whether the same layers are important for all tasks, or whether each task favors a distinct subset. If the former, a single fixed layer selection could serve all tasks; if the latter, task-specific selection is essential. We compare the top-7 layers (30% of 24) selected for each dataset pair using Jaccard similarity:

J(Li,Lj)=∣Li∩Lj∣∣Li∪Lj∣(3)J(\mathcal{L}_i,\mathcal{L}_j) = \frac{|\mathcal{L}_i \cap\mathcal{L}_j|}{|\mathcal{L}_i \cup\mathcal{L}_j|} \tag*{(3)}

Figure 3 reveals distinct task-specific patterns. The mean pairwise Jaccard similarity is 39.5%, averaging intersection-over-union across the six task pairs. Code-Alpaca selects middle layers L10, L12, and L14; Alpaca-ZH includes late layers L17 and L23; Dolly selects L0, L1, L2, L7, L10, L12, and L21. Only L0–L2 are shared by all four top-7 sets. Thus the measured rankings combine a common core with task-specific selections, motivating calibration for each target distribution rather than a single fixed subset.

Finding 4: Task-Specific Layer Importance. Important layers vary substantially across tasks (Jaccard = 39.5%), implying that a fixed layer subset cannot optimally serve all downstream tasks.

Adaptive-LoRA

Motivated by Finding 4, that important layers vary significantly across tasks, we propose Adaptive-LoRA, a calibration-driven method for task-specific layer selection. Unlike static heuristics or exhaustive search, our approach automatically identifies critical layers via frozen-base zero-ablation on a small calibration set with N=30N=30 inputs by default. The approximately 30%-layer experiments achieve competitive performance with about 3×3 \times fewer adapter parameters.

Problem Formulation

Let M\mathcal{M} denote a pre-trained transformer with LL layers, and Dtrain\mathcal{D}_{\mathrm{train}} a task-specific training set. Standard LoRA applies adapters to all layers {0,…,L−1}\{0,\ldots,L-1\}, yielding a trainable parameter count proportional to LL at fixed rank and per-layer target dimensions. Our goal is to identify a subset Lsel⊂{0,…,L−1}\mathcal{L}_{\mathrm{sel}} \subset\{0,\ldots,L-1\} of size k<Lk < L such that adapting only these layers achieves comparable performance while reducing parameters by ≈L/k\approx L/k.

The key challenge is efficiently estimating layer importance without exhaustive search over (Lk)\binom{L}{k} subsets. We address this through base-model interventions that measure each layer’s prediction sensitivity.

Method Overview

Adaptive-LoRA operates in three sequential phases, as summarized in Algorithm 1. Given base model M\mathcal{M}, calibration set Dcal\mathcal{D}_{\mathrm{cal}}, and budget kk, we: (1) measure layer importance on the frozen base model, (2) select the top-kk layers, and (3) fine-tune LoRA adapters on only the selected layers.

Algorithm 1 Adaptive-LoRA: Calibration-Driven Layer Selection for LoRA

Require: Base model M\mathcal{M} with LL layers, calibration set DcalD_{\mathrm{cal}}, budget kk, training set DtrainD_{\mathrm{train}}, rank rr; optional depth threshold d∗d^{*}

Ensure: Adapted model M′\mathcal{M}'

Phase 1: Calibration via Base-Model Zero-Ablation

  1. I←0∈RL\mathcal{I} \leftarrow0 \in\mathbb{R}^{L}

  2. for each input x∈Dcalx \in D_{\mathrm{cal}} do

  3. t∗←arg⁡max⁡tpM(t∣x);q←pM(t∗∣x)t^{*} \leftarrow\arg\max_{t} p_{\mathcal{M}}(t \mid x); q \leftarrow p_{\mathcal{M}}(t^{*} \mid x)

  4. for each layer l=0,…,L−1l = 0, \ldots, L - 1 do

  5. qlA←pM∣al←0(t∗∣x)q^{A}_{l} \leftarrow p_{\mathcal{M}\mid a_{l}\leftarrow0}(t^{*} \mid x) ▷ Attention-only intervention

  6. qlM←pM∣ml←0(t∗∣x)q^{M}_{l} \leftarrow p_{\mathcal{M}\mid m_{l}\leftarrow0}(t^{*} \mid x) ▷ Independent MLP intervention

  7. Il←Il+∣q−qlA∣+∣q−qlM∣\mathcal{I}_{l} \leftarrow\mathcal{I}_{l} + |q - q^{A}_{l}| + |q - q^{M}_{l}|

  8. end for

  9. end for

  10. I←I/∣Dcal∣\mathcal{I} \leftarrow\mathcal{I}/|D_{\mathrm{cal}}|

Phase 2: Task-Specific Layer Selection

  1. C←{0,…,L−1}\mathcal{C} \leftarrow\{0,\ldots,L-1\}; in efficiency mode, C←{l∈C:l≥d∗}\mathcal{C} \leftarrow\{l \in\mathcal{C}: l \ge d^{*}\}

  2. Lsel←TopK⁡l∈C(Il,k)\mathcal{L}_{\mathrm{sel}} \leftarrow\operatorname{TopK}_{l\in\mathcal{C}}(\mathcal{I}_{l}, k)

Phase 3: Selective LoRA Training

  1. Initialize rank-rr adapters in the selected layers; keep base weights frozen

  2. Fine-tune the selected adapters on DtrainD_{\mathrm{train}} to obtain M′\mathcal{M}'

  3. return M′\mathcal{M}'

Phase 1: Calibration via Base-Model Zero-Ablation

Base-Model Interventions. Calibration uses only the frozen base model M\mathcal{M}. For input xx, let t∗=arg⁡max⁡tpM(t∣x)t^{*} = \arg\max_{t} p_{\mathcal{M}}(t \mid x), and let ala_l and mlm_l be the attention and MLP outputs. Two independent runs set one output to zero:

plA(t∣x)=pM∣al←0(t∣x),plM(t∣x)=pM∣ml←0(t∣x).(4)p^{A}_{l}(t \mid x) = p_{\mathcal{M}\mid a_l\leftarrow0}(t \mid x), \qquad p^{M}_{l}(t \mid x) = p_{\mathcal{M}\mid m_l\leftarrow0}(t \mid x). \tag*{(4)}

The layer score sums their absolute effects and averages over inputs:

Il=1∣Dcal∣∑x∈Dcal[∣pM(t∗∣x)−plA(t∗∣x)∣+∣pM(t∗∣x)−plM(t∗∣x)∣].(5)\mathcal{I}_{l} = \frac{1}{|D_{\mathrm{cal}}|} \sum_{x\in D_{\mathrm{cal}}} \left[ |p_{\mathcal{M}}(t^{*} \mid x) - p^{A}_{l}(t^{*} \mid x)| + |p_{\mathcal{M}}(t^{*} \mid x) - p^{M}_{l}(t^{*} \mid x)| \right]. \tag*{(5)}

This requires no trained adapter. The trained-LoRA analysis in Equation (1) instead motivates the selection rule. Across five adapter configurations, base scores correlate with post-training contributions (ρ=0.630\rho= 0.630–0.8930.893; Appendix § 10.§ 3); the fixed-budget comparison in Section 5 tests their downstream selection value.

Efficiency via Small-Sample Calibration. In the Qwen-3B repeated-draw study, selection Jaccard increases from 0.524 at N=10N = 10 to 0.653 at N=30N = 30, then reaches 0.685 at N=100N = 100. The smaller gain beyond 30 inputs supports N=30N = 30 as a practical calibration default (Appendix § 13.§ 3).

A 500-item labeled check also finds similar downstream accuracy with predicted and gold reference tokens (Appendix § 13.4).

Phase 2: Layer Selection

Accuracy mode selects Lsel=TopK⁡(I,k)\mathcal{L}_{\mathrm{sel}} = \operatorname{TopK}(\mathcal{I}, k) over all layers; the main reasoning experiments use k≈0.3Lk \approx0.3L. Efficiency mode uses the same scores but selects within {l:l≥d∗}\{l : l \ge d^{*}\}, with at least kk eligible layers. The actual first selected layer is d=min⁡Lsel≥d∗d = \min\mathcal{L}_{\mathrm{sel}} \ge d^{*}. This mode exposes a frozen prefix for faster training; its speed–accuracy trade-off is measured separately (Appendix § 13.6). Both modes retain task-specific, calibration-driven selection.

Default Layer Budget. The main reasoning experiments retain the compact budget k≈0.3Lk \approx0.3L. For a new model, selecting the top half of layers provides a practical default. At rank 16 with 5,000 training examples, the same 50% layer ratio improves accuracy over full-layer LoRA on all eight evaluated models by 0.2–1.9 percentage points (Appendix § 14.4). This layer-count budget is distinct from the minimum-depth constraint d∗d^{*} used for efficiency mode.

Phase 3: Selective Adaptation

For a targeted projection Wl,p∈Rdout×dinW_{l,p} \in\mathbb{R}^{d_{\mathrm{out}} \times d_{\mathrm{in}}} in a selected layer, we initialize ΔWl,p=Bl,pAl,p\Delta W_{l,p} = B_{l,p} A_{l,p}, where Bl,p∈Rdout×rB_{l,p} \in\mathbb{R}^{d_{\mathrm{out}} \times r} and Al,p∈Rr×dinA_{l,p} \in\mathbb{R}^{r \times d_{\mathrm{in}}}. This accommodates unequal input and output dimensions. The target projection set is an experiment-specific choice; Finding 2 motivates a Q/VQ/V configuration. Unselected layers remain frozen. Training uses AdamW with learning rate 1×10−41 \times10^{-4}, rank r=16r=16, α=32\alpha=32, for 3 epochs with effective batch size 64. At equal rank and adapter capacity per layer, selecting kk of LL layers reduces adapter parameters by L/kL/k (Appendix § 11). The rank-16 main comparison adapts 8 of 28 layers, a 3.5×3.5\times reduction in LoRA-factor capacity at the same rank and projection targets (Table 1).

MethodRankLayersBudget %BoolQPIQASIQAHellaWinoARC-eARC-cOBQAAvg.GSM8KSVAMPAQUAAvg.
LoRA [13]r428/2810063.052.445.638.253.054.842.638.048.532.438.628.433.1
r828/2810063.257.852.444.654.260.448.243.853.138.242.831.637.5
r1628/2810063.464.858.646.855.465.252.448.656.942.646.434.241.1
r3228/2810063.866.262.448.256.072.660.853.260.448.250.637.845.5
LoRA+ [11]r428/2810063.454.846.239.453.455.643.839.249.534.239.429.034.2
r828/2810063.058.453.645.254.861.849.644.853.940.644.233.439.4
r1628/2810063.265.659.248.455.866.854.049.857.944.848.035.842.9
r3228/2810063.667.463.049.856.473.261.654.061.150.452.838.647.3
DoRA [21]r428/2810062.451.844.236.852.652.440.236.447.130.236.826.831.3
r828/2810063.056.450.642.853.658.846.442.051.736.441.230.436.0
r1628/2810063.262.456.445.254.663.850.847.255.540.844.633.239.5
r3228/2810063.465.060.847.455.870.458.251.859.146.048.836.443.7
Oursr48/2828.662.854.247.240.053.656.444.039.849.834.840.229.634.9
r88/2828.663.459.654.846.255.063.251.846.455.140.444.633.839.6
r168/2828.663.665.460.850.256.868.456.251.659.145.248.836.043.3
r328/2828.664.067.864.252.657.473.862.455.062.250.853.439.247.8

Table 1. Budget is the fraction of adapted layers. Accuracy comparison of LoRA variants on Qwen2.5-1.5B across 11 reasoning tasks. Commonsense: 10,000 training samples, 1,000 eval samples each (OBQA: 500). Math: 5,000 training samples with chain-of-thought prompting. Adaptive-LoRA selects top 30% layers (8/28) via frozen-base calibration with N=250N=250 inputs. Results use seed 42. Best average at each rank in bold.

Computational Overhead. Calibration uses O(NL)O(NL) gradient-free forward evaluations; the default N=30N=30 study takes 20–62 s on one A100. This is separate from trained-model mechanistic analysis. Training speed depends on selected-layer depth: all backbone layers remain in an uncached forward pass, while a fully frozen prefix needs no backward pass and can be reused across epochs for fixed inputs. The H200 study separately reports uncached steps, cached steady epochs, and three-epoch totals (Appendix § 13.6).

Experiments

We evaluate Adaptive-LoRA against strong baselines across diverse reasoning tasks, addressing three research questions: (RQ1) Does Adaptive-LoRA achieve superior performance with significantly fewer parameters? (RQ2) What is the optimal layer budget? (RQ3) Does calibration-based selection outperform heuristic baselines?

Experimental Setup

Model and Training. We use Qwen2.5-1.5B [31] (1.54B parameters, 28 layers) as the primary evaluation model. Training data: commonsense_170k (10,000 samples) and math_10k (5,000 samples) [14], 3 epochs, max 512 tokens.

Evaluation Tasks. We evaluate on 11 reasoning tasks: 8 commonsense (BoolQ [3], PIQA [1], Social IQa [33], HellaSwag [38], WinoGrande [32], ARC-Easy [4], ARC-Challenge [4], Open-BookQA [24]) and 3 mathematical (GSM8K [5], SVAMP [29], AQUA [20]). Results are reported as task-level accuracies.

Baselines. We compare against three LoRA variants: LoRA [13] (standard, r∈{4,8,16,32}r \in\{4, 8, 16, 32\}), LoRA+ [11] (differential learning rates), and DoRA [21] (weight-decomposed LoRA). We use these three baselines apply to Q, K, V in all 28 layers. Adaptive-LoRA uses frozen-base zero-ablation on 250 calibration samples to select the top 8 layers (30%).

Main Results

Table 1 presents comprehensive results across all methods and ranks on both commonsense and mathematical reasoning tasks. Despite using only 30% of trainable layers, Adaptive-LoRA consistently achieves the highest average accuracy at each rank while consuming ~3.5× fewer parameters. At r16, Adaptive-LoRA achieves 59.1% commonsense accuracy with adapters in 8 of 28 layers, outperforming the all-layer configurations of LoRA (56.9%), LoRA+ (57.9%), and DoRA (55.5%). At identical rank and target projections, the LoRA-factor capacity is 28/8 = 3.5 times smaller. At r32, Adaptive-LoRA reaches 62.2% commonsense average, surpassing LoRA+ r32 (61.1%) and LoRA r32 (60.4%) with the same 8-of-28 layer budget. For mathematical reasoning, Adaptive-LoRA achieves 47.8% at r32, exceeding LoRA+ (47.3%) and LoRA (45.5%) by a clear margin. On PIQA at r32, Adaptive-LoRA also exceeds LoRA+ (67.8% vs. 67.4%). Its consistent advantage in average performance across all ranks demonstrates the value of concentrating adaptation on informative layers, yielding better generalization with a fraction of the parameters.

At the same r16 setting, a supplementary single-task comparison gives 59.1% for Adaptive-LoRA versus 58.7% for TreeLoRA, 58.5% for GainLoRA, and 58.0% for InfLoRA, while adapting fewer layers (Appendix § 10.§ 18).

Adaptive-LoRA improves commonsense accuracy over full-layer LoRA by 1.3, 2.0, 2.2, and 1.8 percentage points at r4, r8, r16, and r32, respectively. The improvement persists across the tested capacity range and is largest at r16. These controlled downstream results support concentrating adaptation capacity on selected layers and are consistent with an implicit-regularization interpretation.

Generalization Across Model Scales

Table 2 evaluates the primary Qwen2.5-1.5B model (28 layers) alongside five additional models spanning two architecture families and multiple parameter scales. Qwen2.5-3B has 36 layers and is a separate configuration. For each model, we compare Adaptive-LoRA (adapting top 30% layers at r16) against full-layer LoRA (all layers at r16) on the same 8 commonsense reasoning benchmarks using 10,000 training samples.

ModelMethod#LayersBudget %RatioBoolQPIQASIQAHellaWinoARC-eARC-cOBQAAvg.Δ\Delta
Qwen2.5-0.5BLoRA24/24100.01.00×63.049.435.025.849.223.223.427.637.1–
Ours7/2429.23.43×63.250.837.428.250.625.024.828.438.6+1.5
Qwen2.5-1.5BLoRA28/28100.01.00×63.464.858.646.855.465.252.448.656.9–
Ours8/2828.63.50×63.665.460.850.256.868.456.251.659.1+2.2
Qwen2.5-3BLoRA36/36100.01.00×61.060.659.856.655.484.877.867.466.9–
Ours11/3630.63.27×62.473.859.468.259.683.677.266.468.8+1.9
LLaMA-3-3BLoRA28/28100.01.00×63.067.455.262.857.068.660.255.461.2–
Ours8/2828.63.50×63.668.860.664.257.869.461.857.262.4+1.2
Qwen3-8BLoRA36/36100.01.00×69.685.677.280.872.897.691.085.082.5–
Ours11/3630.63.27×68.886.276.481.273.497.290.885.682.5+0.0
Qwen3-32BLoRA64/64100.01.00×72.290.077.289.881.698.895.493.487.3–
Ours20/6431.23.20×73.891.476.287.881.298.694.693.087.1−0.2

Table 2. Budget is 100k/L100k/L; ratio is L/kL/k, the LoRA-factor reduction at equal per-layer capacity. Cross-model generalization of Adaptive-LoRA at rank 16. Each model uses top 30% layers selected via frozen-base calibration (N=30N=30 samples). Training: 10,000 samples, 3 epochs, seed 42. Adaptive-LoRA uses approximately 30% of the trainable parameters; supplementary three-run results are in Appendix § 10.§ 17. Best reported average per model in bold, including ties.

The Qwen3 models use the Qwen3 family [37].

Table 2 shows strong accuracy across model scales with approximately 30% of the trainable parameters. At seed 42, Adaptive-LoRA improves average accuracy by 1.2–2.2 percentage points on the four 0.5B–3B models. Qwen3-8B matches the reported full-layer average (82.5%), while Qwen3-32B reaches 87.1% versus 87.3%, a difference of 0.2 percentage points. These results demonstrate parameter-efficient adaptation: improved accuracy on the smaller models and close preservation of full-layer performance at 8B and 32B. Supplementary three-run evaluations in Appendix § 10.§ 17 report positive mean gains across all eight configurations in the paired-gain panel; the Qwen3-32B mean is +0.5 points at 5K training samples and approximately +0.2 points at 10K. An interesting trend emerges from the per-task breakdown: on knowledge-intensive tasks (ARC-Easy, ARC-Challenge), Adaptive-LoRA closely matches full-layer LoRA even at large scales, while on reasoning-heavy tasks (SIQA, HellaSwag), the selective approach occasionally surpasses the full-model baseline. This pattern is consistent with our hypothesis that reasoning tasks benefit from concentrated adaptation on core processing layers, whereas knowledge tasks distribute learning more uniformly. These results validate that mechanistic layer selection is a general principle applicable across model families, scales, and training regimes.

Ablation Study: Layer Selection Analysis

To address RQ2 and RQ3, we vary the layer budget kk from 4 to 36 and compare four strategies: Adaptive (calibration-based, ours), Random (uniform sampling), Sequential (first kk layers), and Reverse (last kk layers). Evaluation uses perplexity on Alpaca-ZH with identical hyperparameters (r=16r=16, 3 epochs, 3,000 samples). Figure 4 reveals that Adaptive-LoRA has its largest measured relative gain over Random at 12 layers, improving over Random by 25.0%, Sequential by 12.4%, and Reverse by 17.8%. Detailed numerical results are provided in Table 9 (Appendix).

Layer selection strategy comparison

Figure 4. Layer selection strategies. Largest relative gain over Random at 12 layers; all converge at 36. Shading marks intermediate budgets, not the minimum-PPL configuration.

# LayersAdaptiveRandomSeq.Rev.
421.0524.6221.0624.86
817.1119.4817.3219.79
1212.3216.4214.0615.00
1610.3111.8511.7112.03
209.049.849.7710.33
247.918.367.929.07
286.977.306.558.04
326.326.306.037.08
365.755.755.755.75

Table 9.

At 12 layers, the reported PPL gap to Random is 4.10. All strategies converge to identical performance (PPL=5.75) at 36 layers, providing the expected control when all strategies select the same complete layer set.

Calibration Cost and Stability. Across five independent calibration draws, four models attain selection Jaccard of 0.653–0.747. Calibration takes 29–62 s, or 4.1–6.3% of the reference full-layer training time. Appendix § 13 reports the costs (Table 13), six-model stability, and sample-size sweep. Appendix § 13.5 also evaluates calibration across instruction distributions.

ModelkkJaccardCalibration (s)Full LoRA (s)
Qwen-3B100.65345713
Qwen-7B160.682621,240
LLaMA-3B80.74729714
LLaMA-7B160.738581,180

Table 13. Calibration cost and stability. Selection Jaccard is evaluated across five calibration draws with N=30N = 30 inputs. Times use one A100; full-layer training uses 1,000 examples and three epochs. These are the layer budgets used in this study.

Finding 5: Selection at Limited Budgets. At 12 of 36 layers, Adaptive-LoRA reduces perplexity by 25.0% relative to Random at the same budget, its largest measured relative gain. Calibration is especially valuable when adaptation capacity is limited.

Discussion

Our results reveal several notable properties of calibration-based layer selection that merit further analysis. First, the parameter-efficient performance across six models spanning 0.5B to 32B parameters (Table 2) suggests that layer importance heterogeneity is a fundamental property of transformer architectures rather than an artifact of any particular model family. The fact that Adaptive-LoRA achieves competitive or superior performance with 30% of layers across both Qwen and LLaMA families indicates that the calibration signal captures genuinely causal structure in the adaptation process. Second, the gains of 1.3–2.2 percentage points across all four ranks in Table 1 support the value of selective adaptation across capacity levels. This pattern is consistent with implicit regularization through task-specific allocation of adapter capacity. Third, calibration uses N=30N = 30 inputs and gradient-free interventions, taking 20–62 s in the default A100 study. Smaller adapters directly reduce storage. Training acceleration additionally depends on depth and prefix reuse: accuracy mode prioritizes the unrestricted ranking, while efficiency mode offers a measured speed–accuracy operating point (Appendix § 13.6).

Conclusion

This work addresses a fundamental question in parameter-efficient fine-tuning: do all layers contribute equally to LoRA adaptation? Through systematic mechanistic analysis, we demonstrate that LoRA’s adaptation effect is highly non-uniform, with MLP components dominating over attention and contributions concentrated in early layers. Most critically, important layers are highly task-dependent, challenging the prevailing uniform adaptation paradigm. Motivated by these insights, we propose Adaptive-LoRA, a calibration-driven method that automatically identifies task-specific layers via frozen-base zero-ablation. Extensive experiments across 11 diverse reasoning tasks demonstrate that selective adaptation on 30% of carefully chosen layers outperforms full-layer LoRA baselines while achieving a 3.5× reduction in adapter parameters. By bridging mechanistic interpretability and practical efficiency, this work provides both fundamental insights into LoRA’s internal mechanisms and a principled approach for adaptive fine-tuning.

Acknowledgments and Disclosure of Funding

YZ. Huang and K. Wang would like to acknowledge the support in part from UKRI Innovate UK COMET projects (10099265), and UKRI under the Horizon Europe funding guarantee (EP/Y028031/1, EP/Y03743X/1), as part of the European Commission MSCA staff exchange COVER and HarmonicAI projects (101086228 and 101131117), as well as the NVIDIA Academic Grant Program. K. Wang would like to acknowledge the support in part by the Royal Society Industry Fellow scheme (IF\R2\23200104). YZ. Huang would like to appreciate the support of Research Conference Support Scheme (12334207). We also thank the anonymous reviewers for their constructive feedback.

This appendix provides supplementary material organized into seven sections. Appendix A discusses limitations and future directions. Appendix B presents detailed experimental results including per-task breakdowns, parameter efficiency analysis, layer selection ablations, and training dynamics in their specified model configurations. Appendix C provides full implementation details for reproducibility, covering hardware, training configuration, evaluation protocol, and statistical methodology. Appendix D contains additional visualizations of Q/K/VQ/K/V projection contributions and task-specific layer-selection overlap. Appendix E analyzes the sensitivity and cost of the calibration procedure. Appendix F compares layer importance patterns across model families and scales. Appendix G discusses broader societal impact.

Limitations and Future Directions

Selection Granularity. Our current approach operates at the layer level, selecting entire transformer blocks for adaptation. Layer-level selection achieves 3.5×3.5 \times efficiency over full-layer LoRA. However, Findings 1 and 2 suggest finer-grained selection could yield further gains: since MLP contributes 58.4% and V-projection dominates within attention, a component-level or even attention head-level selection strategy could explore further parameter savings by choosing projections within the selected layers. Implementing such fine-grained selection would require extending the calibration procedure to score individual sub-components, which increases the calibration cost from O(L)O(L) to O(L×C)O(L \times C) where CC is the number of components per layer. We leave this exploration to future work.

Task Coverage. We evaluate on 8 commonsense reasoning and 3 math reasoning tasks, which primarily test factual recall and logical reasoning. Generative tasks (dialogue, creative writing, summarization) may exhibit different layer importance patterns, as they rely more heavily on language modeling capacity in later layers. Similarly, long-context scenarios (document QA, multi-turn dialogue) could shift importance toward middle layers that handle cross-position information routing. Extending our mechanistic analysis to these task families would test the generality of Findings 3–4 and potentially reveal new adaptation patterns.

Theoretical Foundations. While our empirical findings are consistent and reproducible across architectures, we currently lack a formal theoretical framework explaining why early layers dominate adaptation importance (Finding 3) or why task-specific patterns emerge at specific depths (Finding 4). Information-theoretic analysis of the residual stream, possibly through the lens of mutual information between layer activations and task labels, could formalize these observations. Such theoretical grounding would enable principled layer selection without calibration data, further reducing the adaptation overhead.

Dynamic Layer Selection. Our method selects layers once before training and keeps them fixed throughout. An adaptive approach that adjusts layer selection during training (e.g., re-calibrating every kk epochs) could potentially capture evolving importance patterns as the model learns. The repeated-draw results in Table 14 measure robustness to calibration sampling; evaluating evolving rankings during training is a separate direction.

ModelNNkkJaccard
Qwen-0.5B3070.653
Qwen-1.5B3080.617
Qwen-3B30100.653
Qwen-7B30160.682
LLaMA-3B3080.747
LLaMA-7B30160.738
Qwen-3B10100.524
Qwen-3B30100.653
Qwen-3B50100.671
Qwen-3B100100.685

Table 14. Selection stability across five independent calibration draws. The first group compares six models at N=30N = 30; the second varies NN for Qwen-3B at k=10k = 10. The shared N=30N = 30 row is repeated for comparison.

Detailed Experimental Results

Detailed Task-wise Performance

Table Table 3 shows detailed commonsense reasoning results across all four ranks for the four methods on Qwen2.5-3B (the generalization experiment model). Adaptive-LoRA achieves the highest average accuracy across all ranks, outperforming baselines that use 3.3×3.3\times more parameters.

MethodRankLayersBudget %BoolQPIQASIQAHellaWinoARC-eARC-cOBQAAvg.
LoRA [13]r436/3610062.868.247.268.058.880.671.863.465.1
LoRA [13]r836/3610064.276.053.867.254.074.469.259.464.8
LoRA [13]r1636/3610061.060.659.868.655.484.877.867.466.9
LoRA [13]r3236/3610060.874.261.866.660.086.480.268.869.9
LoRA+ [11]r436/3610064.272.450.267.859.476.268.460.264.9
LoRA+ [11]r836/3610063.875.855.665.457.678.072.662.466.4
LoRA+ [11]r1636/3610063.072.456.268.660.280.674.864.067.5
LoRA+ [11]r3236/3610062.273.257.867.460.882.476.065.868.2
DoRA [21]r436/3610060.068.848.662.258.669.468.659.662.0
DoRA [21]r836/3610062.274.656.466.656.877.273.862.866.3
DoRA [21]r1636/3610061.667.858.865.257.279.477.065.266.5
DoRA [21]r3236/3610061.072.861.466.858.680.276.465.467.8
Oursr411/3630.663.274.652.867.459.279.871.462.866.4
Oursr811/3630.664.476.255.267.658.080.273.663.067.3
Oursr1611/3630.662.473.859.468.259.683.677.266.468.8
Oursr3211/3630.664.476.262.069.061.086.680.469.071.1

Table 3. Budget is the fraction of adapted layers. Detailed commonsense reasoning results on Qwen2.5-3B (generalization experiment, Section 5). All tasks evaluated on 1,000 samples (Open-BookQA: 500).

Table Table 4 presents math reasoning breakdown on Qwen2.5-3B. Adaptive-LoRA achieves strong math performance (62.7% at r32) while using only 30% of parameters, outperforming LoRA (58.5%) and DoRA (58.1%) at the same rank.

MethodRankGSM8KSVAMPAQUAAvg.
LoRA [13]r458.257.448.554.7
LoRA [13]r863.662.852.459.6
LoRA [13]r1654.657.649.854.0
LoRA [13]r3262.060.453.258.5
LoRA+ [11]r458.257.448.854.8
LoRA+ [11]r862.861.652.058.8
LoRA+ [11]r1663.262.452.659.4
LoRA+ [11]r3264.663.853.260.5
DoRA [21]r443.853.242.646.5
DoRA [21]r861.662.651.858.7
DoRA [21]r1652.455.647.251.7
DoRA [21]r3260.261.852.458.1
Oursr461.260.450.857.5
Oursr865.064.253.660.9
Oursr1664.665.054.861.5
Oursr3266.465.856.062.7

Table 4. Detailed math reasoning results on Qwen2.5-3B (generalization experiment). Chain-of-thought prompting.

Per-Task Performance Analysis

Table Table 5 summarizes the best score over the four tested ranks for each method and task on Qwen2.5-3B. These values are drawn from Table 3 and Table 4, rather than an additional experiment. Adaptive-LoRA has the highest per-task best score on all 11 tasks, with a mean gap of approximately 0.8 percentage points over the strongest baseline. The rank producing each score is displayed explicitly.

TaskLoRARankLoRA+RankDoRARankOursRankBestGap
BoolQ64.2r864.2r462.2r864.4r8Ours+0.2
PIQA76.0r875.8r874.6r876.2r8Ours+0.2
Social IQa61.8r3257.8r3261.4r3262.0r32Ours+0.2
HellaSwag68.6r1668.6r1666.8r3269.0r32Ours+0.4
WinoGrande60.0r3260.8r3258.6r461.0r32Ours+0.2
ARC-Easy86.4r3282.4r3280.2r3286.6r32Ours+0.2
ARC-Chall.80.2r3276.0r3277.0r1680.4r32Ours+0.2
OBQA68.8r3265.8r3265.4r3269.0r32Ours+0.2
GSM8K63.6r864.6r3261.6r866.4r32Ours+1.8
SVAMP62.8r863.8r3262.6r865.8r32Ours+2.0
AQUA53.2r3253.2r3252.4r3256.0r32Ours+2.8
Avg. Gap+0.8

Table 5. Per-task best performance (any rank) for each method on Qwen2.5-3B (generalization experiment). Gap = Adaptive-LoRA − best baseline. Note: baselines use 3.3× more parameters at each rank.

Base Importance and Adaptation Value

Base calibration and trained-adapter analysis measure different interventions. On the same 100 inputs across 24 layers of Qwen2.5-0.5B, their layer rankings agree across five adapter configurations (Table 6). Recovery is the intersection of two top-kk sets divided by kk, with k=7k = 7 and 12; it differs from Jaccard’s union denominator. Pearson correlations of the raw scores range from 0.935 to 0.975.

TargetsRankSpearman ρ\rhoppTop-7 recoveryTop-12 recovery
Q/K/V80.6309.6×10−49.6 \times10^{-4}5/79/12
Q/K/V160.6358.6×10−48.6 \times10^{-4}5/710/12
Q/V40.8934.4×10−94.4 \times10^{-9}5/710/12
Q/V160.8915.2×10−95.2 \times10^{-9}5/710/12
Q/V320.8674.3×10−84.3 \times10^{-8}5/710/12

Table 6. Base-model importance and trained-adapter contribution agree. Qwen2.5-0.5B, 24 layers, 100 common inputs. The first row uses the original analysis configuration (r=8r = 8, α=16\alpha= 16).

The downstream test holds the training and layer budgets fixed. At 12 of 36 layers on Alpaca-ZH, calibrated selection reaches 12.32 PPL versus 16.42 for Random, 14.06 for Sequential, and 15.00 for Reverse (Table 9). At full coverage all methods select the same set and reach 5.75. Together, the ranking agreement and fixed-budget result support using base importance to identify layers worth adapting.

MLP Mediation under Adapter-Disabled Output Replacement

To isolate the path through a block, we record its output in an adapter-disabled reference run on the same input sequence. In the fully adapted run, we replace only that block’s output with the recorded reference and recompute the downstream prediction, retaining the other adapted computations. Thus an MLP block can mediate adaptation even though its own weights have no adapter: Q/K/V updates change the residual stream entering it. This replacement is distinct from zeroing a base-model block during layer selection.

Table 7 gives the MLP shares from this experiment on Qwen2.5-0.5B with Q/K/V adapters, alongside the no-LoRA reference measurements. The primary configuration is r=8r = 8, α=16\alpha= 16; an independent r=16r = 16, α=32\alpha= 32 adapter reproduces the MLP-majority pattern. Each dataset uses 100 inputs. The rank-8 comparisons to the no-LoRA reference use 5,000 paired input-bootstrap resamples; their 95% intervals are intervals for the gap, in percentage points, not for the original Figure 1 shares. The original component analysis and this replacement experiment are reported separately.

Parameter Efficiency Analysis

DatasetMLP share (%) r8MLP share (%) r16No-LoRA referenceGap (pp) r8Gap (pp) r16r8 gap 95% CI
Alpaca-EN62.462.558.1+4.3+4.4[2.6, 6.1]
Code-Alpaca63.863.060.0+3.8+3.0[2.4, 5.2]
Dolly62.562.256.2+6.3+6.0[4.6, 8.0]

Table 7. MLP mediation in two adapter configurations. Shares are percentages. Gaps compare each adapted measurement with the no-LoRA reference; intervals apply only to the rank-8 gap.

Table 8 combines the reported commonsense scores with a structural count of the Q/K/V LoRA factors at rank 32 on Qwen2.5-1.5B.

MethodAdapted layersA/B factors (M)CS Avg. (%)
LoRA28/285.96460.4
LoRA+28/285.96461.1
DoRA28/285.96459.1
Adaptive-LoRA8/281.70462.2

Table 8. LoRA-factor accounting for Qwen2.5-1.5B at r=32r = 32 with Q/K/V targets. Counts are calculated from the model dimensions, not measured training-time parameter inventories. Accuracies are from Table 1.

A/B counts exclude method-specific extra parameters such as DoRA’s magnitude vectors.

For the published Qwen2.5-1.5B configuration [31], the hidden width is 1536, with 12 query heads and 2 key/value heads of width 128. Thus Q has output width 1536 and K/V each have width 256. The low-rank factor count per layer is r[(1536+1536)+2(1536+256)]=6656rr[(1536 + 1536) + 2(1536 + 256)] = 6656r. At rank 16 this gives 2,981,888 factor parameters across 28 layers or 851,968 across 8 layers; at rank 32 the counts double. These architecture-derived counts quantify LoRA-factor capacity and exclude method-specific additional parameters.

The L/kL/k reduction also applies to storage of equally quantized LoRA factors. For example, 100 rank-32 Q/K/V adapters under the configuration above require approximately 1.19 GB for full-depth A/B factors versus 0.34 GB for eight-layer factors in FP16. This calculation excludes base-model weights, KV caches, and method-specific parameters.

Training Dynamics

We examine the training loss curves to understand how selective layer adaptation affects convergence behavior. Training on the commonsense split (10,000 samples, 3 epochs) with Qwen2.5-1.5B at r16, we record per-step training loss for all four methods.

Adaptive-LoRA converges faster than full-layer baselines during the first epoch, reaching a training loss of 0.82 after 500 steps compared to 0.91 for LoRA and 0.89 for LoRA+. This accelerated convergence is consistent with the implicit regularization hypothesis discussed in Section 6: by restricting adaptation to causally important layers, Adaptive-LoRA avoids the initial exploration phase where full-layer methods allocate capacity to non-productive parameters. After epoch 1, all methods converge to similar final training loss values (0.42–0.45), but the evaluation accuracy divergence persists and widens, suggesting that Adaptive-LoRA achieves better generalization rather than merely better training fit.

Notably, the gradient norms for selected layers in Adaptive-LoRA are 1.4–1.8× larger than the corresponding layers in full LoRA, consistent with the concentrated adaptation hypothesis. The non-selected layers in full LoRA receive gradient magnitudes 2–5× smaller, indicating uneven gradient magnitudes across the measured layers.

Layer Selection Analysis

Table 9: Layer selection PPL comparison. The largest relative reduction over Random is 25.0% at 12 layers. Absolute perplexity continues to decrease, reaching 5.75 for all strategies at 36 layers.

Multi-Seed Evaluation

Table 10 supplements the single-seed main results with repeated-run evaluations. In panel (a), all eight configurations have positive mean gains, ranging from 0.17 to 1.57 percentage points. For Qwen3-32B, the 5K and 10K training configurations yield mean gains of 0.50 and 0.23 points, respectively; these are distinct from the seed-42 value in Table 2. Panel (b) reports absolute scores: Adaptive-LoRA reaches mean accuracies of 59.63%, 82.50%, and 71.50%, compared with 58.13%, 80.87%, and 70.30% for full-layer LoRA. The two panels retain their respective configuration and run labels and are summarized separately.

(a) Paired gains: seeds 42, 123, and 7
Model / rank / samplesSeed 42Seed 123Seed 7Mean ± SD
Qwen-3B / r16 / 10K+1.5+1.2+1.8+1.50 ± 0.30
Qwen-7B / r4 / 10K+1.6+1.3+1.8+1.57 ± 0.25
Qwen-7B / r16 / 10K+1.7+1.4+1.5+1.53 ± 0.15
Qwen3-8B / r16 / 10K+0.0+0.3+0.2+0.17 ± 0.15
Qwen3-32B / r16 / 5K+0.0+0.8+0.7+0.50 ± 0.44
Qwen3-32B / r16 / 10K-0.2+0.5+0.4+0.23 ± 0.38
LLaMA-7B / r4 / 10K+1.2+1.0+1.4+1.20 ± 0.20
LLaMA-13B / r4 / 10K+1.2+1.0+1.3+1.17 ± 0.15
(b) Absolute accuracy: three reported runs
ModelMethodRun 1Run 2Run 3Mean ± SD
Qwen-3BFull LoRA58.357.858.358.13 ± 0.29
Adaptive-LoRA59.859.060.159.63 ± 0.57
Qwen-7BFull LoRA80.881.080.880.87 ± 0.12
Adaptive-LoRA82.582.282.882.50 ± 0.30
LLaMA-7BFull LoRA70.270.570.270.30 ± 0.17
Adaptive-LoRA71.571.271.871.50 ± 0.30

Table 10. Supplementary three-run evaluations.

Table 10: Supplementary three-run evaluations. (a) Paired accuracy gains of Adaptive-LoRA over full-layer LoRA, in percentage points. (b) Absolute accuracy (%) at r16 with 10K training samples. Summaries are mean ± sample SD, computed separately from each row’s displayed values.

Panel (a) retains the reported seed IDs; panel (b) retains run labels 1–3. The panels are summarized independently, without assuming a run-by-run correspondence between them. SD describes variation across three runs, not a confidence interval. Bold marks the higher mean in each model pair in (b).

Comparison with Continual-Learning LoRA Methods

We compare with InfLoRA, GainLoRA, and TreeLoRA in a single-task setting (t=1t = 1), using Qwen-1.5B, rank 16, and 10,000 training samples. These methods structure low-rank updates differently, whereas Adaptive-LoRA selects layers before training. Table 11 reports commonsense average accuracy and adapted-layer counts. Adaptive-LoRA reaches 59.1%, exceeding the strongest of these three baselines, TreeLoRA (58.7%), while adapting 8 rather than 28 layers.

MethodAdapted layersAvg. (%)
Full LoRA28/2856.9
InfLoRA [18]28/2858.0
GainLoRA [19]28/2858.5
TreeLoRA [30]28/2858.7
Adaptive-LoRA8/2859.1

Table 11. Single-task comparison with continual-learning LoRA methods. Qwen-1.5B, r=16r = 16, 10,000 training samples; commonsense average accuracy.

Single-task instantiation. With no earlier tasks, InfLoRA retains its data-derived, fixed input projection and trains the output projection; TreeLoRA retains layer-wise adapters and sparse updates, with no earlier branch to regularize against; GainLoRA retains a single adapter branch and its input-dependent gate. Previous-task protection has no history to act on at t=1t = 1. This comparison evaluates the resulting adaptation parameterizations on the present task.

Implementation and Reproducibility

Software and Hardware Environment

The experiments use PyTorch, Hugging Face Transformers, and PEFT. TransformerLens [25] supports the mechanistic analysis. The H200 measurements use the separately specified software configuration in Appendix § 13.6. Code release is planned under the MIT license.

Hardware. The original training and evaluation are conducted on 4×4 \times NVIDIA A100 40GB GPUs connected via NVLink, with 512GB system RAM and AMD EPYC 7763 CPUs. Frozen-base calibration runs on a single A100 GPU; the depth-controlled H200 timing protocol is specified in Appendix § 13.6.

Calibration Configurations and Parameter Counts. NN counts calibration inputs: the deployment default uses N=30N = 30, while the primary Qwen2.5-1.5B comparison in Table 1 uses N=250N = 250. Let TlT_l denote the target projections at selected layer ll. Counting the two low-rank factors gives

Padapter=∑l∈Lsel∑p∈Tlrl,p(din,l,p+dout,l,p).P_{\mathrm{adapter}} = \sum_{l \in\mathcal{L}_{\mathrm{sel}}} \sum_{p \in T_l} r_{l,p}(d_{\mathrm{in},l,p} + d_{\mathrm{out},l,p}).

Equal rank and adapter capacity across layers give the reduction factor L/kL/k. Target sets differ between the Q/K/V mechanism study and the Q/K/V or Q/V bridge configurations in Appendix § 10.§ 3.

Training Details

Random Seeds. The original main results in Table 1 and Table 2 use seed 42 for PyTorch, NumPy, and Python’s random module. CUDA deterministic mode is enabled via torch.backends.cudnn.deterministic = True. The supplementary evaluations in Appendix § 10.§ 17 list their seed or run labels explicitly.

Optimizer. AdamW optimizer with learning rate 1×10−41 \times10^{-4}, weight decay 0.01, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=1×10−8\epsilon= 1 \times10^{-8}. No learning rate warmup is used. Training runs for 3 epochs on all configurations.

Batch Size. Per-device batch size of 8 with gradient accumulation over 8 steps, yielding an effective batch size of 64 per training step (256 across 4 GPUs with DDP).

Precision. FP16 mixed precision via PyTorch AMP. LoRA adapters are initialized in FP32 and cast to FP16 during forward/backward passes. The base model weights remain frozen in FP16 throughout.

Data Preprocessing. All training samples are tokenized with a maximum sequence length of 512 tokens. Samples exceeding this length are truncated; shorter samples are padded to the nearest multiple of 8 for efficiency. Commonsense tasks use a multiple-choice format with answer extraction; math tasks use chain-of-thought prompting with answer parsing via regex.

Evaluation Protocol

Evaluation uses no in-context examples. Commonsense results use answer extraction and task-level accuracy. Mathematical reasoning uses chain-of-thought prompting followed by task-specific answer parsing; GSM8K and SVAMP require numerical answers, whereas AQUA is multiple choice. Decoding is greedy (temperature=0).

Statistical Methodology

Across-Run Summaries. In Table 10, for three displayed observations x1,x2,x3x_1, x_2, x_3, we report xˉ=13∑ixi\bar{x} = \frac{1}{3}\sum_i x_i and sample SD s=12∑i(xi−xˉ)2s = \sqrt{\frac{1}{2}\sum_i (x_i-\bar{x})^2}. Panel (a) summarizes paired accuracy differences; panel (b) summarizes each method’s accuracy. These across-run SDs are distinct from bootstrap intervals over evaluation examples and do not by themselves imply statistical significance.

Additional Visualizations

Q/K/V Projection Analysis

Figure 5 provides a detailed breakdown of how Q, K, and V projection adapters contribute to attention-mediated adaptation. The V projection consistently dominates across all four datasets, contributing 44.3%–47.7% of the total attention effect. The common ordering across these four datasets complements their different layer-selection patterns.

Task-Specific Layer Selection

Figure 6 visualizes the overlap between task-specific layer selections. Panel (a) shows the pairwise Jaccard matrix: the highest overlap is 75.0% between Alpaca-ZH and Alpaca-EN. The minimum, 27.3%, occurs for Alpaca-ZH/Code-Alpaca, Alpaca-ZH/Dolly, and Alpaca-EN/Code-Alpaca. The average over all six task pairs is 39.5%. Panel (b) provides a binary heatmap of which layers are selected for each task, making the task-specific divergence visually apparent.

Figure 6. (a) Jaccard similarity matrix of top-7 layers across tasks (avg. 39.5%). (b) Binary heatmap of layer selection: only L0–L2 are universally selected; task-specific layers emerge at different depths.

Calibration Analysis

Calibration Sample Sensitivity

We investigate the sensitivity of our frozen-base calibration to the number of calibration inputs NN. Using Qwen2.5-1.5B on the commonsense split, we vary N∈{10,20,30,50,100}N \in\{10, 20, 30, 50, 100\} and measure both the resulting layer selection and downstream task performance at r16.

Downstream accuracy spans 0.5 percentage points across this sample-size sweep, supporting N=30N = 30 as a useful cost–accuracy balance. Selection consistency across independent calibration draws is reported separately in Table 14.

Calibration Cost Breakdown

The calibration overhead consists of three components: (1) sampling NN calibration examples from the training set (negligible), (2) running O(NL)O(NL) gradient-free forward evaluations with independent component zero-ablation to compute layer importance scores (∼40 seconds for N=30N=30 on Qwen2.5-1.5B with a single A100), and (3) sorting and selecting top-kk layers (<1 ms). The approximately 40–42 s calibration cost is less than 1% of the 1.5–2 hours of fine-tuning in this configuration. Table 13 separately reports calibration alongside a shorter, 1,000-example training run. Importantly, the calibration requires no gradient computation and can be performed in FP16 with minimal memory footprint.

Stability across Calibration Draws

Stability across Calibration Draws

Table 14 repeats calibration with five independent draws. At N=30N = 30, selection Jaccard ranges from 0.617 to 0.747 across six models. The Qwen-3B sweep gains 0.129 from N=10N = 10 to N=30N = 30, versus 0.032 from N=30N = 30 to N=100N = 100. This provides additional evidence for a small calibration set. The layer budget kk is shown for each model; these experiments are separate from the fixed half-layer-budget study in Appendix § 14.4. Unlike Table 12, which compares each selection with an N=30N = 30 reference, this study measures stability across repeated calibration draws.

NNTime (s)CS Avg.Δ\Delta vs. N=30N=30
101459.4−0.4
202859.6−0.2
304259.80.0
506959.9+0.1
10013759.7−0.1

Table 12. Effect of calibration sample size NN on calibration time and downstream accuracy (Qwen2.5-1.5B, r16, commonsense average).

Table 14: Selection stability across five independent calibration draws. The first group compares six models at N=30N = 30; the second varies NN for Qwen-3B at k=10k = 10. The shared N=30N = 30 row is repeated for comparison.

Reference-Token Sensitivity

Calibration uses the frozen base model’s predicted token t∗t^{*} in Equation (5), so it requires no answer labels. On 500 labeled items, replacing this token with the gold-answer token yields a top-kk selection Jaccard of 0.81. The two reference choices yield downstream accuracies of 58.7% and 58.5%, differing by 0.2 percentage points. This check supports the predicted-token reference used by the method.

Cross-Distribution Calibration

We vary the calibration source and training distribution for Qwen-3B at rank 16 with 5,000 training samples (Table 15). Across the three cross-distribution settings, layer-selection Jaccard relative to matched calibration is 0.453–0.512, and reported accuracy is 72.8–73.5%. These results extend the calibration analysis beyond repeated draws from one distribution and show that useful layer selections transfer across the tested instruction distributions.

Calibration sourceTraining targetJaccardAccuracy (%)
Alpaca-ENAlpaca-EN (matched)1.00075.2
Alpaca-ENCode-Alpaca0.45372.8
Alpaca-ENDolly0.51273.1
Code-AlpacaAlpaca-EN0.48273.5
Random selection——70.2

Table 15. Cross-distribution calibration on Qwen-3B (r=16r = 16, 5K training samples). Jaccard compares the selected layers with calibration matched to the training distribution. The final row gives the reported random-selection reference.

Table 15: Cross-distribution calibration on Qwen-3B (r=16r = 16, 5K training samples). Jaccard compares the selected layers with calibration matched to the training distribution. The final row gives the reported random-selection reference.

Depth-Constrained Selection and Training Time

Both operating modes rank layers by the frozen-base score in Equation (5). Accuracy mode selects over all layers. Efficiency mode restricts the candidates to l≥d∗l \ge d^{*}, allowing the frozen prefix below the actual first selected layer d=min⁡Lseld = \min L_{\mathrm{sel}} to be reused. An uncached forward still evaluates all backbone layers. Frozen layers above an adapted layer must propagate gradients; only the entirely frozen prefix needs no backward graph.

For fixed processed inputs and a deterministic, dropout-free prefix, the depth-dd states are cached during the first epoch and reused subsequently. The host-memory cache scales as ntrainTdmodeln_{\mathrm{train}}T d_{\mathrm{model}} elements and is streamed per batch; ntrainn_{\mathrm{train}} denotes training examples, not the calibration size NN. Removing unselected adapter branches also saves their own forward, backward, and optimizer costs.

The controlled study uses a single H200, PyTorch 2.10, PEFT 0.18, bf16, AdamW, batch size eight, and sequence length 512. There is no gradient checkpointing, gradient accumulation, or DDP. Step measurements include forward, backward, and optimizer computation and are medians of 15 steps after five warm-ups. Table 16 separates same-batch steps without prefix reuse, steady epochs with reuse, and three-epoch training totals including first-epoch cache construction. These training timings do not include the separately measured A100 calibration stage.

ModelLLFull step (ms)d=0d=0 Stepd=0d=0 Steady ep.d=L/2d=L/2 Stepd=L/2d=L/2 Steady ep.d=L/2d=L/2 3 ep.
Qwen2.5-0.5B2462.221.1311.4121.4611.444
Qwen2.5-3B-Instruct36168.781.0971.4461.9001.720
Llama-3.2-3B28178.361.1051.4481.9141.729
Llama-3.1-8B32325.851.0781.4121.9341.722

Table 16. Training speed depends on selected-layer depth. Single H200; nominal k≈L/3k \approx L/3. Speedups are relative to full LoRA. Step columns exclude prefix reuse; steady epochs include it; three-epoch totals include cache construction.

At d=0d=0, speedups are 1.08–1.13×. At d=L/2d=L/2, the 3B and 8B models reach 1.90–1.93× in steady epochs and 1.72–1.73× over three epochs. These are efficiency-mode operating points; the unconstrained accuracy results use their own selected sets.

ModeddCode-Alpaca PPLAlpaca-EN PPLSpeedup
Accuracy (unconstrained)02.2944.8211.13×
Efficiency, d∗=L/4d^*=L/472.3014.8541.35×
Efficiency, d∗=L/2d^*=L/2122.3084.9071.46×
Deepest-kk reference162.3435.0311.61×

Table 17. Speed–accuracy operating points on Qwen2.5-0.5B (k=8k=8, r=16r=16, three epochs, two seeds). d∗d^* is the minimum allowed depth and dd the actual first selected layer. PPL is lower-is-better; speedup is steady-epoch training relative to full LoRA.

For d∗=L/4=6d^*=L/4=6, the actual first selected layer is d=7d=7. The two PPL increases relative to accuracy mode are 0.3% and 0.7%. At d∗=L/2d^*=L/2, the selected set is {12,13,14,15,20,21,22,23}\{12, 13, 14, 15, 20, 21, 22, 23\} and the increases are 0.6% and 1.8%. The deepest-eight reference has larger increases (2.2%, 4.4%) and higher speed; its actual depth is different. These operating points show the quality–speed choice enabled by the common calibration ranking.

Cross-Architecture Comparison

Layer Importance Patterns Across Families

We compare layer importance patterns across three architecture families: Qwen2.5 (0.5B, 1.5B, 3B), LLaMA-3.2 (3B), and LLaMA-2 (7B). Despite differences in depth, width, and training data, several consistent patterns emerge:

  1. Early layer dominance. In all architectures, the first 10–15% of layers consistently rank in the top-5 by importance score, supporting Finding 3. This includes the embedding projection and early attention layers that establish token representations.

  2. Middle layer task specialization. Layers at 30–60% depth show the highest variance in importance across tasks, consistent with Finding 4. These layers encode task-specific reasoning patterns that vary between commonsense and mathematical domains.

  3. Depth-scaled selection. The main comparison in Table 2 uses 7/24, 8/28, 11/36, and 8/28 layers for the 0.5B, 1.5B, Qwen2.5-3B, and LLaMA 3B models, respectively. These are approximately 30% layer budgets. The separate study below tests a fixed 50% budget.

Rank Scaling Behavior

Rank Scaling Behavior

On Qwen2.5-1.5B, Table 1 compares four ranks and yields commonsense gains of 1.3, 2.0, 2.2, and 1.8 percentage points over full-layer LoRA. Selective adaptation therefore improves accuracy throughout the tested rank range, with the largest gain at r16. The cross-model comparison in Table 2 evaluates r16. Together, these experiments establish complementary evidence across ranks and model scales while retaining the parameter savings of layer selection.

MLP vs. Attention Contribution by Depth

An important question for cross-architecture generalization is whether the MLP-dominance pattern (Finding 1) holds uniformly across layer depth or varies systematically. We decompose per-layer importance into MLP and attention components for Qwen2.5-1.5B and LLaMA-3-3B.

In both architectures, early layers (first 20%) show the highest MLP/Attention ratio (∼\sim65/35), consistent with the hypothesis that early layers primarily modify feed-forward features. Middle layers (20–60% depth) exhibit a more balanced ratio (∼\sim55/45), reflecting the co-importance of both information routing (attention) and feature transformation (MLP) for task-specific processing. Late layers (last 40%) return to MLP dominance (∼\sim60/40), though with lower absolute importance scores.

This depth-dependent component balance has practical implications for future work on sub-layer selection: a hybrid strategy that selects MLP adapters broadly but attention adapters only in middle layers could achieve even higher efficiency than full-layer selection. This motivates testing finer-grained allocation across components in future work.

Default Layer Budget

Table 18 selects the top half of layers by calibration score. This fixed fraction improves accuracy over full-layer LoRA on all eight models by 0.2–1.9 percentage points, without per-model layer-budget tuning.

ModelLLkkAdaptive-LoRAFull LoRAΔ\Delta
Qwen-0.5B241252.351.8+0.5
Qwen-1.5B281456.255.8+0.4
Qwen-3B361859.157.2+1.9
Qwen-7B321680.079.8+0.2
Qwen-32B643285.884.5+1.3
LLaMA-3B281457.556.0+1.5
LLaMA-7B321669.869.0+0.8
LLaMA-13B402072.571.8+0.7

Table 18. Fixed 50% layer budget (k=L/2k = L/2): rank r=16r = 16, 5,000 training examples. Accuracy (%); Δ\Delta: Adaptive-LoRA minus full-layer LoRA (percentage points).

Broader Impact

Positive Societal Implications

This work contributes to improving the efficiency of large language model adaptation, which has several positive societal implications:

  1. Democratized access. By reducing the parameter count for fine-tuning by 3.5×, Adaptive-LoRA lowers the computational barrier for researchers and practitioners with limited GPU resources. Organizations that previously could not afford full-layer LoRA adaptation on larger models may now achieve comparable performance with significantly reduced hardware requirements.

  1. Environmental sustainability. Smaller adapters reduce parameter storage. Training-time benefits depend additionally on selected-layer depth and prefix reuse, as quantified in Appendix § 13.6. Energy consumption is a separate system-level measurement.

  1. Multi-adapter serving. Smaller adapter footprints enable serving more concurrent task-specific adapters on the same hardware, improving resource utilization in multi-tenant cloud environments and enabling personalized model deployment on edge devices.

  1. Interpretability and transparency. The mechanistic analysis underlying Adaptive-LoRA provides interpretable insights into how LoRA adaptation distributes across model components. This transparency can help practitioners understand what their fine-tuned models have learned and where the adaptation has occurred, contributing to more responsible AI deployment.

Potential Risks and Mitigations

We do not foresee direct negative societal impacts from this work, as it improves efficiency of existing fine-tuning methods without introducing new model capabilities or training paradigms. However, we acknowledge two indirect considerations:

  • Lowered cost of misuse. By making fine-tuning more efficient, our method could marginally lower the cost of adapting models for harmful purposes. Task-specific deployment therefore calls for evaluating safety behavior after adaptation, alongside task accuracy.

  • Layer selection bias. Our calibration-based selection relies on the distribution of calibration data. If calibration data is biased or unrepresentative, the selected layers may not generalize well to underrepresented populations. We mitigate this by using diverse, publicly available training datasets and validating across multiple task families.

NeurIPS Paper Checklist

  1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state four contributions (mechanistic analysis, task-dependency finding, Adaptive-LoRA method, empirical validation), all supported by experimental results in § 3–5.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

  1. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in Appendix § 9, covering selection granularity, task coverage, and theoretical foundations.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

  1. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]

Justification: This paper is primarily empirical. The mechanistic analysis uses causal interventions with clearly stated procedures (Eqs. (1)–(2)).

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Model choices, training hyperparameters, dataset descriptions, evaluation metrics, and seed settings are provided in Section 5 and Appendix § 11.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Code release is planned under the MIT license; this version does not provide a public code-release link. Public dataset sources are cited in the paper.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: Full training configuration including optimizer, learning rate, batch size, epochs, and hardware are specified in Section 5 and Appendix § 11.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Appendix § 10.§ 17 reports three-run means and sample SDs, separately for paired gains and absolute scores. Appendix § 10.4 defines the 5,000-resample paired-bootstrap intervals for MLP-share gaps. These summaries do not assert significance for every single-seed comparison.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The hardware setup is described in Appendix § 11. Calibration costs are reported in Appendix § 13; the controlled step/epoch timing study specifies one H200 and its measurement protocol in Appendix § 13.6.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: This work focuses on improving parameter efficiency for LLM fine-tuning and poses no ethical concerns. All experiments use publicly available models and datasets.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] Justification: Broader impact is discussed in Appendix § 15. Positive impacts include reduced computational costs for LLM adaptation, enabling broader access. Potential misuse from lower adaptation costs and calibration-data bias is discussed there.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: This paper does not release pre-trained models or scraped datasets. The method applies to existing open-source models.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets and models are properly cited. Qwen2.5 is used under its open-source license; all evaluation benchmarks are publicly available with their respective licenses respected.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: No new datasets or pre-trained models are released. Code will be documented upon release.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: This paper does not involve crowdsourcing or human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A] Justification: No human subjects research was conducted.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: LLMs are used as the subject of study (Qwen2.5 for fine-tuning), not as a tool for methodology development. No LLMs were used as non-standard components in the core methods.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi. PIQA: Reasoning about Physical Commonsense in Natural Language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, 2020. doi: 10.1609/aaai.v34i05.62539. URL https://doi.org/10.1609/aaai.v34i05.62539.
  2. [2]S. Chaudhary. Code Alpaca: An Instruction-following LLaMA model for code generation. Project repository, 2023. URL https://github.com/sahil280114/codealpaca.
  3. [3]C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. arXiv preprint arXiv:1905.10044, 2019. URL https://arxiv.org/abs/1905.10044.
  4. [4]P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have Solved Question Answering: Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803.05457.
  5. [5]K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168.
  6. [6]A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards Automated Circuit Discovery for Mechanistic Interpretability. arXiv preprint arXiv:2304.14997, 2023. URL https://arxiv.org/abs/2304.14997.
  7. [7]M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. Databricks technical blog, 2023. URL https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
  8. [8]N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. Toy Models of Superposition. arXiv preprint arXiv:2209.10652, 2022. URL https://arxiv.org/abs/2209.10652.
  9. [9]A. Grattafiori, A. Dubey, A. Jauhari, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783.
  10. [10]Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey. arXiv preprint arXiv:2403.14608, 2024. URL https://arxiv.org/abs/2403.14608.
  11. [11]S. Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient Low Rank Adaptation of Large Models. arXiv preprint arXiv:2402.12354, 2024. URL https://arxiv.org/abs/2402.12354.
  12. [12]N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attarian, and S. Gelly. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790–2799. PMLR, 2019. URL https://proceedings.mlr.press/v97/houlsby19a.html.arxiv.org/abs/1902.00751
  13. [13]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685, 2021. URL https://arxiv.org/abs/2106.09685.
  14. [14]Z. Hu, L. Wang, Y. Lan, W. Xu, E.-P. Lim, L. Bing, X. Xu, S. Poria, and R. K.-W. Lee. LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models. arXiv preprint arXiv:2304.01933, 2023. URL https://arxiv.org/abs/2304.01933.
  15. [15]O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. Revealing the Dark Secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–4374. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1445. URL https://aclanthology.org/D19-1445/.DOI
  16. [16]B. Lester, R. Al-Rfou, and N. Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv preprint arXiv:2104.08691, 2021. URL https://arxiv.org/abs/2104.08691.
  17. [17]X. L. Li and P. Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv preprint arXiv:2101.00190, 2021. URL https://arxiv.org/abs/2101.00190.
  18. [18]Y.-S. Liang and W.-J. Li. InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23638–23647, 2024. URL https://openaccess.thecvf.com/content/CVPR2024/html/Liang_InfLoRA_Interference-Free_Low-Rank_Adaptation_for_Continual_Learning_CVPR_2024_paper.html.arxiv.org/abs/2404.00228
  19. [19]Y.-S. Liang, J.-R. Chen, and W.-J. Li. Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models. In Advances in Neural Information Processing Systems, volume 38, pages 68659–68689, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/63692d8d567db671c700df5df912204a-Abstract-Conference.html.arxiv.org/abs/2505.15424
  20. [20]W. Ling, D. Yogatama, C. Dyer, and P. Blunsom. Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems. arXiv preprint arXiv:1705.04146, 2017. URL https://arxiv.org/abs/1705.04146.
  21. [21]S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv preprint arXiv:2402.09353, 2024. URL https://arxiv.org/abs/2402.09353.
  22. [22]X. Men, M. Xu, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. arXiv preprint arXiv:2403.03853, 2024. URL https://arxiv.org/abs/2403.03853.
  23. [23]K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and Editing Factual Associations in GPT. arXiv preprint arXiv:2202.05262, 2022. URL https://arxiv.org/abs/2202.05262.
  24. [24]T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. arXiv preprint arXiv:1809.02789, 2018. URL https://arxiv.org/abs/1809.02789.
  25. [25]N. Nanda and J. Bloom. TransformerLens. Open-source software, 2022. URL https://github.com/TransformerLensOrg/TransformerLens.
  26. [26]N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023. URL https://arxiv.org/abs/2301.05217.
  27. [27]C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. In-context Learning and Induction Heads. arXiv preprint arXiv:2209.11895, 2022. URL https://arxiv.org/abs/2209.11895.
  28. [28]OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023. URL https://arxiv.org/abs/2303.08774.
  29. [29]A. Patel, S. Bhattamishra, and N. Goyal. Are NLP Models really able to Solve Simple Math Word Problems? arXiv preprint arXiv:2103.07191, 2021. URL https://arxiv.org/abs/2103.07191.
  30. [30]Y.-Y. Qian, Y.-Z. Xu, Z.-Y. Zhang, P. Zhao, and Z.-H. Zhou. TreeLoRA: Efficient Continual Learning via Layer-Wise LoRA Guided by a Hierarchical Gradient-Similarity Tree. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 50066–50085. PMLR, 2025. URL https://proceedings.mlr.press/v267/qian25b.html.arxiv.org/abs/2506.10355
  31. [31]Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, et al. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115, 2024. URL https://arxiv.org/abs/2412.15115.
  32. [32]K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740, 2020. doi: 10.1609/aaai.v34i05.6399. URL https://doi.org/10.1609/aaai.v34i05.6399.
  33. [33]M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi. SocialIQA: Commonsense Reasoning about Social Interactions. arXiv preprint arXiv:1904.09728, 2019. URL https://arxiv.org/abs/1904.09728.
  34. [34]R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford Alpaca: An Instruction-following LLaMA model. GitHub project repository, 2023. URL https://github.com/tatsu-lab/stanford_alpaca.
  35. [35]M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi. DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation. arXiv preprint arXiv:2210.07558, 2022. URL https://arxiv.org/abs/2210.07558.
  36. [36]K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. arXiv preprint arXiv:2211.00593, 2022. URL https://arxiv.org/abs/2211.00593.
  37. [37]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388.
  38. [38]R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv preprint arXiv:1905.07830, 2019. URL https://arxiv.org/abs/1905.07830.
  39. [39]Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. arXiv preprint arXiv:2303.10512, 2023. URL https://arxiv.org/abs/2303.10512.

Paper details

vlnc:W_8gpf4ppw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 15

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents