betaFeedback

WorksComputer Science

Direction or Magnitude? Learning Scalars over Task-Calibrated Directions

Abstract

Parameter-efficient fine-tuning (PEFT) aims to preserve adaptation quality while reducing training costs. We propose SCALAR, which achieves performance comparable to LoRA with substantially fewer trainable parameters, lower peak training memory, and higher throughput. SCALAR constructs a fixed input basis from task activations and assigns a random output dictionary to each input direction. Fine-tuning learns only layer-specific scalar coefficients, reducing gradient and optimizer-state storage. Grouped computation aggregates output atoms before applying the adapter to token activations, while dictionary sharing across compatible layers reduces fixed-basis storage. Experiments on commonsense reasoning, mathematical reasoning, code generation, and knowledge evaluation after instruction tuning demonstrate effective adaptation at small trainable budgets. On commonsense reasoning, SCALAR achieves accuracy comparable to LoRA with about 4% of its trainable parameters. System measurements on Qwen2.5-7B show 20% lower peak training memory and 6% higher throughput than LoRA. The combination of adaptation quality, a compact trainable state, and training efficiency makes SCALAR a promising approach to resource-constrained fine-tuning.

Introduction

Large language models (LLMs) achieve strong general-purpose performance, but full-parameter fine-tuning incurs substantial training memory and optimization costs. Parameter-efficient fine-tuning (PEFT) alleviates these costs by updating a small subset of parameters, with LoRA [17] serving as a widely adopted baseline. For an m×nm \times n linear layer, LoRA optimizes R(m+n)R(m+n) parameters together with their gradients and optimizer states. These costs are especially relevant to on-device personalization, where local model updates must operate within limited memory and compute budgets. Retaining adaptation quality with a smaller trainable state, lower peak memory, and higher throughput can make such local updates more practical.

Recent methods further reduce the optimization budget by learning compact coefficients over fixed bases [20, 19, 1] (Bałazy et al. 2024). Their effectiveness depends on both the choice of update directions and the computation required to apply them. Random bases provide inexpensive candidate directions, while weight- or activation-derived subspaces use information from pretrained models or downstream data [26, 40, 30]. Recent evidence also suggests that useful task-improving perturbations can be abundant around pretrained weights [9]. These observations motivate the following question.

How can we construct a task-relevant fixed update space that maintains adaptation quality with few trainable coefficients, low memory use, and high throughput?

We first investigate this question through a learned-direction reweighting diagnostic. We train a rank-one LoRA adapter, freeze its learned update direction for each adapted matrix, and re-optimize one scalar coefficient per matrix. On a 7B model, the reweighting stage optimizes 196 parameters compared with LoRA’s 2,523,136, and achieves similar optimization behavior and downstream accuracy. This observation motivates constructing informative directions before fine-tuning and adapting through a small set of coefficients.

We propose SCALAR: Subspace-Calibrated Adaptation via Layerwise Atom Reweighting. SCALAR constructs a fixed input basis from the leading singular vectors of downstream activations and assigns a separate random output dictionary to each input direction. Learning the coefficients selects an effective output vector within each of these fixed output subspaces. With RR input directions and HH output atoms per direction, each adapted matrix has K=RHK = RH trainable coefficients and an update rank of at most RR. The output atoms in each group are aggregated before application to the activations, so the token-dependent matrix multiplications operate on RR effective directions. Compatible layers can also share the frozen output dictionaries.

Experiments on commonsense reasoning, mathematical reasoning, and code generation show competitive performance with compact trainable budgets. On Qwen2.5-7B and LLaMA3-8B, SCALAR approaches rank-8 LoRA accuracy on commonsense reasoning using about 4% of its trainable parameters. System measurements demonstrate lower peak training memory and higher throughput, giving the compact parameterization practical value during training. Our analysis builds on activation coverage to examine the initial gradient signal and connects it to the coefficients trained by SCALAR.

Our main contributions are summarized as follows.

Task-calibrated fixed-subspace adaptation. We introduce SCALAR, which combines an activation-derived input basis with a separate random output subspace for each input direction. Its grouped parameterization learns RHRH coefficients and applies the update through RR aggregated output vectors. Compatible layers share their fixed output dictionaries to reduce basis storage.

Analysis and diagnostic evidence. Building on activation-subspace analysis, we characterize the effect of gradient-weighted activation geometry on the initial learning signal and derive its connection to SCALAR’s coefficients. A learned-direction reweighting diagnostic motivates adaptation within informative fixed update spaces.

Competitive performance and training efficiency. SCALAR achieves competitive adaptation performance with substantially fewer trainable parameters across commonsense reasoning, mathematical reasoning, code generation, and general-domain knowledge evaluation. System measurements on Qwen2.5-7B show lower peak training memory and higher throughput than LoRA, NOLA, and RandLoRA. Structural ablations examine the benefits of grouped computation and shared output dictionaries. The experiments cover three model families and multiple parameter scales up to 32B, with additional large-model evaluations in the Appendix.

Related Work

LoRA [17] adapts pretrained weights through two trainable low-rank factors. Subsequent methods improve rank allocation, parameterization, optimization, and routing [44, 45, 25, 36, 38, 37, 13, 28]. Initialization also shapes the adaptation subspace. PiSSA [26] uses pretrained weight directions, EVA [30] uses dominant input activation directions, and CorDA [40] combines weights with task-dependent activation statistics. Astra [24] initializes trainable LoRA factors from the tail eigenspaces of output activations.

LoRA-FA [43] freezes a randomly initialized input factor A\mathbf{A} and trains only B\mathbf{B}, reducing adapter activation storage. Zhu et al. [47] study the distinct roles of the two factors and the effectiveness of training only the output factor. TLoRA [22] constructs A\mathbf{A} from pretrained weights and input activation statistics, then freezes it while training B\mathbf{B}.

Fixed-basis methods further compress the trainable state. LoRA-XS [3] learns a small core between fixed SVD-derived projections, while NoRA [21] freezes outer factors initialized by activation-aware SVD and trains a nested low-rank structure. SVFT [23] learns sparse coefficients over outer products of weight singular vectors. Other methods use random, frequency-domain, or tensor bases [10, 12] (Nooralinejad et al. 2023). NOLA [19] learns combinations of random matrices, VeRA [20] scales shared random projections, and RandLoRA [1] combines random low-rank bases with diagonal scalings. SCALAR pairs activation-derived input directions with separate fixed random output dictionaries. Its coefficients determine an effective output vector for each input direction, supporting compact adaptation and grouped computation.

Motivation

We motivate SCALAR through a learned-direction reweighting diagnostic and an analysis of input subspaces. The diagnostic examines adaptation through scalar coefficients after informative directions have been obtained. The analysis then connects dominant activation directions to input coverage and the initial gradient signal, providing the basis for constructing a fixed input subspace before fine-tuning.

Learned-Direction Reweighting

We begin with a diagnostic experiment that separates adaptation directions from their magnitudes. Any trained rank-rr LoRA update can be decomposed through singular value decomposition as

ΔWLoRA=∑i=1rσiuiviT,\Delta\mathbf{W}_{\mathrm{LoRA}}=\sum_{i=1}^{r}\sigma_i\mathbf{u}_i\mathbf{v}_i^{\mathsf{T}},

where each pair (ui,vi)(\mathbf{u}_i,\mathbf{v}_i) defines a rank-one adaptation direction and σi\sigma_i specifies its magnitude. We freeze these directions and re-optimize only their scalar coefficients:

ΔW=∑i=1rγiuiviT.\Delta\mathbf{W}=\sum_{i=1}^{r}\gamma_i\mathbf{u}_i\mathbf{v}_i^{\mathsf{T}}.

We report the Qwen2.5-7B results in Table 2. In this setting, a rank-one LoRA adapter contains 2,523,136 trainable parameters, whereas the learned-direction reweighting model optimizes only 196 coefficients. Despite this substantial reduction, it closely follows the optimization behavior of LoRA and achieves slightly higher average accuracy. The same trend is observed on LLaMA3-8B; detailed training trajectories and additional results are provided in Appendix § 9.§ 1.

MethodParamsBoolQPIQASIQAHSWGARC-EARC-COBQAAvg.
LoRA2,523,13674.190.381.895.788.796.289.491.888.5
Learned-direction reweighting19673.990.181.896.089.096.489.992.488.7

Table 2. Accuracy of LoRA and learned-direction reweighting on 8 commonsense reasoning tasks. Learned-direction reweighting achieves comparable performance while optimizing fewer than 0.01% as many parameters as LoRA.

These observations show that reweighting informative learned directions can recover similar fitting behavior and downstream accuracy. They motivate the practical design of SCALAR, which constructs its input directions from activation statistics before fine-tuning and learns coefficients within the resulting fixed update space.

Theoretical Analysis

We analyze the role of the input subspace through activation coverage and the initial gradient energy of a low-rank output factor. The analysis characterizes the effect of weighting activations by the task gradient and motivates the fixed input directions used by SCALAR.

Input-Side Subspace in Low-Rank Adaptation

Consider a frozen linear transformation with a LoRA branch:

z=Wx+BAx,z = W x + B A x,

where x∈Rnx \in\mathbb{R}^{n} is the input activation, W∈Rm×nW \in\mathbb{R}^{m \times n} is the frozen pretrained weight, and A∈RR×nA \in\mathbb{R}^{R \times n} and B∈Rm×RB \in\mathbb{R}^{m \times R} are the input- and output-side factors, respectively. An invertible change of coordinates in the latent space can be absorbed into the two factors while preserving their product. We therefore compare input subspaces through normalized representations. The assumptions used in the analysis are stated below.

Assumption 3.1 (Finite second moments). The activation x∈Rnx \in\mathbb{R}^{n} and the back-propagated gradient g∈Rmg \in\mathbb{R}^{m} satisfy

E[∥x∥22]<∞,E[∥g∥22∥x∥22]<∞.\mathbb{E}\left[\lVert x\rVert_{2}^{2}\right] < \infty, \qquad\mathbb{E}\left[\lVert g\rVert_{2}^{2}\lVert x\rVert_{2}^{2}\right] < \infty.

Define the activation second-moment matrix as

Σ:=E[xx⊤].\Sigma:= \mathbb{E}\left[x x^{\top}\right].

Assumption 3.2 (Normalized subspace representation). The RR-dimensional input subspace is represented by A∈RR×nA \in\mathbb{R}^{R \times n} with orthonormal rows:

AA⊤=IR,1≤R<n.A A^{\top} = I_{R}, \qquad1 \le R < n.

Any full-row-rank factor admits a row-orthonormal representative with the same row subspace. This normalization compares subspaces at a common scale, and the activation-SVD basis used by SCALAR has orthonormal rows by construction.

Assumption 3.3 (Standard zero-output initialization). The loss ℓ(z,y)\ell(z,y) is differentiable with respect to zz, and the output-side factor is initialized as B0=0B^{0}=0. Define

g:=∇zℓ(z,y)∣z=Wx.g := \left.\nabla_{z}\ell(z,y)\right|_{z=Wx}.

Assumption 3.4 (Covariance alignment). Define the gradient-weighted activation second moment as

M:=E[∥g∥22xx⊤].(1)M := \mathbb{E}\left[\lVert g\rVert_{2}^{2}x x^{\top}\right]. \tag*{(1)}

There exist c>0c>0 and ε≥0\varepsilon\ge0 such that

∥M−cΣ∥2≤ε.\lVert M-c\Sigma\rVert_{2}\le\varepsilon.

The quantity ε\varepsilon bounds the change in activation geometry induced by gradient weighting. This condition allows dependence between xx and gg through the resulting second moments. For example, E[∥g∥22∣x]=c\mathbb{E}[\lVert g\rVert_{2}^{2}\mid x]=c implies M=cΣM=c\Sigma even when xx and gg are dependent.

Why Activation-SVD?

For a normalized input subspace, define the captured activation energy as

Φ(A):=E[∥Ax∥22].\Phi(A) := \mathbb{E}\left[\lVert Ax\rVert_2^2\right].

Let

Σ=UDiag⁡(λ1,…,λn)UT,λ1≥⋯≥λn≥0,\Sigma= U\operatorname{Diag}(\lambda_1,\ldots,\lambda_n)U^{\mathsf{T}}, \qquad\lambda_1 \ge\cdots\ge\lambda_n \ge0,

and let UR∈Rn×RU_R \in\mathbb{R}^{n \times R} contain the leading RR eigenvectors. We define Asvd:=URTA_{\mathrm{svd}} := U_R^{\mathsf{T}}. The classical principal-subspace variational principle underlies activation-based initialization methods such as EVA [30]. We state this activation-coverage property together with its comparison to a uniformly random subspace.

Theorem 3.5 (Optimal activation coverage). Under Assumptions 3.1 and 3.2, we have

Φ(A)=Tr⁡(AΣAT),max⁡AAT=IRΦ(A)=∑i=1Rλi.\Phi(A) = \operatorname{Tr}\left(A\Sigma A^{\mathsf{T}}\right), \qquad\max_{AA^{\mathsf{T}}=I_R}\Phi(A) = \sum_{i=1}^{R}\lambda_i.

The maximum is attained by AsvdA_{\mathrm{svd}}. If ArandA_{\mathrm{rand}} represents a uniformly random RR-dimensional row subspace, then

EArand[Φ(Arand)]=RnTr⁡(Σ).\mathbb{E}_{A_{\mathrm{rand}}}\left[\Phi(A_{\mathrm{rand}})\right] = \frac{R}{n}\operatorname{Tr}(\Sigma).

Consequently,

Φ(Asvd)≥EArand[Φ(Arand)].\Phi(A_{\mathrm{svd}}) \ge\mathbb{E}_{A_{\mathrm{rand}}}\left[\Phi(A_{\mathrm{rand}})\right].

The inequality is strict when R<nR<n and Σ\Sigma is not a scalar multiple of InI_n.

For SCALAR, Theorem 3.5 motivates allocating the input-subspace budget to the leading activation directions. The random-subspace comparison describes the activation-energy benefit of this choice when the input second moment is anisotropic.

At the initialization in Assumption 3.3, define the expected gradient energy received by the output-side factor as

G(A):=E[∥∇Bℓ∥F2].\mathcal{G}(A) := \mathbb{E}\left[\lVert\nabla_B\ell\rVert_F^2\right].

Theorem 3.6 (Initial gradient signal under covariance mismatch). Under Assumptions 3.1, 3.2, 3.3, and 3.4, we have

G(A)=Tr⁡(AMAT).\mathcal{G}(A) = \operatorname{Tr}\left(AMA^{\mathsf{T}}\right).

Let A⋆∈arg⁡max⁡AAT=IRG(A)A_\star\in\arg\max_{AA^{\mathsf{T}}=I_R}\mathcal{G}(A). Then

0≤G(A⋆)−G(Asvd)≤2Rε.0 \le\mathcal{G}(A_\star)-\mathcal{G}(A_{\mathrm{svd}}) \le2R\varepsilon.

In particular, AsvdA_{\mathrm{svd}} is exactly optimal when ε=0\varepsilon=0. If G(A⋆)>0\mathcal{G}(A_\star)>0, the relative gap satisfies

G(A⋆)−G(Asvd)G(A⋆)≤2RεG(A⋆).\frac{\mathcal{G}(A_\star)-\mathcal{G}(A_{\mathrm{svd}})}{\mathcal{G}(A_\star)} \le \frac{2R\varepsilon}{\mathcal{G}(A_\star)}.

Moreover,

EArand[G(Arand)]=RnTr⁡(M).\mathbb{E}_{A_{\mathrm{rand}}}\left[\mathcal{G}(A_{\mathrm{rand}})\right] = \frac{R}{n}\operatorname{Tr}(M).

Theorem 3.6 quantifies how covariance mismatch controls the initial gradient-energy gap of activation-SVD. Its relative form expresses this gap on the scale of the optimal initial signal.

Appendix § 9.§ 2 provides complementary empirical observations of activation coverage in learned LoRA input subspaces. These results motivate a fixed input basis formed from the leading activation directions. The next section combines this basis with a grouped random output dictionary and learns layer-specific coefficients within the resulting update subspace.

SCALAR

SCALAR uses the activation geometry analyzed in § 3 to construct a fixed input basis. Each input direction is paired with a group of random output atoms, and adaptation learns their combination coefficients. The groups define separate output subspaces while keeping the effective update rank bounded by the number of input directions. This construction keeps the trainable state compact and supports efficient evaluation through a small set of effective output vectors.

Consider a frozen linear layer W∈Rm×nW \in\mathbb{R}^{m \times n}. SCALAR parameterizes its update as

ΔW=s∑r=1R∑h=1Hγr,hbr,harT,(2)\Delta W = s \sum_{r=1}^{R} \sum_{h=1}^{H} \gamma_{r,h} b_{r,h} a_r^{\mathsf{T}}, \tag*{(2)}

where ar∈Rna_r \in\mathbb{R}^{n} is a fixed task-aware input direction, br,h∈Rmb_{r,h} \in\mathbb{R}^{m} is a fixed random output atom, γr,h∈R\gamma_{r,h} \in\mathbb{R} is trainable, and ss is a fixed scaling factor. Each adapted layer contains K=RHK = RH trainable coefficients. All coefficients are initialized at zero, so fine-tuning starts from the pretrained model.

Task-Aware Input Directions

We collect SS input activations from a small calibration set and stack them into X∈RS×nX \in\mathbb{R}^{S \times n}. Following the second-moment analysis in § 3, XX is not mean-centered. Following activation-based initialization [30], we compute the SVD and retain the leading RR right singular vectors:

X=UDVT,A=V:,1:RT∈RR×n.X = UDV^{\mathsf{T}}, \qquad A = V_{:,1:R}^{\mathsf{T}} \in\mathbb{R}^{R \times n}.

The resulting directions form an orthonormal basis of the dominant empirical activation subspace. We freeze this input basis during fine-tuning, following fixed-input-factor designs [43, 22]. It defines the input coordinates for coefficient learning over SCALAR’s fixed output dictionaries.

Grouped and Shared Random Output Dictionaries

For each input direction ara_r, we sample HH fixed output atoms and collect them into

Br=[br,1 ⋯ br,H]∈Rm×H.B_r = [b_{r,1} \ \cdots\ b_{r,H}] \in\mathbb{R}^{m \times H}.

The atoms are sampled once from a zero-mean isotropic distribution and remain frozen. Let γr=[γr,1,…,γr,H]T\gamma_r = [\gamma_{r,1}, \ldots, \gamma_{r,H}]^{\mathsf{T}} and b~r=Brγr\tilde{b}_r = B_r\gamma_r. During training, b~r\tilde{b}_r varies within the fixed span of BrB_r. Thus, each input direction has its own output subspace, and equation (2) becomes

ΔW=s∑r=1Rb~rarT=sB~A,rank⁡(ΔW)≤R.\Delta W = s \sum_{r=1}^{R} \tilde{b}_r a_r^{\mathsf{T}} = s\tilde{B}A, \qquad\operatorname{rank}(\Delta W) \le R.

Here, RR determines the input-subspace dimension and maximum update rank, while HH controls the number of candidate output directions associated with each input direction. Since the HH output atoms can be merged before multiplication with the activations, the token-dependent computation scales with RR rather than K=RHK = RH. Appendix § 9.6.1 provides the corresponding complexity analysis and empirical throughput comparison. For an input xx, the adapter output is

ΔWx=s∑r=1R(arTx)Brγr.\Delta W x = s \sum_{r=1}^{R} (a_r^{\mathsf{T}}x) B_r\gamma_r.

At zero coefficient initialization, isotropic random output atoms relate the expected scalar-gradient energy to the gradient-weighted activation energy G(A)\mathcal{G}(A). The covariance-alignment condition connects this quantity to activation coverage. Appendix § 7.§ 3 provides the derivation for the trainable coefficients.

Each adapted layer maintains its own activation-derived basis AA and trainable coefficient matrix Γ\Gamma. Layers of the same module type with compatible output dimensions may share the fixed output dictionaries {Br}r=1R\{B_r\}_{r=1}^{R}.

Layer-specific input bases and coefficients produce distinct updates from the shared atoms. The sharing ablation in Appendix § 9.6.1 shows lower memory consumption with similar accuracy.

The construction reduces three sources of training cost. Learning only the coefficient set reduces gradient and optimizer-state storage. Grouped evaluation limits token-dependent adapter computation, while dictionary sharing reduces the memory occupied by fixed output bases. § 5.2 reports the resulting peak memory and throughput measurements. These properties are useful for on-device personalization under limited memory and compute budgets.

Experimental Results

We evaluate adaptation quality at compact trainable budgets and measure the corresponding training memory and throughput. The experiments cover several language adaptation tasks. The Appendix includes additional diagnostic analyses, comparisons with enhanced LoRA variants, general instruction fine-tuning, larger-model evaluations, and extensive ablations on the parameterization, calibration, rank, initialization, and cross-layer sharing, together with full implementation details and parameter-budget comparisons.

Training Configuration. We evaluate SCALAR on commonsense reasoning with Qwen2.5 [39] and LLaMA3 [11], and further test transfer to mathematical reasoning and code generation. We also ablate the initialization of the fixed bases AA and BB.

Baselines. We compare SCALAR with full-parameter fine-tuning and five representative PEFT methods: Full Fine-tuning (FT) as a full-capacity reference; LoRA [17], the standard trainable low-rank parameterization; VeRA [20], which shares random low-rank projections and learns layer-specific scaling vectors; LoRA-XS [3], which optimizes a compact matrix between fixed projections; NOLA [19], which learns combinations of fixed random bases; and RandLoRA [1], which aggregates random low-rank bases with trainable diagonal scalings.

Commonsense Reasoning Benchmarks

We present the main accuracy results in Table 3 and the parameter–performance comparison in Figure 1. We fine-tune all methods on the Commonsense170K training set [18] and evaluate on 8 commonsense reasoning benchmarks. We conduct experiments on both LLaMA3-8B and Qwen2.5-7B, and report the mean accuracy over three independent runs. A more comprehensive comparison covering multiple trainable-parameter budgets for NOLA, LoRA-XS, and the other PEFT baselines is provided in Appendix Table 10.

Figure 1. Average accuracy on 8 commonsense reasoning tasks across trainable parameter budgets. SCALAR approaches LoRA-level accuracy with substantially fewer trainable parameters.

ModelMethodParams (%)BoolQPIQASIQAHSWGARC-EARC-COBQAAvg.
Qwen2.5-7BFT100.0075.389.782.096.189.795.688.493.188.7±0.188.7 \pm0.1
Qwen2.5-7BLoRA (r=1r = 1)0.033174.390.381.995.888.796.189.291.788.5±0.188.5 \pm0.1
Qwen2.5-7BLoRA (r=8r = 8)0.264374.690.481.996.389.395.989.592.388.8±0.188.8 \pm0.1
Qwen2.5-7BVeRA0.020873.790.281.595.687.796.388.891.588.2±0.288.2 \pm0.2
Qwen2.5-7BNOLA0.010573.690.082.095.888.696.489.291.588.4±0.088.4 \pm0.0
Qwen2.5-7BLoRA-XS0.010573.489.882.495.687.296.588.590.988.0±0.288.0 \pm0.2
Qwen2.5-7BRandLoRA0.017573.989.581.295.888.396.189.190.888.1±0.288.1 \pm0.2
Qwen2.5-7BSCALAR (K=1024K = 1024)0.002673.490.382.496.087.596.889.092.388.5±0.188.5 \pm0.1
Qwen2.5-7BSCALAR (K=2048K = 2048)0.005273.790.082.596.188.296.789.291.988.6±0.088.6 \pm0.0
Qwen2.5-7BSCALAR (K=4096K = 4096)0.010574.090.382.296.289.396.589.593.088.9±0.188.9 \pm0.1
LLaMA3-8BFT100.0076.290.483.096.789.494.185.188.888.0±0.288.0 \pm0.2
LLaMA3-8BLoRA (r=1r = 1)0.032674.389.981.996.389.593.383.389.087.2±0.287.2 \pm0.2
LLaMA3-8BLoRA (r=8r = 8)0.260475.990.083.096.689.493.383.187.887.4±0.387.4 \pm0.3
LLaMA3-8BVeRA0.019974.888.981.695.987.192.882.187.586.3±0.386.3 \pm0.3
LLaMA3-8BNOLA0.011474.789.582.896.488.392.982.987.486.9±0.186.9 \pm0.1
LLaMA3-8BLoRA-XS0.011473.388.382.796.088.192.481.987.786.2±0.286.2 \pm0.2
LLaMA3-8BRandLoRA0.020375.189.582.395.586.793.281.788.486.6±0.086.6 \pm0.0
LLaMA3-8BSCALAR (K=1024K = 1024)0.002873.589.282.496.288.092.882.387.686.5±0.386.5 \pm0.3
LLaMA3-8BSCALAR (K=2048K = 2048)0.005774.289.782.596.387.593.383.588.386.9±0.186.9 \pm0.1
LLaMA3-8BSCALAR (K=4096K = 4096)0.011474.089.682.896.888.893.183.989.287.3±0.187.3 \pm0.1

Table 3. Commonsense reasoning accuracy of LLaMA3-8B and Qwen2.5-7B across 8 benchmarks. Results are averaged over three independent runs. The SCALAR configurations are highlighted.

Parameter Efficiency.

Table 3 shows that SCALAR approaches LoRA-level accuracy with substantially smaller trainable budgets. With K=4096K = 4096, it reaches 88.9 average accuracy on Qwen2.5-7B and 87.3 on LLaMA3-8B, compared with 88.8 and 87.4 for rank-8 LoRA. The corresponding trainable parameter counts are approximately 1/25 and 1/23 of LoRA’s, respectively. Across the evaluated coefficient budgets, SCALAR records higher mean accuracy than LoRA-XS and remains competitive with NOLA. Figure 1 shows that the compact coefficient parameterization sustains strong task performance across the evaluated budgets.

Training Memory and Throughput

Figure 2 compares peak GPU memory and training token throughput when fine-tuning Qwen2.5-7B. SCALAR achieves the lowest peak memory among the evaluated PEFT methods, using 20% less memory than LoRA, 15% less than NOLA, and 33% less than RandLoRA. Training throughput increases by 6% over LoRA, 34% over NOLA, and approximately 130% over RandLoRA. Together with the accuracy results, these measurements show that SCALAR combines a compact trainable budget with effective adaptation, lower peak memory, and higher throughput. The structural ablations in Appendix § 9.6.1 examine how grouped computation and cross-layer dictionary sharing balance accuracy and training cost.

Figure 2. Memory and throughput for fine-tuning Qwen2.5-7B on commonsense reasoning. We report peak GPU memory with a device batch size of 16 and token throughput on 2 NVIDIA H20 GPUs. SCALAR uses less memory and achieves higher throughput than the compared baselines.

Mathematical Reasoning and Code Generation

We further evaluate the transferability of our method to mathematical reasoning and code generation tasks. For mathematical reasoning, all methods are trained on MetaMath [41] and evaluated on GSM8K [8] and MATH [16]. For code generation, the models are trained on CodeFeedback [46] and evaluated on HumanEval [5] and MBPP [2]. The results are summarized in Table 4.

Table 4: Comparison on mathematical and code generation benchmarks on Qwen2.5-7B. SCALAR achieves comparable performance on mathematical reasoning and code generation.

MethodParams (%)GSM8KMATHAvg.HumanEvalMBPPAvg.
FT100.0084.648.866.774.482.578.4
LoRA (r=1r = 1)0.033184.848.366.674.481.578.0
LoRA (r=8r = 8)0.264384.848.266.570.183.376.7
VeRA0.018984.448.766.672.682.577.6
NOLA0.001384.347.666.073.282.077.6
LoRA-XS0.001483.047.165.172.681.076.8
RandLoRA0.017584.348.066.272.084.478.2
SCALAR (K=128K = 128)0.000383.548.566.073.281.777.5
SCALAR (K=256K = 256)0.000684.348.566.476.882.079.4
SCALAR (K=512K = 512)0.001385.048.766.973.884.179.0

Table 4.

Across both domains, SCALAR remains competitive with full fine-tuning and LoRA while using substantially fewer trainable parameters. The K=512K = 512 configuration reaches a mathematical average of 66.9 with 0.0013% trainable parameters, and K=256K = 256 reaches a code-generation average of 79.4 with 0.0006% trainable parameters. These results extend the effectiveness of the fixed-subspace parameterization to mathematical and code adaptation.

On the Initialization Methods

We investigate the effects of different initialization strategies for the input-side basis AA and the output-side dictionary BB. As shown in Table 5, activation-SVD consistently yields the strongest initialization for AA, outperforming both random and weight-SVD initialization under either choice of BB. This result supports our design of extracting AA from the dominant activation subspace, which better captures the downstream input geometry.

Table 5: Commonsense reasoning Acc. on Qwen2.5-3B with different initialization methods.

Random-AAWeight-SVD-AAActivation-SVD-AA
Random-BB84.384.485.2
Weight-SVD-BB84.584.685.1

Table 5.

With activation-SVD input directions, random and weight-SVD output dictionaries achieve similar accuracy. Random output atoms additionally support sharing across compatible layers, reducing the storage of layer-specific output bases as examined in Appendix § 9.6.1. We therefore use activation-SVD AA and random, shareable BB as the default construction.

Conclusion

We introduced SCALAR, a PEFT method that combines competitive adaptation performance with substantially fewer trainable parameters, lower peak training memory, and higher throughput. The method learns layer-specific scalar coefficients over activation-derived input directions and grouped random output dictionaries. Compact coefficient updates reduce the trainable state, grouped evaluation limits token-dependent computation, and shared dictionaries reduce fixed-basis storage. Our analysis relates the task-calibrated input geometry to the initial learning signal, while the experiments and structural ablations demonstrate its parameter and training efficiency. These properties make SCALAR a promising approach for on-device personalization under limited memory and compute budgets.

Proofs and Supporting Analysis

This section proves the activation-coverage and initial-gradient results in § 3. The supporting analysis then connects the activation-derived input subspace to the gradients of the trainable coefficients of SCALAR.

Proof of Theorem 3.5

Proof. Using E[xxT]=Σ\mathbb{E}[x x^{\mathsf{T}}] = \boldsymbol{\Sigma} and cyclic invariance of the trace,

Φ(A)=E[∥Ax∥22]=E[xTATAx]=E[Tr⁡(AxxTAT)]=Tr⁡(AΣAT).\begin{aligned} \Phi(A) &= \mathbb{E}[\lVert Ax\rVert_2^2] = \mathbb{E}[x^{\mathsf{T}} A^{\mathsf{T}} A x] \\ &= \mathbb{E}[\operatorname{Tr}(A x x^{\mathsf{T}} A^{\mathsf{T}})] = \operatorname{Tr}(A\boldsymbol{\Sigma}A^{\mathsf{T}}). \end{aligned}

Let

Σ=UΛUT,Λ=Diag⁡(λ1,…,λn),λ1≥⋯≥λn≥0,\boldsymbol{\Sigma} = U\Lambda U^{\mathsf{T}}, \qquad\Lambda= \operatorname{Diag}(\lambda_1,\ldots,\lambda_n), \qquad\lambda_1 \ge\cdots\ge\lambda_n \ge0,

and define C:=AUC := AU. Since AAT=IRAA^{\mathsf{T}} = I_R and UU is orthogonal,

CCT=AUUTAT=IR,CC^{\mathsf{T}} = AUU^{\mathsf{T}}A^{\mathsf{T}} = I_R,
Φ(A)=Tr⁡(CΛCT)=∑i=1nλi∥ci∥22,\Phi(A) = \operatorname{Tr}(C\Lambda C^{\mathsf{T}}) = \sum_{i=1}^{n} \lambda_i \lVert c_i\rVert_2^2,

where cic_i is the ii-th column of CC.

Set

qi:=∥ci∥22,P:=CTC.q_i := \lVert c_i\rVert_2^2, \qquad P := C^{\mathsf{T}}C.

Then

P2=CTCCTC=CTC=P,P^2 = C^{\mathsf{T}}CC^{\mathsf{T}}C = C^{\mathsf{T}}C = P,
qi=eiTPei,0≤qi≤1,q_i = e_i^{\mathsf{T}}Pe_i, \qquad0 \le q_i \le1,
∑i=1nqi=Tr⁡(P)=Tr⁡(CCT)=R.\sum_{i=1}^{n}q_i = \operatorname{Tr}(P) = \operatorname{Tr}(CC^{\mathsf{T}}) = R.

Thus,

Φ(A)=∑i=1Rλiqi+∑i=R+1nλiqi≤∑i=1Rλiqi+λR∑i=R+1nqi=∑i=1Rλiqi+λR(R−∑i=1Rqi)=RλR+∑i=1R(λi−λR)qi≤RλR+∑i=1R(λi−λR)=∑i=1Rλi.(3)\begin{aligned} \Phi(A) &= \sum_{i=1}^{R}\lambda_iq_i + \sum_{i=R+1}^{n}\lambda_iq_i \le\sum_{i=1}^{R}\lambda_iq_i + \lambda_R\sum_{i=R+1}^{n}q_i \\ &= \sum_{i=1}^{R}\lambda_iq_i + \lambda_R\left(R-\sum_{i=1}^{R}q_i\right) = R\lambda_R + \sum_{i=1}^{R}(\lambda_i-\lambda_R)q_i \\ &\le R\lambda_R + \sum_{i=1}^{R}(\lambda_i-\lambda_R) = \sum_{i=1}^{R}\lambda_i. \tag*{(3)} \end{aligned}

For Asvd=URTA_{\mathrm{svd}} = U_R^{\mathsf{T}},

AsvdΣAsvdT=Diag⁡(λ1,…,λR),A_{\mathrm{svd}}\boldsymbol{\Sigma}A_{\mathrm{svd}}^{\mathsf{T}} = \operatorname{Diag}(\lambda_1,\ldots,\lambda_R),
Φ(Asvd)=∑i=1Rλi\Phi(A_{\mathrm{svd}}) = \sum_{i=1}^{R}\lambda_i

so the bound in equation 3 is attained.

For the uniformly random RR-dimensional row subspace, let

Prand:=ArandTArand.P_{\mathrm{rand}} := A_{\mathrm{rand}}^{\mathsf{T}}A_{\mathrm{rand}}.

This is a rank-RR orthogonal projector. Rotational invariance gives, for every orthogonal Q∈Rn×nQ \in\mathbb{R}^{n\times n},

Prand=dQTPrandQ,P_{\mathrm{rand}} \mathrel{\mathop{=}^{\mathrm{d}}} Q^{\mathsf{T}}P_{\mathrm{rand}}Q,
S:=E[Prand]=QTSQ,∀Q∈O(n).S := \mathbb{E}[P_{\mathrm{rand}}] = Q^{\mathsf{T}}SQ, \qquad\forall Q \in O(n).

Hence S=aInS=aI_n for some scalar aa, and

an=Tr⁡(S)=E[Tr⁡(Prand)]=R, and E[Prand]=RnIn.an=\operatorname{Tr}(S)=\mathbb{E}[\operatorname{Tr}(P_{\mathrm{rand}})]=R,\text{ and }\mathbb{E}[P_{\mathrm{rand}}]=\frac{R}{n}I_n.

Therefore,

EArand[Φ(Arand)]=E[Tr⁡(ArandΣArandT)]=E[Tr⁡(PrandΣ)]=Tr⁡(E[Prand]Σ)=RnTr⁡(Σ)=Rn∑i=1nλi.(4)\begin{aligned} \mathbb{E}_{A_{\mathrm{rand}}}[\Phi(A_{\mathrm{rand}})]&=\mathbb{E}\left[\operatorname{Tr}\left(A_{\mathrm{rand}}\Sigma A_{\mathrm{rand}}^{\mathsf{T}}\right)\right]\\ &=\mathbb{E}[\operatorname{Tr}(P_{\mathrm{rand}}\Sigma)]\\ &=\operatorname{Tr}(\mathbb{E}[P_{\mathrm{rand}}]\Sigma)\\ &=\frac{R}{n}\operatorname{Tr}(\Sigma)=\frac{R}{n}\sum_{i=1}^{n}\lambda_i. \tag*{(4)} \end{aligned}

Finally, for R<nR<n,

1R∑i=1Rλi≥1n∑i=1nλi≥1n−R∑i=R+1nλi⟹∑i=1Rλi≥Rn∑i=1nλi.\frac{1}{R}\sum_{i=1}^{R}\lambda_i\ge\frac{1}{n}\sum_{i=1}^{n}\lambda_i\ge\frac{1}{n-R}\sum_{i=R+1}^{n}\lambda_i\Longrightarrow\sum_{i=1}^{R}\lambda_i\ge\frac{R}{n}\sum_{i=1}^{n}\lambda_i.

Together with equation (3) and equation (4),

Φ(Asvd)=∑i=1Rλi≥EArand[Φ(Arand)].\Phi(A_{\mathrm{svd}})=\sum_{i=1}^{R}\lambda_i\ge\mathbb{E}_{A_{\mathrm{rand}}}[\Phi(A_{\mathrm{rand}})].

Equality for R<nR<n requires equality between the average of the first RR eigenvalues and that of the remaining n−Rn-R eigenvalues. Since λ1≥⋯≥λn\lambda_1\ge\cdots\ge\lambda_n, this occurs only when λ1=⋯=λn\lambda_1=\cdots=\lambda_n, equivalently Σ=aIn\Sigma=aI_n for some a≥0a\ge0. Hence the comparison is strict whenever R<nR<n and Σ\Sigma is not a scalar multiple of InI_n. The proof is completed.

Proof of Theorem 3.6

Proof. At B=0B=0, for an arbitrary perturbation dB\mathrm{d}B,

dz=dBAx,\mathrm{d}z=\mathrm{d}BAx,
dℓ=gTdz=gTdBAx=Tr⁡(gTdBAx)=Tr⁡[(g(Ax)T)TdB].\mathrm{d}\ell=g^{\mathsf{T}}\mathrm{d}z=g^{\mathsf{T}}\mathrm{d}BAx=\operatorname{Tr}(g^{\mathsf{T}}\mathrm{d}BAx)=\operatorname{Tr}\left[(g(Ax)^{\mathsf{T}})^{\mathsf{T}}\mathrm{d}B\right].

Hence

∇Bℓ=g(Ax)T,\nabla_B\ell=g(Ax)^{\mathsf{T}},
∥∇Bℓ∥F2=∥g(Ax)T∥F2=∥g∥22∥Ax∥22.\|\nabla_B\ell\|_F^2=\|g(Ax)^{\mathsf{T}}\|_F^2=\|g\|_2^2\|Ax\|_2^2.

Therefore, using equation (1),

G(A)=E[∥∇Bℓ∥F2]=E[∥g∥22∥Ax∥22]=E[∥g∥22xTATAx]=Tr⁡(AE[∥g∥22xxT]AT)=Tr⁡(AMAT).\begin{aligned} \mathcal{G}(A)&=\mathbb{E}\left[\|\nabla_B\ell\|_F^2\right]=\mathbb{E}\left[\|g\|_2^2\|Ax\|_2^2\right]=\mathbb{E}\left[\|g\|_2^2x^{\mathsf{T}}A^{\mathsf{T}}Ax\right]\\ &=\operatorname{Tr}\left(A\mathbb{E}\left[\|g\|_2^2xx^{\mathsf{T}}\right]A^{\mathsf{T}}\right)=\operatorname{Tr}(AMA^{\mathsf{T}}). \end{aligned}

Since the row-orthonormal feasible set is compact and G\mathcal{G} is continuous, a maximizer A∗A_* exists. Define

E:=M−cΣ.E:=M-c\Sigma.

By Assumption 3.4, ∥E∥2≤ε\|E\|_2\le\varepsilon. Since EE is symmetric, for any feasible AA,

−εIn⪯E⪯εIn⟹−εAAT⪯AEAT⪯εAAT⟹−εIR⪯AEAT⪯εIR⟹∣Tr⁡(AEAT)∣≤Rε.(5)-\varepsilon I_n\preceq E\preceq\varepsilon I_n \Longrightarrow-\varepsilon AA^{\mathsf{T}}\preceq AEA^{\mathsf{T}}\preceq\varepsilon AA^{\mathsf{T}} \Longrightarrow-\varepsilon I_R\preceq AEA^{\mathsf{T}}\preceq\varepsilon I_R \Longrightarrow\left|\operatorname{Tr}(AEA^{\mathsf{T}})\right|\le R\varepsilon. \tag*{(5)}

Using M=cΣ+EM=c\Sigma+E, Theorem 3.5, and equation (5),

G(A∗)−G(Asvd)=c[Tr⁡(A∗ΣA∗T)−Tr⁡(AsvdΣAsvdT)]+Tr⁡(A∗EA∗T)−Tr⁡(AsvdEAsvdT)≤Tr⁡(A∗EA∗T)−Tr⁡(AsvdEAsvdT)≤2Rε.\begin{aligned} \mathcal{G}(A_{\ast})-\mathcal{G}(A_{\mathrm{svd}}) &=c\left[\operatorname{Tr}\left(A_{\ast}\Sigma A_{\ast}^{\mathsf{T}}\right)-\operatorname{Tr}\left(A_{\mathrm{svd}}\Sigma A_{\mathrm{svd}}^{\mathsf{T}}\right)\right] \\ &\quad+\operatorname{Tr}\left(A_{\ast}EA_{\ast}^{\mathsf{T}}\right)-\operatorname{Tr}\left(A_{\mathrm{svd}}EA_{\mathrm{svd}}^{\mathsf{T}}\right) \\ &\le\operatorname{Tr}\left(A_{\ast}EA_{\ast}^{\mathsf{T}}\right)-\operatorname{Tr}\left(A_{\mathrm{svd}}EA_{\mathrm{svd}}^{\mathsf{T}}\right) \\ &\le2R\varepsilon. \end{aligned}

Since A∗A_{\ast} maximizes G\mathcal{G},

0≤G(A∗)−G(Asvd)≤2Rε.0\le\mathcal{G}(A_{\ast})-\mathcal{G}(A_{\mathrm{svd}})\le2R\varepsilon.

In particular, when ε=0\varepsilon=0,

G(A∗)=G(Asvd),\mathcal{G}(A_{\ast})=\mathcal{G}(A_{\mathrm{svd}}),

so AsvdA_{\mathrm{svd}} is also a maximizer.

For a uniformly random RR-dimensional row subspace, let

Prand:=ArandTArand.P_{\mathrm{rand}}:=A_{\mathrm{rand}}^{\mathsf{T}}A_{\mathrm{rand}}.

From the proof of Theorem 3.5,

E[Prand]=RnIn.\mathbb{E}[P_{\mathrm{rand}}]=\frac{R}{n}I_n.

Thus,

EArand[G(Arand)]=E[Tr⁡(ArandMArandT)]=E[Tr⁡(PrandM)]=Tr⁡(E[Prand]M)=RnTr⁡(M).\begin{aligned} \mathbb{E}_{A_{\mathrm{rand}}}\left[\mathcal{G}(A_{\mathrm{rand}})\right] &=\mathbb{E}\left[\operatorname{Tr}\left(A_{\mathrm{rand}}MA_{\mathrm{rand}}^{\mathsf{T}}\right)\right] =\mathbb{E}\left[\operatorname{Tr}\left(P_{\mathrm{rand}}M\right)\right] \\ &=\operatorname{Tr}\left(\mathbb{E}[P_{\mathrm{rand}}]M\right) =\frac{R}{n}\operatorname{Tr}(M). \end{aligned}

When G(A∗)>0\mathcal{G}(A_{\ast})>0, dividing the suboptimality bound by G(A∗)\mathcal{G}(A_{\ast}) gives

G(A∗)−G(Asvd)G(A∗)≤2RεG(A∗).\frac{\mathcal{G}(A_{\ast})-\mathcal{G}(A_{\mathrm{svd}})}{\mathcal{G}(A_{\ast})} \le\frac{2R\varepsilon}{\mathcal{G}(A_{\ast})}.

This completes the proof.

Gradient Signal of the Scalar Coefficients

We provide the detailed connection between the activation-derived input basis and the optimization of the scalar coefficients.

For a single adapted linear layer, let x∈Rnx\in\mathbb{R}^{n} denote its input and g∈Rmg\in\mathbb{R}^{m} the gradient of the loss with respect to its output. We analyze the local gradient contribution at zero-initialized coefficients, Γ=0\Gamma=0. From equation (2), the gradient of an individual coefficient is

∂ℓ∂γr,h=s(br,hTg)(arTx).\frac{\partial\ell}{\partial\gamma_{r,h}}=s\left(b_{r,h}^{\mathsf{T}}g\right)\left(a_r^{\mathsf{T}}x\right).

Therefore, the gradient received by each scalar factorizes into the activation along its input direction and the projection of the output gradient onto its random output atom.

Let Γ∈RR×H\Gamma\in\mathbb{R}^{R\times H} collect all trainable scalar coefficients. Assume that the output atoms are independent of (x,g)(x,g) and satisfy

E[br,hbr,hT]=σb2Im.\mathbb{E}\left[b_{r,h}b_{r,h}^{\mathsf{T}}\right]=\sigma_b^2I_m.

Define

J(A):=EBE[∥∇Γℓ∥F2].\mathcal{J}(A):=\mathbb{E}_{B}\mathbb{E}\left[\left\|\nabla_{\Gamma}\ell\right\|_{F}^{2}\right].

The expected initial coefficient-gradient energy is

J(A)=s2∑r=1R∑h=1HE[(br,hg)2(arTx)2]=s2Hσb2E[∥g∥22∑r=1R(arTx)2]=s2Hσb2E[∥g∥22∥Ax∥22]=s2Hσb2G(A).(6)\begin{aligned} \mathcal{J}(A) &= s^2 \sum_{r=1}^{R}\sum_{h=1}^{H}\mathbb{E}\left[\left(\mathbf{b}_{r,h}\mathbf{g}\right)^2\left(\mathbf{a}_r^{\mathsf{T}}\mathbf{x}\right)^2\right] \\ &=s^2H\sigma_b^2\mathbb{E}\left[\|\mathbf{g}\|_2^2\sum_{r=1}^{R}\left(\mathbf{a}_r^{\mathsf{T}}\mathbf{x}\right)^2\right] \\ &=s^2H\sigma_b^2\mathbb{E}\left[\|\mathbf{g}\|_2^2\|A\mathbf{x}\|_2^2\right] \\ &=s^2H\sigma_b^2\mathcal{G}(A). \tag*{(6)} \end{aligned}

Under the stated isotropy and independence assumptions, equation (6) gives the exact relation

J(A)=s2Hσb2Tr⁡(AMAT).\mathcal{J}(A)=s^2H\sigma_b^2\operatorname{Tr}\left(AMA^{\mathsf{T}}\right).

Thus, the expected initial coefficient-gradient energy is governed by the gradient-weighted activation second moment MM. When M=cΣM=c\Sigma, it is proportional to the captured activation energy Φ(A)\Phi(A). More generally, for the maximizer A∗A_\ast in Theorem 3.6,

0≤J(A∗)−J(Asvd)≤2s2Hσb2Rε.0\leq\mathcal{J}(A_\ast)-\mathcal{J}(A_{\mathrm{svd}})\leq2s^2H\sigma_b^2R\varepsilon.

This bound quantifies how the covariance-alignment error affects the initial gradient energy of the coefficients of SCALAR. The expectation measures the squared norm of individual local gradient contributions. Figure 3 complements the analysis by comparing the empirical gradients of the trainable adapter parameters under two input-side initializations.

Figure 3. Gradient norms of the trainable adapter parameters when the input-side factor AA uses activation-SVD or random initialization during Qwen2.5-7B fine-tuning on commonsense reasoning. The two input-side initializations are matched in scale. Activation-SVD initialization yields larger observed gradient norms over most of this run.

Implementation Details

This section supplements § 4 with implementation details, training hyperparameters, baseline implementations, and the complete commonsense reasoning results. Unless stated otherwise, all implementation choices are shared across methods whenever their parameterizations permit.

Implementation Optimizations

We summarize the main implementation choices used in our experiments. Notation follows § 4. For each adapted layer, SCALAR maintains a task-aware input matrix A∈RR×nA\in\mathbb{R}^{R\times n} and a trainable coefficient matrix Γ∈RR×H\Gamma\in\mathbb{R}^{R\times H}, where RR controls the maximum rank of the aggregated update and HH is the number of random output atoms per input direction. The random output dictionaries {Br}r=1R\{B_r\}_{r=1}^{R} are frozen and may be shared across layers with the same module type. Coefficient-only optimization keeps adapter gradients and optimizer states compact. Grouping reuses each input projection across its output atoms, reducing token-dependent work. Cross-layer sharing limits the resident storage of the frozen output dictionaries.

Optional Sharing of Output Dictionaries

Because the output atoms are sampled independently of pretrained weights, compatible layers may reuse the same frozen dictionaries {Br}r=1R\{\mathbf{B}_r\}_{r=1}^{R}. For example, query-projection layers with the same output dimension can reuse one dictionary while retaining their own activation-derived A\mathbf{A} and trainable Γ\mathbf{\Gamma}. We use this option in the default implementation to reduce the storage of frozen random buffers. Appendix C.6.1 compares the shared and unshared variants.

Fast SVD for Input Calibration

To construct A\mathbf{A}, we collect input activations from a small calibration set. Instead of storing the full activation matrix X∈RS×n\mathbf{X} \in\mathbb{R}^{S \times n} on the accelerator, we accumulate the Gram matrix

C=XTX\mathbf{C} = \mathbf{X}^{\mathsf{T}}\mathbf{X}

over streaming token blocks. We then extract the leading RR right singular directions using randomized eigendecomposition [14]. This avoids dense SVD on X\mathbf{X} and is efficient when R≪nR \ll n. Modules that receive the same hidden state within a transformer layer, such as q_proj, k_proj, and v_proj, share one calibration statistic and one decomposition per layer.

To validate this Fast SVD against a dense (full) eigendecomposition of C\mathbf{C}, we compare both the wall-clock cost of the top-RR solve and the recovered RR-dimensional subspaces on the same Gram matrices (Qwen2.5-7B; R=16R = 16; averaged over 84 unique input groups). Let Vfast,Vfull∈Rn×R\mathbf{V}_{\mathrm{fast}}, \mathbf{V}_{\mathrm{full}} \in\mathbb{R}^{n \times R} collect the top-RR right singular vectors, and let Qfast,Qfull\mathbf{Q}_{\mathrm{fast}}, \mathbf{Q}_{\mathrm{full}} be orthonormal bases of their column spaces obtained by QR. The principal angles {θi}i=1R\{\theta_i\}_{i=1}^{R} satisfy

cos⁡θi=σi(QfastTQfull),i=1,…,R,\cos\theta_i = \sigma_i\left(\mathbf{Q}_{\mathrm{fast}}^{\mathsf{T}}\mathbf{Q}_{\mathrm{full}}\right), \qquad i = 1,\ldots,R,

where σi(⋅)\sigma_i(\cdot) denotes the ii-th singular value. We report the mean cosine alignment

cos⁡‾=1R∑i=1Rcos⁡θi(7)\overline{\cos} = \frac{1}{R}\sum_{i=1}^{R}\cos\theta_i \tag*{(7)}

and the principal-angle affinity 1R∑i=1Rcos⁡2θi\frac{1}{R}\sum_{i=1}^{R}\cos^2\theta_i. In this implementation, Table 6 shows that the randomized solver reduces the measured top-RR eigensolve time by 19.6×\times on the collected Gram matrices. The recovered subspaces are also close under the reported principal-angle metrics (cos⁡‾=0.998\overline{\cos} = 0.998). The timing comparison covers the eigensolves on these collected Gram matrices.

Fast SVDFull SVD
Eigen­decomposition time (s)6.29123.0
Speedup vs. full19.6×\times1×\times
Mean cosine cos equation 70.998
Affinity 1R∑icos⁡2θi\frac{1}{R}\sum_i\cos^2\theta_i0.996
Mean principal angle (deg)0.500

Table 6. Fast (randomized) vs. full eigendecomposition for activation Gram C\mathbf{C} on Qwen2.5-7B (R=16R = 16). Time is the summed top-RR solve over all groups; subspace metrics compare Fast and full on the same Grams.

Variance-Preserving Atom Scaling

SCALAR represents each layer update as a sum of K=RHK = RH rank-one atoms:

ΔW=s∑r=1R∑h=1Hγr,hbr,harT.\Delta W = s \sum_{r=1}^{R} \sum_{h=1}^{H} \gamma_{r,h} \mathbf{b}_{r,h} \mathbf{a}_{r}^{\mathsf{T}}.

Here, the overall update has rank at most RR, while each atom br,harT\mathbf{b}_{r,h}\mathbf{a}_{r}^{\mathsf{T}} has rank 1. To keep the update scale stable when RR or HH changes, we use

s=αK,α=1,K=RH.(8)s = \frac{\alpha}{\sqrt{K}}, \qquad\alpha= 1, \qquad K = RH. \tag*{(8)}

We fix α=1\alpha= 1 across tasks and models. The factor 1/K1/\sqrt{K} normalizes the aggregate scale of the KK output atoms.

The motivation is variance preservation. For fixed coefficients and input projections of comparable scale, independent zero-mean output atoms with matched second moments have vanishing cross terms in expectation. The squared norm of the unscaled atom sum then grows linearly with KK, up to the coefficient- and activation-dependent factor:

E[∥∑r=1R∑h=1Hγr,hbr,h(arTx)∥22]=O(K).\mathbb{E}\left[\left\lVert\sum_{r=1}^{R}\sum_{h=1}^{H}\gamma_{r,h}\mathbf{b}_{r,h}\left(\mathbf{a}_{r}^{\mathsf{T}}x\right)\right\rVert_{2}^{2}\right] = O(K).

Under these conditions, the typical unscaled update norm grows as O(K)O(\sqrt{K}). Multiplication by 1/K1/\sqrt{K} compensates for this dependence on dictionary size.

The initial coefficient-gradient energy gives a second interpretation of this scaling. At zero-initialized coefficients, the gradient of one scalar is

∂ℓ∂γr,h=1K(br,hTg)(arTx).\frac{\partial\ell}{\partial\gamma_{r,h}} = \frac{1}{\sqrt{K}}\left(\mathbf{b}_{r,h}^{\mathsf{T}}g\right)\left(\mathbf{a}_{r}^{\mathsf{T}}x\right).

Substituting s=1/RHs = 1/\sqrt{RH} into equation (6) gives

J(A)=σb2RTr⁡(AMAT).\mathcal{J}(A) = \frac{\sigma_b^2}{R}\operatorname{Tr}\left(AMA^{\mathsf{T}}\right).

For fixed RR and output-atom variance, this expected initial gradient energy is independent of HH. Increasing HH therefore expands the output dictionary while maintaining the same expected initial coefficient-gradient energy under the stated assumptions.

Efficient Adapter Forward Pass

The adapter output can be evaluated as

ΔWx=s∑r=1R(arTx)Brγr.\Delta W x = s \sum_{r=1}^{R}\left(\mathbf{a}_{r}^{\mathsf{T}}x\right)\mathbf{B}_{r}\gamma_{r}.

During training, we first compute

h=Ax,h = Ax,

then combine the random output atoms using the trainable coefficients γr\gamma_r. The resulting effective output vectors are reused across the tokens in the batch. This keeps dictionary assembly outside the token-dependent matrix products and avoids explicitly forming the full update matrix ΔW\Delta W during training. Once training is finished, we may cache the effective output directions

b~r=Brγr\widetilde{\mathbf{b}}_r = \mathbf{B}_r\gamma_r

or merge ΔW\Delta W into the pretrained weight, preserving the inference-time behavior of standard LoRA-style adapters.

Hyperparameters

We detail the hyperparameters used to reproduce our experimental results. All methods are optimized with AdamW using β1=0.9\beta_{1}=0.9, β2=0.999\beta_{2}=0.999, and ϵ=1e−8\epsilon=1\mathrm{e}{-8}. We use BF16 precision to reduce memory consumption and train with Distributed Data Parallel (DDP). Gradient synchronization is implemented using PyTorch’s [31] torch.distributed framework. Following a widely used configuration for LLM adaptation, we apply all PEFT methods to the same set of Transformer linear layers. This all-linear setting provides sufficient adaptation capacity and ensures that differences across methods arise from their parameterizations rather than from different target-module selections.

Different PEFT methods have different preferred learning-rate ranges and scaling conventions. For baselines with a LoRA-style scaling factor, we use their default setting, namely α=rank⁡\alpha=\operatorname{rank}. For SCALAR, we use the fixed scaling in equation (8), s=1/RHs=1/\sqrt{RH}, equivalently α/K\alpha/\sqrt{K} with α=1\alpha=1 and K=RHK=RH. This scaling is fixed across all tasks and models.

For compactness, define the learning-rate search spaces as

Speft={1.0e−5,2.5e−5,5.0e−5,1.0e−4,2.5e−4,5.0e−4,1.0e−3}.S_{\mathrm{peft}}=\{1.0\mathrm{e}{-5},2.5\mathrm{e}{-5},5.0\mathrm{e}{-5},1.0\mathrm{e}{-4},2.5\mathrm{e}{-4},5.0\mathrm{e}{-4},1.0\mathrm{e}{-3}\}.
Ssmall={1.0e−6,2.5e−6,5.0e−6,1.0e−5,2.5e−5,5.0e−5,1.0e−4}.S_{\mathrm{small}}=\{1.0\mathrm{e}{-6},2.5\mathrm{e}{-6},5.0\mathrm{e}{-6},1.0\mathrm{e}{-5},2.5\mathrm{e}{-5},5.0\mathrm{e}{-5},1.0\mathrm{e}{-4}\}.

and

Slarge={1.0e−4,2.5e−4,5.0e−4,1.0e−3,2.5e−3,5.0e−3,1.0e−2}.S_{\mathrm{large}}=\{1.0\mathrm{e}{-4},2.5\mathrm{e}{-4},5.0\mathrm{e}{-4},1.0\mathrm{e}{-3},2.5\mathrm{e}{-3},5.0\mathrm{e}{-3},1.0\mathrm{e}{-2}\}.

We tune LoRA, NOLA, LoRA-XS, and SCALAR over SpeftS_{\mathrm{peft}}, RandLoRA and full-parameter fine-tuning over SsmallS_{\mathrm{small}}, and VeRA over SlargeS_{\mathrm{large}}.

All methods are fine-tuned for three epochs with batch size 32 and a cosine learning-rate scheduler. The first 100 steps are used for warm-up. We report results over three random seeds, (0, 1, 2). The maximum sequence length is set to 384, which is sufficient for commonsense reasoning inputs. For our method, we evaluate three default settings: KK in {1024,2048,4096}\{1024,2048,4096\}. A larger KK corresponds to higher model capacity and a larger number of trainable parameters. The trainable parameter counts for different values of KK are reported in Table 7. Table 8 summarizes the main hyperparameter settings.

ModelK=128K=128K=256K=256K=512K=512
Qwen2.5-7B25,088 (0.0003%)50,176 (0.0006%)100,352 (0.0013%)
LLaMA3-8B28,672 (0.0003%)57,344 (0.0006%)114,688 (0.0014%)
ModelK=1024K=1024K=2048K=2048K=4096K=4096
Qwen2.5-7B200,704 (0.0026%)401,408 (0.0052%)802,816 (0.0105%)
LLaMA3-8B229,376 (0.0028%)458,752 (0.0057%)917,504 (0.0114%)

Table 7. Trainable params (%) with different KK on Qwen2.5-7B and LLaMA3-8B.

Table 8: Hyperparameters for fine-tuning LLaMA3-8B and Qwen2.5-7B on commonsense reasoning tasks.

HyperparametersFTLoRAVeRANOLALoRA-XSRandLoRASCALAR
OptimizerAdamW
LR schedulerCosine
PrecisionBF16
Batch size32
Epochs3
Warmup steps100
Seeds{0,1,2}\{0, 1, 2\}
Max sequence length384
Target modulesAll linear projection modules
Rank (rr)-{1,8}\{1, 8\}{1024,2048}\{1024, 2048\}{1,8}\{1, 8\}{32,48,64}\{32, 48, 64\}{2048,4096}\{2048, 4096\}{8,16}\{8, 16\}
Scaling (α\alpha)-rrDefaultrrrrrrequation 8
LR (Qwen)2.5e−52.5\mathrm{e}{-5}1.0e−41.0\mathrm{e}{-4}2.5e−32.5\mathrm{e}{-3}1.0e−31.0\mathrm{e}{-3}5.0e−45.0\mathrm{e}{-4}2.5e−52.5\mathrm{e}{-5}5.0e−45.0\mathrm{e}{-4}
LR (LLaMA)1.0e−51.0\mathrm{e}{-5}1.0e−41.0\mathrm{e}{-4}2.5e−32.5\mathrm{e}{-3}5.0e−45.0\mathrm{e}{-4}5.0e−45.0\mathrm{e}{-4}2.5e−52.5\mathrm{e}{-5}5.0e−45.0\mathrm{e}{-4}

Table 8.

For mathematical reasoning and code generation, we fine-tune on MetaMath [41] and CodeFeedback [46], respectively, for one epoch. We use a batch size of 256, a maximum sequence length of 1024, a 100-step warm-up, and cosine learning-rate decay. The complete hyperparameter settings are provided in Table 9.

Table 9: Hyperparameters for fine-tuning Qwen2.5-7B on mathematical reasoning and code generation tasks.

HyperparametersFTLoRAVeRANOLALoRA-XSRandLoRASCALAR
LR schedulerCosine
PrecisionBF16
Batch size256
Epochs1
Warmup steps100
Max sequence length1024
Target modulesAll linear projection modules
Rank (rr)-{1,8}\{1, 8\}256124409616
Scaling (α\alpha)-rrDefaultrrrrrrequation 8
LR (Math)5.0e−65.0\mathrm{e}{-6}1.0e−51.0\mathrm{e}{-5}2.5e−42.5\mathrm{e}{-4}5.0e−55.0\mathrm{e}{-5}2.5e−42.5\mathrm{e}{-4}2.5e−52.5\mathrm{e}{-5}2.5e−52.5\mathrm{e}{-5}
LR (Code)2.5e−62.5\mathrm{e}{-6}{1.0e−5,1.0e−4}\{1.0\mathrm{e}{-5}, 1.0\mathrm{e}{-4}\}5.0e−45.0\mathrm{e}{-4}1.0e−41.0\mathrm{e}{-4}1.0e−41.0\mathrm{e}{-4}2.5e−52.5\mathrm{e}{-5}5.0e−45.0\mathrm{e}{-4}

Table 9.

Baseline Implementations

All experiments use a common training stack based on Hugging Face transformers and peft (v0.18.0), launched with torchrun and the Hugging Face Trainer. LoRA [17] and VeRA [20] use the standard PEFT APIs (LoraConfig and VeraConfig). RandLoRA [1] uses the upstream RandLoraConfig integration. LoRA-XS [3] is implemented on top of PEFT LoRA with SVD-derived frozen AA and BB and a trainable r×rr \times r matrix MM. NOLA [19] uses an in-house adapter module checked against the reference parameterization because it is not available as a native PEFT tuner. Commonsense reasoning uses generative decoding with regex-based answer extraction following the DoRA/LoRA-XS evaluation style; mathematical reasoning follows the PiSSA setup [26]; and code generation uses the corresponding prompts with evalplus pass@1 on the base test suites.

Complete Commonsense Reasoning Results

The main text reports a compact comparison at representative parameter budgets. Table 10 provides the corresponding per-task results and variation across runs for the evaluated budgets.

ModelMethodParams (%)BoolQPIQASIQAHSWGARC-EARC-COBQAAvg.
Qwen2.5-7BFT100.0075.389.782.096.189.795.688.493.188.7±0.1
Qwen2.5-7BLoRA (r=1r = 1)0.033174.390.381.995.888.796.189.291.788.5±0.1
Qwen2.5-7BLoRA (r=8r = 8)0.264374.690.481.996.389.395.989.592.388.8±0.1
Qwen2.5-7BVeRA0.020873.790.281.595.687.796.388.891.588.2±0.2
Qwen2.5-7BVeRA0.023573.690.081.895.888.096.189.591.288.2±0.1
Qwen2.5-7BNOLA0.002673.289.382.595.888.096.589.091.288.2±0.1
Qwen2.5-7BNOLA0.005273.389.682.195.888.896.689.191.388.3±0.2
Qwen2.5-7BNOLA0.010573.690.082.095.888.696.489.291.588.4±0.0
Qwen2.5-7BLoRA-XS0.002672.589.282.595.486.196.189.491.587.8±0.2
Qwen2.5-7BLoRA-XS0.005272.689.682.795.586.596.588.591.287.9±0.1
Qwen2.5-7BLoRA-XS0.010573.489.882.495.687.296.588.590.988.0±0.2
Qwen2.5-7BRandLoRA0.017573.989.581.295.888.396.189.190.888.1±0.2
Qwen2.5-7BRandLoRA0.022573.689.681.095.788.396.289.291.888.2±0.2
Qwen2.5-7BSCALAR (K=1024K = 1024)0.002673.490.382.496.087.596.889.092.388.5±0.1
Qwen2.5-7BSCALAR (K=2048K = 2048)0.005273.790.082.596.188.296.789.291.988.6±0.0
Qwen2.5-7BSCALAR (K=4096K = 4096)0.010574.090.382.296.289.396.589.593.088.9±0.1
LLaMA3-8BFT100.0076.290.483.096.789.494.185.188.888.0±0.2
LLaMA3-8BLoRA (r=1r = 1)0.032674.389.981.996.389.593.383.389.087.2±0.2
LLaMA3-8BLoRA (r=8r = 8)0.260475.990.083.096.689.493.383.187.887.4±0.3
LLaMA3-8BVeRA0.019974.888.981.695.987.192.882.187.586.3±0.3
LLaMA3-8BVeRA0.022875.389.082.495.788.193.382.185.586.4±0.0
LLaMA3-8BNOLA0.002874.089.181.896.287.893.082.788.386.6±0.3
LLaMA3-8BNOLA0.005774.789.582.596.488.493.083.288.287.0±0.3
LLaMA3-8BNOLA0.011474.789.582.896.488.392.982.987.486.9±0.1
LLaMA3-8BLoRA-XS0.002873.088.781.295.586.091.980.784.585.1±0.0
LLaMA3-8BLoRA-XS0.006473.088.682.595.986.992.581.787.186.0±0.0
LLaMA3-8BLoRA-XS0.011473.388.382.796.088.192.481.987.786.2±0.2
LLaMA3-8BRandLoRA0.020375.189.582.395.586.793.281.788.486.6±0.0
LLaMA3-8BRandLoRA0.026975.489.182.395.687.292.982.587.186.5±0.2
LLaMA3-8BSCALAR (K=1024K = 1024)0.002873.589.282.496.288.092.882.387.686.5±0.3
LLaMA3-8BSCALAR (K=2048K = 2048)0.005774.289.782.596.387.593.383.588.386.9±0.1
LLaMA3-8BSCALAR (K=4096K = 4096)0.011474.089.682.896.888.893.183.989.287.3±0.1

Table 10. Accuracy comparison (3 independent runs) of Qwen2.5-7B and LLaMA3-8B with various PEFT methods on eight commonsense reasoning benchmarks. This table supplements the main commonsense reasoning table by reporting the performance of each baseline under different trainable parameter budgets.

At the largest reported coefficient budgets, SCALAR obtains average accuracies of 88.9 on Qwen2.5-7B and 87.3 on LLaMA3-8B, compared with 88.8 and 87.4 for rank-8 LoRA. The trainable-parameter percentages are 0.0105% and 0.0114% for SCALAR, compared with 0.2643% and 0.2604% for LoRA. These results show comparable accuracy on both backbones with substantially smaller trainable-parameter budgets.

Additional Results and Ablations

This section provides the diagnostic experiments and ablations referenced in the main text. The results examine direction reweighting, activation coverage, and the accuracy and computational cost of the proposed parameterization.

Learned-Direction Reweighting

This subsection expands the learned-direction reweighting diagnostic introduced in § 3. For each adapted matrix, we first train a rank-one LoRA adapter, freeze its learned input and output directions, discard the learned magnitude, and initialize a new scalar coefficient. The second stage optimizes only these scalars using the same data and training protocol as the rank-one LoRA run. This construction examines coefficient reweighting with task-informed directions obtained from the trained adapter.

Figure 4 shows that the scalar-reweighted model follows a similar training-loss trajectory to rank-one LoRA while optimizing several orders of magnitude fewer parameters. Re-optimizing the magnitudes of these learned directions recovers similar fitting behavior.

Figure 4. Optimization behavior of rank-one LoRA and learned-direction reweighting. The reweighting model freezes the directions learned by LoRA and re-optimizes one scalar coefficient for each adapted matrix. It optimizes 224 coefficients for LLaMA3-8B and 196 for Qwen2.5-7B, and follows a similar loss-reduction trajectory in these runs.

Table 11 reports comparable average accuracy for direction reweighting on both model families. Together with the training curves, these results show that re-optimizing the magnitudes of learned rank-one LoRA directions recovers similar fitting behavior and downstream accuracy. This motivates SCALAR’s construction of a calibrated input basis before fine-tuning, followed by optimization of layer-specific atom weights.

ModelMethodParamsAvg. Acc.
LLaMA3-8BLoRA (r=1r = 1)2,621,44087.2
LLaMA3-8BDirection reweighting22487.5
Qwen2.5-7BLoRA (r=1r = 1)2,523,13688.5
Qwen2.5-7BDirection reweighting19688.7

Table 11. Average accuracy of rank-one LoRA and learned-direction reweighting on eight commonsense reasoning tasks. The reweighting model uses fewer than 0.01% as many trainable parameters as rank-one LoRA and attains similar reported averages in these two settings.

Does LoRA Learn High-Energy Input Subspaces?

We examine how the activation coverage of LoRA’s input subspace changes during fine-tuning. This diagnostic uses the coverage criterion defined in § 3 to describe the learned subspaces.

We train LoRA adapters on a 7B language model. For an adapted linear layer, let At∈RR×nA^t \in\mathbb{R}^{R \times n} denote the input-side LoRA factor at training step tt. Directly measuring ∥Atx∥22\lVert A^t x\rVert_2^2 is not reliable because the LoRA factorization is scale-indeterminate:

BtAt=(c−1Bt)(cAt)B^t A^t = (c^{-1}B^t)(cA^t)

for any nonzero scalar cc. We therefore remove the scale effect by evaluating the row-normalized representative

Aˉt=(At(At)⊤+ϵIR)−12At,\bar{A}^t = \left(A^t(A^t)^\top+ \epsilon I_R\right)^{-\frac{1}{2}} A^t,

where ϵ\epsilon is a small constant for numerical stability. Left multiplication by the positive-definite factor preserves the row subspace of AtA^t. When ϵ=0\epsilon= 0 and AtA^t has full row rank, the rows of Aˉt\bar{A}^t are orthonormal and the construction is exactly invariant to positive global rescaling of AtA^t. The small regularizer used in computation makes this invariance approximate near rank-deficient states.

During training, we reuse the input activations from the normal forward pass and compute the captured activation-energy ratio

ρt=Ex[∥Aˉtx∥22]Ex[∥x∥22].(9)\rho^t = \frac{\mathbb{E}_x\left[\left\lVert\bar{A}^t x\right\rVert_2^2\right]}{\mathbb{E}_x\left[\lVert x\rVert_2^2\right]}. \tag*{(9)}

To compare different layers and module types, we report the relative ratio

ρ^t=ρtρ0,\hat{\rho}^{t} = \frac{\rho^{t}}{\rho^{0}},

where ρ0\rho^{0} is the value at the random-initialized LoRA state. Thus, ρ^t>1\hat{\rho}^{t} > 1 indicates that the current LoRA input subspace captures more normalized activation energy than its initialization. Figure 5 reports ρ^t\hat{\rho}^{t} across adapted module types. The trajectories vary across layers and training steps. Most module groups show higher normalized activation coverage than at initialization, while one down-projection layer falls below its initial value. These measurements describe the activation coverage of subspaces learned by optimizing the task loss. They provide empirical motivation for constructing an activation-calibrated input basis before fine-tuning, as in SCALAR.

Figure 5. Diagnostic training dynamics of LoRA on Qwen2.5-7B. We report the relative normalized captured activation-energy ratio ρ^t=ρt/ρ0\hat{\rho}^{t} = \rho^{t}/\rho^{0}, where ρt\rho^{t} is defined in equation (9). Values above 1 indicate stronger activation alignment than the random-initialized LoRA input subspace. Most module groups in this run show higher normalized activation coverage, with variation across layers and training steps.

Comparison with Enhanced LoRA Variants

We further compare SCALAR with DoRA [25], PiSSA [26], and EVA [30], which represent three complementary improvements over standard LoRA. DoRA introduces a magnitude–direction weight decomposition, PiSSA initializes and optimizes the principal singular components of the pretrained weights, and EVA initializes LoRA from high-variance activation directions with optional adaptive rank allocation. These comparisons place SCALAR alongside methods that improve the LoRA parameterization or its initialization. EVA is particularly relevant because it also exploits activation geometry; however, unlike EVA, which subsequently trains both LoRA factors, SCALAR freezes the activation-derived input basis and learns only scalar combinations of fixed random output atoms.

All methods are evaluated on Qwen2.5-7B using the same training and evaluation protocol. We use identical adapted modules and LoRA ranks for LoRA, DoRA, PiSSA, and EVA, and independently tune their learning rates using the same search budget.

Table 12 reports average accuracies from 88.5 to 88.9 across the evaluated methods. SCALAR reaches 88.9 using approximately one-third of the trainable parameters of LoRA, PiSSA, and EVA, and about one-fifth of those used by DoRA. The comparison supports competitive accuracy at a smaller trainable-parameter budget.

MethodParams (%)Avg. Acc.
LoRA0.033188.5
DoRA0.051388.7
PiSSA0.033188.5
EVA0.033188.7
SCALAR0.010588.9

Table 12. Comparison with enhanced LoRA variants on Qwen2.5-7B. SCALAR achieves comparable reported accuracy with fewer trainable parameters than the evaluated LoRA variants.

EVA and SCALAR both use activation-derived input directions. EVA then optimizes the LoRA factors, while SCALAR retains the calibrated input basis and learns coefficients over its fixed random output dictionary. The reported results compare the resulting parameter–performance trade-offs under the stated configurations.

General Instruction Fine-tuning

We evaluate post-tuning general-domain knowledge performance after instruction fine-tuning on three model families spanning different parameter scales: Gemma3-1B [35], LLaMA3.2-3B [11], and Qwen2.5-7B [39]. For each model, we fine-tune on the 52K-example Alpaca-GPT4 dataset [32] for 3 epochs and evaluate the resulting models on MMLU [15] (5-shot). We report accuracy over the four MMLU subject categories, including STEM, humanities, social sciences, and other domains, together with the overall accuracy. Full-parameter fine-tuning, LoRA (r=8r = 8), and two parameter configurations of SCALAR use the same training data, number of epochs, target modules, and optimization budget for a fair comparison.

Table 13 shows comparable post-tuning MMLU accuracy across the evaluated adaptation methods. The higher-scoring SCALAR configurations obtain 27.13 on Gemma3-1B, 56.79 on LLaMA3.2-3B, and 74.58 on Qwen2.5-7B, using 0.0373%, 0.0250%, and 0.0052% trainable parameters, respectively.

ModelMethodParams (%)STEMHumanitiesSocial SciencesOtherOverall
Gemma3-1BFT100.0027.3727.0424.5425.9726.33
Gemma3-1BLoRA0.648127.8526.8224.6726.6526.54
Gemma3-1BSCALAR (K=1024K = 1024)0.018627.6626.8426.0626.6226.81
Gemma3-1BSCALAR (K=2048K = 2048)0.037327.7827.1025.6727.9727.13
LLaMA3.2-3BFT100.0048.3351.9465.6864.3456.89
LLaMA3.2-3BLoRA0.377048.0251.8264.9764.1556.57
LLaMA3.2-3BSCALAR (K=2048K = 2048)0.012547.7051.9264.9363.7656.44
LLaMA3.2-3BSCALAR (K=4096K = 4096)0.025047.8052.1865.4264.3756.79
Qwen2.5-7BFT100.0071.7768.0684.2477.2174.46
Qwen2.5-7BLoRA0.264371.3968.3384.1177.0574.41
Qwen2.5-7BSCALAR (K=2048K = 2048)0.005271.5568.1284.6977.4174.58
Qwen2.5-7BSCALAR (K=4096K = 4096)0.010571.4968.2084.3077.0574.43

Table 13. Post-tuning knowledge performance on MMLU. Params denotes the percentage of trainable parameters. The best parameter-efficient result for each model is highlighted in bold.

The relative performance of the two SCALAR configurations varies across model families. The larger configuration has a higher reported average on Gemma3-1B and LLaMA3.2-3B, while the smaller configuration has a higher average on Qwen2.5-7B. Across these settings, SCALAR achieves post-tuning knowledge performance close to LoRA and full-parameter fine-tuning with a smaller trainable-parameter budget.

Scaling to Larger Models

A key property of SCALAR is that the number of trainable parameters introduced for each adapted matrix is decoupled from its input and output dimensions. For a weight matrix W∈Rm×nW \in\mathbb{R}^{m \times n}, a rank-rr LoRA adapter introduces r(m+n)r(m+n) trainable parameters. In contrast, SCALAR optimizes only KK scalar coefficients for the corresponding fixed atoms, independent of mm and nn. Its total trainable parameter count may still depend on the number of adapted modules and Transformer layers, but it does not grow with the width of each linear transformation.

To examine whether this dimension-decoupled parameterization remains effective on larger backbones, we extend the commonsense reasoning experiments to Qwen2.5-14B and Qwen2.5-32B. All methods are trained on Commonsense170K and evaluated on the same eight benchmarks used in the main experiments. We use the same target modules and training protocol, and compare two configurations of LoRA and SCALAR.

As shown in Table 14, SCALAR remains competitive as the backbone size increases. On Qwen2.5-14B, SCALAR with K=8192K = 8192 achieves the best average accuracy of 91.9 while training only 0.0186% of the model parameters. The K=4096K = 4096 configuration also slightly outperforms both LoRA settings, despite using approximately 3.1× fewer trainable parameters than rank-one LoRA.

ModelMethodParams (%)BoolQPIQASIQAHSWGARC-EARC-COBQAAvg.
Qwen2.5-14BLoRA (r=1r = 1)0.029178.693.184.997.492.397.793.495.891.7
Qwen2.5-14BLoRA (r=8r = 8)0.232477.893.783.897.692.397.993.496.891.7
Qwen2.5-14BSCALAR (K=4096K = 4096)0.009377.894.084.997.491.997.993.896.691.8
Qwen2.5-14BSCALAR (K=8192K = 8192)0.018678.093.484.397.692.797.993.497.891.9
Qwen2.5-32BLoRA (r=1r = 1)0.025580.094.784.097.792.798.795.697.492.6
Qwen2.5-32BLoRA (r=8r = 8)0.204480.694.384.598.093.898.596.097.292.9
Qwen2.5-32BSCALAR (K=4096K = 4096)0.005679.294.685.597.893.498.795.897.092.8
Qwen2.5-32BSCALAR (K=8192K = 8192)0.011279.794.585.797.794.198.695.597.292.9

Table 14. Commonsense reasoning results on larger Qwen2.5 models. Params denotes the percentage of trainable parameters. LoRA introduces r(m+n)r(m+n) trainable parameters for an m×nm \times n weight matrix, whereas SCALAR assigns KK trainable scalar coefficients to each adapted module, independent of mm and nn.

On Qwen2.5-32B, SCALAR with K=8192K = 8192 matches rank-eight LoRA at an average accuracy of 92.9, while reducing the trainable parameter percentage from 0.2044% to 0.0112%, corresponding to an approximately 18.3× reduction. The K=4096K = 4096 configuration remains within 0.1 points of the best result while training only 0.0056% of the model parameters.

For a fixed value of KK, the trainable parameter percentage of SCALAR decreases as the backbone grows from 14B to 32B. This behavior follows from the fact that the scalar budget of each adapted module is independent of the matrix dimensions mm and nn. By contrast, LoRA requires r(m+n)r(m+n) parameters per adapted matrix and therefore allocates more trainable parameters as the model width increases. These results show that SCALAR preserves competitive adaptation performance on larger models while becoming proportionally more parameter-efficient.

Ablation Study

We examine how grouping, cross-layer dictionary sharing, update rank, calibration size, and output-atom initialization affect accuracy and computational cost. The following subsections describe the corresponding configurations.

Grouped Parameterization and Cross-Layer Sharing

Grouped parameterization. As introduced in § 4, SCALAR parameterizes the weight update as

ΔW=s∑r=1R∑h=1Hγr,hbr,harT,K=RH,\Delta W = s \sum_{r=1}^{R} \sum_{h=1}^{H} \gamma_{r,h} \boldsymbol{b}_{r,h} \boldsymbol{a}_{r}^{\mathsf{T}}, \qquad K = RH,

where the HH components in each group share the same input-side direction ar\boldsymbol{a}_{r}, while retaining distinct output-side directions br,h\boldsymbol{b}_{r,h}. The update can therefore be equivalently expressed as

ΔW=s∑r=1Rb~rarT,b~r=∑h=1Hγr,hbr,h.(10)\Delta W = s \sum_{r=1}^{R} \widetilde{\boldsymbol{b}}_{r} \boldsymbol{a}_{r}^{\mathsf{T}}, \qquad\widetilde{\boldsymbol{b}}_{r} = \sum_{h=1}^{H} \gamma_{r,h} \boldsymbol{b}_{r,h}. \tag*{(10)}

Although the formulation contains K=RHK = RH scalar-weighted atoms, the HH output atoms within each group can be merged into a single effective output vector b~r\widetilde{\boldsymbol{b}}_{r} before being applied to the input activations. Consequently, the resulting update has rank at most RR, rather than KK.

This grouped structure also reduces the cost of applying the update. For a batch of activations X∈Rn×L\boldsymbol{X} \in\mathbb{R}^{n \times L}, where LL denotes the total number of tokens, the grouped update can be evaluated as

ΔWX=sB~AX,\Delta W \boldsymbol{X} = s \widetilde{\boldsymbol{B}} \boldsymbol{A} \boldsymbol{X},

where A∈RR×n\boldsymbol{A} \in\mathbb{R}^{R \times n} and B~∈Rm×R\widetilde{\boldsymbol{B}} \in\mathbb{R}^{m \times R}. Constructing B~\widetilde{\boldsymbol{B}} requires weighted accumulation of the KK fixed output atoms, with cost O(mK)\mathcal{O}(mK). The subsequent matrix multiplications cost O(LR(n+m))\mathcal{O}(LR(n+m)), giving a total complexity of

O(mK+LR(n+m)).\mathcal{O}(mK + LR(n+m)).

Since K=RHK = RH, the token-dependent matrix-multiplication cost scales with RR rather than KK. The merging cost is independent of LL and can be amortized over the batch and sequence dimensions. The grouped formulation also reduces the intermediate activation size from O(LK)\mathcal{O}(LK) to O(LR)\mathcal{O}(LR).

To examine the structural trade-off between the number of distinct input directions and the per-token computation, we compare the grouped parameterization with an ungrouped variant at the same coefficient budget:

ΔW=s∑k=1KγkbkakT,(11)\Delta W = s \sum_{k=1}^{K} \gamma_{k} \boldsymbol{b}_{k} \boldsymbol{a}_{k}^{\mathsf{T}}, \tag*{(11)}

where each scalar coefficient is associated with an independent pair of input- and output-side directions. For A∈RK×n\boldsymbol{A} \in\mathbb{R}^{K \times n} and B∈Rm×K\boldsymbol{B} \in\mathbb{R}^{m \times K}, this update is evaluated as

ΔWX=sB diag⁡(γ)AX,\Delta W \boldsymbol{X} = s \boldsymbol{B}\,\operatorname{diag}(\boldsymbol{\gamma}) \boldsymbol{A}\boldsymbol{X},

with dominant complexity

O(LK(n+m)).\mathcal{O}(LK(n+m)).

The ungrouped variant can produce an update with rank as high as KK, but it requires all KK input and output directions to be evaluated for every token. In contrast, grouping retains KK candidate output atoms while reducing the token-dependent computation by approximately a factor of HH.

As shown in Table 15, the grouped formulation increases training throughput from 0.58 to 0.75 iterations per second, corresponding to an improvement of approximately 29%, without increasing peak memory. This result is consistent with the complexity analysis: the grouped implementation first merges the HH output atoms associated with each input direction through weighted vector accumulation, and then performs matrix multiplications over only RR effective directions. By contrast, the ungrouped variant evaluates KK independent input and output directions for every token.

MethodIter./sMem. (GB)Avg. Acc.
Grouped, equation (10) (K=1024K = 1024, R=16R = 16)0.7549.188.5
Ungrouped, equation (11) (K=1024K = 1024)0.5849.188.3

Table 15. Ablation on the grouped update parameterization. Results are reported on the commonsense reasoning tasks using Qwen2.5-7B. The variants use the same coefficient budget and differ in the number of input directions, maximum update rank, and token-dependent computation.

At this coefficient budget, the grouped and ungrouped variants obtain average accuracies of 88.5 and 88.3, respectively. The grouped variant combines comparable accuracy with higher throughput by allocating KK output atoms across RR input directions. The coefficient budget is unchanged, while the grouped computation applies fewer distinct input and output projections to each token. This comparison shows the system benefit of organizing the dictionary for efficient application.

Cross-layer sharing of BB. We next examine the cross-layer sharing strategy for the output-side bases. In the default implementation, layers corresponding to the same projection module share the same collection of fixed output directions {br,h}\{b_{r,h}\}. For example, all query-projection layers reuse one output basis, while different module types retain separate bases. Layer-specific adaptation is still preserved because each layer learns its own coefficients γr,h\gamma_{r,h}.

Table 16 reports nearly identical throughput and average accuracy for the shared and unshared dictionaries at all three values of KK. The memory benefit of sharing grows substantially with KK. At K=1024K = 1024, sharing reduces peak memory from 51.6 GB to 49.1 GB, while at K=4096K = 4096, the reduction goes from 59.4 GB to 49.7 GB. Reusing the frozen output dictionaries reduces their storage while retaining each layer’s calibrated input basis and trainable coefficients.

MethodIter./sMem. (GB)Avg. Acc.
Shared BB (K=1024K = 1024)0.7549.188.5
Unshared BB (K=1024K = 1024)0.7551.688.5
Shared BB (K=2048K = 2048)0.7549.388.6
Unshared BB (K=2048K = 2048)0.7554.288.6
Shared BB (K=4096K = 4096)0.7449.788.9
Unshared BB (K=4096K = 4096)0.7459.488.8

Table 16. Ablation on cross-layer sharing of BB. Sharing the output-side bases maintains nearly identical accuracy and throughput while substantially reducing memory consumption, especially as KK increases.

For the tested module dimensions and values of KK, the shared and unshared variants differ in average accuracy by at most 0.1 point. Each layer forms a distinct update through its calibrated input basis and scalar coefficients. The two ablations connect grouping to higher training throughput and dictionary sharing to lower peak memory at comparable reported accuracy.

Effect of the Update Rank

We investigate the effect of the maximum update rank RR while keeping the total number of trainable coefficients K=RHK = RH fixed. For each K∈{1024,2048,4096}K \in\{1024, 2048, 4096\}, we vary R∈{8,16,32,64}R \in\{8, 16, 32, 64\} and set H=K/RH = K/R. Increasing RR expands the activation-derived input subspace and raises the maximum update rank, whereas decreasing RR allocates more output atoms to each input direction.

As shown in Table 17, R=16R = 16 achieves the highest or tied-highest reported accuracy under all three parameter budgets. The variation across R=8R = 8 to 64 is at most 0.2 points within each value of KK. These results support R=16R = 16 as a default allocation between input-subspace coverage and the number of output atoms per input direction.

K=1024K = 1024K=2048K = 2048K=4096K = 4096
R=8R = 888.488.688.7
R=16R = 1688.588.688.9
R=32R = 3288.588.688.8
R=64R = 6488.488.588.7

Table 17. Effect of the update rank RR under fixed trainable-parameter budgets. We report the average accuracy on the commonsense reasoning benchmarks using Qwen2.5-7B. Performance remains stable across a broad range of ranks, with R=16R = 16 giving the strongest overall results.

Effect of the Calibration Size

SCALAR estimates the activation-derived input subspace from a calibration set collected before fine-tuning. We therefore study how the number of calibration examples affects both downstream performance and the one-time cost of subspace construction. Table 18 reports the average accuracy over the eight commonsense reasoning benchmarks together with the calibration time.

Calibration sizeAverage accuracyCalibration time (s)
1688.373.483
3288.5108.18
6488.4147.67
128 (default)88.5237.44
25688.5373.28
51288.4927.95
102488.51886.6

Table 18. Effect of the calibration-set size on downstream performance and calibration cost on the 7B model. Average accuracy is computed over the eight commonsense reasoning benchmarks. Calibration time denotes the one-time cost of collecting activation statistics and computing the input subspaces.

The downstream accuracy ranges from 88.3 to 88.5 across the evaluated calibration sizes. Several sizes, including 32 and 128 examples, reach the highest reported average of 88.5. The results show similar downstream accuracy across a broad range of calibration-set sizes in this setting.

In contrast, the end-to-end calibration cost increases substantially with the number of examples, from 73.4 seconds at a calibration size of 16 to 1886.6 seconds at 1024. This timing includes activation collection through forward passes, distributed reduction of the activation Gram matrices, and the subsequent SVD computation and basis application. We use 128 examples by default, with an average accuracy of 88.5 and a calibration time of 237.44 seconds. This one-time calibration cost is reported separately from the per-step training throughput in the main experiments.

Effect of the Output-Atom Initialization

In SCALAR, the input basis AA is obtained from the leading right singular vectors of downstream activations and remains fixed during fine-tuning. The output atoms BB are randomly initialized and are also kept frozen, while only the scalar coefficients are optimized. Under the default Gaussian initialization, the entries of BB are sampled independently as

Bij∼N(0,σ2),B_{ij} \sim\mathcal{N}(0,\sigma^2),

where σ\sigma denotes the element-wise standard deviation.

We examine whether performance is sensitive to the initialization distribution or scale of the fixed output atoms. The upper block of Table 19 compares Gaussian, Rademacher, and Kaiming initialization for BB. The lower block retains Gaussian initialization and varies σ\sigma, while keeping the activation-derived input basis and all other training hyperparameters unchanged.

Output-atom initializationK=1024K = 1024K=2048K = 2048
Gaussian (default)85.185.2
Rademacher85.185.2
Kaiming85.185.1
Gaussian, σ=0.05\sigma= 0.0585.085.2
Gaussian, σ=0.10\sigma= 0.1085.185.2
Gaussian, σ=0.20\sigma= 0.2085.085.0

Table 19. Effect of the initialization distribution and element-wise scale of the fixed output atoms BB. The upper block compares different initialization rules, while the lower block varies the Gaussian standard deviation σ\sigma.

Gaussian, Rademacher, and Kaiming initialization yield reported accuracies within 0.1 percentage points across the two values of KK. Varying the Gaussian standard deviation from 0.05 to 0.2 changes accuracy by at most 0.2 points. We use Gaussian initialization with σ=0.1\sigma= 0.1 by default.

Adapter Storage Analysis

Table 1 compares trainable parameter counts. Here we report the storage associated with the trainable coefficients, frozen bases, and random dictionaries, together with the representation used after merging. We distinguish trainable parameters, compact adapter checkpoints, and deployable adapters. The checkpoint comparison covers adapter tensors and basis-reconstruction metadata.

LoRALoRA-XSVeRANOLARandLoRASCALAR
ΔW\Delta WBABABMABMAΛbBΛdA\Lambda_b B \Lambda_d A(∑i=1kBαiBi)(∑j=1kAβjAj)\left(\sum_{i=1}^{k_B}\alpha_i B_i\right)\left(\sum_{j=1}^{k_A}\beta_j A_j\right)∑j=1qBjΛjAΓj\sum_{j=1}^{q} B_j\Lambda_j A\Gamma_j∑r=1R∑h=1Hγr,hbr,harT\sum_{r=1}^{R}\sum_{h=1}^{H}\gamma_{r,h}b_{r,h}a_r^{\mathsf{T}}
Trainable paramsR(m+n)R(m+n)R2R^2m+Rm+RkA+kBk_A+k_B⌈d/R⌉(d+R)\lceil d/R\rceil(d+R)RHRH

Table 1. Comparison of trainable parameters across methods. Let W∈Rm×nW \in\mathbb{R}^{m \times n} and d=min⁡(m,n)d=\min(m,n). LoRA, LoRA-XS, VeRA, NOLA, and SCALAR use rank RR. NOLA uses kAk_A and kBk_B random bases, while SCALAR uses HH random output atoms per input direction.

For a single adapted matrix W∈Rm×nW \in\mathbb{R}^{m \times n}, standard rank-RR LoRA stores

ΔW=BA,B∈Rm×R,A∈RR×n.\Delta W = BA,\qquad B \in\mathbb{R}^{m \times R},\qquad A \in\mathbb{R}^{R \times n}.

Thus, both its trainable parameter count and deployable adapter storage are R(m+n)R(m+n).

SCALAR uses a different training-time parameterization:

ΔW=s∑r=1R∑h=1Hγr,hbr,harT,\Delta W = s \sum_{r=1}^{R}\sum_{h=1}^{H}\gamma_{r,h}b_{r,h}a_r^{\mathsf{T}},

where ss is the fixed atom-scaling factor defined in Appendix § 8.1.3. Only the RHRH scalar coefficients are trainable. The input directions A∈RR×nA \in\mathbb{R}^{R \times n} are fixed task-specific buffers obtained from activation statistics. During training, these input directions require RnRn frozen values per adapted matrix. The output dictionary contains mRHmRH frozen values per compatible sharing group, while the coefficients and their optimizer states remain layer-specific. The random output atoms can be regenerated from seeds for compact checkpoint storage. After training, the output atoms associated with the same input direction can be merged:

b~r=∑h=1Hγr,hbr,h=Brγr.\tilde{b}_r = \sum_{h=1}^{H}\gamma_{r,h}b_{r,h} = B_r\gamma_r.

Therefore,

ΔW=s∑r=1Rb~rarT=sB~A,B~=[b~1⋯b~R].\Delta W = s\sum_{r=1}^{R}\tilde{b}_r a_r^{\mathsf{T}} = s\tilde{B}A,\qquad\tilde{B} = \begin{bmatrix}\tilde{b}_1 & \cdots& \tilde{b}_R\end{bmatrix}.

The deployable SCALAR adapter is thus a standard rank-RR adapter and requires R(m+n)R(m+n) values, with the fixed scale ss absorbed into either factor. During training, SCALAR optimizes RHRH coefficients per adapted matrix. Its compact adapter checkpoint stores the calibrated AA, the learned Γ\Gamma, and the seeds and sampling configuration used to regenerate the random dictionary.

Table 20: Storage comparison for a single adapted matrix W∈Rm×nW \in\mathbb{R}^{m \times n}. Here, RR is the adapter rank, HH is the number of SCALAR output atoms per input direction, K=RHK=RH, and NscaleN_{\mathrm{scale}} denotes the method-dependent number of stored scaling parameters for RandLoRA. Fixed random bases are assumed to be recoverable from seeds when applicable.

MethodLoRALoRA-XSVeRANOLARandLoRASCALAR
Compact adapter checkpointR(m+n)R(m+n)R2+R(m+n)R^2+R(m+n)m+R+seedsm+R+\text{seeds}kA+kB+seedsk_A+k_B+\text{seeds}Nscale+seedsN_{\mathrm{scale}}+\text{seeds}Rn+RH+seedsRn+RH+\text{seeds}

Table 20. Storage comparison for a single adapted matrix W∈Rm×nW \in\mathbb{R}^{m \times n}.

For direct loading, the LoRA-XS checkpoint includes the trainable R×RR \times R matrix and the frozen SVD-derived factors. Although these frozen factors are not trainable, they are needed for direct adapter loading unless one recomputes the truncated SVD from the pretrained weight. For NOLA, random bases may be regenerated from seeds, so its compact checkpoint need only retain the learned combination coefficients and the corresponding seeds. SCALAR has a similar seed-based variant: instead of saving the random output dictionary, we save only the activation-derived AA, the trained scalar matrix Γ\Gamma, and the seeds required to regenerate {Br}r=1R\{B_r\}_{r=1}^{R}. When a deployable adapter is required, Γ\Gamma and the regenerated random dictionary are merged into Bˉ\bar{B}, resulting in the standard rank-RR storage R(m+n)R(m+n).

We next instantiate these quantities on Qwen2.5-3B and Qwen2.5-7B. We adapt all linear projection modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. Storage is reported under BF16 precision, where each value occupies two bytes.

On Qwen2.5-7B, the compact SCALAR checkpoints occupy 18.92 MB and 37.04 MB at ranks 8 and 16, respectively. The corresponding LoRA adapters occupy 40.37 MB and 80.74 MB. Each compact SCALAR checkpoint stores the calibrated input basis and learned coefficients together with dictionary-reconstruction metadata. The trainable scalar coefficients account for 0.80 MB in both cases because K=2048K=2048 is fixed; the remaining tensor storage is the calibrated input basis. This compact representation reduces the persistent storage required per adapter. After materialization, both rank-RR adapters require R(m+n)R(m+n) values and use the standard low-rank execution form.

Benchmark Details

We evaluate the adaptation methods on three representative domains: commonsense reasoning, mathematical reasoning, and code generation. For each domain, we fine-tune the pretrained models on a corresponding instruction dataset and evaluate the resulting checkpoints on held-out benchmarks. The descriptions below summarize the data used in our experiments; evaluation details and hyperparameters are given in Appendix § 8.§ 2.

Fine-Tuning Datasets

  • Commonsense170K. Commonsense170K [18] is a collection of approximately 170K instruction-formatted examples constructed from the training sets of eight commonsense reasoning benchmarks: BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA. The original tasks are converted into a unified instruction–response format and combined for multi-task fine-tuning. Following prior PEFT studies, we train on the combined dataset and evaluate separately on the test set of each constituent benchmark.

  • MetaMathQA. MetaMathQA [41] is a mathematical instruction-tuning dataset containing approximately 395K examples. It is constructed by rewriting existing mathematical questions from multiple perspectives, including forward, backward, and answer-conditioned transformations, while preserving the underlying mathematical knowledge. The dataset contains step-by-step solutions and is

Storage typeQwen2.5-3BQwen2.5-7B
Rank-8 LoRA deployable adapter29.93 MB40.37 MB
SCALAR compact checkpoint (R=8R = 8)14.45 MB18.92 MB
SCALAR deployable adapter (R=8R = 8)29.93 MB40.37 MB
Rank-16 LoRA deployable adapter59.86 MB80.74 MB
SCALAR compact checkpoint (R=16R = 16)27.87 MB37.04 MB
SCALAR deployable adapter (R=16R = 16)59.86 MB80.74 MB

Table 21. Practical adapter storage on Qwen2.5-3B and Qwen2.5-7B under BF16 precision. We report LoRA and SCALAR with R∈{8,16}R \in\{8, 16\}, while fixing K=RH=2048K = RH = 2048 for SCALAR. Storage is reported in MB with 1MB=1061\text{MB} = 10^{6} bytes.

designed to improve multi-step mathematical reasoning. We fine-tune the models on MetaMathQA and evaluate them on GSM8K and MATH.

  • Code-Feedback. Code-Feedback [46] contains approximately 66K code-related interactions involving code generation, execution feedback, and iterative refinement. Its examples cover both initial solutions and subsequent corrections based on execution results, providing supervision for generating functionally correct programs. We use Code-Feedback for code instruction tuning and evaluate the resulting models on HumanEval and MBPP.

Evaluation Benchmarks

Commonsense Reasoning. We evaluate commonsense reasoning using the eight benchmarks included in Commonsense170K:

  • BoolQ [6] consists of naturally occurring yes-or-no questions paired with passages and evaluates reading comprehension and implicit reasoning.

  • PIQA [4] evaluates physical commonsense reasoning by asking models to select the more plausible solution to an everyday physical problem.

  • SIQA [34] contains multiple-choice questions about social interactions and evaluates reasoning about people’s intentions, reactions, and social consequences.

  • HellaSwag [42] evaluates commonsense natural-language inference by requiring models to select the most plausible continuation of a given context.

  • WinoGrande [33] is a large-scale Winograd-style benchmark that tests commonsense reasoning and coreference resolution through binary fill-in-the-blank questions.

  • ARC-Easy and ARC-Challenge [7] are the easy and challenging subsets of the AI2 Reasoning Challenge, respectively. Both contain grade-school-level multiple-choice science questions, with ARC-Challenge including questions that are more difficult for retrieval- and word-matching-based systems.

  • OpenBookQA [27] contains multiple-choice science questions that require combining elementary scientific facts with broader commonsense knowledge.

We report accuracy on each benchmark and use their unweighted mean as the overall commonsense reasoning score.

Mathematical Reasoning. We evaluate mathematical reasoning on GSM8K and MATH:

  • GSM8K [8] contains 8.5K linguistically diverse grade-school mathematics word problems. Solving these problems generally requires several arithmetic reasoning steps before producing a final numerical answer.

  • MATH [16] consists of competition-level mathematics problems spanning algebra, counting and probability, geometry, intermediate algebra, number theory, pre-algebra, and pre-calculus. Compared with GSM8K, it requires more advanced mathematical knowledge and longer reasoning chains.

We extract the final answer from each model response and report exact-match accuracy following the standard evaluation protocol.

Code Generation. We evaluate code generation on HumanEval and MBPP:

  • HumanEval [5] contains 164 hand-written Python programming problems. Each problem provides a function signature, a natural-language specification, and hidden unit tests used to assess functional correctness.

  • MBPP [2] contains short Python programming problems described in natural language, together with reference solutions and test cases. The tasks primarily assess the ability to synthesize functionally correct programs for basic programming problems.

For both benchmarks, generated programs are executed against the provided unit tests, and we report pass@1 as the primary metric.

General-Domain Knowledge Evaluation. We evaluate post-tuning general-domain knowledge performance using Massive Multitask Language Understanding (MMLU) [15]. MMLU consists of multiple-choice questions from 57 academic and professional subjects, covering STEM, humanities, social sciences, and other domains. The benchmark evaluates the breadth of a model’s pretrained knowledge and its ability to apply that knowledge across heterogeneous subject areas.

We report accuracy for the four aggregated subject categories and the overall accuracy across all MMLU tasks. These scores characterize post-tuning knowledge performance after fine-tuning on Alpaca-GPT4 [32].

References

  1. [1]Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. RandLoRA: Full rank parameter-efficient fine-tuning of large models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=Hn5eoTunHN.arxiv.org/abs/2502.00987
  2. [2]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
  3. [3]Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604, 2024.
  4. [4]Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.arxiv.org/abs/1911.11641
  5. [5]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  6. [6]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers), pages 2924–2936, 2019.arxiv.org/abs/1905.10044
  7. [7]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  8. [8]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  9. [9]Yulu Gan and Phillip Isola. Neural thickets: Diverse task experts are dense around pretrained weights. arXiv preprint arXiv:2603.12228, 2026.arxiv.org/abs/2603.12228
  10. [10]Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2405.03003
  11. [11]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  12. [12]Yuxuan Gu, Wuyang Zhou, Giorgos Iacovides, and Danilo Mandic. Tera: Vector-based random tensor network for high-rank adaptation of large language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2314–2329, 2026.
  13. [13]Chenlu Guo, Yi Chang, and Yuan Wu. NLoRA: Nyström-initiated low-rank adaptation for large language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 1371–1385, Suzhou, China, November 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.findings-emnlp.72/.arxiv.org/abs/2502.14482
  14. [14]Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217–288, 2011.arxiv.org/abs/0909.4061
  15. [15]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  16. [16]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021b.
  17. [17]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
  18. [18]Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 5254–5276, 2023.arxiv.org/abs/2304.01933
  19. [19]Soroush Abbasi Koohpayegani, Navaneet K L, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiyavash. NOLA: Compressing lora using linear combination of random basis. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=TjfXcDgvzk.
  20. [20]Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.11454
  21. [21]Lujun Li, Cheng Lin, Dezhi Li, You-Liang Huang, Wei Li, Tianyu Wu, Jie Zou, Wei Xue, Sirui Han, and Yike Guo. Efficient fine-tuning of large models via nested low-rank adaptation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22252–22262, 2025. doi: 10.1109/ICCV51701.2025.02066.
  22. [22]Weicheng Lin, Yi Zhang, Jiawei Dang, and Liang-Jie Zhang. Tlora: Task-aware low rank adaptation of large language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29252–29269, 2026.
  23. [23]Vijay Lingam, Atula Tejaswi Neerkaaje, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Eunsol Choi, Alex Dimakis, Aleksandar Bojchevski, and sujay sanghavi. SVFT: Parameter-efficient fine-tuning with singular vectors. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024.arxiv.org/abs/2405.19597
  24. [24]Kainan Liu, Yong Zhang, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, and Jing Xiao. Astra: Activation-space tail-eigenvector low-rank adaptation of large language models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 31833–31854, 2026.
  25. [25]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.09353
  26. [26]Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038–121072, 2024.
  27. [27]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391, 2018.
  28. [28]John X Morris, Niloofar Mireshghallah, Mark Ibrahim, and Saeed Mahloujifar. Learning to reason in 13 parameters. arXiv preprint arXiv:2602.04118, 2026.
  29. [30]Fabian Paischer, Lukas Hauzenberger, Thomas Schmied, Benedikt Alkin, Marc Peter Deisenroth, and Sepp Hochreiter. Parameter efficient fine-tuning via explained variance adaptation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=movsqor65f.DOI
  30. [31]Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
  31. [32]Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.arxiv.org/abs/2304.03277
  32. [33]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
  33. [34]Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 4463–4473, 2019.arxiv.org/abs/1904.09728
  34. [35]Gemma Team. Gemma 3. 2025. URL https://arxiv.org/abs/2503.19786.
  35. [36]Chunlin Tian, Zhan Shi, Zhiqiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37:9565–9584, 2024.arxiv.org/abs/2404.19245
  36. [37]Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient llm fine-tuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4823–4836, 2025.
  37. [38]Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. Advances in Neural Information Processing Systems, 37:54905–54931, 2024.arxiv.org/abs/2407.05000
  38. [39]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Daiyiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024a. doi: 10.48550/ARXIV.2412.15115. URL https://doi.org/10.48550/arXiv.2412.15115.
  39. [40]Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shauiwen L Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning. Advances in Neural Information Processing Systems, 37:71768–71791, 2024b.arxiv.org/abs/2406.05223
  40. [41]Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12884, 2023.
  41. [42]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.arxiv.org/abs/1905.07830
  42. [43]Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Efficient and effective low rank representation fine-tuning. arXiv preprint arXiv:2308.03303, 2023a.arxiv.org/abs/2308.03303
  43. [44]Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=lq62uWRJiJY.arxiv.org/abs/2303.10512
  44. [45]Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5048–5060, 2024.arxiv.org/abs/2403.09113
  45. [46]Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12834–12859, 2024.arxiv.org/abs/2402.14658
  46. [47]Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez De Ocariz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Urochkhin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. arXiv preprint arXiv:2402.16842, 2024.arxiv.org/abs/2402.16842

Paper details

vlnc:W_e8xawr44
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents