betaFeedback

WorksComputer Science

Model Merging Does Not Require Uniform LoRA Ranks

Abstract

Model merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging. We train 36 experts on three domains (mathematics, instruction following, multilingual) at four ranks and three model scales, then merge with six methods at uniform ranks and at each domain's own best rank. Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp. The split has a simple cause. Merging keeps only a fixed share of the pooled spectral components, and methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp. Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8×\times rank range (0.7 pp, against 6.7 pp for TIES at 4B scale). Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.

Introduction

Post-training on specialized domains produces experts that excel individually but cannot be deployed together without multiplying inference cost. Model merging is the post-training step that resolves this, combining the experts into one model that handles all tasks. The technique has matured rapidly: from model soups [1] to TIES [2], DARE [3] and KnOTS [4], each method improves on simple averaging, and toolkits like MergeKit [5] make experimentation accessible. Yet across this line of work the LoRA rank is fixed uniformly across experts, and merging adapters of differing rank is largely unexplored.

This convention has a cost. Rank sets adapter capacity, and how much is needed varies with the task, since domains differ in how far they shift the base model and in how much training data they have. A shared rank therefore forces at least one expert away from the rank that suits it. Rank is fixed during fine-tuning, yet the constraint on it comes from the later merging step, so the two stages cannot be designed independently. We measure the per-domain optima in § 3 and then ask whether merging really requires this coupling at all.

The answer depends on the merge method, and the reason is structural. When experts share rank rr, the truncation budget R=max⁡iriR = \max_i r_i admits a fixed fraction of the pooled spectrum however large rr is. Diverse ranks relax that budget, but magnitude-based selection cannot use the extra room, because it keeps the largest singular values whether or not they reduce loss. We formalize a first-order utility criterion and find that most of the components it rejects would have reduced loss, so the operator discards useful directions by construction rather than by bad tuning.

We make two contributions. First, we show that giving experts different ranks costs averaging and utility-based selection nothing and gains them up to 1.8 pp, while conflict-resolution methods such as TIES can lose up to 4 pp with inconsistent performance, and we trace that asymmetry to how each operator picks spectral components. Second, we identify what the failure destroys rather than only how much: magnitude truncation pushes a single expert out of the kept subspace, and the loss lands almost entirely on instruction following. Together these say that rank coordination across experts is a constraint of the merge method, so changing the method removes it.

Related Work

Model merging began with simple averaging [1] and task arithmetic [6], which scales and adds task vectors. These work well when expert updates are compatible and degrade when experts learn conflicting directions, which later methods address by resolving the conflict explicitly: TIES [2] by trimming and sign voting, DARE [3] by random dropping, and KnOTS [4] in spectral space. A separate line weights parameters by importance, using curvature [7], feature covariance [8], or learned coefficients [9]. All of them select by the size of a weight or singular value.

Our criterion follows Optimal Brain Damage [10] and the pruning results that succeeded it, where movement pruning [11] showed magnitude to be a particularly poor saliency signal in the fine-tuning regime we work in. Applying that idea to merging, Park et al. [12] score task vectors with connectivity-based saliency and also decompose LoRAs rank-wise, and Yang et al. [13] vary sparsity per singular vector by importance instead of truncating by singular value. We differ in evaluating utility with a first-order gradient taken at the merged model rather than at individual experts, which is valid because the merged model sits at no single task’s minimum, and in using the criterion to ask whether experts must share a rank at all.

Rank heterogeneity has been studied in adjacent settings, though for different reasons. Federated methods aggregate adapters of unequal rank because client hardware forces it [14], and adaptive methods allocate rank across layers within one adapter [15]. Neither treats rank as a per-expert choice interacting with the merge operator. Closest to us, He et al. [16] also identify the R=max⁡iriR = \max_i r_i bottleneck and respond by compressing before merging, where we keep the budget and change which components fill it. LoRA [17] is agnostic to rank, but merging is not. MergeBench [18] benchmarks merging methods and our setup follows its protocol, while concurrent work [19] finds expert training duration matters too, so both choices should be made jointly.

Methods

We merge with six operators spanning the two selection strategies at issue. Simple averaging and greedy soup [1] combine weights without selecting components, the latter adding experts only when validation accuracy improves. TIES [2] trims low-magnitude entries and resolves sign conflicts, KnOTS [4] truncates the SVD of the stacked update, and plain SVD truncation keeps the top-RR singular directions. The last three all rank components by singular value, so they are magnitude-based.

Against these we use a gradient-guided selection method (GGSM), which keeps the same truncation budget but changes the criterion. Stacking the expert updates and taking the SVD gives components (σj,uj,vj)(\sigma_j, u_j, v_j). We score each by projecting the combined task gradient GG onto it,

utility⁡j=−σj(ujTGvj),(1)\operatorname{utility}_j = -\sigma_j (u_j^{\mathsf{T}} G v_j), \tag*{(1)}

and keep the RR components of highest utility rather than largest σj\sigma_j. Positive utility means the component reduces loss to first order. To validate this criterion empirically, we sample 50 random components from a merged 8.0B model and measure the actual loss change from adding each. The predicted utility correlates with ground-truth loss change (Spearman ρ=0.49\rho= 0.49, p<0.001p < 0.001), outperforming magnitude (ρ=0.45\rho= 0.45). However, the first-order approximation can overestimate utility for some components, particularly those with extreme predicted scores, suggesting room for higher-order corrections in future work. To compute GG, we first merge with plain SVD (top-σ\sigma), then compute per-domain gradients Gt=∂Lt/∂WG_t = \partial L_t/\partial W at the merged model using 64 validation samples per domain. Each gradient is normalized to unit Frobenius norm (G^t=Gt/∥Gt∥F\hat{G}_t = G_t/\lVert G_t\rVert_F) and summed with equal weight: G=∑tG^tG = \sum_t \hat{G}_t. This merged-point approximation is more accurate than evaluating at the base model because the first-order Taylor expansion is valid locally. Obtaining GG takes one backward pass per domain, so unlike the data-free baselines GGSM needs data access, at negligible cost.

We use the Qwen3.5 dense checkpoints at three scales, 0.8B, 2B and 4B, following the Qwen3 training recipe [20]. All experts are fine-tuned with QLoRA [21] on a 4-bit quantized base, targeting all attention and MLP projection modules. We train with learning rate 2×10−42 \times10^{-4}, 100-step linear warmup, effective batch size 8, and sequence length 2048, selecting checkpoints by minimum validation loss. Rank is the only factor we vary: each domain is trained at ranks 4, 8, 16, and 32, giving 36 experts. Merges use a single seed, so we treat small score differences as ties throughout.

Following MergeBench [18], we adopt three task domains. Mathematics uses DART-Math [22] for training and GSM8K [23] for evaluation. Instruction following uses TULU-3 [24] for training and IFEval [25] for evaluation. Multilingual uses Aya [26] for training and reports the mean of MMLU [27], ARC [28], and MMLU-zh for evaluation. Each domain has its own optimal rank for single-task performance, and these optima differ across domains and scales (Table 1), so any uniform choice is suboptimal for at least one domain.

ModelMathematicsInstructionMultilingual
Qwen3.5-0.8B8, 16, 32164
Qwen3.5-2B3284
Qwen3.5-4B4328

Table 1. Single-task optimal LoRA rank per domain and model. Optima differ across domains and shift with scale, so any uniform choice is suboptimal for at least one domain. Note that within each domain the variance across ranks is modest (typically ≤2\le2 pp), so while optimum differs, the penalty for mismatched rank is small for single-task use; the larger effect comes from merging.

Results

Uniform ranks minimize what survives truncation

Uniform ranks put the merge operator in the worst position it can occupy, and raising the rank does not help. With TT experts at a shared rank rr, the pool holds ≈Tr\approx Tr components while the budget admits R=rR = r, so a fixed 1/T1/T survives through much capacity as each expert was given. Table 2 shows this pinned near one-third across an 8×8\times range of ranks. Diverse ranks break the pattern, because the

RegimeRanks (m/i/l)RRpoolkeptE@R
uniform4/4/4412.30.3260.794
uniform8/8/8824.10.3320.827
uniform16/16/161648.20.3320.869
uniform32/32/323296.20.3320.876
diverse8/16/41628.40.5650.898
diverse4/16/323252.40.6110.891
diverse4/32/83244.20.7250.966

Table 2. Spectral structure of the merged update (0.8B, averaged over 20–64 modules, two independent samples agreeing to ∼2\sim2 decimals). POOL is the mean numerical rank of the merged ΔW\Delta W (count of singular values ≥10−6σmax⁡\ge10^{-6}\sigma_{\max}, which is approximately ∑iri\sum_i r_i but varies slightly due to near-zero components). R=max⁡iriR = \max_i r_i is the truncation budget; KEPT = RR/pool; and E@R is the Frobenius energy retained by the top-RR components. Uniform assignments sit at 1/31/3 kept at every rank, while diverse assignments retain substantially more.

budget follows the largest expert while the pool follows the sum: assigning r=4r = 4 to math, 32 to instruction and 8 to multilingual leaves ≈44\approx44 components with R=32R = 32, so 73% survive. A shared rank costs no capacity inside any expert, only the share of the combined update that survives.

This bottleneck applies only to methods that truncate the merged spectrum (SVD, KnOTS, TIES, and GGSM). Simple averaging keeps the full sum ∑iΔWi\sum_i \Delta W_i without truncation, so rank heterogeneity is free by construction—there is no RR budget to exhaust. The empirical results in § 4.2 confirm this: averaging tolerates diverse ranks as well as GGSM does, even though GGSM actively selects components while averaging does not.

Equation (1) lets us ask whether the discarded components were worth keeping, and by that measure the budget is not merely tight, it is spent on the wrong components. Table 3 shows that 65 to 84% of pooled components carry positive utility while the budget admits 33% under uniform ranks and at most 62% under diverse ones, so useful directions must be dropped whatever the hyperparameters, and the waste grows with rank: the count of high-utility components magnitude keeps out rises from about 1 to nearly 10 per matrix between r=4r = 4 and r=32r = 32, tracking the decline of TIES while utility-based selection stays flat (Figure 2). Utility selection retains less spectral energy (Eutil<EσE_{\mathrm{util}} < E_{\sigma}) and still merges better, so what matters is which components survive, not how much energy they carry.

RegimeRankskeptoverlapclippedfrac. pos.EσE_{\sigma}EutilE_{\mathrm{util}}
uniform4/4/40.3330.7471.010.7950.7970.694
uniform8/8/80.3330.7142.290.7550.8270.686
uniform16/16/160.3330.6695.300.7200.8660.665
uniform32/32/320.3330.6949.780.6520.8760.701
diverse8/16/40.5710.7444.090.8430.8960.783
diverse4/16/320.6150.68710.020.7800.8920.718

Table 3. Utility versus magnitude selection at identical budget RR (0.8B, base-point gradients, 96 target matrices). FRAC. POS. exceeds KEPT in every configuration: the budget must reject loss-reducing directions. CLIPPED is the mean count per matrix that utility keeps and magnitude discards.

Figure 2. Left: the fraction of pooled components carrying positive first-order utility far exceeds the fraction the budget admits, which is fixed at 1T\frac{1}{T} for TT tasks under any uniform assignment. Right: the number of high-utility components that magnitude selection discards grows roughly tenfold from r=4r = 4 to r=32r = 32 (left axis), tracking the decline of TIES while utility-based selection stays flat (right axis).

Magnitude methods lose more on diverse ranks

If the structural argument holds, methods should divide by how they choose components rather than by how sophisticated they are, and Table 4 shows exactly that. Moving each method from its best uniform assignment to its best diverse one separates the six operators into two groups. For GGSM, SVD and Simple averaging every delta lies between −0.3 and +1.8 pp, inside our interval, and the direction is not random: GGSM improves at all three scales by 0.4 to 1.4 pp, and the largest single gain is 1.8 pp for Simple averaging at 0.8B. Diverse ranks are free for these three, with a hint of a gain our sample sizes cannot resolve. Greedy soup loses ground at 0.8B and 2B, while TIES and KnOTS are inconsistent; their best diverse configurations beat uniform at 2B but trail at 0.8B and 4B.

Table 4: Method comparison across scales, best diverse configuration per method, where Δ=\Delta= (best diverse) −- (best uniform). GGSM, SVD and averaging hold within noise, TIES and KnOTS perform inconsistently across scales while greedy soup degrades. IFEval drives the spread.

ScaleMethodMathIFEvalMultiAvgΔ\Delta
0.8BGGSM40.955.150.748.9+1.4
0.8BSVD42.448.651.247.4+0.3
0.8BSimple avg44.949.750.948.5+1.8
0.8BGreedy soup45.339.948.044.4−4.1
0.8BTIES37.639.747.941.7−0.5
0.8BKnOTS37.934.247.639.9−2.1
2BGGSM61.567.558.262.4+0.7
2BSVD64.164.558.362.3+0.2
2BSimple avg62.563.458.361.4+0.3
2BGreedy soup64.561.255.160.3−1.0
2BTIES62.854.355.957.7+0.4
2BKnOTS62.951.952.855.9+2.3
4BGGSM82.582.370.478.4+0.4
4BSVD84.278.770.477.8−0.3
4BSimple avg83.979.370.277.8−0.1
4BGreedy soup83.878.970.477.70.0
4BTIES83.861.268.271.1−4.0
4BKnOTS82.654.068.568.4−1.1

Table 4.

Rank sensitivity tells the same story from a different angle. Across the uniform-rank sweep at 4B, GGSM spans 0.7 pp while TIES spans 6.7, about 9×9\times more (Figure 1, left). A method that ignores utility does not merely mishandle diverse ranks, it is unstable under the choice of rank in general. The ordering (GGSM < SVD < averaging < KnOTS < TIES) tracks reliance on magnitude. Appendix § 7 gives the full sweep, and Appendix § 8 shows that at 4B, every diverse configuration costs TIES 4 to 9 pp, at 0.8B under 1 pp, and only at 2B does the best diverse setup edge ahead.

Figure 1. Left: sensitivity to uniform rank at 4B, where utility selection spans 0.7 pp against 6.7 pp for TIES. Right: share of the retained top-RR subspace per expert under magnitude truncation (SVD), where one expert is often dropped almost entirely (e.g. 16/16/16 → 0.93/0.01/0.06). GGSM selects by utility rather than magnitude, so the eviction pattern differs; we leave that analysis to future work.

Which of these differences we can trust depends on a resolution floor, which we derive in Appendix § 6 as roughly 1.4 pp for the paired comparisons we report. Three results clear it: the TIES penalty of −4 pp, the sensitivity ranges of 6.7 against 0.7 pp, and the IFEval gaps of 16 to 25 pp. The gaps among the four leading methods—GGSM, SVD, averaging, and greedy soup—do not clear this floor, so we treat those as statistically tied. In particular, GGSM’s advantage over SVD is small (0.6 pp at 4B, 0.1 pp at 2B, 1.5 pp at 0.8B) and within noise; the value of GGSM lies in its rank stability (0.7 pp range vs 1.3 pp for SVD at 4B) rather than in higher peak accuracy.

Instruction following is suppressed

An average score hides the shape of the damage, which is concentrated rather than spread. Attributing the retained top-RR subspace to individual experts through ⟨ΔWd,Cj⟩\langle\Delta W_d, C_j \rangle over kept components jj shows it is badly unbalanced: in 6 of 7 configurations one expert holds at least 0.64 of the retained energy while another is driven near zero, reaching 0.93/0.01/0.06 at uniform rank 16 (Figure 1, right). Merging is meant to combine experts, but under magnitude truncation it can quietly delete one, a capability loss introduced by the merge rather than absent beforehand.

Table 4 compares each method’s best diverse-rank configuration against the single-task experts of Table 10. On the Math task, every method exceeds its expert at all three scales, combining experts yields consistent positive transfer for Math capability. On Multilingual task, GGSM and SVD exceed the single-task experts at every scale while the remaining methods, TIES and KnOTS in particular,

ScaleDomainBest RankScore
0.8BMath (GSM8K)r8/r16/r3234.5
0.8BInstruction (IFEval)r1654.2
0.8BMultilingualr448.0
2BMath (GSM8K)r3256.6
2BInstruction (IFEval)r867.3
2BMultilingualr457.7
4BMath (GSM8K)r479.7
4BInstruction (IFEval)r3281.7
4BMultilingualr870.3

Table 10. Single-task expert baselines across scales (best rank per domain and its score).

ScaleMethodMathIFEvalMultiAvg
0.8BGGSM (utility)38.655.548.547.5
0.8BSVD (magnitude)39.747.750.846.1
0.8BSimple averaging42.747.150.346.7
0.8BGreedy soup44.750.850.048.5
0.8BTIES39.339.248.142.2
0.8BKnOTS41.936.048.242.1
2BGGSM (utility)58.667.156.560.7
2BSVD (magnitude)63.863.858.762.1
2BSimple averaging62.263.457.360.9
2BGreedy soup64.661.657.661.3
2BTIES60.750.858.156.5
2BKnOTS57.142.155.751.6
4BGGSM (utility)81.782.370.078.0
4BSVD (magnitude)83.780.869.778.1
4BSimple averaging84.178.769.977.6
4BGreedy soup84.079.170.077.7
4BTIES84.266.068.773.0
4BKnOTS80.956.667.768.4

Table 5. Per-domain scores at uniform rank 4 at 0.8B, 2B and 4B. IFEval collapses for TIES and KnOTS at every scale.

fall below. The Instruction following task is the discriminating axis: GGSM is the only method that surpasses the single-task IFEval expert at every scale (0.8B: 55.1 vs. 54.2; 2B: 67.5 vs. 67.3; 4B: 82.3 vs. 81.7), whereas all other methods trail it and TIES and KnOTS collapse (e.g. 2B: 54.3 and 51.9). GGSM is thus the only method to improve on the single-task expert across all three domains simultaneously, which is why it attains the highest average at every scale (48.9, 62.4, 78.4) and stands as the strongest method overall.

Conclusion

Established practice in model merging trains every expert at the same LoRA rank. Our results suggest this convention reflects a property of the merge operator rather than a requirement of merging. It comes specifically from conflict-resolution methods that select components by magnitude: TIES and KnOTS are sensitive to rank heterogeneity while SVD truncation is not, even though all three rank components by singular value. The difference is that TIES and KnOTS apply additional operations (trimming, sign election) that interact with the magnitude distribution in ways that break under heterogeneous ranks. With mismatched ranks, the low-rank expert has fewer but denser parameters compared to a high-rank expert with many but sparser parameters; when TIES trims by magnitude percentile and elects signs by majority vote, the low-rank expert’s concentrated magnitudes get disproportionately trimmed while its smaller vote share loses sign elections. The combined effect pushes the low-rank expert out of the merged result. With averaging or utility-based selection, diverse ranks cost nothing and we measure gains of up to 1.8 pp, so rank becomes a per-expert decision that needs no coordination. Evaluated with TIES the same change costs 4 pp at 4B, and KnOTS is inconsistent. For TIES and KnOTS, the safer choice is matching ranks, but that conclusion does not generalize to merging or even to all magnitude-based methods. This matters most when experts come from separate teams or arrive at different times. The practical recommendation is to train each expert at the rank that suits its domain, and merge with an operator that is rank-insensitive—whether by keeping the full sum (averaging), by selecting components on effect rather than size (GGSM), or by truncating without conflict resolution (SVD).

Our study has limitations we plan to address in future work. The small gains from diverse ranks sit below our 1.4 pp resolution floor, so confirming their significance needs more seeds and larger evaluations. We also cover three domains, focus on LoRA fine-tuning only, and use one model family.

Resolution Floor for Score Comparisons

Per-benchmark standard errors propagate as SEdomain=∑isei2/k\mathrm{SE}_{\mathrm{domain}} = \sqrt{\sum_i \mathrm{se}_i^2/k} over the kk benchmarks in a domain, and SEavg=∑dSEd2/3\mathrm{SE}_{\mathrm{avg}} = \sqrt{\sum_d \mathrm{SE}_d^2/3} over the three domains. This gives SEavg≈0.93\mathrm{SE}_{\mathrm{avg}} \approx0.93 to 1.04 pp, so a 95% interval spans roughly ±1.9\pm1.9 pp on any single multi-domain score. The diverse-versus-uniform comparisons are paired, since both configurations are evaluated on identical test items, and a paired bootstrap over those shared items roughly halves the interval. We therefore treat 1.4 pp as the smallest difference we can resolve, and read anything below it as a tie.

How Much the Merge Method Depends on Rank

Table 6 gives the full uniform-rank sweep at 0.8B, 2B and 4B Qwen3.5 models. At every scale GGSM has the most stable performance across 8× range while TIES drops by 4.3 at 0.8B scale and by 6.7 at 4B scale. The ordering by range (GGSM < SVD < averaging < KnOTS < TIES) matches reliance on magnitude-based selection. Figure 2 shows the mechanism behind that ordering: the

ScaleMethodr4r8r16r32Range
0.8BGGSM (utility)47.547.546.946.31.2
0.8BSVD (magnitude)46.146.847.145.91.2
0.8BSimple averaging46.746.546.244.91.8
0.8BKnOTS42.137.735.034.67.5
0.8BTIES42.241.039.737.94.3
2BGGSM (utility)60.761.261.760.61.1
2BSVD (magnitude)62.161.061.959.82.3
2BSimple averaging60.961.160.658.52.6
2BKnOTS51.653.652.251.02.6
2BTIES56.557.355.553.63.7
4BGGSM (utility)78.077.877.977.20.7
4BSVD (magnitude)78.177.177.276.81.3
4BSimple averaging77.677.877.575.82.1
4BKnOTS68.469.568.466.33.2
4BTIES73.075.170.468.46.7

Table 6. Uniform-rank sweep across scales (same-rank merges). GGSM is the most rank-stable at every scale; KnOTS and TIES are the most sensitive.

ScaleRanks (m/i/l)MathIFEvalMultiAvg
0.8B8/32/4 (optimal)37.639.747.941.7
4/8/1639.339.446.241.6
8/16/439.038.446.941.4
4/32/835.638.450.141.4
32/8/439.935.348.841.3
2B16/8/4 (optimal)62.854.355.957.7
16/4/861.954.355.857.3
16/32/860.952.356.756.6
32/16/461.753.454.556.5
32/8/461.553.454.356.4
4B4/32/8 (optimal)83.861.268.271.1
4/16/882.956.068.169.0
4/8/1681.457.568.169.0
4/16/3280.751.667.766.7
4/8/3281.050.167.466.1

Table 7. TIES on diverse ranks across scales. At every size the best diverse configuration underperforms the best uniform-rank baseline (0.8B: 42.2 at r4; 2B: 57.3 at r8; 4B: 75.1 at r8) except at 2B where the diverse-rank setup is marginally higher.

share of pooled components carrying positive utility far exceeds what the budget admits, and the number of useful components discarded grows with rank.

The Cost of Diverse Ranks

Table 7 reports TIES across diverse-rank configurations at all three model scales, from which two observations follow. First, the natural heuristic of assigning each domain its individually optimal rank (0.8B: 8/16/4; 2B: 32/8/4; 4B: 4/32/8) yields no improvement: it underperforms the best uniform-rank baseline at every scale (0.8B: 41.4 vs. 42.2, −0.8; 2B: 56.4 vs. 57.3, −0.9; 4B: 71.1 vs. 75.1, −4.0). Second, an exhaustive search over diverse assignments does recover a configuration that matches or marginally exceeds the uniform baseline at the two smaller scales (best diverse at 0.8B, 41.7, −0.5; at 2B, 57.7, +0.4), whereas at 4B even the best diverse configuration falls short by 4.0 pp. For TIES, therefore, the benefit of rank diversity is marginal and attainable only through search; the per-domain-optimal heuristic is consistently inferior to a single shared rank, and this deficit grows with scale. Where degradation occurs, it is attributable to IFEval, which deteriorates sharply under diverse-rank TIES.

Table 8 extends this comparison to all methods evaluated at the per-domain-optimal ranks across scales. The utility-, magnitude-, and averaging-based methods (GGSM, SVD, simple averaging, and Greedy soup) lie within approximately 1pp in average score at 2B (60.3–61.3) and 4B (77.7–78.4), inside our interval, so we treat them as tied, while GGSM establishes a wider margin of roughly 2pp at 0.8B (48.5 vs. 46.6). TIES and KnOTS trail this group by 4 to 10pp at every scale, again owing to the collapse of IFEval: GGSM preserves IFEval performance at 56.7, 66.7, and 82.3 for 0.8B, 2B, and 4B respectively, whereas TIES falls to 38.4, 53.4, and 61.2. This pattern is thus consistent across scales rather than specific to any single model size.

Where the Loss Lands

Table 10 lists the single-task baselines the merged models are measured against. Merged models can exceed a single-task expert on its own domain through positive transfer—for instance, stronger instruction following may improve how math answers are formatted and thereby extracted. Figure 3 breaks the uniform-rank sweep down by domain and shows that the method spread is dominated by instruction following (at 4B, a 33-point range across methods versus 3.3 for math and 3.9 for multilingual), while Figure 4 compares IFEval against the single-task baseline and sweeps TIES across diverse configurations.

Figure 3. Per-domain scores across uniform ranks at 4B. IFEval (center) shows the starkest method divergence: GGSM and SVD maintain 75.1 to 82.3 while TIES and KnOTS collapse to 49.2 to 70.4. Math (left) and Multilingual (right) vary little across methods.

Figure 4. Left: IFEval scores at 4B, uniform rank 4. TIES and KnOTS lose 16 to 25 pp relative to the single-task expert baseline (dashed line), while GGSM slightly exceeds it. Right: TIES performance across diverse rank configurations vs best uniform (r8). Every diverse configuration underperforms uniform on aggregate, primarily due to IFEval collapse.

ScaleMethodMathIFEvalMultiAvg
0.8BGGSM (utility)39.056.749.848.5
0.8BSimple averaging41.648.249.846.5
0.8BSVD (magnitude)41.347.351.246.6
0.8BGreedy soup45.339.948.044.4
0.8BKnOTS37.934.247.639.9
0.8BTIES39.038.446.941.4
2BGGSM (utility)61.966.755.261.3
2BSimple averaging63.563.655.961.0
2BSVD (magnitude)64.461.055.860.4
2BGreedy soup64.561.255.160.3
2BKnOTS62.951.952.855.9
2BTIES61.553.454.356.4
4BGGSM (utility)82.582.370.478.4
4BSimple averaging83.879.370.277.8
4BSVD (magnitude)84.278.770.477.8
4BGreedy soup83.878.970.477.7
4BKnOTS82.654.068.568.4
4BTIES83.861.268.271.1

Table 8. All methods on the per-domain optimal ranks at each scale (0.8B: 8/16/4; 2B: 32/8/4; 4B: 4/32/8). GGSM leads on Avg at every scale; TIES and KnOTS trail because IFEval collapses.

MethodIngredientsMathIFEvalMultiAvg
GGSM (same)16/16/1661.467.156.761.7
Greedy (same)m4, i464.661.657.661.3
GGSM (diverse)8/16/461.567.558.262.4
Greedy (diverse)m32, i4, l864.561.255.160.3

Table 9. 2B results: Greedy soup vs GGSM on same-rank and diverse-rank configurations.

References

  1. [1]Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), 2022.arxiv.org/abs/2203.05482
  2. [2]Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.01708
  3. [3]Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03099, 2024.arxiv.org/abs/2311.03099
  4. [4]George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. KnOTS: Model merging with SVD to tie the Knots. arXiv preprint arXiv:2410.19735, 2024.
  5. [5]Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meade, Hailey Zeng, and Elie Fung. Arcee’s MergeKit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257, 2024.arxiv.org/abs/2403.13257
  6. [6]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hanneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2212.04089
  7. [7]Michael S. Matena and Colin Raffel. Merging models with Fisher-weighted averaging. Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2111.09832
  8. [8]Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2212.09849
  9. [9]Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.02575
  10. [10]Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems (NeurIPS), 1989.
  11. [11]Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2005.07683
  12. [12]Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Saliency-aware model merging. arXiv preprint arXiv:2606.00511, 2026.arxiv.org/abs/2606.00511
  13. [13]Yan Yang, Yixia Li, Hongru Wang, Xuetao Wei, Jianqiao Yu, Yun Chen, and Guanhua Chen. ImPart: Importance-aware delta-sparsification for improved model compression and merging in LLMs. arXiv preprint arXiv:2504.13237, 2025.arxiv.org/abs/2504.13237
  14. [14]Muhammad Waseem, Nurbek Tastan, Andrej Jovanovic, Nicholas D. Lane, Nils Lukas, Karthik Nandakumar, and Samuel Horvath. PreLoRT: Prefix-nested LoRA for federated fine-tuning under rank heterogeneity. arXiv preprint arXiv:2606.15963, 2026.arxiv.org/abs/2606.15963
  15. [15]Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdoLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. International Conference on Learning Representations (ICLR), 2023.DOI
  16. [16]Zhengbao He, Ruiqi Ding, Zehao Huang, Ruikai Yang, Tao Li, and Xiaolin Huang. Compress then merge: From multiple LoRAs into one low-rank adapter. arXiv preprint arXiv:2606.03723, 2026.arxiv.org/abs/2606.03723
  17. [17]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2106.09685
  18. [18]Yifei He, Siqi Zeng, Yuzheng Hu, Rui Yang, Tong Zhang, and Han Zhao. MergeBench: A benchmark for merging domain-specialized LLMs. arXiv preprint arXiv:2505.10833, 2025.arxiv.org/abs/2505.10833
  19. [19]Nikita Kozodoi, Zainab Afolabi, and Jack Butler. Are we merging the right models? impact of expert training duration on model merging for LLMs. arXiv preprint arXiv:2607.11997, 2026.arxiv.org/abs/2607.11997
  20. [20]An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  21. [21]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient fine-tuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
  22. [22]Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. DART-math: Difficulty-aware rejection tuning for mathematical problem-solving. arXiv preprint arXiv:2407.13690, 2024.arxiv.org/abs/2407.13690
  23. [23]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  24. [24]Nathan Lambert et al. TŰLU 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024.arxiv.org/abs/2411.15124
  25. [25]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.arxiv.org/abs/2311.07911
  26. [26]Shivalika Singh et al. Aya dataset: An open-access collection for multilingual instruction tuning. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024.
  27. [27]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference on Learning Representations (ICLR), 2021.
  28. [28]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. In arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457

Paper details

vlnc:W_qz47f3t2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents