WorksComputer Science
Watch Where You Linearize: Predicting and Correcting Model-Merging Interference
Abstract
Model merging combines separately fine-tuned models by adding their task vectors to a shared backbone. Interference is typically assessed only after constructing and evaluating a candidate merge, making coefficient search expensive. We show that a first-order directional derivative of task A's outputs along task B's vector can predict interference, but only when evaluated at the appropriate model. At the receiver model, this derivative forecasts nearby merges per task and per example, halves the accuracy error of a no-change baseline, and identifies most examples whose predictions flip. For full multi-task merges, a single evaluation is insufficient; two evaluations along the merge path are needed. At the shared pretrained backbone, however, the derivative is nearly orthogonal to realized changes in our vision experiments and only partly predictive in LoRA language-model pilots. Controls link this failure to displacement along the receiver's own task vector, not simply parameter distance; the mismatch appears within the first few dozen fine-tuning steps. Freezing receiver-local responses produces a surrogate affine in the merge coefficients. We use it to construct LR-Merge, a label-free correction that optimizes coefficients through convex teacher matching on unlabeled probes, without backpropagation through merged networks. LR-Merge improves all tested closed-form, isotropic, and sparsified base mergers, with gains generally increasing as more tasks are merged. Using pretrained rather than receiver-local responses instead degrades the base merge. LR-Merge matches exact teacher matching through a merged network, which takes roughly three times longer to converge and performs similarly at an equal compute budget. Once built, the surrogate answers new coefficient and task-subset queries in seconds.
Introduction
Model merging composes separately fine-tuned models into one network, without joint training or the original data. Each model contributes its parameter difference from the shared pretrained starting point, its task vector ([30, 11]). The updates interfere. A vector learned for one task changes predictions on another and costs accuracy. Closed-form mergers reduce the damage by altering the composition analytically ([12, 20, 32]). Adaptive mergers tune coefficients on unlabeled data, building and running the merged network at every step ([34]). In both cases the effect of a merge is known only after the merged model is built and evaluated, and every hyperparameter a pipeline tunes (coefficients, sparsity levels, task subsets) multiplies that cost. Yet everything the merge is made of is known in advance: the task vectors and the models that will receive them. We ask whether the effect of adding a task vector can be predicted before any merge is built, and whether the prediction can replace the build inside coefficient search. The natural object is the cross-task directional response : the first-order change task A’s outputs undergo when task B’s vector is added at , one Jacobian–vector product (JVP). First-order expansion is standard. What is not settled is where the response should be read when task vectors come from ordinary nonlinear fine-tuning. Task arithmetic defines every vector relative to the pretrained , tangent-space methods linearize there (Ortiz-Jimenez et al., [22]), and kernel-regime accounts suggest the initial linearization can persist through fine-tuning (Malladi et al., [19]). All three invite reading the response at , the one point shared by all tasks. A merge, however, is applied to a fine-tuned receiver.
Figure 1. Interference is receiver-local. : logits on task A’s probes; ; : task B’s vector. Black: realized drift ; blue: forecast at the receiving model, ; orange: forecast at the pretrained model, , which points in a nearly unrelated direction (formal setup in § 3).
We find that the anchor is decisive. Read at the receiving model, forecasts nearby merges accurately. Read at , it carries almost no usable signal in the vision settings we study. What separates the two anchors is direction, not distance. The gap forms at the very beginning of fine-tuning, when the network re-aims its cross-task responses while their norm barely changes.
The finding becomes useful through a method. If receiver-local responses are accurate near a merge, freezing them at that merge gives a surrogate whose outputs are affine in the merge coefficients. Coefficient selection then becomes a convex, label-free problem: match the fine-tuned teachers’ predictions on unlabeled probes. The search never builds or backpropagates through a candidate merged network. We call the method Linear-Response Merge (LR-Merge). It needs no labels and no joint training. It is not data-free, since it uses a few unlabeled probes from each task’s distribution and the fine-tuned models to compute pseudo-labels. Because a derivative is local, LR-Merge is designed as a cheap correction of a strong base merge rather than a merger from scratch. The base merger supplies a good starting point, and the frozen responses replace the search around it with convex solves whose build is reused across queries.
Contributions.
A pre-merge interference forecast, with its range of validity. One JVP at the receiving model predicts, before the merge is built, how logits drift, which examples flip, and each task’s accuracy at the strengths that coefficient search visits (Table 1). Used to score candidate coefficients, it ranks them as well as building and evaluating each, and once the surrogate is built it scores a candidate about a thousand times faster. For a complete merge one read degrades, and two reads along the path recover nearly all of the realized drift (Table 2).
Table 1: Pairwise forecast, 30 ordered pairs, checkpoint set 0, three probe seeds (other sets in Table 9). Flips are rare at (1.0% of probe–pair cases, and 17 of 30 pairs have none). Rate-matched label-free low-margin and random flaggers reach precision 0.57 and 0.06 (0.42, 0.11 at ). At the per-pair accuracy drop correlates at 0.75 with the forecast and at 0.12 with task-vector cosine.
| forecast | no change | flip recall | flip prec. | damage | drift cos | |
| 0.2 | 0.13 | 0.26 | 0.82 | 0.79 | 0.917 | 0.946 |
| 0.4 | 0.30 | 0.60 | 0.75 | 0.68 | 0.824 | 0.905 |
Table 1.
Table 2: Forecasting complete merges () produced by five base mergers: six receivers, mean over three checkpoint sets. One JVP is too coarse at this distance, and two along the path are accurate. At one JVP at reaches cosine 0.90–0.98 for every merger, against −0.14 to 0.17 at .
| merger | one JVP at : cos | one JVP at : damage | one JVP at : acc MAE | two JVPs on the path (midpoint): cos | two JVPs on the path (midpoint): damage | two JVPs on the path (midpoint): acc MAE | one JVP at : cos | one JVP at : damage |
| RegMean | 0.756 | 0.668 | 4.0 | 0.978 | 0.973 | 1.1 | −0.209 | −0.004 |
| Iso-C | 0.793 | 0.537 | 3.5 | 0.989 | 0.978 | 2.1 | 0.160 | 0.104 |
| Iso-CTS | 0.788 | 0.470 | 5.3 | 0.986 | 0.971 | 2.8 | 0.044 | 0.039 |
| TIES | 0.674 | 0.479 | 11.8 | 0.959 | 0.950 | 3.2 | 0.191 | 0.243 |
| Task arithmetic | 0.721 | 0.509 | 11.3 | 0.977 | 0.964 | 3.1 | 0.243 | 0.207 |
Table 2.
Where the response must be read, and why. The pretrained anchor fails in vision. On three CLIP backbones its forecast is nearly orthogonal to the realized drift, while the receiver-local forecast is closely aligned with it (Table 3 and Table 6). In LoRA language pilots it is weaker but keeps some signal. Anchor controls tie the failure to displacement along the receiver’s task vector rather than to distance, and training trajectories place its onset in the first few dozen steps (Figure 2).
| anchor | what it tests | mean cos | min cos | dist. from | disp. along |
| (receiving model) | the forecast as used | 0.949±0.008 | 0.296 | 0 | 0 |
| isotropic, matched distance | is distance the cause? | 0.945±0.009 | 0.315 | 0.00 | |
| isotropic, 1.41× distance | farther still random | 0.942±0.009 | – | 0.00 | |
| span of other task vectors | is the direction structured? | 0.665±0.036 | −0.320 | 0.07 | |
| toward , matched distance | third model, confound removed | 0.461±0.014 | −0.007 | 0.67 | |
| isotropic tilted to | direction dose–response | 0.352±0.018 | – | 0.71 | |
| , a third fine-tuned model | does any fine-tuned point work? | 0.349±0.014 | −0.650 | 0.94 | |
| (pretrained) | the natural alternative | 0.122±0.011 | −0.719 | 1.00 | |
| + isotropic kick | is the point special? | 0.116±0.022 | – | 1.00 |
Table 3. Forecast drift cosine by anchor: 30 ordered pairs at , mean±sd over six checkpoint sets, min = worst pair. Last column: for anchor displacement from . This displacement orders the nine anchors (Spearman −0.98, ), and distance from does not (−0.16, ).
Figure 2. Cross-task responses re-aim at the start of fine-tuning. (A) Angle between and along the receiver’s fine-tuning. (B) The -anchored forecast collapses over the same window, while the forecast at the current checkpoint keeps rising. Receivers SVHN and GTSRB, five sources each, fixed at the source’s final checkpoint, , no labels. Lines are means over pairs and three fine-tuning seeds, and the band is the across-seed spread (, seed 0 beyond).
| arm | cos @ receiver | cos @ |
| CLIP ViT-B/32 (six sets) | 0.949 | 0.122 |
| CLIP ViT-B/32, canonical eight (56 pairs) | 0.975 | 0.180 |
| CLIP ViT-B/16 | 0.966 | 0.152 |
| CLIP ViT-L/14 | 0.924 | 0.090 |
| RoBERTa-base/LoRA (3 GLUE tasks) | 0.77 | 0.11 |
| Pythia-410M/LoRA (5 tasks) | 0.92 | 0.23–0.68 |
| Pythia-1.4B/LoRA (2 pairs) | 0.97–0.98 | 0.53–0.54 |
Table 6. Forecast drift cosine at the receiver vs at (vision at ; language in teacher-forced logit space, LoRA fine-tunes). Language rows are pilots.
LR-Merge, a cheap convex correction built on the forecast. It improves every closed-form, isotropic and sparsified base we test on the canonical eight-task benchmark, the 14- and 20-task suites and larger backbones, and the gain generally grows with the number of tasks (Table 4). It matches exact teacher matching through the merged network and AdaMerging’s objective run on the same base, at a third or less of their cost to convergence (at an equal budget, exact matching is at parity). It beats gradient-free search at matched wall-clock, and it amortizes, since each further coefficient query is a convex solve of a few seconds (§ 6.2).
| suite (backbone) | checkpoints | base merger | base | +LR-Merge | |
| canonical eight (B/32) | ours, 3 sets | RegMean | |||
| Iso-C | $85.36\pm0.03 | | ||||
| Iso-CTS | |||||
| six-task (B/32) | ours, 6 sets | RegMean | $90.58\pm0.20 | | ||
| Iso-C | $91.28\pm0.27 | | ||||
| Iso-CTS | |||||
| TIES | $86.08\pm0.45 | | ||||
| ours, 3 sets | DOGE (adaptive) | $89.53\pm0.32 | | |||
| 14 tasks (B/32) | released, 1 set | RegMean | |||
| Iso-CTS | |||||
| 20 tasks (B/32) | released, 1 set | RegMean | |||
| Iso-CTS | |||||
| six-task (B/16) | ours, 1 set | RegMean / Iso-C / Iso-CTS | / / | / / | / / |
| six-task (L/14) | ours, 1 set | RegMean / Iso-C / Iso-CTS | / / | / / | / / |
| canonical eight (B/32) | released, 1 run | AdaRank (adaptive) |
Table 4. LR-Merge as a correction of base mergers (average accuracy, %). Rows with one checkpoint set are pilots without across-set spread. The correction cost excludes the base merger (Iso-C: 54 s). Iso-C, Iso-CTS and TIES select their global scale by labeled probe accuracy, while RegMean, DOGE and LR-Merge use no labels. Per-set values and the eight-task suite with CIFAR-10 in Appendix D.
Related Work
Model merging. Task arithmetic (Ilharco et al., [11]) adds task vectors at chosen coefficients. One line edits the vectors before adding them: sparsifying and resolving sign conflicts ([32, 36, 3, 5]), masking parameters per task ([28, 24, 10]), or restructuring them in low-rank and spectral subspaces ([31, 1, 18, 20, 37]). A second spends calibration data ([21, 12, 34, 15, 29, 33]). Both learn what a composition did by building it, whereas LR-Merge corrects their outputs through a frozen first-order surrogate. Output-space projection merging (Evans et al., [6]) also works in output space, but constructs new layer-wise updates, whereas we keep the task vectors fixed and optimize only their coefficients.
Linearized task arithmetic. Tangent-space fine-tuning makes linearity true by construction, training within a linearization fixed at ([22, 17, 27]), and Ortiz-Jimenez et al. [22] show that post-hoc linearization of a nonlinear fine-tune at loses accuracy. Regularizers make vectors interfere less or behave more linearly: Jacobian-product penalties (, Yoshida et al., [35]), curvature-based disentanglement (Porrello et al., [23]), and distillation of linearized behavior (Sommariva et al., [26]). These change how checkpoints are trained. We take ordinary checkpoints as given and ask post hoc where their response must be read. For a fully linearized model the two anchors coincide by construction, but on our six tasks such fine-tunes lose several points of single-task accuracy (Table 7).
| measurement | value |
| Section 2: related approaches | |
| single-task accuracy, fully linearized vs ordinary fine-tunes (six tasks) | 87.5 vs 93.8 |
| ranking all 57 task subsets by merged accuracy, one frozen build | Spearman 0.88 |
| ranking pairwise interference: gradient-mismatch bound, receiver-side term / full mismatch / our forecast | 0.66 / 0.34 / 0.75 |
| Section 3: forecast | |
| flip precision at =0.2, 95% CI | 0.79, [0.72, 0.91] |
| acc. MAE at =1.0, forecast vs no change | 2.38 vs 4.70 |
| Section 4: anchor | |
| crossing step, median (range) over 30 pairseed curves | 16 (12.9–22.3) |
| weight displacement at saturation (step 50), fraction of final | 20% |
| response norm ratio, final vs initialization (median) | 0.97 |
| cosine of the two responses, sharing a top-32 output subspace | 0.11 |
| spread of cross-task angles at =500, task vs random directions | sd vs |
| Section 5: method | |
| smallest distance of a solved coefficient to the box | 0.48 |
| standalone task-wise selection by objective (set 0): hard-label CE / KL to soft labels / logit MSE / closed-form ridge | 80.7 / 80.6 / 75.0 / 69.3 |
| Section 6: experiments | |
| gain vs number of tasks, Spearman over 21 (, set) points | 0.92 |
| AdaRank: held-out teacher-matching loss reduction / largest accuracy gain | 5–17% / 0.3 |
| AdaRank with 256 probes vs RegMean + LR-Merge (canonical eight) | 82.00 vs 83.59 |
| wrong-anchor correction, change on SVHN / GTSRB | −24.5 / −7.9 |
| LR-Merge minus converged AdaMerging objective, Iso-C / Iso-CTS base (three sets) | +0.33 / +0.46 |
| wall-clock of those runs, LR-Merge vs AdaMerging objective | 950 s vs 3357–5610 s |
| random search at matched wall-clock (580 candidates), minus LR-Merge, six / grouped coefficients | −2.6 / −5.1 |
| 300 further exact teacher-matching steps from the LR-Merge solution | +0.20 0.07 |
| DTD at 32 probes, entropy vs teacher matching | 47.8 vs 66.0 |
| reuse: build / per-query solve (RTX 8000) | 926 s / 5.4 s |
| twenty queries: LR-Merge / per-configuration AdaMerging / exact matching | 1033 / 40,133 / 70,867 s |
| fast AdaMerging (500 steps, bf16) vs RegMean + LR-Merge, canonical eight | 78.68 in 123 s vs 83.59 in 774 s |
| Pythia-410M: Spearman with teacher-forced loss change / with free-generation degradation | 0.77 / 0.02 |
| Section 7: limitations | |
| one-JVP curvature estimate, Spearman with forecast failure | 0.991 |
Table 7. Additional quantitative results, grouped by the main-text section they support. ViT-B/32, six-task suite unless noted.
Predicting the effect of a merge. Zhou et al. [38] predict a scalar, whether a merge will succeed, from model-pair metrics, and SyMerge (Jung et al., [13]) relates cross-task performance to merging quality and, like LR-Merge, uses expert pseudo-labels, but backpropagates through the merged network. MAP (Li et al., 2025) amortizes coefficient search by fitting quadratic surrogates of task metrics to a set of evaluated merges, for Pareto-front estimation. LR-Merge instead linearizes model outputs with receiver-local JVPs and solves a convex teacher-matching problem, without evaluating each trial coefficient setting through the full merged network. Our forecast is per example and per coefficient. It answers the scalar question as a by-product, since one frozen build ranks all 57 task subsets of our six-task suite by real merged accuracy with high rank correlation. Daheim et al. [2] bound merging error through gradient mismatch. On our pairs only the receiver-side term of their bound ranks interference, and less well than our label-free forecast (Table 7). Influence functions ([14, 8]) and tangent overlap in continual learning ([4, 7, 9]) estimate effects of parameter changes more broadly, and Zhou et al. [39] observe cross-task linearity of features between endpoints.
Predicting Merge Interference
Setup. Unless stated otherwise, measurements use CLIP ViT-B/32 (Radford et al., [25]) on six tasks (MNIST, SVHN, GTSRB, EuroSAT, DTD, CIFAR-10), fine-tuned independently under the standard task-vector protocol (Ilharco et al., [11]). Each task has 256 unlabeled probes and an index-disjoint 1000-example evaluation split, and every measurement is replicated over six independently fine-tuned checkpoint sets. This suite uses CIFAR-10 where the canonical suite uses Stanford Cars, only because the Cars host was unavailable when the study began. The forecast and the correction were repeated on the canonical eight tasks (MNIST, SVHN, GTSRB, EuroSAT, DTD, Cars, RESISC45, SUN397), and those are the numbers to compare with prior work. Table 18 (Appendix F) lists every suite and its checkpoints.
| suite | tasks | checkpoints; used in |
| six-task | MNIST, SVHN, GTSRB, EuroSAT, DTD, CIFAR-10 | ours, six sets (one on B/16, L/14); § 3–§ 6 |
| eight-task | six-task + RESISC45, SUN397 | ours, six sets; Table 11, Table 12 |
| canonical eight | MNIST, SVHN, GTSRB, EuroSAT, DTD, Cars, RESISC45, SUN397 | ours, three sets; Table 4, Table 5; released [11]: AdaRank, fast AdaMerging |
| 14 tasks | canonical eight + CIFAR-100, STL-10, Flowers102, Oxford-IIIT Pet, PCAM, FER2013 | released [28], one set |
| 20 tasks | 14 tasks + EMNIST, CIFAR-10, Food-101, Fashion-MNIST, Rendered SST-2, KMNIST | released [28], one set |
Table 18. Task suites and checkpoints (ViT-B/32 unless noted).
Table 11. Eight-task suite with CIFAR-10 in place of Cars, six checkpoint sets.
| base merger (8-task) | base | +ours | sets | cost (s) | |
| RegMean | 83.100.25 | 86.190.20 | +3.09 0.34 | 6 | 1685 |
| Iso-C | 85.920.22 | 88.280.19 | +2.36 0.15 | 6 | 1700 |
| Iso-CTS | 86.180.46 | 88.660.22 | +2.47 0.52 | 6 | 1688 |
| TIES | 73.620.44 | 80.620.65 | +7.00 0.31 | 6 | 1693 |
The forecasting problem. Fine-tuning task from gives and task vector . For a target task A with probes , let be the stacked logits ( classes, fixed heads absorbed) and . Adding task B’s vector to the model fine-tuned on A displaces its outputs by , and first-order expansion at the receiver gives
which defines the receiver-local forecast : one JVP, no merge built. Appendix B bounds by curvature along the merge segment.
Metrics. For each ordered pair and strength , on the probes of A: drift cosine is the cosine between the flattened -dimensional and . Damage is the Spearman correlation over probes between forecast and realized change of the true-class logit. Acc. MAE is the absolute difference between forecast accuracy ( of ) and realized merged accuracy. A flip is a probe whose differs from that of , pooled over pairs. Drift cosine, damage and MAE are averaged over the 30 ordered pairs, then over three probe seeds. Labels enter only damage and accuracy. The no-change reference predicts the receiver’s own accuracy.
Pairwise forecasts. Table 1 reports the forecast at the strengths coefficient search visits. It predicts per-task accuracy to within a few tenths of a point, recovers about four in five flipping examples at similar precision, and orders which examples are hurt. Against the no-change reference it halves the accuracy error at every strength we test. Its larger added value is per example, where no-change predicts nothing. The forecast replicates on all six checkpoint sets and on all 56 pairs of the canonical eight tasks (Table 9). Scores computed from the task vectors alone do not substitute for it. Task-vector cosine barely tracks the per-pair accuracy drop, which the forecast tracks closely (Table 7). When the receiver is itself a scaled model , the forecast read at that receiver stays accurate and the forecast stays poor (Table 8), so the anchor is whichever model receives the update. Used to score candidate coefficients, the forecast ranks them as well as building and evaluating each (Appendix § 13).
| 0.3 | 0.83 / 0.42 / 0.29 | 0.69 / 0.43 / 0.23 | 0.51 / 0.45 / 0.15 | 0.37 / 0.45 / 0.05 |
| 0.5 | 0.91 / 0.66 / 0.21 | 0.82 / 0.63 / 0.20 | 0.68 / 0.59 / 0.16 | 0.56 / 0.55 / 0.10 |
| 0.7 | 0.94 / 0.83 / 0.15 | 0.88 / 0.79 / 0.15 | 0.77 / 0.73 / 0.14 | 0.66 / 0.65 / 0.12 |
| 1.0 | 0.95 / – / 0.11 | 0.91 / – / 0.12 | 0.83 / – / 0.11 | 0.74 / – / 0.11 |
Table 8. Drift cosine for a scaled receiver : forecast at the actual receiver / at / at , mean over 30 pairs and three probe seeds (seed spread ). At the receiver is .
Table 9. Forecast and anchor gap over six independently fine-tuned checkpoint sets, 30 ordered pairs each at . Last row: the canonical eight tasks, all 56 ordered pairs of one checkpoint set.
| checkpoint set | cos @ | cos @ | damage | acc. MAE (pp) |
| 0 | 0.946 | 0.124 | 0.916 | 0.13 |
| 1 | 0.954 | 0.126 | 0.925 | 0.17 |
| 2 | 0.958 | 0.107 | 0.927 | 0.16 |
| 3 | 0.946 | 0.109 | 0.916 | 0.09 |
| 4 | 0.953 | 0.132 | 0.930 | 0.13 |
| 5 | 0.936 | 0.133 | 0.916 | 0.10 |
| mean sd | 0.9490.008 | 0.1220.011 | 0.9220.006 | 0.130.03 |
| canonical eight, 56 pairs | 0.975 | 0.180 | 0.946 | 0.22 |
Complete merges: the forecast is local. The forecast applies to any merger as a black box. A merge displaces the receiver by , and forecasts that displacement. A fifth of the way to the merge it is well aligned for all five mergers, while the forecast is close to orthogonal (Table 7). At the merge itself a single read degrades and misses per-task accuracy by several points (Table 2), so one JVP does not replace evaluating a complete merge. Reading the response at two points along the path, the midpoint rule for (Appendix B), recovers nearly all of the realized drift. This shapes the design of LR-Merge, which reads the response at the current merge and uses it only for nearby corrections.
Why the Forecast Must Be Receiver-Local
Both forecasts are first-order expansions. The forecast is a Taylor expansion at the point where the update is applied. The forecast transports a derivative from another point, and its error is the change of along the anchor displacement, a curvature term. The question for merging is whether the error grows with distance, so that any nearby anchor will do, or depends on direction, so that the anchor inherited from task arithmetic is the wrong one. We find the latter, locate when it forms, and show in § 6.1 that it decides whether a correction helps or harms.
Direction, not distance. Table 3 moves the anchor. Isotropic anchors as far from as , or farther, forecast as well as the receiver. Tilting an isotropic anchor toward cuts the drift cosine by almost two thirds. A third fine-tuned model does no better, and a full-size isotropic kick away from does not revive it. Displacement along the receiver’s own task vector orders all nine anchors almost perfectly, while distance does not order them (Table 7). Displacement along other tasks’ vectors also costs, but less. The span-of-others anchor barely moves along and still loses about a third of its alignment, so this coordinate is the dominant factor but not a complete characterization. We therefore use “receiver-local” only for the rule read the response at the model being updated. That some other anchors also work is an empirical observation about our settings.
An early re-aiming. We track at checkpoints of the receiver’s fine-tuning (Figure 2). Its angle to the initialization response passes within roughly the first twenty steps and saturates by step 50, when the weights have covered only a fifth of their final displacement. The forecast collapses over the same window. Two further receivers replicate the early crossing at different plateau heights. The response rotates rather than shrinks. Its norm stays almost constant, and the initial and final responses keep most of their energy in the same low-dimensional output subspace while becoming nearly orthogonal. Norm-matched random directions rotate on the same schedule, so the onset is generic to the evolving Jacobian. What is pair-specific is the plateau, where cross-task angles spread about three times more widely than random ones (Table 7, Appendix C).
LR-Merge: Freezing the Local Response
Forecasting one candidate is not yet a merging method, because coefficient selection compares many nearby candidates. LR-Merge reads the responses once at a merge, freezes them, and searches coefficients inside the resulting surrogate.
Coefficients and reference model. A base merger (RegMean, Iso-C, TIES, . . .) outputs . LR-Merge searches over total coefficients on the task vectors added to it, , starting from (the base itself). In the grouped variant each block-restricted piece ( zeroes parameters outside group ) receives its own coefficient, so , with groups on ViT-B (one per transformer block plus embedding/projection) and 25 on ViT-L/14. The reported corrections use the grouped variant, and everything below reads unchanged with replaced by .
One round. Round holds coefficients and (i) forms the anchor ; (ii) reads and freezes the responses for every target and source , giving the affine surrogate
(iii) fits by minimizing
where are pseudo-labels from the fine-tuned teacher on unlabeled probes (so LR-Merge needs the teachers and unlabeled task-distribution probes, but no labels); (iv) applies the clipped coefficients ; and (v) moves the anchor there and rebuilds the responses for the next round. We use two rounds throughout (Algorithm 1).
Why it is cheap, and convex. Once frozen, every candidate’s logits are linear combinations of stored vectors, so no merged network is built, run, or backpropagated through during the search. Because the surrogate logits are affine in , is convex and, with the ridge, -strongly convex with a unique minimizer (Appendix B). We minimize it with a fixed Adam schedule. Clipping after an unconstrained solve is not a projected method. In every deployed run the coefficients stay strictly inside the box (Table 7), so clipping Algorithm 1 LR-Merge
Require: task vectors , teachers , unlabeled probes , base , box , , rounds
; ▷ pseudo-labels, once
for do
▷ anchor
▷ one JVP each; frozen
▷ objective (3) on surrogate (2)
end for
return
leaves the optimizer’s output unchanged. We match hard pseudo-labels with cross-entropy, as does the exact teacher-matching baseline of § 6.2. This works as well as KL to the teacher’s soft labels and clearly better than regressing teacher logits (Table 7). The ridge keeps each move small, and the second round re-reads the geometry after the first. This is how the method respects the locality of § 3, and at the deployed anchors the forecast stays well aligned (Appendix § 11).
Cost, and a low-rank build. Reading the responses costs JVPs per anchor, which grows quadratically with the number of tasks. A rank- output projection removes that growth. For each target, the projected responses are inner products of the task vectors with vector–Jacobian products that depend on the target alone (Appendix § 10), so the build costs passes. Rank 16 matches the exact build’s accuracy (Table 5) and is used for the 14- and 20-task suites. A deployed correction keeps the probes, their cached pseudo-labels (not the teachers), the task vectors, and the response tensors, about 10 MB per anchor for six tasks (Table 14).
| correction of the same RegMean base | accuracy | paired vs. LR-Merge | wall-clock (s) | sets |
| six-task suite | ||||
| exact teacher matching, converged | +3.24±0.23 | +0.14 ± 0.24 | 3447±78 | 6 |
| exact teacher matching, stopped at LR-Merge’s budget | +3.18±0.30 | +0.08 ± 0.14 | ≈900 | 6 |
| AdaMerging objective on the same base, converged | +3.13±0.52 | +0.03 ± 0.22 | 3432±32 | 6 |
| coordinate descent on the same objective, matched time | +2.87±0.27 | −0.41 ± 0.28 | 964 | 3 |
| LR-Merge, exact responses | +3.10±0.37 | – | 952±15 | 6 |
| LR-Merge, rank-16 sketch | +3.18±0.33 | +0.08 ± 0.12 | 704±6 | 6 |
| LR-Merge with responses read at (wrong anchor) | −3.78 | −6.88 ± 0.76 | ≈952 | 6 |
| canonical eight-task benchmark | ||||
| exact teacher matching | +1.75 | −0.28 ± 0.27 | 5290 | 3 |
| AdaMerging objective on the same base, converged | +0.91 | −1.12 ± 0.21 | 5087 | 3 |
| LR-Merge | +2.03±0.08 | – | 1685 | 3 |
Table 5. Correcting the same RegMean base by different optimizers. Paired differences are (row − LR-Merge) over the listed checkpoint sets.
| backbone | params | groups | build (s) | solve (s) | peak GB | cache MB |
| ViT-B/32 | 88M | 13 | 201 | 1.5 | 6.5 | 10 |
| ViT-B/16 | 86M | 13 | 838 | 1.6 | 6.8 | 10 |
| ViT-L/14 | 304M | 25 | 8704 | 1.4 | 19.8 | 19 |
Table 14. Cost of one response build and one convex solve, six tasks, 256 probes per task, one L40S GPU. ViT-L/14 uses a JVP batch of 8.
Experiments
All wall-clocks are end-to-end on one GPU with identical probe budgets (RTX 8000 unless noted; Appendix § 14).
LR-Merge Improves Closed-Form, Isotropic and Sparsified Mergers
Table 4 collects the correction results. On the canonical eight-task benchmark LR-Merge improves RegMean, Iso-C and Iso-CTS on every checkpoint set. On the six-task suite it improves all four bases on every set, and it gives a smaller but significant gain on the recent adaptive merger DOGE [29]. The gain generally grows with the number of merged tasks. Over a nested ordering of two to eight tasks it is negligible at two tasks, exceeds three points from six tasks on, and is significant for every (Table 12). With released 14- and 20-task checkpoints (Wang et al., [28]) it is larger still and largest at 20 tasks, concentrated on tasks where the bases collapse (EMNIST, KMNIST). The same direction holds on ViT-B/16 and ViT-L/14.
Table 12. Average accuracy vs number of merged tasks on the eight-task suite (tasks added in the order MNIST, SVHN, GTSRB, EuroSAT, DTD, CIFAR-10, RESISC45, SUN397), meansd over three checkpoint sets; task arithmetic at , and from a paired test of the correction against its base.
| task arithmetic | RegMean base | RegMean + ours | gain | ||
| 2 | 90.500.39 | 98.230.21 | 98.280.15 | +0.05 0.17 | 0.67 |
| 3 | 86.060.48 | 97.090.16 | 97.410.19 | +0.32 0.24 | 0.15 |
| 4 | 81.730.25 | 96.180.21 | 97.030.21 | +0.85 0.35 | 0.052 |
| 5 | 75.230.12 | 88.820.20 | 91.050.15 | +2.23 0.34 | 0.0077 |
| 6 | 75.800.15 | 87.370.25 | 90.650.10 | +3.28 0.35 | 0.0037 |
| 7 | 75.770.29 | 86.530.10 | 89.980.15 | +3.45 0.25 | 0.0018 |
| 8 | 73.280.45 | 82.980.19 | 86.290.08 | +3.31 0.23 | 0.0015 |
AdaRank (Lee et al., [15]) adapts its merge with 10k entropy steps on the test images and is already close to its teachers. LR-Merge still lowers its held-out teacher-matching loss, but no label-free setting we tried moves its accuracy by more than a few tenths of a point (Appendix § 13). Given the same 256 unlabeled probes instead of the test pool, AdaRank falls below RegMean with LR-Merge (Table 7). LR-Merge pulls a merge toward its teachers, so it helps in proportion to how far the base is from them.
The anchor is part of the algorithm. The wrong-anchor ablation changes one thing. It reads the responses at , , instead of at the current anchor , and keeps the objective, probes, pseudo-labels, solver, rounds and wall-clock. The RegMean correction then loses about seven points on every checkpoint set and ends below the uncorrected base, with most of the damage on SVHN and GTSRB (Table 5; per set in Table 10). In standalone selection the same swap costs about as much. A wrong anchor thus turns a helpful first-order correction into a harmful one.
Table 10. Wrong-anchor ablation (six-task, six checkpoint sets): responses read at the current anchor vs at , all else fixed, in standalone selection and in the RegMean correction. The correction drop has .
| responses read at | s0 | s1 | s2 | s3 | s4 | s5 | mean sd |
| standalone selection | |||||||
| current anchor | 80.65 | 80.18 | 80.72 | 79.98 | 80.83 | 80.00 | 80.390.38 |
| pretrained | 73.77 | 73.83 | 73.80 | 74.00 | 74.65 | 73.65 | 73.950.36 |
| paired drop | 6.88 | 6.35 | 6.92 | 5.98 | 6.18 | 6.35 | 6.440.38 |
| correction of RegMean (base 87.49 0.22) | |||||||
| current anchor | 90.53 | 90.68 | 90.73 | 90.23 | 90.80 | 90.52 | 90.580.20 |
| pretrained | 84.12 | 83.67 | 84.07 | 84.45 | 82.90 | 83.03 | 83.710.63 |
| paired drop | 6.41 | 7.01 | 6.66 | 5.78 | 7.90 | 7.49 | 6.880.76 |
Cost and Comparisons on Equal Footing
Same base, same objective. Table 5 isolates what the surrogate buys by holding the base fixed. Exact teacher matching backpropagates the same objective through the real merged network. On the six-task suite, run to convergence, it is within noise of LR-Merge at about 3.6 times the wall-clock, and stopped at LR-Merge’s budget it is at parity. On the canonical eight tasks it trails LR-Merge at three times the cost. AdaMerging’s entropy objective run to convergence on the same base ties on the six-task suite and trails by about a point on the canonical eight. From the stronger Iso-C and Iso-CTS bases LR-Merge is also ahead of it, in under a third of its wall-clock (Table 7). Gradient-free search over the same coefficients at matched wall-clock does not substitute. Coordinate descent trails slightly, and random search over several hundred evaluated candidates trails by three to five points (Table 7). The surrogate therefore reaches the accuracy of exact optimization from cheap gradients, with no stopping step to choose. Its optimum is close to the exact one, since continuing exact optimization from it adds only a fifth of a point. In standalone selection, output-space projection merging (Evans et al., [6]) is about ten points less accurate than task-wise LR-Merge and about 25 times slower (Table 16).
| method (six-task, from scratch) | avg | sel. cost (s) |
| Iso-C (closed-form) | 88.9±0.3 | 54 |
| AdaMerging (task-wise) | 80.1±0.5 | 1185 |
| AdaMerging (layer-wise) | 85.2±0.8 | 2007 |
| Exact teacher matching (layer-wise) | 85.6 | 2160 |
| OSP, per-row, all layers (reimpl.; set 0) | 70.8 | 2161 |
| OSP, final MLP only (reimpl.; set 0) | 65.9 | 49 |
| LR-Merge (task-wise) | 80.4±0.4 | 84 |
| LR-Merge (layer-wise) | 85.4±0.5 | 933 |
Table 16. Standalone selection (six-task, from scratch). LR-Merge matches the optimization-based methods at a fraction of their cost. Closed-form Iso-C is stronger standalone, which is why correction is the intended use. With six re-anchoring rounds LR-Merge reaches 86.17 ± 0.61.
Few probes. Figure 3A compares pipelines at matched probe budgets. RegMean is built from probes and corrected by LR-Merge with the same , and standalone AdaMerging uses the same pool. At 32 probes per task LR-Merge still improves its RegMean base, and it beats AdaMerging given eight times as many probes at about 26 times less compute (Table 13). The arms start from different points, and Table 5 gives the equal-footing comparison. Part of the gap is the objective. With 32 probes over DTD’s 47 classes, entropy-based selection falls far behind teacher matching (Table 7).
Figure 3. (A) Accuracy vs unlabeled probe budget on identical pools: RegMean built from the same probes (dashed), LR-Merge on that RegMean, and standalone layer-wise AdaMerging (values in Table 13). (B) Cumulative cost of repeated coefficient queries. At twenty queries LR-Merge is 39–69× cheaper. Mean±sd over three checkpoint sets.
| probes | RegMean@b | + LR-Merge | AdaMerging | gap | LR / Ada (s) |
| 32 | 86.75±0.22 | 88.47±0.94 | 82.86±1.08 | +5.61 | 130 / 2378 |
| 64 | 87.18±0.25 | 89.18±0.61 | 84.42±0.99 | +4.76 | 246 / 3416 |
| 128 | 87.27±0.19 | 90.18±0.18 | 85.14±0.75 | +5.04 | 480 / 3435 |
| 256 | 87.35±0.26 | 90.66±0.11 | 85.41±0.96 | +5.25 | 943 / 3437 |
Table 13. Per-budget values behind Figure 3A, checkpoint sets 0–2. RegMean@b is built from the same b probes that LR-Merge uses, and AdaMerging is standalone. Building RegMean adds under 5 s.
Reuse. The build is a cache. Once built, it answers each further coefficient query (a new ridge, clamp, or task subset) with a convex solve of a few seconds. Twenty queries then cost about 40 times less than per-configuration AdaMerging and about 70 times less than per-configuration exact matching (Figure 3B). A fast AdaMerging implementation (500 steps, bf16) is quicker for a single configuration but about five points less accurate on the canonical eight tasks (Table 7). On an L40S a build takes minutes on ViT-B/32 and under two and a half hours on ViT-L/14, and a solve takes under two seconds (Table 14). Re-anchoring is a new build. When a sixth task arrives at a deployed five-task merge, computing only its responses and re-solving is several times cheaper than a full rebuild and keeps the five deployed tasks. The arriving task pays for the stale anchor, and a re-anchor recovers part of that loss (Table 15).
| accuracy (6 tasks) | deployed 5 tasks | new task (CIFAR-10) | cost (s) | |
| incremental (new JVPs + re-solve) | 89.54±0.34 | 90.7 | 83.7±2.4 | 146±1 |
| + surrogate re-anchor | 90.09±0.33 | 90.6 | 87.5±2.0 | 613 |
| full rebuild (base + correction) | 90.65±0.10 | 89.9 | 94.5 | 964 |
| AdaMerging from scratch† | 85.2±0.8 | – | – | 2007 |
Table 15. A sixth task arrives at a deployed five-task corrected merge (three checkpoint sets). †Six-task standalone run of Table 16, whose per-task CIFAR-10 accuracy is not reported.
Transfer to Other Backbones and to Language Models
The anchor gap holds on larger CLIP backbones and narrows, without closing, on LoRA language models (Table 6). There the pretrained anchor keeps partial signal, consistent with kernel-regime descriptions of fine-tuning at scale [19]. As a correction of a task-arithmetic merge of five Pythia-410M LoRA fine-tunes, LR-Merge lowers held-out teacher-forced NLL and raises token accuracy in about a minute, matching exact teacher matching through the merged network at the same budget (Table 17). These are pilots on one fine-tune with teacher-forced outputs. The forecast ranks teacher-forced loss changes but not which free generations degrade (Table 7), and we make no claim about generation.
| arm (Pythia-410M/LoRA, five tasks) | held-out NLL | top-1 token acc. (%) | wall (s) |
| base merge (TA ) | – | ||
| LR-Merge (ours) | 78 | ||
| exact TM at ours’ budget | |||
| exact TM, oracle best step | – | ||
| exact TM, final (1000 steps) | 1227 |
Table 17. Pythia-410M/LoRA correction, mean±sd over three probe/optimizer draws (one LoRA fine-tune, teacher-forced held-out metrics). Both methods beat the base ( NLL, token accuracy), and oracle stopping steps differ across draws (500/50/100).
Limitations
One read of the forecast is accurate for nearby candidates, not for a complete merge (Table 2). A one-JVP curvature estimate ranks where it fails (Table 7) but is no certificate. The anchor controls are empirical, not causal, and checkpoints trained to stay linear (, TAK, DELTA) were not measured. LR-Merge assumes one shared initialization and architecture and needs a few tens of unlabeled probes per task. It matches rather than exceeds exact teacher matching at matched compute, and it does not help mergers already close to their teachers. The larger-backbone, 14/20-task, task-arrival (one arriving task) and language results each rest on one checkpoint set or fine-tune.
Conclusion
A single directional derivative, read at the model receiving a task vector, forecasts the interference that vector causes on nearby merges, per task and per example. Read at the pretrained model it does not, in vision, because fine-tuning re-aims the cross-task response within its first steps. Frozen at a merge, the same derivative makes coefficient selection a cheap convex problem. LR-Merge corrects closed-form, isotropic and sparsified merges, most strongly when many tasks are merged. It is as accurate as exact teacher matching run to convergence at a third of its cost, and it answers later queries in seconds. For the nonlinear fine-tunes studied here, what a task vector does depends strongly on the model that receives it, so watch where you linearize.
Reproducibility statement
Section 3 and Appendix § 14 specify models, data splits, probe/evaluation disjointness, fine-tuning and selection hyperparameters, and hardware, and Appendix § 12 reports per-set results. Code, per-experiment scripts, stored result files and the fine-tuned checkpoint sets will be released with the paper.
AI use statement
We used generative AI tools as coding and writing assistants for generating and refactoring experimental and analysis code, for checking the proofs in Appendix § 10, and for refining text. All AI-assisted code was reviewed and its outputs verified by the authors (re-running key configurations and validating tables and figures against the result files), and all AI-assisted text was reviewed and edited by the authors. We did not use generative AI for research ideation.
Additional Quantitative Results
Table 7 reports the measurements behind statements in the main text that no other table covers, grouped by section.
Bounds, Convexity and the Low-Rank Identity
Pairwise remainder. For with and , let , , so is the forecast. Taylor’s theorem with integral remainder gives the forecast error exactly, and bounding the integrand by the worst output acceleration gives
Merge paths. For a reference , candidate , and , . The -node composite midpoint forecast has error at most for three times continuously differentiable. On six-way merges at , going from one node to four raises drift cosine from 0.878 to 0.995 and cuts accuracy MAE from 3.97 to 0.35 points over six checkpoint sets.
Convexity. For fixed target and probe the surrogate logits are affine, , with and -th column of equal to , the -th correction direction ( or ). Cross-entropy has logit Hessian , so each per-example term has Hessian , and the ridge adds : is -strongly convex with a unique minimizer. The clamp is applied after the solve and is not part of the convex problem.
Low-rank identity. For an orthonormal spanning the leading output directions, , where is vector–Jacobian products for target alone. is found by a randomized range finder with power iterations and oversampling , for passes in all (, ).
Receiver-Locality: Additional Measurements
Deployed anchors. LR-Merge reads responses at a base merge far from every receiver ( times for RegMean) but nearly orthogonal to the task-vector span (projection fraction 0.015, per-unit-distance projection on −0.009), and 99.9% of its displacement from lies in the first two transformer blocks, where RegMean’s least-squares solve is ill-conditioned. Measured directly with forecast and realized drift both around , the drift cosine is 0.858 0.006 at and 0.909 0.017 at the typical solved correction magnitude. Interpolating Table 3 linearly in per-unit-distance projection, between the isotropic row (0.945 at 0) and the span row (0.665 at 0.07), gives 0.909 at ’s 0.009, matching the measurement.
Scope controls. A class-split control (two disjoint halves of CIFAR-100) and a from-scratch ResNet pilot place the anchor effect in fine-tuning of pretrained encoders, not in distribution shift.
Onset details. Per-seed 45° crossings on the primary receivers are 16.19/15.82/16.15. Until saturation the angle tracks weight displacement (Spearman +0.94 for ). Norm-matched random Gaussian directions cross at against for task vectors. At cross-task angles spread to sd 27° against for random directions (Levene ). MNIST and EuroSAT receivers cross at and 11.7 and plateau near 97° and 75°, and their forecast collapses from 0.83 to 0.07 while the current-anchor forecast reaches 0.975. Along the straight segment the response rotates smoothly with , so the checkpoint result is not an artifact of training time.
Per-Set Results
Additional Experiments
Ranking candidate coefficients. One build of 36 JVPs at a merged model (), then 60 random coefficient vectors within of it, scored by the forecast and, separately, by building and evaluating each on the same 256 probes (checkpoint set 0). Against held-out accuracy the forecast ranks them at Spearman 0.947 (direct evaluation 0.955), with the same top-1 regret (0.28 points), at 1.3 ms instead of 1.56 s per candidate. Anchored at over a wider box, which the candidates move away from, it drops to 0.50–0.67.
AdaRank. Canonical eight tasks, released checkpoints [11], one L40S GPU, 1000 evaluation images per task. Ported into our evaluation code with its official CART configuration (10k entropy steps on the test pool, bf16), AdaRank reaches 86.56 at its final step and 87.69 at its best step (chosen by us with labels), below its reported 89.2. LR-Merge on top gives 86.56 → 86.34 and 87.69 → 87.26 at 256 probes. A label-free search over 56 settings (trust region or proximal penalty around AdaRank’s coefficients, one or two rounds, per-task gating, 256 or 512 probes), selected by held-out teacher-matching loss, gains at most 0.3 points.
DOGE. Adaptive projective gradient descent [29] with AdaMerging-style coefficients on the same 256 probes per task reaches 88.73 ± 0.18, and LR-Merge raises it to 89.53 ± 0.32. With 32 probes per task the gain disappears (−0.81 ± 0.98, not significant).
Full test sets. Re-evaluating the six-task models of checkpoint set 0 on the complete official test sets changes each average by at most 0.65 points and no ordering (RegMean 87.74, with LR-Merge 90.99, Iso-C 88.76, with LR-Merge 91.35).
Language-model arm. RoBERTa-base with rank-8 LoRA on three GLUE tasks, and Pythia-410M/1.4B with rank-16 LoRA. Forecasts use held-out teacher-forced token blocks and target the full token-by-vocabulary logit displacement. At 1.4B, one JVP takes 14.6 s and 20 GB against 6.2 s and 5.5 GB for a forward pass (about 2.4 forward passes), and exact teacher matching needs 1227 s to converge. RoBERTa uses SST-2, MRPC and QNLI, with LoRA rank 8, on the query and value projections, AdamW, learning rate , 1000 steps of batch 32, maximum length 128. Pythia-410M: math (GSM8K), code (MBPP), French (opus_books en–fr), question answering (SQuAD v1.1) and topic (AG News), with LoRA rank 16, on the attention and MLP projections, AdamW, learning rate with cosine decay, 800 steps of batch 8, 256-token packed blocks. Forecasts use 64 held-out probe blocks per task, and each correction draw fits on 16 blocks and evaluates on 16 disjoint blocks. Pythia-1.4B: the two ordered pairs math↔French at .
Reproducibility Details
Splits. Each task uses 256 unlabeled probes for JVPs, forecasting and coefficient selection, and an index-disjoint 1000-example evaluation split, and probe seeds vary independently of fine-tuning seeds. Released 14/20-task checkpoints match their published accuracies within 0.11 points. Standard deviations over six checkpoint sets are sample deviations, while three-set controls report population deviations.
Fine-tuning. AdamW, learning rate , weight decay 0.1, batch size 128, 2000 iterations, cosine decay, frozen zero-shot text heads, and the non-QuickGELU OpenAI-weight CLIP variant of open_clip. Tangent-space fine-tunes use the same settings.
JVPs. Forward-mode automatic differentiation in evaluation mode, with no full Jacobians or per-example gradients. The math scaled-dot-product-attention backend is used because the fused backend does not support forward-mode derivatives. Software: PyTorch 2.8.0 (CUDA 12.8) and open_clip 3.3.0. Merged parameters are evaluated with functional_call, without building new modules.
LR-Merge. Ridge , Adam with learning rate 0.01 for 800 steps (task-wise and grouped correction) or 1500 steps (grouped standalone selection). Initial coefficients 0 for corrections and 0.3 for standalone selection, box for corrections and for selection, two rounds. Within to the box changes no coefficient and no accuracy, while a sign-constrained box clips 46% of coordinates and costs points. Reported cost covers both builds, the solver and re-anchoring, and pseudo-labels are computed once.
Base mergers. Task arithmetic: with . RegMean [12]: per linear layer, with input Gram matrices computed on each task’s 256 probes through its own fine-tuned model, off-diagonal reduction and ridge . Other parameters (LayerNorm, embeddings, biases) are averaged. Because nn.MultiheadAttention calls its output projection functionally, a forward hook on that module never fires, so we route the attention context through the module (verified equivalent to ) so that the 12 output projections also receive the RegMean solve, and the reported RegMean bases use this fix. Iso-C and Iso-CTS [20]: ports of the official code (isotropic spectrum per 2-D weight, Iso-CTS with common-subspace fraction 0.8), with the global scale chosen from by average accuracy on the labeled probes (early stopping after three non-improving steps), as in the authors’ protocol but with our probes as the validation split. TIES [32]: keep the top 20% of per task, elect signs, average agreeing entries, with the scale chosen from by labeled probe accuracy, as for Iso-C and Iso-CTS. DOGE [29]: adaptive projective gradient descent with AdaMerging-style coefficients on the same 256 probes per task. AdaRank [15]: official CART configuration (Appendix § 13).
Comparison optimizers (Table 5). All use the same base, the same grouped coefficients on the task vectors (), the same 256 unlabeled probes and pseudo-labels, and the same 1000-example evaluation split. Exact teacher matching: cross-entropy to the pseudo-labels through the real merged network, Adam (learning rate ) from zero coefficients, one batch of 64 probes per task per step, 2000 steps, accuracy logged at 100, 200, 300, 500, 1000, 1500, 2000 steps (the matched-budget point is read off this curve). AdaMerging objective on the same base: identical, with prediction entropy in place of cross-entropy. Random search: coefficient vectors drawn uniformly in the correction box , each scored by the teacher-matching objective on the real merged network, best kept, until the wall-clock budget (952s, LR-Merge’s) is spent. Coordinate descent: cycle over coordinates, line-search each on 5 evenly spaced values in the same box with the others fixed, same objective and budget. Selection in both uses no labels, and held-out accuracy is computed once for the selected point. AdaMerging (standalone): entropy descent (Adam, learning rate , batch 64, initialization 0.3) on the same unlabeled probe pool as LR-Merge. Costs exclude single-task fine-tuning, shared by every method.
Forecast experiments. Pairwise forecasts (Table 1, Table 3) evaluate on all 30 ordered pairs, each probe seed redrawing the probe and evaluation splits on the same checkpoints. Full-merge forecasts (Table 2) take the merged model of each base merger, set for each receiver, and compare with , and one JVP covers every , since the forecast is linear in . The two-JVP column uses the midpoint rule at .
References
- [1]Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan. Whoever started the interference should end it: Guiding data-free model merging via task vectors. In ICML, 2025.
- [2]Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In ICLR, 2024.arxiv.org/abs/2310.12808
- [4]Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume R abusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the NTK overlap matrix. In AISTATS, 2021.arxiv.org/abs/2010.04003
- [5]Guodong Du, Junlin Lee, Jing Li, et al. Parameter competition balancing for model merging. In NeurIPS, 2024.arxiv.org/abs/2410.02396
- [6]Bethan Evans, Benjamin Etheridge, Stephen Roberts, and Jared Tanner. Model merging by output-space projection. arXiv preprint arXiv:2605.29101, 2026.arxiv.org/abs/2605.29101
- [7]Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In COLT, 2022.arxiv.org/abs/2205.09588
- [8]Roger Grosse, Juhan Bae, Cem Anil, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023.arxiv.org/abs/2308.03296
- [9]Ido Nitzan Hidekel and Dan Raviv. Catastrophic forgetting is low-rank: A function-space theory for continual adaptation. In ICML 2026 Workshop on Continual Adaptation at Scale: Towards Sustainable AI (CATS), 2026.
- [10]Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. EMR-merging: Tuning-free high-performance model merging. In NeurIPS, 2024.arxiv.org/abs/2405.17461
- [11]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.arxiv.org/abs/2212.04089
- [12]Xisen Jin, Xiang Ren, Daniel Pretiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2023.arxiv.org/abs/2212.09849
- [13]Aecheon Jung, Seunghwan Lee, Dongyoon Han, and Sungeun Hong. SyMerge: From non-interference to synergistic merging via single-layer adaptation. In ICML, 2026.
- [14]Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.arxiv.org/abs/1703.04730
- [15]Chanhyuk Lee et al. AdaRank: Adaptive rank pruning for enhanced model merging. In ICLR, 2026.arxiv.org/abs/2503.22178
- [17]Tian Yu Liu and Stefano Soatto. Tangent model composition for ensembling and continual fine-tuning. In ICCV, 2023.arxiv.org/abs/2307.08114
- [18]Zehua Liu, Han Wu, Yuxuan Yao, Xiaojin Fu, Ruifeng She, Xiongwei Han, Tao Zhong, and Mingxuan Yuan. LoRE-merging: Exploring low-rank estimation for large language model merging. In Findings of EMNLP, 2025.arxiv.org/abs/2502.10749
- [19]Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In ICML, 2023.arxiv.org/abs/2210.05643
- [20]Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D. Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task-specific subspaces. In ICML, 2025.
- [21]Michael Matena and Colin Raffel. Merging models with Fisher-weighted averaging. In NeurIPS, 2022.arxiv.org/abs/2111.09832
- [22]Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In NeurIPS, 2023.
- [23]Angelo Porrello et al. Dataless weight disentanglement in task arithmetic via Kronecker-factored approximate curvature. arXiv preprint arXiv:2602.17385, 2026.arxiv.org/abs/2602.17385
- [24]Biqing Qi, Fangyuan Li, Zhen Wang, Junqi Gao, Dong Li, Peng Ye, and Bowen Zhou. Less is more: Efficient model merging with binary task switch. In CVPR, 2025.arxiv.org/abs/2412.00054
- [25]Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.arxiv.org/abs/2103.00020
- [26]Thomas Sommariva et al. Distilling linearized behavior into non-linear fine-tuning for effective task arithmetic. arXiv preprint arXiv:2605.18993, 2026.arxiv.org/abs/2605.18993
- [27]Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In ICLR, 2024.arxiv.org/abs/2310.04742
- [28]Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, Francois Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In ICML, 2024.arxiv.org/abs/2405.07813
- [29]Yongqian Wei, Anke Tang, Li Shen, Zixuan Hu, Chun Yuan, and Xiaochun Cao. Modeling multi-task model merging as adaptive projective gradient descent. In ICML, 2025.arxiv.org/abs/2501.01230
- [30]Mitchell Wortsman, Gabriel Ilharco, Samir Yadav, et al. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.arxiv.org/abs/2203.05482
- [31]Feng Xiong, Runxi Cheng, Wang Chen, Zhangqiu Zhang, Yiwen Guo, Chun Yuan, and Ruifeng Xu. Multi-task model merging via adaptive weight disentanglement. arXiv preprint arXiv:2411.18729, 2024.arxiv.org/abs/2411.18729
- [32]Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In NeurIPS, 2023.arxiv.org/abs/2306.01708
- [33]Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In ICML, 2024a.arxiv.org/abs/2402.02705
- [34]Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. AdaMerging: Adaptive model merging for multi-task learning. In ICLR, 2024b.arxiv.org/abs/2310.02575
- [35]Kotaro Yoshida et al. Mastering task arithmetic: τJp as a key indicator for weight disentanglement. In ICLR, 2025.
- [36]Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super Mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024.arxiv.org/abs/2311.03099
- [37]Shenghe Zheng and Hongzhi Wang. FREE-merging: Fourier transform for efficient model merging. In ICCV, 2025.arxiv.org/abs/2411.16815
- [38]Luca Zhou, Bo Zhao, Rose Yu, and Emanuele Rodolà. Demystifying mergeability: Interpretable properties to predict model merging success. In ICML, 2026.arxiv.org/abs/2601.22285
- [39]Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. On the emergence of cross-task linearity in pretraining-finetuning paradigm. In ICML, 2024.arxiv.org/abs/2402.03660


