betaFeedback

WorksComputer Science

Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging

Abstract

Multimodal model merging aims to consolidate task experts into a single model that retains their complementary capabilities. Most unimodal model merging methods combine expert updates within individual layers, and multimodal approaches largely follow this design. However, an expert update changes the representations passed to subsequent layers, allowing its influence to propagate across depth and affect how visual and textual information interact. When visual and language updates are combined, later updates act on inputs already modified by earlier ones, coupling their effects. This poses two challenges: (1) how to characterize the multimodal influence of individual expert updates across depth, and (2) how to jointly combine expert updates based on their multimodal influence. To address these challenges, we propose TAC-Merge for tracing and coordinating cross-layer influence in multimodal model merging. It contains two modules, i.e., multimodal influence mapping (MIM) and coupled merge control (CMC). MIM constructs graphs of update effects and uses Ricci curvature together with expert predictions to define a shared fusion objective. CMC models interactions among coefficient adjustments and jointly optimizes regional weights to synthesize one shared model. Experiments across diverse multimodal tasks demonstrate the effectiveness of TAC-Merge in consolidating complementary expert capabilities and supporting generalization to unseen tasks.

Introduction

Large pretrained models are often adapted into experts for different tasks and domains ([13]; [39]). Storing and deploying each expert separately incurs substantial overhead ([36]). Model merging offers an efficient alternative by combining expert updates into one deployable model ([35]; [14]). This is particularly valuable for multimodal large language models (MLLMs), whose experts span question answering, visual reasoning, chart understanding, and object localization ([34]). The objective is to consolidate these complementary capabilities in one model while preserving the interactions through which visual evidence and textual instructions jointly support task predictions.

Most model merging methods adopt a layer-wise design. Their strategies include direct parameter-space aggregation ([35]; [14]), interference-aware update transformation ([36]; [38]; [7]; [?]), and data-dependent merge estimation ([21]; [15]; [32]; [37]). Multimodal merging methods largely retain this layer-wise formulation, adapting how updates are combined to the architectural components and task requirements of multimodal models ([3]; [5]; [39]; [27]; [34]).

Some recent methods move beyond this layer-wise view by modeling dependencies across model depth. Chain of Merges refreshes activation statistics as layers are merged sequentially ([2]), whereas RegMean++ introduces explicit correlations between layers ([24]). These designs reflect a basic property of deep models: changing an intermediate representation can alter downstream computation ([8]). Their formulations, however, do not distinguish how these effects are distributed across modalities.

This perspective presents two challenges. Firstly, how to characterize the multimodal influence of individual expert updates across depth (CH1). An expert update can alter representations far beyond the region where it is applied, making local parameter changes insufficient to characterize its influence. At later depths, effects from different regions can overlap, obscuring the contribution of each update. Multimodal fusion adds ambiguity because visual evidence and textual instructions jointly shape hidden representations. For example, a change at an image-token position may reflect altered visual processing, a different response to the instruction, or both. Neither token labels nor final predictions alone reveal these distinctions. The challenge is therefore to trace effects across depth back to individual expert updates while distinguishing how they alter image- and instruction-dependent computation within shared multimodal representations.

Secondly, how to jointly combine expert updates based on their multimodal influence (CH2). Given the influence information, fusion requires determining how much of each expert update to apply in each region. The weights assigned to visual and language updates must be chosen together to form one model. A shared objective evaluates both multimodal influence structure and agreement with expert predictions. The preferred weight for one update depends on the weights assigned to the others. The measured influence must therefore support modeling complete combinations within a coefficient neighborhood to determine the regional weights of one shared model.

To address these challenges, we propose TAC-Merge, a framework that traces multimodal influence across depth and uses it to coordinate expert updates. It consists of two modules, i.e., multimodal influence mapping (MIM) and coupled merge control (CMC). MIM constructs influence graphs linking regional expert updates to hidden representations and conditional image–instruction responses at different depths and token positions. It evaluates candidate merges by measuring agreement with both the Ricci curvature of task-expert graphs and expert predictions. CMC uses the measured responses to identify a compact coefficient subspace and fit a local model of interactions among coefficient adjustments. It then jointly selects regional weights within a bounded neighborhood and combines the original expert updates into one static model shared across tasks.

We summarize our contributions as follows: (1) We identify a limitation of layer-wise multimodal merging, where local parameter changes alone cannot fully capture how expert updates influence downstream representations and image–instruction interactions. (2) We propose TAC-Merge, combining multimodal influence mapping with coupled merge control to trace expert influence and coordinate regional update weights. (3) We evaluate TAC-Merge on two multimodal benchmarks across eight seen and four unseen tasks. TAC-Merge achieves the best average performance among the compared merging methods on both task groups. Its gains on unseen tasks further demonstrate that the consolidated capabilities extend beyond the datasets used for expert training.

Multimodal Model Merging. Model merging consolidates task-specialized models into one shared set of parameters. On single-modality vision and language tasks, existing approaches can be grouped by how they construct the merged parameters. Direct parameter-space aggregation averages model weights or adds scaled task-specific updates to a common initialization [35, 14]. Interference-aware update transformation prunes or rescales update entries, resolves conflicting signs, or aligns singular subspaces before combining updates [36, 7, ?]. Data-dependent merge estimation uses parameter-importance statistics, feature-matching objectives, or calibration losses to determine parameter values and mixing coefficients [21, 15, 32, 37]. Multimodal merging extends these strategies to models with modality-specific encoders and shared language components, including compositions that reuse encoders while merging language parameters [3]. Architectural mappings support interpolation between heterogeneous multimodal models [5], while uncertainty estimated from visual and textual inputs guides expert selection and composition order [27]. Other methods normalize low-rank updates to preserve direction robustness [39], or denoise task vectors and optimize their interactions to combine task and modality capabilities [34].

Layer and Modality Dependencies. Dependencies across depth arise because parameter updates change the intermediate representations used by subsequent layers. Sequential merging accounts for this propagation by refreshing activation statistics after preceding layers are merged (Buzzega et al., 2026). Regression-based formulations incorporate correlations within and between layers when estimating merged parameters (Nguyen et al., 2026). Related output-space formulations optimize coefficients against calibration responses and extend the procedure sequentially across layers (Evans et al., 2026). In multimodal models, depth-dependent computation also determines where visual and textual information interact. Attention interventions localize information transfer among visual tokens, textual tokens, and answer-prediction positions (Zhang et al., 2025). Extensions to free-form generation trace these interactions during image captioning and chain-of-thought reasoning, revealing variation in visual–textual integration across architectures and tasks (Saporita et al., 2026).

Method

Problem Statement

Let f(⋅,θ0)f(\cdot, \theta_0) be a pretrained multimodal model, and let {θk}k=1K\{\theta_k\}_{k=1}^{K} be KK expert models adapted from the same base model for different multimodal tasks. Each expert contributes a task-specific update, such as the effective weight update of a LoRA adapter:

Δθk=θk−θ0,k=1,…,K,(1)\Delta\theta^k = \theta_k - \theta_0, \qquad k = 1,\ldots,K, \tag*{(1)}

where Δθk\Delta\theta^k is expert kk’s full update. Following data-dependent merging methods (Du et al., 2025; Zhang et al., 2026a; Buzzega et al., 2026), we assume access to a small calibration set C=⋃t=1KCt\mathcal{C} = \bigcup_{t=1}^{K} \mathcal{C}_t of multimodal inputs. Each task-specific subset is Ct={xit}i=1Nt\mathcal{C}_t = \{x_i^t\}_{i=1}^{N_t}, where xit=(vit,qit)x_i^t = (v_i^t, q_i^t) contains a visual input vitv_i^t and its textual instruction qitq_i^t. Our objective is to select a single model from the regional weighted family:

θ(α)=θ0+Concat⁡ℓ=0L−1(∑k=1KαℓkΔθℓk),(2)\theta(\boldsymbol{\alpha}) = \theta_0 + \operatorname{Concat}_{\ell=0}^{L-1}\left(\sum_{k=1}^{K}\alpha_{\ell k}\Delta\theta_{\ell}^{k}\right), \tag*{(2)}

where LL is the number of parameter regions, Δθℓk\Delta\theta_{\ell}^{k} is expert kk’s update block in region ℓ\ell, and αℓk\alpha_{\ell k} is its weight. Concatenation follows the original parameter order. The vector α\boldsymbol{\alpha} collects these coefficients. Given C\mathcal{C}, we seek coefficients that preserve the complementary multimodal capabilities represented by the experts in one model shared across tasks.

Framework Overview

Figure 1 illustrates TAC–Merge, which combines multimodal influence mapping (MIM) with coupled merge control (CMC). MIM constructs influence graphs from original readouts and responses to image and instruction changes, tracing regional update effects across depth and token positions. Matching task-expert curvature and predictions defines the reference objective. CMC uses measured responses to identify a compact coefficient subspace and fits a quadratic model from individual and paired coefficient variations. It jointly optimizes these coefficients around a uniform merge and accepts the proposal only if it improves the measured objective. The selected weights combine regional expert updates into one shared model with fixed inference coefficients.

Multimodal Influence Mapping

Multimodal Influence Graph. Visual influence varies across depth and token positions (Zhang et al., 2025), motivating a source-specific account of how expert updates alter image- and instruction-dependent computation. The LL disjoint parameter regions are ordered along the visual and language pathways, with boundaries and readout positions fixed throughout coefficient selection. The regional coefficients in Eq. (2) are shared across inputs and collected in α∈RLK\boldsymbol{\alpha} \in\mathbb{R}^{LK}. The uniform reference αˉ\bar{\boldsymbol{\alpha}} has every entry equal to 1/K1/K. Let hv(v,q,α)h_v(v,q,\boldsymbol{\alpha}) denote a pooled hidden readout, where rr specifies a region boundary and the group of token positions pooled there. The readouts cover visual representations and language image-token and instruction-token positions carrying fused information (Saporita et al., 2026). For each task tt, a fixed seeded permutation forms disjoint calibration pairs (i,j)(i,j) with i≠ji \ne j. Both pair orders are reused throughout coefficient selection, and any unpaired input still contributes to original readouts and output agreement. For a fixed pair (i,j)(i,j), suppressing task superscripts, define the image-change and instruction-change responses separately:

Figure 1. Overview of TAC-Merge. MIM maps multimodal expert influence, and CMC coordinates regional coefficients using the resulting reference objective to synthesize one shared model.

hij,rv(α)=(hr(vi,qi,α)−hr(vj,qi,α)),hij,rq(α)=(hr(vi,qi,α)−hr(vi,qj,α)),(3)h_{ij,r}^{v}(\alpha) = \left(h_r(v_i,q_i,\alpha) - h_r(v_j,q_i,\alpha)\right), \qquad h_{ij,r}^{q}(\alpha) = \left(h_r(v_i,q_i,\alpha) - h_r(v_i,q_j,\alpha)\right), \tag*{(3)}

Here hij,rvh_{ij,r}^{v} varies the image at a fixed instruction, whereas hij,rqh_{ij,r}^{q} varies the instruction at a fixed image. Both comparisons hold the readout location rr and coefficients α\alpha fixed. We retain the original representation as hi,r0(α)=hr(vi,qi,α)h_{i,r}^{0}(\alpha) = h_r(v_i,q_i,\alpha) and index these three readout channels by c∈{0,v,q}c \in\{0,v,q\}. Channel 0 is always retained. Channel vv or qq is retained when at least one fixed pair changes that modality, with this choice fixed throughout coefficient selection. Identical instructions give hij,rq=0h_{ij,r}^{q}=0 for every coefficient configuration, while the image-change response remains available. For each available channel, a bipartite graph links expert–region sources to observations labeled by depth and token-position group. An edge (ℓ,k,r)(\ell,k,r) connects expert kk in region ℓ\ell to observation rr reachable for channel cc in the forward computation. The architecture and attention mask fix these connections, excluding observations structurally independent of the changed modality. Its task-specific influence measures the sensitivity of the corresponding readout channel:

wℓk,rt,c(α)=E[∥∂hrc(α)∂αℓk∥22],(4)w_{\ell k,r}^{t,c}(\alpha) = \mathbb{E}\left[\left\lVert\frac{\partial h_r^c(\alpha)}{\partial\alpha_{\ell k}}\right\rVert_2^2\right], \tag*{(4)}

where sample indices are suppressed in hrch_r^c, and ∥⋅∥2\lVert\cdot\rVert_2 is the Euclidean norm. The expectation averages uniformly over all original inputs of task tt for c=0c=0 and over its fixed ordered pairs for c=v,qc=v,q. The separate graphs preserve the distinction between the three sensitivities. To characterize the allocation of influence around each node, incident weights are normalized into a probability distribution. For a source, neighbor masses describe relative influence across observations, while for an observation they describe relative sensitivity to expert–region updates. For nodes with positive total incident weight, suppressing task, channel, and coefficient arguments, we retain mℓk(ℓ,k)=mr(r)=1/2m_{\ell k}(\ell,k)=m_{r}(r)=1/2 at the source and observation nodes themselves and distribute the remaining mass among their architectural neighbors:

mℓk(r)=wℓk,r2∑r′wℓk,r′,mr(ℓ,k)=wℓk,r2∑ℓ′,k′wℓ′k′,r.(5)m_{\ell k}(r) = \frac{w_{\ell k,r}}{2\sum_{r'}w_{\ell k,r'}}, \qquad m_r(\ell,k) = \frac{w_{\ell k,r}}{2\sum_{\ell',k'}w_{\ell'k',r}}. \tag*{(5)}

The sums range over architecturally connected observations and sources, respectively. Other nodes receive zero probability mass. A node with zero incident influence places all its probability mass at itself. The equal self and neighbor masses retain endpoint identity and relative influence allocation for comparison across coefficient configurations.

Joint Influence Measurement. Joint updates change downstream computation [2], so selecting their coefficients requires characterizing their combined effect on multimodal influence. For edge (ℓ,k,r)(\ell,k,r) in channel cc, let mℓkm_{\ell k} and mrm_r denote the endpoint neighborhood distributions defined above, with task, channel, and coefficient arguments suppressed. Distances use the fixed shortest-path metric on the channel’s undirected architectural support with unit edge lengths. Since an edge’s endpoints are at unit distance, its Ollivier–Ricci curvature [25] is:

κℓk,rt,c(α)=1−W1(mℓk,mr),(6)\kappa_{\ell k,r}^{t,c}(\alpha) = 1 - W_1(m_{\ell k},m_r), \tag*{(6)}

where W1W_1 is the minimum cost of transporting one endpoint distribution to the other using this graph distance. For a fixed graph support, changes in curvature summarize changes in the relative influence patterns around each source–observation pair. The task-tt expert corresponds to coefficients equal to one for expert tt and zero for the other experts. We evaluate its channel graphs on CtC_t using the same fixed pairs and record κℓk,rt,c,⋆\kappa_{\ell k,r}^{t,c,\star} as the task-specific curvature target. For each original input, let pitp_i^t and pi(α)p_i(\alpha) denote expert and merged next-token distributions on fixed continuation prefixes generated by expert tt. Combining curvature matching with expert-output agreement gives:

E(α)=Et,iDKL(pit∥pi(α))+λEt,c,ℓ,k,r[(κℓk,rt,c(α)−κℓk,rt,c,⋆)2],(7)\mathcal{E}(\alpha) = \mathbb{E}_{t,i}D_{\mathrm{KL}}\left(p_i^t \mathbin\Vert p_i(\alpha)\right) + \lambda\mathbb{E}_{t,c,\ell,k,r}\left[\left(\kappa_{\ell k,r}^{t,c}(\alpha)-\kappa_{\ell k,r}^{t,c,\star}\right)^2\right], \tag*{(7)}

where DKLD_{\mathrm{KL}} is Kullback–Leibler divergence averaged over continuation positions and λ≥0\lambda\ge0 weights the geometric reference. Both expectations assign equal weight to tasks. Within each task, the first weights original inputs equally, and the second weights available channels equally and architecturally valid edges equally within each channel. Crossed inputs supply response measurements without answer targets. At each queried coefficient configuration α\alpha, this module computes the readout derivatives and (7) on CC. Curvature matching constrains the relative organization of influence across regions and modalities, while output agreement penalizes deviations from the expert’s next-token distributions on the original calibration inputs. Optimizing both terms links representation-level influence preservation to the predictive behavior needed for each task.

Coupled Merge Control

Joint Response Modeling. Selecting regional weights requires modeling how expert updates jointly affect the reference objective, since the effect of one update can depend on the others. Directly modeling these interactions over all LKLK coefficients is costly, so we construct a compact control subspace from measured multimodal responses and fit a local quadratic model within it. To select this subspace, we stack the readout derivatives returned by MIM and αˉ\bar{\alpha} into a response matrix, with one column per expert–region source and one row per scalar readout component across tasks, samples, channels, and readout locations. The squared stacking weights assign equal mass to tasks, then to available channels, and uniform mass to samples and readout locations within each channel. If the response matrix is zero, we retain the uniform merge. For a nonzero matrix, let σ1≥σ2≥⋯\sigma_1 \ge\sigma_2 \ge\cdots denote its singular values. We select the control dimension dd based on the stable rank ∑aσa2/σ12\sum_a \sigma_a^2/\sigma_1^2, which reflects how response energy is distributed across singular directions. The leading dd right singular vectors form an orthonormal basis V∈RLK×dV \in\mathbb{R}^{LK\times d}, with each column specifying a coordinated adjustment of expert weights across regions. The control vector u∈Rdu \in\mathbb{R}^d adjusts these directions within radius R>0R>0 of the uniform reference:

α=αˉ+Vu,∥u∥2≤R.(8)\alpha= \bar{\alpha} + Vu,\qquad\lVert u\rVert_2 \le R. \tag*{(8)}

Orthonormality makes RR a bound on the aggregate coefficient displacement from αˉ\bar{\alpha}. Within this neighborhood, a quadratic model approximates the objective change relative to αˉ\bar{\alpha}:

E^(u)=gTu+12uTHu,(9)\widehat{\mathcal{E}}(u) = g^\mathsf{T}u + \frac{1}{2}u^\mathsf{T}Hu, \tag*{(9)}

where g∈Rdg \in\mathbb{R}^d represents linear trends and the symmetric matrix H∈Rd×dH \in\mathbb{R}^{d\times d} represents quadratic effects within and between control directions. To fit these terms, we use MIM to evaluate (7) under a fixed set of symmetric coefficient variations. With eae_a denoting a coordinate unit vector, we use signed individual variations u=±Reau=\pm Re_a for 1≤a≤d1\le a\le d. We also use paired variations u=R(±ea±eb)/2u=R(\pm e_a\pm e_b)/\sqrt{2} for 1≤a<b≤d1\le a<b\le d, with all independent sign choices. Least-squares fitting then identifies the model coefficients.

(g,H)∈arg min⁡g, H=HT Eu[(E(αˉ+Vu)−E(αˉ)−gTu−12uTHu)2],(10)(\mathbf{g},H) \in\underset{\mathbf{g},\,H=H^\mathsf{T}}{\operatorname{arg\,min}}\ \mathbb{E}_u\left[\left(\mathcal{E}(\bar{\alpha}+Vu)-\mathcal{E}(\bar{\alpha})-\mathbf{g}^\mathsf{T}u-\frac{1}{2}u^\mathsf{T}Hu\right)^2\right], \tag*{(10)}

where Eu\mathbb{E}_u averages over the evaluated coefficient configurations. Individual variations identify linear trends and quadratic terms within each direction, while paired variations identify interactions between directions through the off-diagonal entries of HH. The interaction terms let the controller evaluate each adjustment in the context of the others, helping identify expert combinations that work together across regions to preserve multimodal influence.

Static Coefficient Synthesis. Because the fitted model is only a local approximation, a predicted improvement may not reduce the measured fusion objective. We therefore limit coefficient changes to the prescribed neighborhood and check the resulting proposal against the uniform reference. We first obtain a joint coefficient proposal by minimizing the fitted objective:

u⋆∈arg⁡min⁡∥u∥2≤RE^(u)(11)u^\star\in\mathop{\arg\min}_{\lVert u\rVert_2 \le R} \widehat{\mathcal{E}}(u) \tag*{(11)}

The reference u=0u = 0 is feasible and predicts zero objective change. We solve (11) using an eigendecomposition of HH and the spectral trust-region conditions [23]:

(H+μId)u⋆=−g,H+μId⪰0(12)(H + \mu I_d)u^\star= -g,\qquad H + \mu I_d \succeq0 \tag*{(12)}

where μ≥0\mu\ge0 is a spectral shift satisfying μ(∥u⋆∥2−R)=0\mu(\lVert u^\star\rVert_2 - R) = 0 and ⪰0\succeq0 denotes positive semidefiniteness. An interior solution has μ=0\mu= 0, while a boundary solution uses a shift consistent with ∥u⋆∥2=R\lVert u^\star\rVert_2 = R. In the eigenbasis, the boundary search reduces to a scalar norm condition, with a nullspace component included when required by a singular boundary solution. Together with feasibility, (12) characterizes a global minimizer of the fixed quadratic model over the prescribed ball. To verify the predicted improvement, we evaluate the proposal using (7) on the same calibration data and expert targets. The final coefficients are α⋆=αˉ+Vu⋆\alpha^\star= \bar{\alpha} + V u^\star if E(αˉ+Vu⋆)<E(αˉ)−τ\mathcal{E}(\bar{\alpha} + V u^\star) < \mathcal{E}(\bar{\alpha}) - \tau, and α⋆=αˉ\alpha^\star= \bar{\alpha} otherwise, where τ≥0\tau\ge0 is a fixed numerical comparison tolerance. This check requires no iterative refitting of the response model. The selected coefficients α⋆\alpha^\star synthesize the shared model:

θ⋆=θ0+Concat⁡ℓ=0L−1(∑k=1Kαℓk⋆Δθℓk)(13)\theta^\star= \theta_0 + \operatorname{Concat}_{\ell=0}^{L-1}\left(\sum_{k=1}^{K}\alpha_{\ell k}^\star\Delta\theta_\ell^k\right) \tag*{(13)}

The regional blocks are concatenated in the original parameter order to obtain f(⋅,θ⋆)f(\cdot,\theta^\star). By selecting regional coefficients jointly, CMC accounts for how updates in one part of the model affect the inputs seen by later updates. This coordination helps reduce interference across layers and modalities and retain the complementary capabilities of different experts.

Experiments

Experimental Setups

Datasets. Following RobustMerge [39], we adopt MM-MergeBench with eight seen datasets: ScienceQA [18], ImageNet [4], VQAv2 [9], REC-COCO [16], Mao et al. [20], OCR-VQA [22], VizWiz-Caption [10], Flickr30K [26], and IconQA [17]. These datasets cover visual question answering, referring expression comprehension, image classification, and image captioning. Generalization is evaluated on four unseen datasets: A-OKVQA [30], ImageNet-R [12], Screen2Words [33], and TabMWP [19]. The seen-task evaluation measures how well one merged model retains the capabilities of the eight adapted experts. Unseen tasks evaluate whether the same coefficients transfer to datasets outside expert adaptation, testing whether the consolidated capabilities remain useful under a change in task distribution.

Baselines. To evaluate the effectiveness of TAC–Merge, we compare it with the following representative baselines. We first consider two reference settings: (1) Zero-shot, which directly evaluates the pretrained model, and (2) Individual, which evaluates each task expert on its corresponding task. For single-modality merging, we include (3) TA [14], which adds the summed expert updates to the base model with a shared scaling coefficient, (4) TIES [36], which trims small values, resolves sign conflicts, and averages the remaining updates with consistent signs, and (5) DC-Merge [41], which smooths singular values and aligns task vectors in a shared orthogonal subspace before aggregation. For cross-layer merging, we compare (6) Chain of Merges [2], which merges layers sequentially while refreshing activation statistics, and (7) RegMean++ [24], which incorporates intra-layer and cross-layer dependencies into regression-based merging. For multimodal merging, we include (8) RobustMerge [39], which prunes and rescales low-rank parameters and normalizes task contributions to preserve direction robustness, and (9) OptMerge [34], which denoises task vectors and optimizes the merged update through an objective over their interactions.

MethodSciQAImage VQARECOCRVizWizFlickrIconQAAvg.AVQAImage-RS2WTabMWPAvg.
Zero-shot67.1338.8172.0076.1570.5823.7425.6525.8649.9978.6987.275.3564.3958.92
Individual97.9297.4774.8591.5371.5172.8060.4499.1583.21–––––
TA94.9959.0372.5273.4868.7852.2158.1179.7969.8686.2090.9515.9569.9565.76
TIES94.1556.3672.7174.2269.1952.4257.8070.3968.4183.6790.9715.5869.5764.95
DC-Merge95.2959.0372.5274.5568.4652.4257.9579.7970.0086.2090.1015.5969.9565.46
CoM95.2963.2172.5474.6268.4652.5958.0479.2570.5084.7290.1015.5971.8265.56
RegMean++95.8485.8669.9275.5567.4153.1657.9594.8575.0787.2585.8315.1970.7864.76
RobustMerge96.0348.1872.1574.5566.7550.7657.5992.1569.7787.0788.8315.7567.8564.88
OptMerge93.8088.4663.2538.7057.0448.7656.5047.8361.7958.0877.8614.9253.8651.18
TAC-Merge97.1288.0873.8683.4269.9358.6859.8497.0778.5088.4790.5718.7274.6468.10

Table 1. Performance comparison on Qwen3.5-4B.

MethodSciQAImage VQARECOCRVizWizFlickrIconQAAvg.AVQAImage-RS2WTabMWPAvg.
Zero-shot94.4035.0372.3477.3269.6127.2628.6644.2056.1079.6590.485.8760.4059.10
Individual95.7994.6974.4589.3374.3269.6859.5490.9481.09–––––
TA95.1935.0772.8777.0872.4348.7856.8672.2966.3287.1690.679.4161.6762.23
TIES96.1354.7770.6584.0770.2451.1258.1882.4970.9686.7289.6014.0463.1463.38
DC-Merge96.5872.0870.5286.5573.2552.7657.6786.9474.5487.3489.9514.4166.5664.56
CoM95.9846.8572.6774.1372.5951.3857.8983.4069.3687.5190.0713.0364.1063.68
RegMean++96.4377.4973.0383.9273.2752.3357.7786.4575.0887.9590.3312.3567.5564.55
RobustMerge95.1434.8572.6869.9071.6149.8358.1780.7866.6287.9590.2214.3564.0564.14
OptMerge96.2358.7770.7684.1570.7750.2557.7882.1671.3687.6089.7513.3361.7563.11
TAC-Merge96.3882.6373.5887.4273.4656.8359.3790.3377.5088.6590.5316.8469.9866.50

Table 2. Performance comparison on Qwen3-VL-8B.

Implementation Details. We use Qwen3.5-4B (Qwen Team, 2026) and Qwen3-VL-8B-Instruct [1] as the base models and train one LoRA expert for each of the eight seen tasks using Unsloth [11]. LoRA adapters are inserted into the attention and MLP modules of both the vision encoder and the language backbone, and the two parts are optimized jointly on each task’s image–instruction examples while the pretrained weights remain frozen. The LoRA rank and scaling factor are both 16. Experts train for one epoch with AdamW, learning rate 2×10−42 \times10^{-4}, linear decay, warmup ratio 0.03, weight decay 0.001, and a 2,048-token sequence limit.

Performance Evaluation

Main Results. Tables 1 and 2 compare performance on seen and unseen tasks using Qwen3.5-4B and Qwen3-VL-8B, respectively. The results yield four main observations. (1) Traditional single-modality merging methods struggle to balance heterogeneous multimodal capabilities. Task Arithmetic, TIES, and DC-Merge determine update compatibility through parameter aggregation, sign consistency, or subspace alignment, leaving the role of each update in visual–language computation implicit. Parameter-compatible updates can still alter the representations through which visual and textual information interact. The resulting trade-offs are evident on the 8B model, where TIES gains 19.70 points on ImageNet over Task Arithmetic while losing 2.22 points on VQA2. (2)

Existing multimodal merging methods remain limited in preserving the propagation of expert capabilities across depth. RobustMerge stabilizes low-rank update directions, and OptMerge optimizes interactions among task vectors within individual layers. These parameter-space objectives provide indirect control over how a visual update changes the input encountered by later language updates. Their lower aggregate performance is consistent with this gap between local update compatibility and preservation of multimodal influence throughout the model. (3) Cross-layer methods improve knowledge consolidation, with remaining limitations in multimodal transfer. Chain of Merges refreshes activation statistics sequentially, and RegMean++ models correlations across layers, accounting for representation changes induced by preceding updates. Their objectives emphasize activation agreement and cross-layer correlations, leaving source-specific image–instruction effects implicit. On the 4B model, RegMean++ improves the seen average over TIES by 6.66 points, while its unseen average remains 1.00 point below Task Arithmetic. This uneven transfer supports extending cross-layer dependency modeling with explicit multimodal influence measurements when jointly selecting expert coefficients. (4) TAC-Merge achieves the best average performance on both seen and unseen tasks across the two backbones. On Qwen3.5-4B, it exceeds the strongest merging baselines by 3.43 points on seen tasks and 2.34 points on unseen tasks. The corresponding gains on Qwen3-VL-8B are 2.42 and 1.94 points, respectively. It also leads the merging methods on seven of eight seen tasks and three of four unseen tasks for each backbone. These results support jointly selecting regional coefficients based on multimodal influence to preserve complementary expert capabilities in a shared model.

Ablation Study. Table 3 presents ablation results organized by module, examining each module and its central design. Results are reported as mean ± standard deviation. All variants share the expert checkpoints, calibration inputs, coefficient radius, and budget of objective evaluations, with the control dimension matched whenever a basis is used. (1) Effect of MIM. The variant w/o MIM uses output KL as its sole objective and a fixed random orthonormal basis for CMC, replacing the influence graphs and readout-response measurements. The variant w/o modality contrasts retains the original readouts h0h^{0} and their influence graphs, removing the image-change and instruction-change channels from both curvature matching and response-based subspace selection. Original readouts capture overall representation changes, while the two contrast channels distinguish update effects associated with changing the image or instruction at fixed observation locations. Removing MIM reduces seen averages by 2.38 and 1.47 points on 4B and 8B, respectively, compared with 1.04 and 0.41 points when removing only the contrasts. These results support using explicit modality-dependent responses alongside the original representations to guide influence matching and coefficient-direction selection. (2) Effect of CMC. The variant w/o CMC retains the full reference objective and replaces subspace selection and quadratic control with direct search over fixed-seed regional coefficient candidates sampled uniformly within the same radius, selecting the best candidate together with the uniform reference. The variant w/o directional coupling retains the fitted linear terms and the diagonal entries of the quadratic matrix HH, setting its off-diagonal entries to zero before solving for the coefficients. Its response basis, calibration evaluations, and proposal check remain fixed, isolating the contribution of interactions between control directions. Each direction adjusts multiple regional weights, so these mixed terms describe how one coordinated adjustment changes the effect of another. The unseen averages decrease by 0.52 and 0.24 points after removing CMC, compared with 0.39 and 0.12 points after removing only directional coupling. These results support coordinated coefficient selection that accounts for interactions among regional updates, helping preserve complementary capabilities on unseen tasks. (3) Overall comparison. Across both backbones, the full configuration achieves the highest mean in all four evaluation settings, while removing either module lowers both seen and unseen performance. Retaining MIM but replacing CMC with direct search reduces the seen averages by 1.47 and 0.86 points on 4B and 8B, respectively, suggesting that an informative fusion objective alone may be insufficient under a fixed evaluation budget. Conversely, the larger drops after removing MIM suggest that coefficient control depends on the objective and response measurements used to guide it. These comparisons distinguish two complementary roles. MIM specifies the multimodal influence to preserve through its reference objective, while CMC uses that objective to compare coordinated coefficient adjustments and select regional weights for the merged model.

VariantQwen3.5-4BQwen3-VL-8B
SeenUnseenSeenUnseen
w/o MIM76.12 ± 0.3967.04 ± 0.2776.03 ± 0.4365.88 ± 0.30
MIM: w/o modality contrasts77.46 ± 0.2667.87 ± 0.2177.09 ± 0.2966.29 ± 0.24
w/o CMC77.03 ± 0.4667.58 ± 0.3276.64 ± 0.4866.26 ± 0.34
CMC: w/o directional coupling77.79 ± 0.3067.71 ± 0.2077.02 ± 0.3666.38 ± 0.19
TAC-Merge (Full)78.50 ± 0.2468.10 ± 0.1877.50 ± 0.3166.50 ± 0.22

Table 3. Ablation study.

Hyperparameter Sensitivity. To examine the sensitivity of TAC-Merge, Figure 2 compares changes in seen and unseen average scores relative to the shared default (λ,R)=(1,0.25)(\lambda, R) = (1, 0.25). The scans vary λ∈{0.1,0.5,0.8,1,2,3,10}\lambda\in\{0.1, 0.5, 0.8, 1, 2, 3, 10\} and R∈{1/16,1/8,1/4,1/2,1}R \in\{1/16, 1/8, 1/4, 1/2, 1\} separately, holding the other parameter fixed. Each curve uses its corresponding default mean from the main tables as zero, allowing seen and unseen sensitivity to be compared on the same scale. The sensitivity curves show a broad intermediate plateau for λ\lambda, with small local fluctuations. Smaller weights retain expert-output agreement and the response subspace, while larger weights emphasize the experts’ relative influence patterns. Since these geometric targets are constructed from seen tasks, preserving their influence patterns and transferring to unseen inputs can favor different weights. For RR, smaller neighborhoods restrict coordinated coefficient adjustments, whereas larger neighborhoods expand both the feasible displacement and the configurations used to fit the response model. The large-radius reference points represent accepted proposals with a greater decline on unseen tasks, illustrating a possible difference between calibration-objective improvement and transfer to new task distributions. The acceptance check can also produce plateaus when multiple proposals return the uniform merge.

Figure 2. Sensitivity to (a) λ\lambda and (b) RR. Δ\Delta is the average-score change from (λ,R)=(1,0.25)(\lambda, R) = (1, 0.25) in score points. Settings are equally spaced, and dotted lines mark the default values.

Conclusion

This paper presents TAC-Merge, a framework for multimodal model merging that traces expert influence across depth and jointly coordinates regional updates. Multimodal influence mapping characterizes how each update affects representations at different depths and token positions, using Ricci curvature and expert predictions to define a shared merging objective. Coupled merge control models interactions among coefficient adjustments in a compact subspace and selects regional weights to synthesize one model shared across tasks. The selected coefficients yield one static model that serves all tasks with the base inference architecture. Across the two evaluated backbones, the merged model improves both seen and unseen averages over the compared merging methods, and the ablations support the contribution of modality-specific responses and interactions between control directions. Future work will explore more efficient influence estimation and scalable coordination when merging larger collections of task experts.

References

  1. [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. URL https://arxiv.org/abs/2511.21631.
  2. [2]Pietro Buzzega, Riccardo Salami, Angelo Porrello, and Simone Calderara. Rethinking layer-wise model merging through chain of merges, 2026. URL https://arxiv.org/abs/2508.21421.
  3. [3]Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, and Yang Liu. Model composition for multimodal large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11246–11262, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.606. URL https://aclanthology.org/2024.acl-long.606/.
  4. [4]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255, 2009. doi: 10.1109/CVPR.2009.5206848.DOI
  5. [5]Yiyang Du, Xiaoch en Wang, Chi Chen, Jiabo Ye, Yiru Wang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Zhifang Sui, Maosong Sun, and Yang Liu. Adamms: Model merging for heterogeneous multimodal large language models with unsupervised coefficient optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9413–9422, June 2025.DOI
  6. [7]Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. Task singular vectors: Reducing task interference in model merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18695–18705, June 2025.
  7. [8]Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 15466–15490. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/ghandeharioun24a.html.arxiv.org/abs/2401.06102
  8. [9]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6904–6913, 2017. URL https://arxiv.org/abs/1612.00837.
  9. [10]Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. Captioning images taken by people who are blind. In European Conference on Computer Vision (ECCV), 2020. URL https://arxiv.org/abs/2002.08565.
  10. [11]Daniel Han, Michael Han, and Unsloth team. Unsloth, 2023. URL https://github.com/unslothai/unsloth.
  11. [12]Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8340–8349, 2021. URL https://arxiv.org/abs/2006.16241.
  12. [13]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.
  13. [14]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj.
  14. [15]Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dateless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=FCnohUR6AnM.arxiv.org/abs/2212.09849
  15. [16]Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 787–798. Association for Computational Linguistics, 2014. doi: 10.3115/v1/D14-1086. URL https://aclanthology.org/D14-1086/.DOI
  16. [17]Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. IconQA: A new benchmark for abstract diagram understanding and visual language reasoning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URL https://iconqa.github.io/.arxiv.org/abs/2110.13214
  17. [18]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, volume 35, pp. 2507–2521, 2022. URL https://arxiv.org/abs/2209.09513.
  18. [19]Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Eleventh International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2209.14610.
  19. [20]Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11–20, 2016. URL https://arxiv.org/abs/1511.02283.
  20. [21]Michael S Matena and Colin Raffel. Merging models with fisher-weighted averaging. In S. Koyejо, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17703–17716. Curran Associates, Inc., 2022. doi: 10.52202/068431-1287. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/70c26937fbf3d4600b69a129031b66ec-Paper-Conference.pdf.DOI
  21. [22]Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: Visual question answering by reading text in images. In International Conference on Document Analysis and Recognition (ICDAR), 2019. URL https://ocr-vqa.github.io/.DOI
  22. [23]Jorge J. Moré and D. C. Sorensen. Computing a trust region step. SIAM Journal on Scientific and Statistical Computing, 4(3):553–572, 1983. doi: 10.1137/0904038. URL https://doi.org/10.1137/0904038.
  23. [24]The-Hai Nguyen, Dang Huu-Tien, Takeshi Suzuki, and Le-Minh Nguyen. Regmean++: Enhancing effectiveness and generalization of regression mean for model merging. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=H5LdsSCS9i.arxiv.org/abs/2508.03121
  24. [25]Yann Ollivier. Ricci curvature of Markov chains on metric spaces. Journal of Functional Analysis, 256(3):810–864, 2009. doi: 10.1016/j.jfa.2008.11.001. URL https://doi.org/10.1016/j.jfa.2008.11.001.
  25. [26]Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123(1):74–93, 2017. doi: 10.1007/s11263-016-0957-7.
  26. [27]Huaizhi Qu, Xinyu Zhao, Jie Peng, Kwonjoon Lee, Behzad Dariush, and Tianlong Chen. UQ–Merge: Uncertainty guided multimodal large language model merging. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 1401–1417, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.73. URL https://aclanthology.org/2025.findings-acl.73/.
  27. [30]Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision (ECCV), 2022. URL https://arxiv.org/abs/2206.01718.
  28. [32]Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task parameter sub-spaces. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2312.04339.
  29. [33]Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2Words: Automatic mobile UI summarization with multimodal learning. In Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, pp. 498–510, 2021. doi: 10.1145/3472749.3474765. URL https://arxiv.org/abs/2108.03353.
  30. [34]Yongxian Wei, Runxi Cheng, Weike Jin, Enneng Yang, Li Shen, LU HOU, SiNan Du, Chun Yuan, Xiaochun Cao, and Dacheng Tao. Optmerge: Unifying multimodal llm capabilities and modalities via model merging. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), International Conference on Learning Representations, volume 2026, pp. 61571–61595, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/64741c1e80cc3bbed205b1c2b040dfa8–Paper–Conference.pdf.arxiv.org/abs/2505.19892
  31. [35]Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Marcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965–23998. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/wortsman22a.html.
  32. [36]Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 7093–7115. Curran Associates, Inc., 2023. doi: 10.52202/075280-0310. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/1644c9af28ab7916874f16fd6228a9bcf-Paper-Conference.pdf.
  33. [37]Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In B. Kim, Y. Yue, S. Choudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Learning Representations, volume 2024, pp. 22743–22763, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/62868cc2fc1eb5cdf321d05b4b88510c-Paper-Conference.pdf.
  34. [38]Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 57755–57775. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/yu24p.html.
  35. [39]Fanhu Zheng, Haiyang Guo, Fei Zhu, Li Shen, and Hao Tang. Robustmerge: Parameter-efficient model merging for mllms with directional robustness. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.), Advances in Neural Information Processing Systems, volume 38, Main Conference, pp. 71071–71095. Curran Associates, Inc., 2025. doi: 10.52202/085713-2390. URL https://proceedings.neurips.cc/paper_files/paper/2025/file/67101f97dc23fcc10346091181fff6cb-Paper-Conference.pdf.DOI
  36. [41]Han-Chen Zhang, Zi-Hao Zhou, Mao-Lin Luo, Shimin Di, Min-Ling Zhang, and Tong Wei. DC-Merge: Improving model merging with directional consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22248–22258, June 2026b. URL https://openaccess.thecvf.com/content/CVPR2026/html/Zhang_DC-Merge_Improving_Model_Merging_with_Directional_Consistency_CVPR_2026_paper.html.arxiv.org/abs/2603.06242

Paper details

vlnc:W_xhqwesfe
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents