betaFeedback

WorksComputer Science

Bring Back Language to Actions: Restoring VLM capabilities in VLA Models via Parameter Merging

Abstract

Vision-Language-Action models (VLAs) are obtained by fine-tuning pre-trained Vision-Language Models (VLMs) on robot demonstration data, enabling action prediction from visual observations and language instructions. However, this fine-tuning induces catastrophic forgetting, substantially degrading the semantic and visual grounding capabilities of the original VLM. We address this by merging the parameters of a VLM and its VLA derivative without retraining either model. Concretely, we extract the task vector (the difference between VLA and VLM weights) and decompose it via Singular Value Decomposition. For each component, we learn a scalar mixing coefficient controlling how much of the VLA adaptation is retained, optimised to jointly preserve action prediction and vision-language capabilities. Experiments show that our merged model performs well on both axes simultaneously, consistently outperforming established merging baselines.

Introduction

The ability to reason about complex environments and to plan sequences of actions has long been a central challenge in artificial intelligence. Large Language Models (LLMs) and Vision-Language Models (VLMs) have made significant strides in this direction, combining reasoning, visual perception, and scene understanding into unified models capable of high-level instruction following, task decomposition, and environment description [8, 1] (Wei et al. 2023; Yao et al. 2023; Radford et al. 2021; Liu et al. 2023b) — capabilities that make them compelling building blocks for autonomous robotic agents. However, these models remain confined to perception and reasoning, and cannot directly produce the low-level motor commands required for physical interaction.

Vision-Language-Action models (VLAs) bridge this gap by fine-tuning pre-trained VLMs on robot demonstration data (O’Neill et al., 2025; Walke et al., 2023), enabling them to predict actions directly from visual observations and language instructions [6] (Kim et al. 2024). However, this fine-tuning comes at a significant cost: the action objective progressively overwrites the weight-space directions encoding visual-language understanding (Hancock et al., 2025), yielding a specialised model that excels at low-level control but can no longer reliably perform the high-level semantic reasoning needed for robust, generalist robotic agents.

This tension between action specialisation and semantic preservation is a fundamental obstacle to building robotic systems that are both capable of language modelling and fine-grained motor control (see Figure 1). Existing approaches to mitigating this degradation typically require architectural modifications or costly co-training on mixed datasets (Zawalski et al., 2024; Hancock et al., 2025), making them difficult to apply post-hoc to already fine-tuned VLAs. Model merging offers a compelling alternative (Li et al., 2025; Yadav et al., 2025): by combining the parameters of a VLM and a VLA, one can in principle recover a model that retains both action prediction and vision-language capabilities. However, naive uniform blending treats all parameters equally and fails to account for the heterogeneous roles played by different weight matrices across the two models, yielding suboptimal trade-offs between semantic and action performance.

Figure 1. (Left) Given the same visual scene, a VLM responds accurately to descriptive language prompts but produces no action output. Fine-tuning into a VLA enables action prediction from a task goal, but at the cost of language grounding — the VLA generates incoherent text when prompted with the same descriptive query. Our merged model recovers both capabilities, producing coherent language responses while retaining the ability to predict actions. (Right) The VLM achieves strong language performance but cannot act, while VLAs excel at manipulation but lose language capability. Our merged models are the only ones that achieve the best of both worlds, reaching the upper-right region of the trade-off space.

In this work, we investigate how to merge a VLM and a VLA such that the resulting model retains the action capabilities acquired during fine-tuning while recovering the vision-language understanding of the original VLM. To this end, we extract the task vector by subtracting the VLM weights from those of the VLA and decompose it using Singular Value Decomposition to expose the individual components of this adaptation. For each such direction, we learn a scalar coefficient that controls how much of the VLA adaptation is retained in the merged model, jointly preserving action prediction performance and the original vision-language capabilities of the VLM. We demonstrate that this approach consistently outperforms uniform blending and established merging baselines across both vision-language benchmarks and robotic manipulation tasks, on two distinct VLA architectures.

Our contributions are as follows:

  • A post-hoc, lightweight merging method that decomposes the VLM–VLA task vector via SVD and learns per-component coefficients to jointly recover language understanding and preserve action capabilities, requiring only the optimisation of a small set of scalar merging parameters while keeping both source models fully frozen.

  • An ablation study justifying fine-grained merging, showing per-component coefficients are necessary where global or per-layer scalars fail.

  • A structural analysis of the fine-tuning directions at multiple scales of the SVD decomposition, identifying which components of the adaptation must be preserved to reconcile vision-language understanding and action prediction in the merged model.

Together, these contributions offer a practical and substantially cheaper path toward robotic agents that combine reliable low-level control with the high-level semantic reasoning of their underlying VLM.

Related Work

Vision Language Action Models

Vision-Language-Action (VLA) models extend Vision-Language Models (VLMs) by fine-tuning them on large-scale robot demonstration data ((O’Neill et al., 2025)), enabling them to act as generalist policies that directly generate low-level actions ([6] (Ghosh et al. 2024; Kim et al. 2024)). Actions are produced either autoregressively ((Kim et al., 2024; Pertsch et al., 2025)) or via continuous decoding such as flow matching ([5]). However, this fine-tuning causes them to largely lose their original vision-language capabilities through catastrophic forgetting.

Some works sidestep this issue by keeping the VLM frozen and training only a lightweight action module ((Shukor et al., 2025; Wen et al., 2025)), but this limits the model’s ability to fully adapt its representations for embodied control. Others reintroduce text tasks during training either by training VLAs to generate both actions and text simultaneously ([4] (Zawalski et al. 2024)) or by aligning action representations with the language objective ((Hancock et al., 2025)). Both approaches require large-scale co-training to recover knowledge already present in the initial VLM. In contrast, our approach operates post-hoc on an already fine-tuned VLA, requiring no additional training of either source model.

Model Merging

Model merging has emerged as a cost-effective alternative to full retraining, enabling the combination of specialised models into a single capable model without additional data or compute. Model Soup (Wortsman et al. (2022)) averages the weights of models fine-tuned from the same checkpoint. SLERP (Jang et al. (2024)) interpolates along the spherical geodesic rather than linearly, better preserving parameter norms. TIES (Yadav et al. (2023)) resolves sign conflicts across multiple task vectors before averaging, requiring at least two independently fine-tuned models and thus not directly applicable to our setting. DARE (Yu et al. (2024)) randomly drops a fraction of task vector parameters before rescaling the remainder. Fisher Merging (Matena and Raffel (2022)) weights each parameter by its diagonal Fisher information, but requires computing and storing a matrix as large as the full model, making it costly at billion-parameter scale. AdaMerging (Yang et al. (2024)) learns task-wise or layer-wise merging weights but does not decompose the task vector into finer-grained components. All these methods operate with fixed or coarse-grained coefficients and none have been applied to reconciling a VLM with its VLA derivative. Closer to our work, decomposing task vectors via SVD has already been proposed to reduce interference between models trained on different tasks Gargiulo et al. [9]. We build on this insight and

extend it to the VLM–VLA setting, where the objective is to reconcile two fundamentally different capabilities — language understanding and action prediction — rather than reduce task interference.

Method

Problem Formulation

Let θVLM\theta_{\mathrm{VLM}} and θVLA\theta_{\mathrm{VLA}} denote the parameters of a pre-trained Vision-Language Model and a fine-tuned Vision-Language-Action model, respectively. Both models share a common backbone architecture. The VLM initially possesses strong language and visual grounding capabilities, while the VLA has been further trained to predict robotic actions, albeit at the cost of partial degradation of its original vision-language abilities. Our merging procedure is applied to all layers that are common to both architectures; VLA-specific components such as the action head are left unchanged and carried over directly into the merged model.

Our goal is to construct a merged model θmerged\theta_{\mathrm{merged}} that jointly retains the action prediction capabilities of the VLA and the vision-language understanding of the VLM (Figure 2a). We define it as a weighted combination of both source models, controlled by a set of mixing coefficients α\boldsymbol{\alpha}:

Overview of the VLM–VLA merging method

Figure 2. Overview of our VLM–VLA merging method. (a) The task vector Δ(l)\Delta^{(l)} is extracted by subtracting the VLM weights from the VLA weights for each layer ll. (b) Δ(l)\Delta^{(l)} is decomposed via Singular Value Decomposition, revealing the orthogonal directions of adaptation introduced during VLA fine-tuning. (c) A scalar mixing coefficient αk(l)\alpha_k^{(l)} is assigned to each singular component, controlling how much of each direction is retained in the merged model. The coefficients α\alpha are optimised by minimising a composite objective that jointly penalises degradation of language generation quality and action prediction accuracy with respect to both source models.

θmerged(l)=(1−α(l))θVLM(l)+α(l)θVLA(l),l∈{1,…,L}(1)\theta_{\mathrm{merged}}^{(l)} = (1-\alpha^{(l)})\theta_{\mathrm{VLM}}^{(l)}+\alpha^{(l)}\theta_{\mathrm{VLA}}^{(l)},\quad l \in\{1,\ldots,L\} \tag*{(1)}

where LL is the total number of layers and α(l)∈[0,1]\alpha^{(l)} \in[0,1] controls the contribution of each source model at layer ll. The central question is how to determine α\alpha — and at what granularity — such that the merged model best preserves both capabilities. In the following, we show that operating at the level of individual singular components of the per-layer task vector provides the necessary fine-grained control.

Task Vector and SVD Decomposition

For each layer ll, we define the task vector as the difference between the VLA and VLM weight matrices (Figure 2a):

Δ(l)=θVLA(l)−θVLM(l)(2)\Delta^{(l)} = \theta_{\mathrm{VLA}}^{(l)} - \theta_{\mathrm{VLM}}^{(l)} \tag*{(2)}

Substituting into Eq. (1), the merged model can be equivalently written as:

θmerged(l)=θVLM(l)+α(l)Δ(l)(3)\theta_{\mathrm{merged}}^{(l)} = \theta_{\mathrm{VLM}}^{(l)} + \alpha^{(l)} \Delta^{(l)} \tag*{(3)}

This formulation shows that merging amounts to scaling the task vector by α(l)\alpha^{(l)} at each layer. To obtain finer-grained control, we decompose Δ(l)\Delta^{(l)} into its individual directions via Singular Value Decomposition, as [9] (Figure 2b), allowing a separate coefficient to be assigned to each direction independently:

Δ(l)=∑k=1r(l)σk(l)uk(l)vk(l)T(4)\Delta^{(l)} = \sum_{k=1}^{r^{(l)}} \sigma_k^{(l)} u_k^{(l)} v_k^{(l)\mathsf{T}} \tag*{(4)}

where σ1(l)≥⋯≥σr(l)(l)\sigma_1^{(l)} \ge\cdots\ge\sigma_{r^{(l)}}^{(l)} are the singular values ranked by importance and r(l)r^{(l)} is the rank of Δ(l)\Delta^{(l)}. Each term σk(l)uk(l)vk(l)T\sigma_k^{(l)} u_k^{(l)} v_k^{(l)\mathsf{T}} captures a distinct direction of adaptation introduced during VLA fine-tuning.

Substituting the SVD decomposition of Eq. (4) into Eq. (3), the merged model becomes:

θmerged(l)=θVLM(l)+α(l)∑k=1r(l)σk(l)uk(l)vk(l)T(5)\theta_{\mathrm{merged}}^{(l)} = \theta_{\mathrm{VLM}}^{(l)} + \alpha^{(l)} \sum_{k=1}^{r^{(l)}} \sigma_k^{(l)} u_k^{(l)} v_k^{(l)\mathsf{T}} \tag*{(5)}

We now replace the single scalar α(l)\alpha^{(l)} by a per-component coefficient αk(l)∈[0,1]\alpha_k^{(l)} \in[0,1] (Figure 2c), one for each singular direction, yielding our final merging equation:

θmerged(l)=θVLM Prism(l)+∑k=1r(l)αk(l)σk(l)uk(l)vk(l)T(6)\theta_{\mathrm{merged}}^{(l)} = \theta_{\mathrm{VLM\ Prism}}^{(l)} + \sum_{k=1}^{r^{(l)}} \alpha_k^{(l)} \sigma_k^{(l)} u_k^{(l)} v_k^{(l)\mathsf{T}} \tag*{(6)}

When all αk(l)=0\alpha_k^{(l)} = 0, the layer reduces to the VLM. When all αk(l)=1\alpha_k^{(l)} = 1, it recovers the full VLA adaptation. The global scalar merging of Eq. (3) is recovered as the special case where all coefficients are equal to α\alpha. Importantly, both θVLM\theta_{\mathrm{VLM}} and the SVD decomposition of the task vector are frozen throughout — the only free parameters are the mixing coefficients α\alpha, which are optimised to simultaneously satisfy the language understanding objective of the VLM and the action prediction objective of the VLA, as described in § 3.3.

Training Objective

To optimise the mixing coefficients α\alpha, we define a composite objective that captures both language fidelity and action prediction accuracy, while penalising excessive deviation from either source model:

L(α)=λ1Ltext(α)+λ2⋅DKL(πα∥πVLM)+λ3Lact(α)+λ4⋅DKL(πα∥πVLA)(7)\mathcal{L}(\alpha) = \lambda_1 \mathcal{L}_{\mathrm{text}}(\alpha) + \lambda_2 \cdot D_{\mathrm{KL}}(\pi_\alpha\parallel\pi_{\mathrm{VLM}}) + \lambda_3 \mathcal{L}_{\mathrm{act}}(\alpha) + \lambda_4 \cdot D_{\mathrm{KL}}(\pi_\alpha\parallel\pi_{\mathrm{VLA}}) \tag*{(7)}

where λ1,λ2,λ3,λ4≥0\lambda_1,\lambda_2,\lambda_3,\lambda_4 \ge0 are tunable weighting hyperparameters (details can be found in Appendix § 6.§ 1). Ltext\mathcal{L}_{\mathrm{text}} measures language generation quality via negative log-likelihood over a language dataset Dtext\mathcal{D}_{\mathrm{text}}. Lact\mathcal{L}_{\mathrm{act}} measures action prediction quality over a robot trajectory dataset Dact\mathcal{D}_{\mathrm{act}}; its form depends on the VLA action head — for an autoregressive architecture it is the negative log-likelihood over action tokens, while for flow matching it is the conditional flow matching loss ((Lipman et al., 2023)). The KL divergence terms act as regularisers, penalising deviation from the pure VLM and VLA distributions respectively. Since θmerged(l)\theta_{\mathrm{merged}}^{(l)} is a differentiable function of α\alpha via Eq. (6), L\mathcal{L} is minimised directly with gradient descent using Adam ((Kingma and Ba, 2017)) (details can be found in Appendix § 6.§ 2).

Experiments

Experimental Setup

Models and datasets. We apply our merging method to PaliGemma-3B ([3]) as the VLM, paired with two VLA variants sharing the same architecture: π0-FAST\pi_{0\text{-FAST}} ((Pertsch et al., 2025)), an autoregressive VLA that tokenises actions, and π0.5\pi_{0.5} ([4]), a flow matching VLA that predicts continuous action chunks via iterative denoising. Both VLA variants are obtained by fine-tuning PaliGemma on robot demonstration data. All models are kept frozen — only the mixing coefficients α\alpha are optimised via Adam (Kingma and Ba, 2017) on two complementary datasets built from the LIBERO dataset available on LeRobot (Liu et al., (2023a); Cadene et al. [7]). For the action signal, we relabel existing expert demonstrations from the LIBERO dataset with the pure VLA’s own predicted action chunks, producing Dact\mathcal{D}_{\mathrm{act}}. For the language signal, we prompt PaliGemma to caption each observed frame, producing a paired dataset Dtext={(xi,ci)}\mathcal{D}_{\mathrm{text}} = \{(x_i, c_i)\} of observations and scene descriptions.

Evaluation protocol. We evaluate the merged model along three axes: vision-language capabilities, embodied reasoning, and action capabilities.

For vision-language capabilities, we assess captioning performance on the COCO dataset (Lin et al., 2015) using Rouge-L (Lin, 2004), METEOR (Banerjee and Lavie, [2]) and BERTScore (Zhang et al., 2020) metrics, and visual question answering performance on VQAv2 (Goyal et al., 2017).

For embodied reasoning, we evaluate accuracy on the EQA benchmark across three question categories — Object State, Spatial, and Attribute Recognition — which probe whether the model can answer language questions grounded in a robotic setting, bridging the gap between the general-domain vision-language evaluations above and the action benchmarks below.

For action capabilities, we evaluate task success rate on the LIBERO task suites — LIBERO-Spatial, LIBERO-Goal and LIBERO-Object (Liu et al., 2023a) — and report the average success rate across tasks for each suite (details can be found in Appendix § 7; additional results on the RoboCasa kitchen tasks are also reported in Appendix § 7.§ 1).

Baselines. We compare our merged model against a set of baselines. The VLM (PaliGemma (Beyer et al., [3])) and VLAs (π0-FAST\pi_{0\text{-}\mathrm{FAST}} and π0.5\pi_{0.5}) serve as upper bounds on language and action performance respectively. π0.5\pi_{0.5} (Black et al., [4]) is also a co-trained model that jointly optimises language and action objectives during training. We further include three established merging strategies that all use a fixed, non-learned merging coefficient: Model Soup (Wortsman et al., 2022), which linearly interpolates between VLM and VLA weights, and which we evaluate at four interpolation coefficients α∈{0.3,0.5,0.7,0.9}\alpha\in\{0.3, 0.5, 0.7, 0.9\} to characterise the full trade-

off curve it induces between language and action performance; DARE (Yu et al., 2024), which randomly drops a fraction of the task vector parameters before rescaling and adding the remainder to the VLM; and SLERP (Jang et al., 2024), which interpolates between the two weight vectors along the spherical geodesic rather than linearly. For the merged baselines, π0-FAST\pi_{0\text{-}\mathrm{FAST}} is used as the VLA backbone. We additionally compare against AdaMerging (Yang et al., 2024), which learns a single per-layer coefficient rather than per-component ones; we optimise it with the same objective (Eq. (7)) as our method; we use π0.5\pi_{0.5} as its VLA backbone for direct comparison with Oursπ0.5_{\pi_{0.5}}. All stochastic methods — DARE, AdaMerging, and our merged models — are averaged over 10 seeds, with standard deviations reported in Table 1.

ModelTypeCaptioning RougeLCaptioning MeteorCaptioning BERTScoreVQAv2 AccEQA Obj StateEQA SpatialEQA Attri RecogLibero SpatialLibero GoalLibero Object
PaliGemmaVLM33.623.688.953.844.421.732.40.00.00.0
π0Fast\pi_{0\mathrm{Fast}}VLA1.20.568.29.915.55.73.870.082.088.0
π0.5\pi_{0.5}VLA9.41.576.58.28.80088.090.096.0
Model Soupα=0.3_{\alpha=0.3}Merged14.611.482.732.422.215.122.70.00.00.0
Model Soupα=0.5_{\alpha=0.5}Merged8.35.376.421.719.98.111.60.00.00.0
Model Soupα=0.7_{\alpha=0.7}Merged5.05.173.915.319.94.44.70.00.00.0
Model Soupα=0.9_{\alpha=0.9}Merged2.11.170.19.015.55.82.65.013.017.0
DAREMerged10.4±0.910.4 \pm0.97.6±0.47.6 \pm0.478.3±0.378.3 \pm0.322.5±1.722.5 \pm1.79.2±1.89.2 \pm1.83.6±0.93.6 \pm0.94.1±0.84.1 \pm0.80.0±0.00.0 \pm0.00.0±0.00.0 \pm0.00.0±0.00.0 \pm0.0
SLERPMerged9.97.477.718.517.710.310.50.00.00.0
AdaMergingπ0.5_{\pi_{0.5}}Merged28.1±5.328.1 \pm5.323.1±5.823.1 \pm5.887.5±1.187.5 \pm1.138.2±4.938.2 \pm4.943.4±13.743.4 \pm13.714.0±3.814.0 \pm3.820.3±4.520.3 \pm4.56.2±3.26.2 \pm3.22.9±1.12.9 \pm1.13.7±2.93.7 \pm2.9
Oursπ0Fast_{\pi_{0\mathrm{Fast}}}Ours18.6±2.118.6 \pm2.114.1±1.114.1 \pm1.182.1±0.382.1 \pm0.328.9±1.928.9 \pm1.929.7±4.829.7 \pm4.814.1±5.114.1 \pm5.117.2±6.917.2 \pm6.961.2±2.861.2 \pm2.873.1±3.573.1 \pm3.579.8±3.179.8 \pm3.1
Oursπ0.5_{\pi_{0.5}}Ours29.5±1.829.5 \pm1.822.7±1.722.7 \pm1.787.8±0.587.8 \pm0.548.7±2.148.7 \pm2.132.3±5.632.3 \pm5.617.3±6.417.3 \pm6.419.5±6.719.5 \pm6.789.3±3.189.3 \pm3.188.1±4.288.1 \pm4.293.3±2.693.3 \pm2.6

Table 1. Vision-language, embodied reasoning, and robot manipulation performance of all methods. The pure VLM cannot act, and both VLAs lose language and reasoning ability despite strong action performance. Existing merging baselines largely collapse on manipulation, while our merged models recover strong performance on all three axes simultaneously, without retraining either source model.

Main Results

Vision-language capabilities. Table 1 reports captioning and VQA performance. The pure VLM sets the upper bound (33.68 ROUGE-L, 88.98 BERTScore, 53.87 VQAv2), while both pure VLAs degrade severely (π0-FAST\pi_{0\text{-}\mathrm{FAST}}: 1.25 ROUGE-L, 9.93 VQAv2; π0.5\pi_{0.5}: 9.40 ROUGE-L, 8.22 VQAv2 despite co-training). Model Soup, DARE and SLERP offer only modest, comparable recovery (8.32–10.11 ROUGE-L), confirming that a single coefficient applied uniformly across the task vector cannot restore language grounding. AdaMergingπ0.5_{\pi_{0.5}}, which shares our training objective but learns only a per-layer coefficient, recovers substantially more (28.1 ROUGE-L, 87.5 BERTScore, 38.2 VQAv2) — but still trails Oursπ0.5_{\pi_{0.5}} (29.5/87.8/48.7), and, as shown below, this comes at the cost of near-total collapse on action prediction. Our merged models close most of the remaining gap to the VLM: Oursπ0-Fast_{\pi_{0\text{-}\mathrm{Fast}}} reaches 18.6 ROUGE-L / 28.9 VQAv2, and Oursπ0.5_{\pi_{0.5}} reaches 29.5 ROUGE-L / 87.8 BERTScore / 48.7 VQAv2 — nearly matching the VLM. Recovery is stronger for π0.5\pi_{0.5} than π0-FAST\pi_{0\text{-}\mathrm{FAST}}, likely because the latter entangles language and action within a shared autoregressive head. Figure 3 illustrates this: π0.5\pi_{0.5} retains scene-relevant keywords (e.g. “Kitchen pan”, “Robot”) but loses the language modelling coherence of the VLM needed to form descriptions, whereas our merged model generates grammatically coherent captions that correctly identify objects and their spatial relationships. Additional examples can be found in Appendix § 8.§ 3.

Figure 3. Qualitative captioning examples. Given the same visual observation, the VLM produces coherent scene descriptions, the VLA (π0.5\pi_{0.5}) generates fragmented or incomplete text due to catastrophic forgetting, and Oursπ0.5_{\pi_{0.5}} merged model recovers meaningful captions while retaining action capabilities.

Embodied reasoning. The same pattern holds on EQA (Object State / Spatial / Attribute Recognition). The VLM leads (44.44/21.73/32.46), both VLAs degrade sharply, and fixed-coefficient baselines recover only partially and inconsistently (best: Model Soupα=0.3_{\alpha=0.3} at 22.22/15.19/22.75). AdaMergingπ0.5_{\pi_{0.5}} recovers markedly more (43.4/14.0/20.3), even edging out Oursπ0.5_{\pi_{0.5}} on Object State, indicating that a learned, task-aware objective is the primary driver of embodied-reasoning recovery, with SVD granularity offering more modest further gains on this axis. Our merged models nonetheless outperform all baselines overall: Oursπ0Fast_{\pi_{0\mathrm{Fast}}} reaches 29.7/14.1/17.2, and Oursπ0.5_{\pi_{0.5}} reaches 32.3/17.3/19.5, recovering a substantial share of the VLM’s embodied reasoning capabilities despite having no direct EQA supervision.

Robot manipulation performance. Table 1 also reports success rate on the LIBERO suites. The VLM achieves zero success everywhere, while both VLAs establish strong upper bounds (π0.5\pi_{0.5}: 88/90/96; π0−FAST\pi_{0\mathrm{-FAST}}: 70/82/88). DARE, SLERP and Model Soup (at α≤0.7\alpha\le0.7) collapse entirely to zero success, and Model Soupα=0.9_{\alpha=0.9} only recovers marginally (5/13/17) by sacrificing nearly all language gains — confirming that a fixed, uniform coefficient cannot preserve the action-relevant structure of the VLA. AdaMergingπ0.5_{\pi_{0.5}} collapses almost as severely (6.2/2.9/3.7) despite sharing our training objective, showing that a single per-layer coefficient lacks the resolution to preserve action-relevant directions — consistent with the granularity ablation of Section 4.3, where per-layer α\alpha cannot disentangle language- from action-relevant directions within Δ(l)\Delta^{(l)}. Our merged models retain strong performance on both variants: Oursπ0Fast_{\pi_{0\mathrm{Fast}}} reaches 61.2/73.1/79.8, and Oursπ0.5_{\pi_{0.5}} reaches 89.3/88.1/93.3, closely tracking its pure VLA counterpart while still achieving the language and reasoning gains above — a trade-off no baseline, including AdaMerging, offers.

Ablation study

We conduct three ablations to validate the design choices of our method: the granularity of the mixing coefficients, the contribution of each component of the training objective, and the sensitivity of the method to the rank of the SVD decomposition.

Granularity of mixing coefficients. To analyse the impact of per-component SVD coefficients, we compare two levels of granularity for α\alpha. Under per-layer α\alpha, a single coefficient α(l)∈[0,1]\alpha^{(l)} \in[0,1] is learned for each layer ll and applied uniformly to every singular component of that layer’s task vector Δ(l)\Delta^{(l)} — this is equivalent to the layer-wise AdaMerging protocol Yang et al. (2024), adapted to our setting. Under per-component α\alpha (ours), a separate coefficient αk(l)∈[0,1]\alpha_k^{(l)} \in[0,1] is learned for each individual singular component kk within each layer ll. Figure 4 shows the evolution of Ltext\mathcal{L}_{\mathrm{text}}, Lact\mathcal{L}_{\mathrm{act}}, DKL(πα∥πVLM)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLM}}) and DKL(πα∥πVLA)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLA}}) over optimisation steps for each condition. The results are unambiguous: per-component α\alpha consistently and substantially outperforms the per-layer scalar across all four metrics. On Ltext\mathcal{L}_{\mathrm{text}}, per-layer α\alpha plateaus around 2.0, while per-component α\alpha converges below 1.0. The gap is even more pronounced on action prediction: per-layer α\alpha converges to Lact≈0.7\mathcal{L}_{\mathrm{act}} \approx0.7 and DKL(πα∥πVLA)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLA}}) remains noisy around 0.06–0.08, whereas per-component α\alpha drives both metrics down to roughly a tenth of these values. A single coefficient per layer cannot disentangle the directions within Δ(l)\Delta^{(l)} that encode language degradation from those that encode action prediction, since both are entangled within the same weight matrix and a layer-wide α(l)\alpha^{(l)} must trade them off uniformly. These results confirm that the fine-grained control afforded by SVD decomposition is essential — coarser parameterisations lack the expressivity to reconcile both objectives.

Figure 4. Granularity of mixing coefficients. We compare two levels of granularity for α\alpha — per-layer and per-component (ours) — and report the evolution of Ltext\mathcal{L}_{\mathrm{text}}, Lact\mathcal{L}_{\mathrm{act}}, DKL(πα∥πVLM)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLM}}) and DKL(πα∥πVLA)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLA}}) over training. Per-component α\alpha converges to substantially lower values on every metric, confirming that fine-grained SVD-based control is essential to simultaneously preserve language and action capabilities.

Contribution of each loss component. To understand the role of each term in the training objective L(α)\mathcal{L}(\alpha), we train four variants of Oursπ0.5_{\pi_{0.5}} by removing one component at a time: without Ltext\mathcal{L}_{\mathrm{text}}, Lact\mathcal{L}_{\mathrm{act}}, DKL(πα∥πVLM)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLM}}), and DKL(πα∥πVLA)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLA}}). Results are reported in Table 2. Removing Lact\mathcal{L}_{\mathrm{act}} causes the merged model to collapse on action prediction entirely (0% success rate) while language is the highest of all variants (34.6 ROUGE-L), confirming that without an explicit action signal α\alpha drifts toward the VLM. Conversely, removing DKL(πα∥πVLM)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLM}}) severely degrades language quality (7.9 ROUGE-L) while leaving action performance almost intact (94.1%), showing that this KL term is the primary anchor preserving language fluency and that Ltext\mathcal{L}_{\mathrm{text}} alone is insufficient to fully constrain α\alpha toward the VLM. Removing Ltext\mathcal{L}_{\mathrm{text}} has a comparatively mild effect (21.9 ROUGE-L, 95% success rate), suggesting that DKL(πα∥πVLM)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLM}}) already provides most of the language-preserving signal on its own. Removing DKL(πα∥πVLA)D_{\mathrm{KL}}(\pi_{\alpha}\|\pi_{\mathrm{VLA}}) yields the second-best language score (31.2 ROUGE-L) but a substantial drop in success rate (71.5%), indicating that this term plays an important role in stabilising action prediction beyond what Lact\mathcal{L}_{\mathrm{act}} alone provides. Overall, the full objective achieves the best joint trade-off (29.5 ROUGE-L, 93.3% success rate), confirming that all four terms are complementary: the L\mathcal{L} terms provide the primary task signal while the KLKL terms regularise and stabilise the optimisation, with the VLM-side KL term proving especially critical for language retention.

ObjectiveROUGE-L ↑\uparrowSR ↑\uparrow
w/o Ltext\mathcal{L}_{\mathrm{text}}21.995.6
w/o Lact\mathcal{L}_{\mathrm{act}}34.60
w/o DKL(πα∣πVLM)D_{\mathrm{KL}}(\pi_{\alpha}|\pi_{\mathrm{VLM}})7.994.1
w/o DKL(πα∣πVLA)D_{\mathrm{KL}}(\pi_{\alpha}|\pi_{\mathrm{VLA}})31.271.5
Full objective (ours)29.593.3

Table 2. Contribution of each loss component. We ablate each term of the training objective and evaluate using ROUGE-L for language quality and Success Rate SR for action predictions. Best results in bold.

Effect of SVD rank. We study the sensitivity of the method to the rank rr used in the SVD decomposition by varying r∈{4,16,64,256,1024,full}r \in\{4, 16, 64, 256, 1024,\mathrm{full}\}. For a given rr, each layer’s task vector Δ(l)\Delta^{(l)} is truncated to its top-rr singular components — those associated with the rr largest singular values — and only the corresponding αk(l)\alpha_k^{(l)} are optimised, with the remaining components implicitly set to the VLM. Figure 5 reveals a striking asymmetry between the two objectives: Ltext\mathcal{L}_{\mathrm{text}} is largely insensitive to rank, while Lact\mathcal{L}_{\mathrm{act}} shows a clear separation, with higher rank leading to significantly better action preservation. The KL divergence terms follow the same pattern. This is confirmed by the downstream scatter plot: ROUGE-L is low at the extreme r=4r = 4 but already saturates around 30 from r=16r = 16 onward, while mean LIBERO success rate stays at 0% for r≤16r \le16, jumps to 64% at r=256r = 256, and exceeds 90% only from r=1024r = 1024 onward. Notably, r=1024r = 1024 requires only 269,443 learnable parameters compared to 381,605 at full rank — both roughly four orders of magnitude smaller than the finetuning of the ∼3\sim3B backbone, confirming that near-optimal performance is achieved with an even smaller fraction of the already lightweight parameter budget. This indicates that language recovery only requires a small set of dominant directions of the task vector, whereas action preservation requires a much larger, near complete, set of singular directions — confirming that the VLA adaptation is diffuse in weight space while language-relevant directions are concentrated in a handful of dominant components. More broadly, this asymmetry explains why uniform merging fails to recover action performance while partially recovering language.

Figure 5. Effect of SVD rank. (Left) Training losses for varying rank r∈{4,16,64,256,1024,full}r \in\{4, 16, 64, 256, 1024,\mathrm{full}\}: Ltext\mathcal{L}_{\mathrm{text}} is largely insensitive to rank, while Lact\mathcal{L}_{\mathrm{act}} improves markedly with higher rank, indicating that the VLA adaptation is diffuse across many singular components. (Right) Downstream evaluation confirms this asymmetry: ROUGE-L saturates from r=16r = 16, while LIBERO success rate requires r≥1024r \ge1024 to exceed 90% — language recovery needs far fewer singular directions than action preservation.

More analysis on the α\alpha distribution can be found in Appendix § 8.§ 2.

Conclusion

In this paper, we present a post-hoc method for merging a VLM and a VLA by decomposing the task vector via SVD and learning per-component mixing coefficients through gradient descent. We apply this method to two distinct VLA architectures, an autoregressive and a flow matching model, demonstrating that it generalises across different action decoding strategies. Both source models remain frozen throughout, and only the mixing coefficients are optimised, making the approach substantially cheaper than retraining or co-training a VLA from scratch and broadly applicable to already fine-tuned VLAs. Our approach recovers a model that retains the action prediction capabilities of the VLA while restoring the vision-language understanding of the original VLM, a balance that uniform interpolation fails to achieve. Through extensive ablations, we show that this fine-grained control over the merging process is essential, and our analysis further reveals that language and action capabilities are encoded at different scales in weight space, offering new insight into the nature of catastrophic forgetting in VLA fine-tuning. Future work could leverage the language and reasoning capabilities restored by our method to equip VLAs with high-level planning, enabling them to decompose and reason about long-horizon tasks rather than merely reacting to immediate instructions. Code and merged model checkpoints will be released upon acceptance.

Limitations

Our method assumes the VLM and VLA share a common backbone, so it does not directly apply to mismatched architectures or tokenizers. It also still requires a training loop to optimise α\alpha, albeit a much cheaper one than full fine-tuning. Our action and language datasets are both derived from a single benchmark (LIBERO) and from the source models themselves, so performance on out-of-distribution tasks, embodiments, or real-world deployments remains untested. Finally, our experiments are limited to VLMs and VLAs at the 3–4B parameter scale, and whether our findings hold at larger scales remains untested.

Ethical Considerations

This work aims to mitigate catastrophic forgetting in robot learning models, helping robotic agents retain language-based communication alongside low-level control rather than sacrificing one for the other. All experiments are conducted in simulation using publicly available models and datasets; no human subjects or physical robot deployments are involved. As with any method that enhances action capabilities, thorough safety validation is recommended before real-world deployment.

Acknowledgments

Experiments presented in this paper were carried out using the HPC resources of IDRIS under the allocations 2026-[A0201016980] and 2026-[AD011015093R2] made by GENCI. This work was supported by the European Commission’s Horizon Europe Framework Programme under grant No 101070381 (PILLAR-robots), by RODEO Project (ANR-24-CE23-5886), by PEPR Sharp (ANR-23-PEIA-0008, ANR, FRANCE 2030) and by Cluster PostGenAI@Paris (ANR-23-IACL-0007, FRANCE 2030).

Implementation Details

Hyperparameters

Table Table 3 reports the full set of hyperparameters used to train the mixing coefficients α\alpha for our main results, separately for Oursπ0Fast_{\pi_0\mathit{Fast}} and Oursπ0.5_{\pi_{0.5}}.

HyperparameterOursπ0Fast_{\pi_0\mathit{Fast}}Oursπ0.5_{\pi_{0.5}}
OptimiserAdamAdam
Initial learning rate0.010.01
Learning rate schedulecosine decaycosine decay
Total training steps20,00020,000
λ1\lambda_1 (Ltext\mathcal{L}_{\mathrm{text}})11
λ2\lambda_2 (Lact\mathcal{L}_{\mathrm{act}})15
λ3\lambda_3 (KLVLM\mathrm{KL}_{\mathrm{VLM}})0.010.01
λ4\lambda_4 (KLVLA\mathrm{KL}_{\mathrm{VLA}})0.810
α\alpha initialisationUniform(0, 1)Uniform(0, 1)
SVD power iterations44
Batch size22
Number of GPUs11

Table 3. Hyperparameters used to optimise α\alpha for each VLA variant. λ1\lambda_1, λ2\lambda_2 weight Ltext\mathcal{L}_{\mathrm{text}} and Lact\mathcal{L}_{\mathrm{act}} respectively; λ3\lambda_3, λ4\lambda_4 weight the KL regularisation terms toward the VLM and VLA respectively.

KL Regularisation Terms

Language side. For the VLM side, πVLM\pi_{\mathrm{VLM}} and πα\pi_{\alpha} denote the token-level output distributions of the pure VLM and the merged model, respectively, computed over the language vocabulary at each generated position given the same visual observation and prompt. For π0-FAST\pi_{0\text{-}\mathrm{FAST}}, whose vocabulary jointly contains both language and action tokens, this distribution is restricted to the language sub-vocabulary, masking out action-token logits before computing the divergence. DKL(πα∥πVLM)D_{\mathrm{KL}}(\pi_{\alpha}\Vert\pi_{\mathrm{VLM}}) is the standard categorical KL divergence between these two (vocabulary-restricted) token distributions, averaged over sequence positions, and is well-defined for both VLA variants since the language head is shared and discrete in both π0-FAST\pi_{0\text{-}\mathrm{FAST}} and π0.5\pi_{0.5}.

Action side — autoregressive VLA. For π0-FAST\pi_{0\text{-}\mathrm{FAST}}, actions are represented as discrete tokens drawn from a dedicated action sub-vocabulary, and πVLA\pi_{\mathrm{VLA}}, πα\pi_{\alpha} denote the corresponding token-level distributions restricted to this action sub-vocabulary, with language-token logits masked out. DKL(πα∥πVLA)D_{\mathrm{KL}}(\pi_{\alpha}\Vert\pi_{\mathrm{VLA}}) is therefore also a standard categorical KL divergence, computed analogously to the language term above but over the disjoint action sub-vocabulary, ensuring that each KL term reflects only the corresponding modality.

Action side — flow matching VLA. For π0.5\pi_{0.5}, actions are continuous and generated via iterative denoising, so no closed-form categorical distribution exists over the action space. Following prior work on distribution matching for diffusion- and flow-based visuomotor policies (Jia et al., 2024; Ping et al., 2026), we approximate πVLA\pi_{\mathrm{VLA}} and πα\pi_{\alpha} as Gaussian distributions over the predicted velocity field. Concretely, for a given observation we sample N=16N=16 independent noise draws x0(1),…,x0(N)∼N(0,I)x_0^{(1)},\ldots,x_0^{(N)}\sim\mathcal{N}(0,I), form the corresponding interpolated actions xt(i)=tx1+(1−t)x0(i)x_t^{(i)}=t x_1+(1-t)x_0^{(i)}, and compute the predicted velocities vθVLA(i)v_{\theta_{\mathrm{VLA}}}^{(i)} and vθmerged(i)v_{\theta_{\mathrm{merged}}}^{(i)} under the pure VLA and the merged model respectively. We then estimate the empirical mean and variance of each set of velocity predictions,

μθ=1N∑i=1Nvθ(i),σθ2=1N∑i=1N(vθ(i)−μθ)2,θ∈{θVLA,θmerged},(8)\mu_\theta=\frac{1}{N}\sum_{i=1}^{N}v_\theta^{(i)},\qquad\sigma_\theta^2=\frac{1}{N}\sum_{i=1}^{N}\left(v_\theta^{(i)}-\mu_\theta\right)^2,\qquad\theta\in\{\theta_{\mathrm{VLA}},\theta_{\mathrm{merged}}\}, \tag*{(8)}

and approximate πVLA≈N(μθVLA,σθVLA2)\pi_{\mathrm{VLA}} \approx\mathcal{N}(\mu_{\theta_{\mathrm{VLA}}}, \sigma_{\theta_{\mathrm{VLA}}}^{2}), πα≈N(μθmerged,σθmerged2)\pi_{\alpha} \approx\mathcal{N}(\mu_{\theta_{\mathrm{merged}}}, \sigma_{\theta_{\mathrm{merged}}}^{2}), for which the KL divergence admits the standard closed form:

DKL(πα∥πVLA)=12(σθmerged2σθVLA2+(μθVLA−μθmerged)2σθVLA2−1+log⁡σθVLA2σθmerged2).(9)D_{\mathrm{KL}}(\pi_{\alpha}\Vert\pi_{\mathrm{VLA}}) = \frac{1}{2}\left(\frac{\sigma_{\theta_{\mathrm{merged}}}^{2}}{\sigma_{\theta_{\mathrm{VLA}}}^{2}} + \frac{(\mu_{\theta_{\mathrm{VLA}}} - \mu_{\theta_{\mathrm{merged}}})^{2}}{\sigma_{\theta_{\mathrm{VLA}}}^{2}} - 1 + \log\frac{\sigma_{\theta_{\mathrm{VLA}}}^{2}}{\sigma_{\theta_{\mathrm{merged}}}^{2}}\right). \tag*{(9)}

This Gaussian approximation is computed independently per action dimension and averaged. We note that this is an approximation of the true (intractable) distribution induced by the iterative denoising process, justified by the fact that the velocity field at a fixed timestep is approximately unimodal in practice.

Dataset Collection

Action dataset Dact\mathcal{D}_{\mathrm{act}}. For each VLA variant, we construct Dact\mathcal{D}_{\mathrm{act}} by relabeling existing expert demonstrations from the LIBERO dataset available on LeRobot ([7] (Liu et al. 2023a)). Concretely, we take the visual observations, proprioceptive states, and task instructions from the expert trajectories and query the pure VLA to obtain its predicted action chunks for each observation, storing the resulting pairs. This collection is performed once per VLA variant and reused throughout training and ablation, and yields a total of 5,000 examples.

Language dataset Dtext\mathcal{D}_{\mathrm{text}}. For each visual observation collected above, we additionally query the pure VLM with a captioning prompt to obtain a corresponding scene description, producing paired examples (xi,ci)∈Dtext(x_i,c_i) \in\mathcal{D}_{\mathrm{text}}, also totalling 5,000 examples. Both Dact\mathcal{D}_{\mathrm{act}} and Dtext\mathcal{D}_{\mathrm{text}} are split into disjoint train and test partitions, and all reported metrics are computed on test examples.

Training Cost

Training the mixing coefficients α\alpha for our main results takes approximately 3 hours on a single NVIDIA H100 GPU for 20,000 steps. Both source models (θVLM\theta_{\mathrm{VLM}} and θVLA\theta_{\mathrm{VLA}}) remain frozen throughout, and only α\alpha is updated. At our chosen practical rank, this amounts to a total of 381,587 learnable parameters for Oursπ0Fast_{\pi_{0}\mathrm{Fast}} and 381,605 for Oursπ0.5_{\pi_{0.5}} across all layers weight matrices — in both cases roughly four orders of magnitude smaller than the ∼\sim3B frozen parameters of the VLA backbone alone. This makes our approach substantially cheaper than retraining or co-training a VLA from scratch.

Environment and Task Definitions

LIBERO. LIBERO ((Liu et al., 2023a)) is a simulation benchmark for lifelong robot manipulation built on the Robosuite simulator, comprising several task suites each containing 10 tasks with 50 expert demonstrations per task. We evaluate on three of its suites: LIBERO-Spatial, where the robot must place a bowl on a plate among visually identical bowls that differ only in their spatial location relative to other objects, testing spatial reasoning under a fixed set of objects; LIBERO-Object, where the robot must pick and place a different, unique target object into a basket across tasks while the layout remains fixed, testing object-centric generalisation; and LIBERO-Goal, where both the objects and their spatial layout are held fixed while the task goal varies, testing the transfer of procedural, goal-directed behaviour. Each suite is designed to disentangle a specific axis of generalisation (spatial, object, or goal knowledge), and observations consist of third-person and wrist-camera RGB images.

Robocasa. RoboCasa ((Nasiriany et al., 2026)) is a large-scale simulation benchmark for household robot manipulation in photorealistic kitchen environments, comprising a set of atomic skills (e.g. pick-and-place, opening/closing doors and drawers, turning knobs, pressing buttons) together with longer-horizon composite tasks. We evaluate on three atomic kitchen tasks: Open Fridge, requiring the robot to open a refrigerator door; Setup Cup, requiring the robot to pick up a cup and place it in a target location; and Navigate Kitchen, requiring the robot to move within the kitchen environment to reach a target location before completing a manipulation sub-goal. Compared to LIBERO, RoboCasa tasks are evaluated across a larger number of randomised scene layouts and object instances, making the benchmark substantially more challenging: even strong VLA baselines such as π0.5\pi_{0.5} average only around 39% success rate across these tasks in our setting, reflecting the benchmark’s greater diversity in scene configurations and the longer effective horizon of some tasks (e.g. Navigate Kitchen).

Additional Results on RoboCasa

Table 4 reports task success rate on three RoboCasa kitchen tasks — Open Fridge, Setup Cup, and Navigate Kitchen — complementing the LIBERO results presented in the main paper. The overall trends are consistent with those observed on LIBERO: the pure VLM achieves zero success rate everywhere, both pure VLAs establish strong baselines, and all fixed-coefficient merging methods (Model Soup, DARE, SLERP) collapse entirely to zero success rate across all tasks. Our merged models retain meaningful action performance on both VLA variants: Oursπ0 Fast_{\pi_{0\,Fast}} reaches 56/61/78 (versus 78/68/89 for the pure π0\pi_0-FAST), and Oursπ0.5_{\pi_{0.5}} reaches 65/49/21 (versus 71/58/25 for the pure π0.5\pi_{0.5}). The performance drop relative to the pure VLAs is more pronounced on RoboCasa than on LIBERO, reflecting the greater difficulty of this benchmark: even the pure VLAs average only around 50–78% across tasks, and Navigate Kitchen proves particularly challenging for π0.5\pi_{0.5} (25% for the pure VLA, 21% after merging). Despite this increased difficulty, our method remains the only merging approach that retains non-trivial action performance while simultaneously recovering the vision-language and embodied reasoning capabilities reported in the main paper.

ModelTypeOpen FridgeSetup CupNavigate Kitchen
PaliGemmaVLM000
π0 Fast\pi_{0\,Fast}VLA786889
π0.5\pi_{0.5}715825
Model Soupα=0.3_{\alpha=0.3}Merged000
Model Soupα=0.5_{\alpha=0.5}000
Model Soupα=0.7_{\alpha=0.7}000
Model Soupα=0.9_{\alpha=0.9}000
DARE000
SLERP000
Oursπ0 Fast_{\pi_{0\,Fast}}566178
Oursπ0.5_{\pi_{0.5}}654921

Table 4. Task success rate (%) on three RoboCasa kitchen tasks. All fixed-coefficient merging baselines collapse to zero, while our merged models retain meaningful performance on both VLA variants.

Analysis of the Merged Model

Models Similarity

To better understand what the learned α\alpha coefficients produce in practice, we analyse the per-layer alignment of the merged model with respect to both source models, for both VLA variants. For each layer ll, we compute three cosine similarities: sim⁡(θVLM(l),θVLA(l))\operatorname{sim}(\theta_{\mathrm{VLM}}^{(l)}, \theta_{\mathrm{VLA}}^{(l)}) — the baseline distance between the two source models — sim⁡(θmerged(l),θVLM(l))\operatorname{sim}(\theta_{\mathrm{merged}}^{(l)}, \theta_{\mathrm{VLM}}^{(l)}), and sim⁡(θmerged(l),θVLA(l))\operatorname{sim}(\theta_{\mathrm{merged}}^{(l)}, \theta_{\mathrm{VLA}}^{(l)}), where θmerged\theta_{\mathrm{merged}} denotes the merged model for the corresponding VLA variant. These are computed directly on the reconstructed merged weight matrices using the learned α\alpha.

Figure 6 reports these three curves across all 602 layers for both VLA variants. Several observations emerge, most of which hold consistently across both panels. First, the baseline similarity between VLM and VLA (blue) is consistently the lowest in both cases, confirming that VLA fine-tuning significantly displaces the weights from their original VLM values; this displacement grows with layer depth, dropping from ∼0.80–0.85 in the earliest layers to ∼0.65–0.70 in the deepest ones, indicating that fine-tuning increasingly diverges from the VLM as depth increases, for both the autoregressive and flow matching VLA. Second, and most importantly, the merged model remains significantly closer to both source models than the VLM and VLA are to each other in both cases — orange and green curves lie well above the blue baseline throughout. This indicates that the merged model does not simply interpolate between two distant models but finds a position in weight space that is simultaneously close to both sources, with no significant drop in alignment relative to either, regardless of the underlying VLA architecture. Third, the merged model is consistently and substantially closer to the VLA than to the VLM across all layers in both panels (green above orange throughout), indicating that the learned α\alpha coefficients globally preserve a large share of the VLA adaptation while still maintaining a clear margin of proximity to the VLM. The main difference between the two panels is the magnitude of this gap: for Oursπ0.5_{\pi_{0.5}} (bottom), the green curve sits substantially higher (∼\sim0.97–0.98) and more tightly clustered than for Oursπ0FAST_{\pi_{0\mathrm{FAST}}} (top, ∼\sim0.88–0.93), while the orange curves are comparable in both cases (∼\sim0.83–0.93). This is consistent with the merged π0.5\pi_{0.5} model retaining a stronger and more uniform alignment with the VLA across layers, in line with the higher action success rates observed for Oursπ0.5_{\pi_{0.5}} in Table 1. Both the orange and blue curves exhibit a regular sawtooth pattern in both panels, consistent with the alternation between different weight types (attention and feed-forward) within each transformer block. Additionally, the orange curve partially recovers around layer index 350–400 before declining again toward the end of the network in both cases. This structured behaviour emerges directly from the optimisation of α\alpha, confirming that the learned coefficients successfully navigate the trade-off between both source models rather than collapsing toward either one, irrespective of the VLA’s action decoding strategy.

Figure 6. Per-layer similarity to source models. (Top) Oursπ0FAST_{\pi_{0\mathrm{FAST}}}. (Bottom) Oursπ0.5_{\pi_{0.5}}. For each of the 602 weight matrices, we compute the cosine similarity between the VLM and VLA weights (blue), between the merged model and the VLM (orange), and between the merged model and the VLA (green). In both cases, the merged model is consistently closer to the VLA than to the VLM across all layers, while remaining well above the VLM–VLA baseline throughout.

Merging coefficients analysis

To examine how the learned coefficients are distributed across the singular spectrum, we average αk(l)\alpha_k^{(l)} across all layers at each rank index kk (where k=0k = 0 corresponds to the largest singular value σ0(l)\sigma_0^{(l)} within each layer), for both VLA variants.

Figure 7 shows the same overall pattern for both VLA variants: mean α\alpha is near 0 for the most dominant directions of the task vector and rises monotonically toward the tail of the spectrum. For Oursπ0.5_{\pi_{0.5}} (top), mean α\alpha is exactly 0 for k<200k < 200 and rises to a peak around 0.8 near k≈1050k \approx1050 before declining slightly toward the tail. For Oursπ0Fast_{\pi_{0Fast}} (bottom), the trend is similar, with mean α\alpha starting close to, though not exactly at, 0 (∼0.1\sim0.1 at k=0k = 0) and plateauing around 0.65–0.7 from k≈600k \approx600 onward. The consistency of this monotonic pattern across both VLA architectures indicates that the low-rank/high-rank asymmetry exists between language and action recovery. This pattern directly complements the rank ablation of § 4.3: when the model is restricted to a low rank, only the dominant directions are available, leaving the optimisation with little room to inject VLA-relevant adaptation, which is consistent with action prediction degrading sharply at low rank while language recovers comparatively quickly, for both VLA variants. Conversely, the gradual rise of mean α\alpha toward the tail of the spectrum in both panels shows that the directions necessary for action prediction are spread broadly across the weaker components, consistent with the diffuse nature of the VLA adaptation observed throughout this work.

Figure 7. Figure 7: Mean α\alpha vs. rank index. (Top) Oursπ0.5_{\pi_{0.5}}. (Bottom) Oursπ0Fast_{\pi_{0Fast}}. For each rank index kk, we average αk(l)\alpha_k^{(l)} across all 602 layers. Both VLA variants show the same overall pattern: mean α\alpha is near 0 for the most dominant directions and rises monotonically toward the tail of the spectrum, confirming that this asymmetry is not specific to a single action decoding strategy.

Additional Qualitative Examples

Figure 8 shows six additional captioning examples spanning diverse scenes, from tabletop manipulation to kitchen environments. Across all examples, the pure VLA output degrades into either a single repeated keyword (e.g. “Wood Wood Wood ....”, “Robot Bot Bot ...”), a short fragment stripped of grammatical structure (e.g. “Colored”, “@Water @@”), or garbled non-linguistic tokens (e.g. “p̈w̄Γ f tools”), consistent with the catastrophic forgetting of language modelling capabilities discussed in § 4.2. In contrast, our merged model consistently recovers full, grammatically coherent sentences that correctly identify the objects and setting present in the scene (e.g. “A collection of tools including screwdrivers, wrenches, and bolts.”, “A 3D rendering of a kitchen scene featuring a stovetop and oven, including a sink.”). We note that our merged model’s captions occasionally diverge stylistically from the pure VLM’s — favouring phrases such as “3D rendering of...” — but remain semantically accurate and fluent, indicating that language coherence is recovered even when the exact phrasing differs from the original VLM. These qualitative observations are consistent with the quantitative captioning scores reported in Table 1, where the pure VLA collapses on ROUGE-L, METEOR and BERTScore while our merged model recovers most of the pure VLM’s performance.

Qualitative captioning examples

Figure 8. Qualitative captioning examples. Given the same visual observation, the VLM produces coherent scene descriptions, the VLA (π0.5\pi_{0.5}) generates fragmented or incomplete text due to catastrophic forgetting, and Oursπ0.5_{\pi_{0.5}} merged model recovers meaningful captions while retaining action capabilities.

References

  1. [1]Mohamed Salim Aissi, Clémence Grislain, Clément Romac, Laure Soulier, Mohamed Chetouani, Olivier Sigaud, and Nicolas Thome. 2026. PRISM: Perception reasoning interleaved for sequential decision making. In Forty-third International Conference on Machine Learning.arxiv.org/abs/2605.05407
  2. [2]Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  3. [3]Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, and 16 others. 2024. Paligemma: A versatile 3b vlm for transfer. Preprint, arXiv:2407.07726.arxiv.org/abs/2407.07726
  4. [4]Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Divya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, and 16 others. 2025. π₀.₅: a vision-language-action model with open-world generalization. Preprint, arXiv:2504.16054.arxiv.org/abs/2504.16054
  5. [5]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, and 5 others. 2024. π₀: A vision-language-action flow model for general robot control. Preprint, arXiv:2410.24164.arxiv.org/abs/2410.24164
  6. [6]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, and 35 others. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. Preprint, arXiv:2307.15818.arxiv.org/abs/2307.15818
  7. [7]Remi Cadene, Simon Alibert, Francesco Capuano, Michel Aractingi, Adil Zouitine, Pepijn Kooijmans, Jade Choghari, Martino Russi, Caroline Pascal, Steven Palma, Mustafa Shukor, Jess Moss, Alexander Soare, Dana Aubakirova, Quentin Lhoest, Quentin Gallouedec, and Thomas Wolf. 2026. Lerobot: An open-source library for end-to-end robot learning. Preprint, arXiv:2602.22818.arxiv.org/abs/2602.22818
  8. [8]Yinlong Dai, Benjamin A. Christie, Daniel J. Evans, Dylan P. Losey, and Simon Stepputtis. 2026. Language movement primitives: Grounding language models in robot motion. Preprint, arXiv:2602.02839.arxiv.org/abs/2602.02839
  9. [9]Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. 2025. Task singular vectors: Reducing task interference in model merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18695–18705.arxiv.org/abs/2412.00081

Paper details

vlnc:W_v64whqcm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents