WorksComputer Science
Bring Back Language to Actions: Restoring VLM capabilities in VLA Models via Parameter Merging
Abstract
Vision-Language-Action models (VLAs) are obtained by fine-tuning pre-trained Vision-Language Models (VLMs) on robot demonstration data, enabling action prediction from visual observations and language instructions. However, this fine-tuning induces catastrophic forgetting, substantially degrading the semantic and visual grounding capabilities of the original VLM. We address this by merging the parameters of a VLM and its VLA derivative without retraining either model. Concretely, we extract the task vector (the difference between VLA and VLM weights) and decompose it via Singular Value Decomposition. For each component, we learn a scalar mixing coefficient controlling how much of the VLA adaptation is retained, optimised to jointly preserve action prediction and vision-language capabilities. Experiments show that our merged model performs well on both axes simultaneously, consistently outperforming established merging baselines.
Introduction
The ability to reason about complex environments and to plan sequences of actions has long been a central challenge in artificial intelligence. Large Language Models (LLMs) and Vision-Language Models (VLMs) have made significant strides in this direction, combining reasoning, visual perception, and scene understanding into unified models capable of high-level instruction following, task decomposition, and environment description [8, 1] (Wei et al. 2023; Yao et al. 2023; Radford et al. 2021; Liu et al. 2023b) — capabilities that make them compelling building blocks for autonomous robotic agents. However, these models remain confined to perception and reasoning, and cannot directly produce the low-level motor commands required for physical interaction.
Vision-Language-Action models (VLAs) bridge this gap by fine-tuning pre-trained VLMs on robot demonstration data (O’Neill et al., 2025; Walke et al., 2023), enabling them to predict actions directly from visual observations and language instructions [6] (Kim et al. 2024). However, this fine-tuning comes at a significant cost: the action objective progressively overwrites the weight-space directions encoding visual-language understanding (Hancock et al., 2025), yielding a specialised model that excels at low-level control but can no longer reliably perform the high-level semantic reasoning needed for robust, generalist robotic agents.
This tension between action specialisation and semantic preservation is a fundamental obstacle to building robotic systems that are both capable of language modelling and fine-grained motor control (see Figure 1). Existing approaches to mitigating this degradation typically require architectural modifications or costly co-training on mixed datasets (Zawalski et al., 2024; Hancock et al., 2025), making them difficult to apply post-hoc to already fine-tuned VLAs. Model merging offers a compelling alternative (Li et al., 2025; Yadav et al., 2025): by combining the parameters of a VLM and a VLA, one can in principle recover a model that retains both action prediction and vision-language capabilities. However, naive uniform blending treats all parameters equally and fails to account for the heterogeneous roles played by different weight matrices across the two models, yielding suboptimal trade-offs between semantic and action performance.
Figure 1. (Left) Given the same visual scene, a VLM responds accurately to descriptive language prompts but produces no action output. Fine-tuning into a VLA enables action prediction from a task goal, but at the cost of language grounding — the VLA generates incoherent text when prompted with the same descriptive query. Our merged model recovers both capabilities, producing coherent language responses while retaining the ability to predict actions. (Right) The VLM achieves strong language performance but cannot act, while VLAs excel at manipulation but lose language capability. Our merged models are the only ones that achieve the best of both worlds, reaching the upper-right region of the trade-off space.
In this work, we investigate how to merge a VLM and a VLA such that the resulting model retains the action capabilities acquired during fine-tuning while recovering the vision-language understanding of the original VLM. To this end, we extract the task vector by subtracting the VLM weights from those of the VLA and decompose it using Singular Value Decomposition to expose the individual components of this adaptation. For each such direction, we learn a scalar coefficient that controls how much of the VLA adaptation is retained in the merged model, jointly preserving action prediction performance and the original vision-language capabilities of the VLM. We demonstrate that this approach consistently outperforms uniform blending and established merging baselines across both vision-language benchmarks and robotic manipulation tasks, on two distinct VLA architectures.
Our contributions are as follows:
A post-hoc, lightweight merging method that decomposes the VLM–VLA task vector via SVD and learns per-component coefficients to jointly recover language understanding and preserve action capabilities, requiring only the optimisation of a small set of scalar merging parameters while keeping both source models fully frozen.
An ablation study justifying fine-grained merging, showing per-component coefficients are necessary where global or per-layer scalars fail.
A structural analysis of the fine-tuning directions at multiple scales of the SVD decomposition, identifying which components of the adaptation must be preserved to reconcile vision-language understanding and action prediction in the merged model.
Together, these contributions offer a practical and substantially cheaper path toward robotic agents that combine reliable low-level control with the high-level semantic reasoning of their underlying VLM.
Related Work
Vision Language Action Models
Vision-Language-Action (VLA) models extend Vision-Language Models (VLMs) by fine-tuning them on large-scale robot demonstration data ((O’Neill et al., 2025)), enabling them to act as generalist policies that directly generate low-level actions ([6] (Ghosh et al. 2024; Kim et al. 2024)). Actions are produced either autoregressively ((Kim et al., 2024; Pertsch et al., 2025)) or via continuous decoding such as flow matching ([5]). However, this fine-tuning causes them to largely lose their original vision-language capabilities through catastrophic forgetting.
Some works sidestep this issue by keeping the VLM frozen and training only a lightweight action module ((Shukor et al., 2025; Wen et al., 2025)), but this limits the model’s ability to fully adapt its representations for embodied control. Others reintroduce text tasks during training either by training VLAs to generate both actions and text simultaneously ([4] (Zawalski et al. 2024)) or by aligning action representations with the language objective ((Hancock et al., 2025)). Both approaches require large-scale co-training to recover knowledge already present in the initial VLM. In contrast, our approach operates post-hoc on an already fine-tuned VLA, requiring no additional training of either source model.
Model Merging
Model merging has emerged as a cost-effective alternative to full retraining, enabling the combination of specialised models into a single capable model without additional data or compute. Model Soup (Wortsman et al. (2022)) averages the weights of models fine-tuned from the same checkpoint. SLERP (Jang et al. (2024)) interpolates along the spherical geodesic rather than linearly, better preserving parameter norms. TIES (Yadav et al. (2023)) resolves sign conflicts across multiple task vectors before averaging, requiring at least two independently fine-tuned models and thus not directly applicable to our setting. DARE (Yu et al. (2024)) randomly drops a fraction of task vector parameters before rescaling the remainder. Fisher Merging (Matena and Raffel (2022)) weights each parameter by its diagonal Fisher information, but requires computing and storing a matrix as large as the full model, making it costly at billion-parameter scale. AdaMerging (Yang et al. (2024)) learns task-wise or layer-wise merging weights but does not decompose the task vector into finer-grained components. All these methods operate with fixed or coarse-grained coefficients and none have been applied to reconciling a VLM with its VLA derivative. Closer to our work, decomposing task vectors via SVD has already been proposed to reduce interference between models trained on different tasks Gargiulo et al. [9]. We build on this insight and
extend it to the VLM–VLA setting, where the objective is to reconcile two fundamentally different capabilities — language understanding and action prediction — rather than reduce task interference.
Method
Problem Formulation
Let and denote the parameters of a pre-trained Vision-Language Model and a fine-tuned Vision-Language-Action model, respectively. Both models share a common backbone architecture. The VLM initially possesses strong language and visual grounding capabilities, while the VLA has been further trained to predict robotic actions, albeit at the cost of partial degradation of its original vision-language abilities. Our merging procedure is applied to all layers that are common to both architectures; VLA-specific components such as the action head are left unchanged and carried over directly into the merged model.
Our goal is to construct a merged model that jointly retains the action prediction capabilities of the VLA and the vision-language understanding of the VLM (Figure 2a). We define it as a weighted combination of both source models, controlled by a set of mixing coefficients :

Figure 2. Overview of our VLM–VLA merging method. (a) The task vector is extracted by subtracting the VLM weights from the VLA weights for each layer . (b) is decomposed via Singular Value Decomposition, revealing the orthogonal directions of adaptation introduced during VLA fine-tuning. (c) A scalar mixing coefficient is assigned to each singular component, controlling how much of each direction is retained in the merged model. The coefficients are optimised by minimising a composite objective that jointly penalises degradation of language generation quality and action prediction accuracy with respect to both source models.
where is the total number of layers and controls the contribution of each source model at layer . The central question is how to determine — and at what granularity — such that the merged model best preserves both capabilities. In the following, we show that operating at the level of individual singular components of the per-layer task vector provides the necessary fine-grained control.
Task Vector and SVD Decomposition
For each layer , we define the task vector as the difference between the VLA and VLM weight matrices (Figure 2a):
Substituting into Eq. (1), the merged model can be equivalently written as:
This formulation shows that merging amounts to scaling the task vector by at each layer. To obtain finer-grained control, we decompose into its individual directions via Singular Value Decomposition, as [9] (Figure 2b), allowing a separate coefficient to be assigned to each direction independently:
where are the singular values ranked by importance and is the rank of . Each term captures a distinct direction of adaptation introduced during VLA fine-tuning.
Substituting the SVD decomposition of Eq. (4) into Eq. (3), the merged model becomes:
We now replace the single scalar by a per-component coefficient (Figure 2c), one for each singular direction, yielding our final merging equation:
When all , the layer reduces to the VLM. When all , it recovers the full VLA adaptation. The global scalar merging of Eq. (3) is recovered as the special case where all coefficients are equal to . Importantly, both and the SVD decomposition of the task vector are frozen throughout — the only free parameters are the mixing coefficients , which are optimised to simultaneously satisfy the language understanding objective of the VLM and the action prediction objective of the VLA, as described in § 3.3.
Training Objective
To optimise the mixing coefficients , we define a composite objective that captures both language fidelity and action prediction accuracy, while penalising excessive deviation from either source model:
where are tunable weighting hyperparameters (details can be found in Appendix § 6.§ 1). measures language generation quality via negative log-likelihood over a language dataset . measures action prediction quality over a robot trajectory dataset ; its form depends on the VLA action head — for an autoregressive architecture it is the negative log-likelihood over action tokens, while for flow matching it is the conditional flow matching loss ((Lipman et al., 2023)). The KL divergence terms act as regularisers, penalising deviation from the pure VLM and VLA distributions respectively. Since is a differentiable function of via Eq. (6), is minimised directly with gradient descent using Adam ((Kingma and Ba, 2017)) (details can be found in Appendix § 6.§ 2).
Experiments
Experimental Setup
Models and datasets. We apply our merging method to PaliGemma-3B ([3]) as the VLM, paired with two VLA variants sharing the same architecture: ((Pertsch et al., 2025)), an autoregressive VLA that tokenises actions, and ([4]), a flow matching VLA that predicts continuous action chunks via iterative denoising. Both VLA variants are obtained by fine-tuning PaliGemma on robot demonstration data. All models are kept frozen — only the mixing coefficients are optimised via Adam (Kingma and Ba, 2017) on two complementary datasets built from the LIBERO dataset available on LeRobot (Liu et al., (2023a); Cadene et al. [7]). For the action signal, we relabel existing expert demonstrations from the LIBERO dataset with the pure VLA’s own predicted action chunks, producing . For the language signal, we prompt PaliGemma to caption each observed frame, producing a paired dataset of observations and scene descriptions.
Evaluation protocol. We evaluate the merged model along three axes: vision-language capabilities, embodied reasoning, and action capabilities.
For vision-language capabilities, we assess captioning performance on the COCO dataset (Lin et al., 2015) using Rouge-L (Lin, 2004), METEOR (Banerjee and Lavie, [2]) and BERTScore (Zhang et al., 2020) metrics, and visual question answering performance on VQAv2 (Goyal et al., 2017).
For embodied reasoning, we evaluate accuracy on the EQA benchmark across three question categories — Object State, Spatial, and Attribute Recognition — which probe whether the model can answer language questions grounded in a robotic setting, bridging the gap between the general-domain vision-language evaluations above and the action benchmarks below.
For action capabilities, we evaluate task success rate on the LIBERO task suites — LIBERO-Spatial, LIBERO-Goal and LIBERO-Object (Liu et al., 2023a) — and report the average success rate across tasks for each suite (details can be found in Appendix § 7; additional results on the RoboCasa kitchen tasks are also reported in Appendix § 7.§ 1).
Baselines. We compare our merged model against a set of baselines. The VLM (PaliGemma (Beyer et al., [3])) and VLAs ( and ) serve as upper bounds on language and action performance respectively. (Black et al., [4]) is also a co-trained model that jointly optimises language and action objectives during training. We further include three established merging strategies that all use a fixed, non-learned merging coefficient: Model Soup (Wortsman et al., 2022), which linearly interpolates between VLM and VLA weights, and which we evaluate at four interpolation coefficients to characterise the full trade-
off curve it induces between language and action performance; DARE (Yu et al., 2024), which randomly drops a fraction of the task vector parameters before rescaling and adding the remainder to the VLM; and SLERP (Jang et al., 2024), which interpolates between the two weight vectors along the spherical geodesic rather than linearly. For the merged baselines, is used as the VLA backbone. We additionally compare against AdaMerging (Yang et al., 2024), which learns a single per-layer coefficient rather than per-component ones; we optimise it with the same objective (Eq. (7)) as our method; we use as its VLA backbone for direct comparison with Ours. All stochastic methods — DARE, AdaMerging, and our merged models — are averaged over 10 seeds, with standard deviations reported in Table 1.
| Model | Type | Captioning RougeL | Captioning Meteor | Captioning BERTScore | VQAv2 Acc | EQA Obj State | EQA Spatial | EQA Attri Recog | Libero Spatial | Libero Goal | Libero Object |
| PaliGemma | VLM | 33.6 | 23.6 | 88.9 | 53.8 | 44.4 | 21.7 | 32.4 | 0.0 | 0.0 | 0.0 |
| VLA | 1.2 | 0.5 | 68.2 | 9.9 | 15.5 | 5.7 | 3.8 | 70.0 | 82.0 | 88.0 | |
| VLA | 9.4 | 1.5 | 76.5 | 8.2 | 8.8 | 0 | 0 | 88.0 | 90.0 | 96.0 | |
| Model Soup | Merged | 14.6 | 11.4 | 82.7 | 32.4 | 22.2 | 15.1 | 22.7 | 0.0 | 0.0 | 0.0 |
| Model Soup | Merged | 8.3 | 5.3 | 76.4 | 21.7 | 19.9 | 8.1 | 11.6 | 0.0 | 0.0 | 0.0 |
| Model Soup | Merged | 5.0 | 5.1 | 73.9 | 15.3 | 19.9 | 4.4 | 4.7 | 0.0 | 0.0 | 0.0 |
| Model Soup | Merged | 2.1 | 1.1 | 70.1 | 9.0 | 15.5 | 5.8 | 2.6 | 5.0 | 13.0 | 17.0 |
| DARE | Merged | ||||||||||
| SLERP | Merged | 9.9 | 7.4 | 77.7 | 18.5 | 17.7 | 10.3 | 10.5 | 0.0 | 0.0 | 0.0 |
| AdaMerging | Merged | ||||||||||
| Ours | Ours | ||||||||||
| Ours | Ours |
Table 1. Vision-language, embodied reasoning, and robot manipulation performance of all methods. The pure VLM cannot act, and both VLAs lose language and reasoning ability despite strong action performance. Existing merging baselines largely collapse on manipulation, while our merged models recover strong performance on all three axes simultaneously, without retraining either source model.
Main Results
Vision-language capabilities. Table 1 reports captioning and VQA performance. The pure VLM sets the upper bound (33.68 ROUGE-L, 88.98 BERTScore, 53.87 VQAv2), while both pure VLAs degrade severely (: 1.25 ROUGE-L, 9.93 VQAv2; : 9.40 ROUGE-L, 8.22 VQAv2 despite co-training). Model Soup, DARE and SLERP offer only modest, comparable recovery (8.32–10.11 ROUGE-L), confirming that a single coefficient applied uniformly across the task vector cannot restore language grounding. AdaMerging, which shares our training objective but learns only a per-layer coefficient, recovers substantially more (28.1 ROUGE-L, 87.5 BERTScore, 38.2 VQAv2) — but still trails Ours (29.5/87.8/48.7), and, as shown below, this comes at the cost of near-total collapse on action prediction. Our merged models close most of the remaining gap to the VLM: Ours reaches 18.6 ROUGE-L / 28.9 VQAv2, and Ours reaches 29.5 ROUGE-L / 87.8 BERTScore / 48.7 VQAv2 — nearly matching the VLM. Recovery is stronger for than , likely because the latter entangles language and action within a shared autoregressive head. Figure 3 illustrates this: retains scene-relevant keywords (e.g. “Kitchen pan”, “Robot”) but loses the language modelling coherence of the VLM needed to form descriptions, whereas our merged model generates grammatically coherent captions that correctly identify objects and their spatial relationships. Additional examples can be found in Appendix § 8.§ 3.
Figure 3. Qualitative captioning examples. Given the same visual observation, the VLM produces coherent scene descriptions, the VLA () generates fragmented or incomplete text due to catastrophic forgetting, and Ours merged model recovers meaningful captions while retaining action capabilities.
Embodied reasoning. The same pattern holds on EQA (Object State / Spatial / Attribute Recognition). The VLM leads (44.44/21.73/32.46), both VLAs degrade sharply, and fixed-coefficient baselines recover only partially and inconsistently (best: Model Soup at 22.22/15.19/22.75). AdaMerging recovers markedly more (43.4/14.0/20.3), even edging out Ours on Object State, indicating that a learned, task-aware objective is the primary driver of embodied-reasoning recovery, with SVD granularity offering more modest further gains on this axis. Our merged models nonetheless outperform all baselines overall: Ours reaches 29.7/14.1/17.2, and Ours reaches 32.3/17.3/19.5, recovering a substantial share of the VLM’s embodied reasoning capabilities despite having no direct EQA supervision.
Robot manipulation performance. Table 1 also reports success rate on the LIBERO suites. The VLM achieves zero success everywhere, while both VLAs establish strong upper bounds (: 88/90/96; : 70/82/88). DARE, SLERP and Model Soup (at ) collapse entirely to zero success, and Model Soup only recovers marginally (5/13/17) by sacrificing nearly all language gains — confirming that a fixed, uniform coefficient cannot preserve the action-relevant structure of the VLA. AdaMerging collapses almost as severely (6.2/2.9/3.7) despite sharing our training objective, showing that a single per-layer coefficient lacks the resolution to preserve action-relevant directions — consistent with the granularity ablation of Section 4.3, where per-layer cannot disentangle language- from action-relevant directions within . Our merged models retain strong performance on both variants: Ours reaches 61.2/73.1/79.8, and Ours reaches 89.3/88.1/93.3, closely tracking its pure VLA counterpart while still achieving the language and reasoning gains above — a trade-off no baseline, including AdaMerging, offers.
Ablation study
We conduct three ablations to validate the design choices of our method: the granularity of the mixing coefficients, the contribution of each component of the training objective, and the sensitivity of the method to the rank of the SVD decomposition.
Granularity of mixing coefficients. To analyse the impact of per-component SVD coefficients, we compare two levels of granularity for . Under per-layer , a single coefficient is learned for each layer and applied uniformly to every singular component of that layer’s task vector — this is equivalent to the layer-wise AdaMerging protocol Yang et al. (2024), adapted to our setting. Under per-component (ours), a separate coefficient is learned for each individual singular component within each layer . Figure 4 shows the evolution of , , and over optimisation steps for each condition. The results are unambiguous: per-component consistently and substantially outperforms the per-layer scalar across all four metrics. On , per-layer plateaus around 2.0, while per-component converges below 1.0. The gap is even more pronounced on action prediction: per-layer converges to and remains noisy around 0.06–0.08, whereas per-component drives both metrics down to roughly a tenth of these values. A single coefficient per layer cannot disentangle the directions within that encode language degradation from those that encode action prediction, since both are entangled within the same weight matrix and a layer-wide must trade them off uniformly. These results confirm that the fine-grained control afforded by SVD decomposition is essential — coarser parameterisations lack the expressivity to reconcile both objectives.
Figure 4. Granularity of mixing coefficients. We compare two levels of granularity for — per-layer and per-component (ours) — and report the evolution of , , and over training. Per-component converges to substantially lower values on every metric, confirming that fine-grained SVD-based control is essential to simultaneously preserve language and action capabilities.
Contribution of each loss component. To understand the role of each term in the training objective , we train four variants of Ours by removing one component at a time: without , , , and . Results are reported in Table 2. Removing causes the merged model to collapse on action prediction entirely (0% success rate) while language is the highest of all variants (34.6 ROUGE-L), confirming that without an explicit action signal drifts toward the VLM. Conversely, removing severely degrades language quality (7.9 ROUGE-L) while leaving action performance almost intact (94.1%), showing that this KL term is the primary anchor preserving language fluency and that alone is insufficient to fully constrain toward the VLM. Removing has a comparatively mild effect (21.9 ROUGE-L, 95% success rate), suggesting that already provides most of the language-preserving signal on its own. Removing yields the second-best language score (31.2 ROUGE-L) but a substantial drop in success rate (71.5%), indicating that this term plays an important role in stabilising action prediction beyond what alone provides. Overall, the full objective achieves the best joint trade-off (29.5 ROUGE-L, 93.3% success rate), confirming that all four terms are complementary: the terms provide the primary task signal while the terms regularise and stabilise the optimisation, with the VLM-side KL term proving especially critical for language retention.
| Objective | ROUGE-L | SR |
| w/o | 21.9 | 95.6 |
| w/o | 34.6 | 0 |
| w/o | 7.9 | 94.1 |
| w/o | 31.2 | 71.5 |
| Full objective (ours) | 29.5 | 93.3 |
Table 2. Contribution of each loss component. We ablate each term of the training objective and evaluate using ROUGE-L for language quality and Success Rate SR for action predictions. Best results in bold.
Effect of SVD rank. We study the sensitivity of the method to the rank used in the SVD decomposition by varying . For a given , each layer’s task vector is truncated to its top- singular components — those associated with the largest singular values — and only the corresponding are optimised, with the remaining components implicitly set to the VLM. Figure 5 reveals a striking asymmetry between the two objectives: is largely insensitive to rank, while shows a clear separation, with higher rank leading to significantly better action preservation. The KL divergence terms follow the same pattern. This is confirmed by the downstream scatter plot: ROUGE-L is low at the extreme but already saturates around 30 from onward, while mean LIBERO success rate stays at 0% for , jumps to 64% at , and exceeds 90% only from onward. Notably, requires only 269,443 learnable parameters compared to 381,605 at full rank — both roughly four orders of magnitude smaller than the finetuning of the B backbone, confirming that near-optimal performance is achieved with an even smaller fraction of the already lightweight parameter budget. This indicates that language recovery only requires a small set of dominant directions of the task vector, whereas action preservation requires a much larger, near complete, set of singular directions — confirming that the VLA adaptation is diffuse in weight space while language-relevant directions are concentrated in a handful of dominant components. More broadly, this asymmetry explains why uniform merging fails to recover action performance while partially recovering language.
Figure 5. Effect of SVD rank. (Left) Training losses for varying rank : is largely insensitive to rank, while improves markedly with higher rank, indicating that the VLA adaptation is diffuse across many singular components. (Right) Downstream evaluation confirms this asymmetry: ROUGE-L saturates from , while LIBERO success rate requires to exceed 90% — language recovery needs far fewer singular directions than action preservation.
More analysis on the distribution can be found in Appendix § 8.§ 2.
Conclusion
In this paper, we present a post-hoc method for merging a VLM and a VLA by decomposing the task vector via SVD and learning per-component mixing coefficients through gradient descent. We apply this method to two distinct VLA architectures, an autoregressive and a flow matching model, demonstrating that it generalises across different action decoding strategies. Both source models remain frozen throughout, and only the mixing coefficients are optimised, making the approach substantially cheaper than retraining or co-training a VLA from scratch and broadly applicable to already fine-tuned VLAs. Our approach recovers a model that retains the action prediction capabilities of the VLA while restoring the vision-language understanding of the original VLM, a balance that uniform interpolation fails to achieve. Through extensive ablations, we show that this fine-grained control over the merging process is essential, and our analysis further reveals that language and action capabilities are encoded at different scales in weight space, offering new insight into the nature of catastrophic forgetting in VLA fine-tuning. Future work could leverage the language and reasoning capabilities restored by our method to equip VLAs with high-level planning, enabling them to decompose and reason about long-horizon tasks rather than merely reacting to immediate instructions. Code and merged model checkpoints will be released upon acceptance.
Limitations
Our method assumes the VLM and VLA share a common backbone, so it does not directly apply to mismatched architectures or tokenizers. It also still requires a training loop to optimise , albeit a much cheaper one than full fine-tuning. Our action and language datasets are both derived from a single benchmark (LIBERO) and from the source models themselves, so performance on out-of-distribution tasks, embodiments, or real-world deployments remains untested. Finally, our experiments are limited to VLMs and VLAs at the 3–4B parameter scale, and whether our findings hold at larger scales remains untested.
Ethical Considerations
This work aims to mitigate catastrophic forgetting in robot learning models, helping robotic agents retain language-based communication alongside low-level control rather than sacrificing one for the other. All experiments are conducted in simulation using publicly available models and datasets; no human subjects or physical robot deployments are involved. As with any method that enhances action capabilities, thorough safety validation is recommended before real-world deployment.
Acknowledgments
Experiments presented in this paper were carried out using the HPC resources of IDRIS under the allocations 2026-[A0201016980] and 2026-[AD011015093R2] made by GENCI. This work was supported by the European Commission’s Horizon Europe Framework Programme under grant No 101070381 (PILLAR-robots), by RODEO Project (ANR-24-CE23-5886), by PEPR Sharp (ANR-23-PEIA-0008, ANR, FRANCE 2030) and by Cluster PostGenAI@Paris (ANR-23-IACL-0007, FRANCE 2030).
Implementation Details
Hyperparameters
Table Table 3 reports the full set of hyperparameters used to train the mixing coefficients for our main results, separately for Ours and Ours.
| Hyperparameter | Ours | Ours |
| Optimiser | Adam | Adam |
| Initial learning rate | 0.01 | 0.01 |
| Learning rate schedule | cosine decay | cosine decay |
| Total training steps | 20,000 | 20,000 |
| () | 1 | 1 |
| () | 1 | 5 |
| () | 0.01 | 0.01 |
| () | 0.8 | 10 |
| initialisation | Uniform(0, 1) | Uniform(0, 1) |
| SVD power iterations | 4 | 4 |
| Batch size | 2 | 2 |
| Number of GPUs | 1 | 1 |
Table 3. Hyperparameters used to optimise for each VLA variant. , weight and respectively; , weight the KL regularisation terms toward the VLM and VLA respectively.
KL Regularisation Terms
Language side. For the VLM side, and denote the token-level output distributions of the pure VLM and the merged model, respectively, computed over the language vocabulary at each generated position given the same visual observation and prompt. For , whose vocabulary jointly contains both language and action tokens, this distribution is restricted to the language sub-vocabulary, masking out action-token logits before computing the divergence. is the standard categorical KL divergence between these two (vocabulary-restricted) token distributions, averaged over sequence positions, and is well-defined for both VLA variants since the language head is shared and discrete in both and .
Action side — autoregressive VLA. For , actions are represented as discrete tokens drawn from a dedicated action sub-vocabulary, and , denote the corresponding token-level distributions restricted to this action sub-vocabulary, with language-token logits masked out. is therefore also a standard categorical KL divergence, computed analogously to the language term above but over the disjoint action sub-vocabulary, ensuring that each KL term reflects only the corresponding modality.
Action side — flow matching VLA. For , actions are continuous and generated via iterative denoising, so no closed-form categorical distribution exists over the action space. Following prior work on distribution matching for diffusion- and flow-based visuomotor policies (Jia et al., 2024; Ping et al., 2026), we approximate and as Gaussian distributions over the predicted velocity field. Concretely, for a given observation we sample independent noise draws , form the corresponding interpolated actions , and compute the predicted velocities and under the pure VLA and the merged model respectively. We then estimate the empirical mean and variance of each set of velocity predictions,
and approximate , , for which the KL divergence admits the standard closed form:
This Gaussian approximation is computed independently per action dimension and averaged. We note that this is an approximation of the true (intractable) distribution induced by the iterative denoising process, justified by the fact that the velocity field at a fixed timestep is approximately unimodal in practice.
Dataset Collection
Action dataset . For each VLA variant, we construct by relabeling existing expert demonstrations from the LIBERO dataset available on LeRobot ([7] (Liu et al. 2023a)). Concretely, we take the visual observations, proprioceptive states, and task instructions from the expert trajectories and query the pure VLA to obtain its predicted action chunks for each observation, storing the resulting pairs. This collection is performed once per VLA variant and reused throughout training and ablation, and yields a total of 5,000 examples.
Language dataset . For each visual observation collected above, we additionally query the pure VLM with a captioning prompt to obtain a corresponding scene description, producing paired examples , also totalling 5,000 examples. Both and are split into disjoint train and test partitions, and all reported metrics are computed on test examples.
Training Cost
Training the mixing coefficients for our main results takes approximately 3 hours on a single NVIDIA H100 GPU for 20,000 steps. Both source models ( and ) remain frozen throughout, and only is updated. At our chosen practical rank, this amounts to a total of 381,587 learnable parameters for Ours and 381,605 for Ours across all layers weight matrices — in both cases roughly four orders of magnitude smaller than the 3B frozen parameters of the VLA backbone alone. This makes our approach substantially cheaper than retraining or co-training a VLA from scratch.
Environment and Task Definitions
LIBERO. LIBERO ((Liu et al., 2023a)) is a simulation benchmark for lifelong robot manipulation built on the Robosuite simulator, comprising several task suites each containing 10 tasks with 50 expert demonstrations per task. We evaluate on three of its suites: LIBERO-Spatial, where the robot must place a bowl on a plate among visually identical bowls that differ only in their spatial location relative to other objects, testing spatial reasoning under a fixed set of objects; LIBERO-Object, where the robot must pick and place a different, unique target object into a basket across tasks while the layout remains fixed, testing object-centric generalisation; and LIBERO-Goal, where both the objects and their spatial layout are held fixed while the task goal varies, testing the transfer of procedural, goal-directed behaviour. Each suite is designed to disentangle a specific axis of generalisation (spatial, object, or goal knowledge), and observations consist of third-person and wrist-camera RGB images.
Robocasa. RoboCasa ((Nasiriany et al., 2026)) is a large-scale simulation benchmark for household robot manipulation in photorealistic kitchen environments, comprising a set of atomic skills (e.g. pick-and-place, opening/closing doors and drawers, turning knobs, pressing buttons) together with longer-horizon composite tasks. We evaluate on three atomic kitchen tasks: Open Fridge, requiring the robot to open a refrigerator door; Setup Cup, requiring the robot to pick up a cup and place it in a target location; and Navigate Kitchen, requiring the robot to move within the kitchen environment to reach a target location before completing a manipulation sub-goal. Compared to LIBERO, RoboCasa tasks are evaluated across a larger number of randomised scene layouts and object instances, making the benchmark substantially more challenging: even strong VLA baselines such as average only around 39% success rate across these tasks in our setting, reflecting the benchmark’s greater diversity in scene configurations and the longer effective horizon of some tasks (e.g. Navigate Kitchen).
Additional Results on RoboCasa
Table 4 reports task success rate on three RoboCasa kitchen tasks — Open Fridge, Setup Cup, and Navigate Kitchen — complementing the LIBERO results presented in the main paper. The overall trends are consistent with those observed on LIBERO: the pure VLM achieves zero success rate everywhere, both pure VLAs establish strong baselines, and all fixed-coefficient merging methods (Model Soup, DARE, SLERP) collapse entirely to zero success rate across all tasks. Our merged models retain meaningful action performance on both VLA variants: Ours reaches 56/61/78 (versus 78/68/89 for the pure -FAST), and Ours reaches 65/49/21 (versus 71/58/25 for the pure ). The performance drop relative to the pure VLAs is more pronounced on RoboCasa than on LIBERO, reflecting the greater difficulty of this benchmark: even the pure VLAs average only around 50–78% across tasks, and Navigate Kitchen proves particularly challenging for (25% for the pure VLA, 21% after merging). Despite this increased difficulty, our method remains the only merging approach that retains non-trivial action performance while simultaneously recovering the vision-language and embodied reasoning capabilities reported in the main paper.
| Model | Type | Open Fridge | Setup Cup | Navigate Kitchen |
| PaliGemma | VLM | 0 | 0 | 0 |
| VLA | 78 | 68 | 89 | |
| 71 | 58 | 25 | ||
| Model Soup | Merged | 0 | 0 | 0 |
| Model Soup | 0 | 0 | 0 | |
| Model Soup | 0 | 0 | 0 | |
| Model Soup | 0 | 0 | 0 | |
| DARE | 0 | 0 | 0 | |
| SLERP | 0 | 0 | 0 | |
| Ours | 56 | 61 | 78 | |
| Ours | 65 | 49 | 21 |
Table 4. Task success rate (%) on three RoboCasa kitchen tasks. All fixed-coefficient merging baselines collapse to zero, while our merged models retain meaningful performance on both VLA variants.
Analysis of the Merged Model
Models Similarity
To better understand what the learned coefficients produce in practice, we analyse the per-layer alignment of the merged model with respect to both source models, for both VLA variants. For each layer , we compute three cosine similarities: — the baseline distance between the two source models — , and , where denotes the merged model for the corresponding VLA variant. These are computed directly on the reconstructed merged weight matrices using the learned .
Figure 6 reports these three curves across all 602 layers for both VLA variants. Several observations emerge, most of which hold consistently across both panels. First, the baseline similarity between VLM and VLA (blue) is consistently the lowest in both cases, confirming that VLA fine-tuning significantly displaces the weights from their original VLM values; this displacement grows with layer depth, dropping from ∼0.80–0.85 in the earliest layers to ∼0.65–0.70 in the deepest ones, indicating that fine-tuning increasingly diverges from the VLM as depth increases, for both the autoregressive and flow matching VLA. Second, and most importantly, the merged model remains significantly closer to both source models than the VLM and VLA are to each other in both cases — orange and green curves lie well above the blue baseline throughout. This indicates that the merged model does not simply interpolate between two distant models but finds a position in weight space that is simultaneously close to both sources, with no significant drop in alignment relative to either, regardless of the underlying VLA architecture. Third, the merged model is consistently and substantially closer to the VLA than to the VLM across all layers in both panels (green above orange throughout), indicating that the learned coefficients globally preserve a large share of the VLA adaptation while still maintaining a clear margin of proximity to the VLM. The main difference between the two panels is the magnitude of this gap: for Ours (bottom), the green curve sits substantially higher (0.97–0.98) and more tightly clustered than for Ours (top, 0.88–0.93), while the orange curves are comparable in both cases (0.83–0.93). This is consistent with the merged model retaining a stronger and more uniform alignment with the VLA across layers, in line with the higher action success rates observed for Ours in Table 1. Both the orange and blue curves exhibit a regular sawtooth pattern in both panels, consistent with the alternation between different weight types (attention and feed-forward) within each transformer block. Additionally, the orange curve partially recovers around layer index 350–400 before declining again toward the end of the network in both cases. This structured behaviour emerges directly from the optimisation of , confirming that the learned coefficients successfully navigate the trade-off between both source models rather than collapsing toward either one, irrespective of the VLA’s action decoding strategy.
Figure 6. Per-layer similarity to source models. (Top) Ours. (Bottom) Ours. For each of the 602 weight matrices, we compute the cosine similarity between the VLM and VLA weights (blue), between the merged model and the VLM (orange), and between the merged model and the VLA (green). In both cases, the merged model is consistently closer to the VLA than to the VLM across all layers, while remaining well above the VLM–VLA baseline throughout.
Merging coefficients analysis
To examine how the learned coefficients are distributed across the singular spectrum, we average across all layers at each rank index (where corresponds to the largest singular value within each layer), for both VLA variants.
Figure 7 shows the same overall pattern for both VLA variants: mean is near 0 for the most dominant directions of the task vector and rises monotonically toward the tail of the spectrum. For Ours (top), mean is exactly 0 for and rises to a peak around 0.8 near before declining slightly toward the tail. For Ours (bottom), the trend is similar, with mean starting close to, though not exactly at, 0 ( at ) and plateauing around 0.65–0.7 from onward. The consistency of this monotonic pattern across both VLA architectures indicates that the low-rank/high-rank asymmetry exists between language and action recovery. This pattern directly complements the rank ablation of § 4.3: when the model is restricted to a low rank, only the dominant directions are available, leaving the optimisation with little room to inject VLA-relevant adaptation, which is consistent with action prediction degrading sharply at low rank while language recovers comparatively quickly, for both VLA variants. Conversely, the gradual rise of mean toward the tail of the spectrum in both panels shows that the directions necessary for action prediction are spread broadly across the weaker components, consistent with the diffuse nature of the VLA adaptation observed throughout this work.
Figure 7. Figure 7: Mean vs. rank index. (Top) Ours. (Bottom) Ours. For each rank index , we average across all 602 layers. Both VLA variants show the same overall pattern: mean is near 0 for the most dominant directions and rises monotonically toward the tail of the spectrum, confirming that this asymmetry is not specific to a single action decoding strategy.
Additional Qualitative Examples
Figure 8 shows six additional captioning examples spanning diverse scenes, from tabletop manipulation to kitchen environments. Across all examples, the pure VLA output degrades into either a single repeated keyword (e.g. “Wood Wood Wood ....”, “Robot Bot Bot ...”), a short fragment stripped of grammatical structure (e.g. “Colored”, “@Water @@”), or garbled non-linguistic tokens (e.g. “p̈w̄Γ f tools”), consistent with the catastrophic forgetting of language modelling capabilities discussed in § 4.2. In contrast, our merged model consistently recovers full, grammatically coherent sentences that correctly identify the objects and setting present in the scene (e.g. “A collection of tools including screwdrivers, wrenches, and bolts.”, “A 3D rendering of a kitchen scene featuring a stovetop and oven, including a sink.”). We note that our merged model’s captions occasionally diverge stylistically from the pure VLM’s — favouring phrases such as “3D rendering of...” — but remain semantically accurate and fluent, indicating that language coherence is recovered even when the exact phrasing differs from the original VLM. These qualitative observations are consistent with the quantitative captioning scores reported in Table 1, where the pure VLA collapses on ROUGE-L, METEOR and BERTScore while our merged model recovers most of the pure VLM’s performance.

Figure 8. Qualitative captioning examples. Given the same visual observation, the VLM produces coherent scene descriptions, the VLA () generates fragmented or incomplete text due to catastrophic forgetting, and Ours merged model recovers meaningful captions while retaining action capabilities.
References
- [1]Mohamed Salim Aissi, Clémence Grislain, Clément Romac, Laure Soulier, Mohamed Chetouani, Olivier Sigaud, and Nicolas Thome. 2026. PRISM: Perception reasoning interleaved for sequential decision making. In Forty-third International Conference on Machine Learning.arxiv.org/abs/2605.05407
- [2]Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
- [3]Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, and 16 others. 2024. Paligemma: A versatile 3b vlm for transfer. Preprint, arXiv:2407.07726.arxiv.org/abs/2407.07726
- [4]Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Divya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, and 16 others. 2025. π₀.₅: a vision-language-action model with open-world generalization. Preprint, arXiv:2504.16054.arxiv.org/abs/2504.16054
- [5]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, and 5 others. 2024. π₀: A vision-language-action flow model for general robot control. Preprint, arXiv:2410.24164.arxiv.org/abs/2410.24164
- [6]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, and 35 others. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. Preprint, arXiv:2307.15818.arxiv.org/abs/2307.15818
- [7]Remi Cadene, Simon Alibert, Francesco Capuano, Michel Aractingi, Adil Zouitine, Pepijn Kooijmans, Jade Choghari, Martino Russi, Caroline Pascal, Steven Palma, Mustafa Shukor, Jess Moss, Alexander Soare, Dana Aubakirova, Quentin Lhoest, Quentin Gallouedec, and Thomas Wolf. 2026. Lerobot: An open-source library for end-to-end robot learning. Preprint, arXiv:2602.22818.arxiv.org/abs/2602.22818
- [8]Yinlong Dai, Benjamin A. Christie, Daniel J. Evans, Dylan P. Losey, and Simon Stepputtis. 2026. Language movement primitives: Grounding language models in robot motion. Preprint, arXiv:2602.02839.arxiv.org/abs/2602.02839
- [9]Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. 2025. Task singular vectors: Reducing task interference in model merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18695–18705.arxiv.org/abs/2412.00081





