betaFeedback

WorksComputer Science

Locating and Repairing Domain Shift in VLM Trajectory Planning

Abstract

Vision-Language Models (VLMs) are increasingly adopted as end-to-end driving planners, yet they suffer from cross-city domain shift. Existing methods treat this shift as a single quantity addressed by uniform alignment or single-site editing. We challenge this view on two counts: (i) domain shift in VLM trajectory planners is inherently structured across modality and layer, and (ii) a locate-then-repair recipe matched to this structure outperforms every uniform-objective baseline. We analyse domain shift via activation patching and uncover two regularities. (F1) Modality axis. Image and text tokens exhibit distinct shift patterns in each VLM. (F2) Layer axis. The layer-wise shift partition is architecture-dependent and dataset-stable. These findings recast domain shift as a two-dimensional tensor indexed by modality (image vs. text tokens) and layer, motivating a simple principle: locate where shift occurs, and repair only there. We instantiate this principle in MoLaRx (Modality-Layer Repair), a locate-then-repair framework. The Causal Importance Score (CIS) estimates the domain shift as a modality-layer tensor and partitions the layers into functional zones. CIS-Guided Selective Repair (CISR) then applies a matched operator (Maximum Mean Discrepancy (MMD) / low-rank adaptation (LoRA) / freezing) to repair each zone, with the composition adapting automatically to each architecture. We evaluate MoLaRx on three architecturally distinct VLMs (LFM2-VL-1.6B, Qwen2-VL-2B, and PaliGemma2-3B) across two cross-city benchmarks (nuScenes and Argoverse 2). MoLaRx attains the lowest cross-city trajectory-error gap among adaptation methods in all six main-table settings, while using 50-60% fewer trainable parameters and 8.4-10.9 ms lower inference latency. In contrast, uniform MMD-only adaptation collapses in all six settings, highlighting the importance of structure-guided repair.

Introduction

Vision-Language Models (VLMs) are increasingly being introduced for end-to-end trajectory planning [22, 35, 30, 31, 29] as they naturally integrate visual perception with historical trajectory context for decision-making. However, they suffer from severe cross-city domain shift [Naeniian et al., 2026]: as shown in Figure 1 (a, Figure 1a model trained in Boston incurs a +653% increase in trajectory error when deployed in Singapore (from 0.15 m to 1.13 m). Since collecting and annotating data for every new city is impractical, mitigating this gap is critical for safe and scalable deployment.

Figure 1. Domain shift in VLM trajectory planners is structured along modality and layer. (a) VLMs fine-tuned as trajectory planners suffer cross-city domain shift; (b) the existing diagnostic — DPA: saturates near 1 at every layer of every VLM; (c) F1 (modality axis): image and text shifts differ within every VLM; (d) F2 (layer axis): zone partition is architecture-dependent and dataset-stable.

Two paradigms in the broader literature shape how such cross-domain gaps are tackled in deep models. The first is uniform domain adaptation, a broad class of methods that align source and target distributions, including adversarial training [11], distribution matching [32, 13], image translation [Zhu et al., 2017, Yang and Soatto, 2020], and parameter-efficient adaptation [17]. Despite their differences, these methods apply one common adaptation strategy across layers. The second is locate-then-edit, originally developed for single-fact knowledge editing in language models [23] and later extended to multimodal LLMs [7]. ROME [23] uses causal tracing to identify an MLP site and applies a rank-one weight update there; MMEdit [7] benchmarks model editing in multimodal settings. Across both strategies, the same assumption is encoded: domain shift is treated as a single quantity that uniform alignment or single-site editing can target. We argue that this scalar assumption is fundamentally misaligned with structured domain shift, as it enforces a single objective across heterogeneous components, inevitably disrupting domain-invariant computation. We therefore ask two questions: (Q1) what is the structural form of cross-city domain shift in VLM trajectory planners? and (Q2) given the answer, what repair recipe outperforms uniform alignment? We answer empirically: cross-city domain shift is inherently structured along modality and layer axes (Q1), and a locate-then-repair recipe matched to this structure dominates every uniform-objective baseline (Q2).

To establish this empirically, we need a causal diagnostic, but standard tools fall short. Correlational probes such as domain probing accuracy (DPA) [2, 4, 34] saturate above 95% at every layer of each VLM (Figure 1 (b)), uninformative about causal contribution; attention weights [18] are similarly correlational; gradient attribution [33] conflates causal effect with gradient magnitude in regression. We therefore adopt activation patching [23, 8], an intervention that swaps activations between two forward passes: at each (modality, layer), we replace target-domain activations with source-domain activations at modality-mm token positions and measure the change in predicted trajectory. We call the resulting score the Causal Importance Score (CIS); its values across (m,l)(m, l) form a modality–layer tensor. Measurements use fine-tuned VLM planners (Appendix § 9).

Applying CIS to three architecturally distinct VLMs (LFM2-VL-1.6B [3], Qwen2-VL-2B [38], and PaliGemma2-3B [5]) across two cross-city benchmarks (nuScenes [6] and Argoverse 2 [Wilson et al., 2023]) reveals two regularities. (F1) Modality axis (Figure 1 (c)). Image-token shift dominates the earliest layers in every VLM; text-token shift overtakes it in intermediate layers of LFM2-VL and Qwen2-VL, but never in PaliGemma2. A single uniform alignment cannot target both modalities; this motivates per-modality repair. (F2) Layer axis (Figure 1 (d)). Each VLM partitions into 2 or 3 causal zones (Φalign\Phi_{\mathrm{align}}, Φtune\Phi_{\mathrm{tune}}, Φfreeze\Phi_{\mathrm{freeze}}) with architecture-specific boundaries; within each architecture, the partition is identical (LFM2-VL, PaliGemma2) or differs by at most one layer (Qwen2-VL) on nuScenes and Argoverse 2, making the structure architecture-dependent and dataset-stable. This motivates architecture-adaptive repair.

These findings recast cross-city domain shift as a two-dimensional tensor over modality and layer, naturally motivating a locate-then-repair framework, MoLaRx (Modality–Layer Repair). Specifically, (i) CIS locates the (modality, layer) sources of the cross-city gap and partitions layers into functional zones; (ii) CIS-Guided Selective Repair (CISR) repairs each zone with a matched operator: distribution alignment (MMD) on Φalign\Phi_{\mathrm{align}}, parameter-efficient adaptation (LoRA) on Φtune\Phi_{\mathrm{tune}}, and freezing on Φfreeze\Phi_{\mathrm{freeze}}. The operator composition is determined by the discovered partition, adapting automatically to the underlying architecture. The contributions are summarized as follows:

  • Discovery. We uncover a modality–layer structure of cross-city domain shift: image and text exhibit distinct layer-wise patterns, and the zone partition is architecture-dependent and dataset-stable. To our knowledge, this is the first joint modality–layer characterisation of cross-city shift in VLM trajectory planners.

  • Tool. We introduce CIS, an activation-patching diagnostic that measures how source-to-target interventions at each (modality, layer) slice causally affect predicted trajectories.

  • Method. We propose MoLaRx, a locate-then-repair framework: CIS locates the modality–layer sources of domain shift, and CISR routes existing adaptation operators (MMD, LoRA, and freezing) to the diagnosed zones, with the composition adapting automatically to each architecture.

  • Experiments. On three architecturally distinct VLMs across two benchmarks, MoLaRx attains the smallest cross-domain gap among adaptation methods in all six main-table settings while using 50–60% fewer trainable parameters and achieving 8.4–10.9 ms lower inference latency. In contrast, uniform MMD-only adaptation collapses in all six settings, highlighting the importance of structure-guided repair.

Related Work

Cross-domain trajectory planning has been studied for end-to-end driving [10] (Yasarla et al. 2025); our work draws on two treatment paradigms. (1) Uniform domain adaptation in trajectory planning applies one adaptation strategy uniformly across the model. Feature alignment via adversarial losses [11], kernel MMD [13], or correlation alignment [32], parameter-efficient adapters [17], and test-time normalisation [37] are the dominant variants [26]. Trajectory-prediction work adds auxiliary perception regularisation [14], domain-invariant graph encoders (T-GNN [Xu et al., 2022]), or self-training on pseudo-trajectories across geographies (STraj [Zhang et al., 2025]); VLM-based planners (GPT-Driver [22], DriveVLM [35], LMDrive [30], DriveLM [31], LanguageMPC [29]) are typically evaluated on a single benchmark distribution, which motivates studying cross-city shift within multimodal planning backbones [Naeini et al., 2026]. (2) Locate-then-edit identifies a small set of components causally responsible for a target behaviour and modifies them directly. ROME [23] edits a single causally identified MLP site, while MEMIT [24] distributes knowledge updates across multiple identified MLP layers. MMedIt [7] extends this to multimodal visual-fact editing, and Surgical Fine-Tuning (Surgical FT) [20] adapts a manually chosen layer band by shift type. Related approaches also explore selective treatment along other axes: CaDeT [27] disentangles causal and spurious latent factors, CRiTIC [1] gates inter-agent attention using discovered causal relations, and GALA [Sahoo et al., [2025] selects which layers to update at test time by gradient alignment. MoLaRx diagnoses cross-city shift jointly by modality and layer through activation interventions, and matches different existing repair operators to the diagnosed zones.

Preliminaries

Definitions

Definition 1 (VLM-based Trajectory Planning). A VLM-based trajectory planner is a function fΘ:X→Yf_{\Theta}:\mathcal{X}\to\mathcal{Y} that maps a multimodal input xx (image + text) to a future trajectory Y\mathbf{Y} by passing xx through LL stacked backbone layers and decoding the final-layer hidden state:

fΘ(x)=Y(x)=Dec⁡(H(L)(x)),H(l+1)(x):=gl(H(l)(x);θl),H(0)(x):=Embed⁡(x),(1)f_{\Theta}(\mathbf{x}) = \mathbf{Y}(\mathbf{x}) = \operatorname{Dec}\left(\mathbf{H}^{(L)}(\mathbf{x})\right), \qquad\mathbf{H}^{(l+1)}(\mathbf{x}) := g_l\left(\mathbf{H}^{(l)}(\mathbf{x});\theta_l\right), \qquad\mathbf{H}^{(0)}(\mathbf{x}) := \operatorname{Embed}(\mathbf{x}), \tag*{(1)}

where fΘf_{\Theta} has LL backbone layers parameterised by Θ={θ0,…,θL−1}\Theta=\{\theta_0,\ldots,\theta_{L-1}\}; gl(⋅;θl)g_l(\cdot;\theta_l) is the layer-ll operator and Dec⁡\operatorname{Dec} is the decoder. x∈X=I×Cx\in\mathcal{X}=\mathcal{I}\times\mathcal{C} is the input from the image space I\mathcal{I} and the textual-context space C\mathcal{C}. Output Y(x)∈Y\mathbf{Y}(x)\in\mathcal{Y} is a sequence of KK ego-vehicle 2D positions {(uk,vk)}k=1K\{(u_k,v_k)\}_{k=1}^{K}. H(l)(x)∈RT×d\mathbf{H}^{(l)}(x)\in\mathbb{R}^{T\times d} is the layer-ll hidden state; its modality–layer slicing is given in Definition 2.

Definition 2 (Modality–Layer Hidden States). A modality–layer hidden state HMm(l)(x)\mathbf{H}^{(l)}_{\mathcal{M}_m}(x) is the row-block of H(l)(x)\mathbf{H}^{(l)}(x) at layer l∈{0,…,L−1}l\in\{0,\ldots,L-1\} that collects all tokens of one modality m∈{img,txt}m\in\{\mathrm{img},\mathrm{txt}\}.

HMm(l)(x)=[H:,i(l)(x)]i∈Mm(x)∈R∣Mm(x)∣×d,m∈{img,txt}.(2)\mathbf{H}^{(l)}_{\mathcal{M}_m}(x)=\left[\mathbf{H}^{(l)}_{:,i}(x)\right]_{i\in\mathcal{M}_m(x)}\in\mathbb{R}^{|\mathcal{M}_m(x)|\times d},\qquad m\in\{\mathrm{img},\mathrm{txt}\}. \tag*{(2)}

where Mm(x)⊆[T]\mathcal{M}_m(x)\subseteq[T] is the index set of modality-mm token positions: Mimg(x)={i∈[T]:token⁡i∈Vimg}\mathcal{M}_{\mathrm{img}}(x)=\{i\in[T]:\operatorname{token}_i\in\mathcal{V}_{\mathrm{img}}\} collects the image-token positions, with Vimg\mathcal{V}_{\mathrm{img}} the image-token vocabulary (bracketed by the ⟨img_start⟩,⟨img_end⟩\langle\mathrm{img\_start}\rangle,\langle\mathrm{img\_end}\rangle delimiters), and Mtxt(x)=[T]∖Mimg(x)\mathcal{M}_{\mathrm{txt}}(x)=[T]\setminus\mathcal{M}_{\mathrm{img}}(x) collects the text-token positions.

Definition 3 (Paired Cross-Domain Distribution). Let Ss={(xsi,Ys∗i)}i=1NsS_s=\{(x_s^i,Y_s^{*i})\}_{i=1}^{N_s} be a labelled source dataset and St={xtj}j=1NtS_t=\{x_t^j\}_{j=1}^{N_t} an unlabelled target dataset. The paired cross-domain distribution Ppaired\mathcal{P}_{\mathrm{paired}} is the uniform distribution over scene-matched pairs (xs,xt)(x_s,x_t), where each target input xt∈Stx_t\in S_t is matched with its source nearest neighbour xs∈Ssx_s\in S_s under a task-relevant content vector k(⋅)∈R3\mathbf{k}(\cdot)\in\mathbb{R}^{3}:

Ppaired=Uniform⁡({(xs,xt):xt∈St, xs=arg⁡min⁡x∈Ss ∥k(x)−k(xt)∥2}).(3)\mathcal{P}_{\mathrm{paired}}=\operatorname{Uniform}\left(\left\{(x_s,x_t):x_t\in S_t,\ x_s=\underset{x\in S_s}{\arg\min}\ \left\|\mathbf{k}(x)-\mathbf{k}(x_t)\right\|_2\right\}\right). \tag*{(3)}

Each pair represents the same driving scenario in two cities, isolating domain shift from content shift.

Definition 4 (Domain Shift Gap). The domain shift gap G(f)G(f) measures how much ff’s predicted trajectory differs between the source and target inputs of a paired sample. Concretely, given a scene-matched pair (xs,xt)∼Ppaired(x_s,x_t)\sim\mathcal{P}_{\mathrm{paired}}, ff is run on each input to produce two predicted trajectories, f(xs)f(x_s) and f(xt)f(x_t). G(f)G(f) is the averaged ℓ2\ell_2 distance between the two trajectories over the KK waypoints of all scene-matched pairs in Ppaired\mathcal{P}_{\mathrm{paired}}:

G(f)=E(xs,xt)∼Ppaired[1K∑k=1K∥f(xt)k−f(xs)k∥2]∈R≥0.(4)G(f)=\mathbb{E}_{(x_s,x_t)\sim\mathcal{P}_{\mathrm{paired}}}\left[\frac{1}{K}\sum_{k=1}^{K}\left\|f(x_t)_k-f(x_s)_k\right\|_2\right]\in\mathbb{R}_{\ge0}. \tag*{(4)}

where f(x)k=(uk,vk)∈R2f(x)_k=(u_k,v_k)\in\mathbb{R}^{2} is the kk-th predicted waypoint of ff on input xx. G(f)=0G(f)=0 means ff predicts the same trajectory on both inputs of every pair.

Definition 5 (Modality–Layer Tensor). The modality–layer tensor Ψ\Psi tells us which (m,l)(m,l) slices of the planner’s hidden state are responsible for the cross-domain gap: each entry Ψ[m,l]\Psi[m,l] measures how much slice (m,l)(m,l) contributes to the gap. Formally, Ψ∈R≥02×L\Psi\in\mathbb{R}_{\ge0}^{2\times L} is a non-negative 2×L2\times L grid with rows indexed by modality m∈{img,txt}m\in\{\mathrm{img},\mathrm{txt}\} and columns by layer l∈{0,…,L−1}l\in\{0,\ldots,L-1\}. The (m,l)(m,l)-entry is the expected per-waypoint ℓ2\ell_2 displacement of fΘ(xt)f_{\Theta}(x_t) when the modality-mm block at layer ll is replaced with the source counterpart from a scene-matched pair (xs,xt)∼Ppaired(x_s,x_t)\sim\mathcal{P}_{\mathrm{paired}}:

Ψ[m,l]=E(xs,xt)∼Ppaired[1K∑k=1K∥fΘ(do⁡(HMm(l)(xt)←HMm(l)(xs));xt)k−fΘ(xt)k∥2],(5)\Psi[m,l]=\mathbb{E}_{(x_s,x_t)\sim\mathcal{P}_{\mathrm{paired}}}\left[\frac{1}{K}\sum_{k=1}^{K}\left\|f_{\Theta}\left(\operatorname{do}\left(\mathbf{H}^{(l)}_{\mathcal{M}_m}(x_t)\leftarrow\mathbf{H}^{(l)}_{\mathcal{M}_m}(x_s)\right);x_t\right)_k-f_{\Theta}(x_t)_k\right\|_2\right], \tag*{(5)}

where do⁡(⋅)\operatorname{do}(\cdot) overwrites the modality-mm rows of H(l)(xt)\mathbf{H}^{(l)}(x_t) with those of H(l)(xs)\mathbf{H}^{(l)}(x_s) and the forward pass continues from layer l+1l+1. A large Ψ[m,l]\Psi[m,l] marks the slice as a causal driver of the cross-domain gap; a near-zero entry marks it as domain-invariant. The empirical estimator is given in §4.1.

Problem Statement

Our task is to locate and repair the domain shift gap GG of a VLM trajectory planner. Concretely, given a source-trained planner fΘ′f_{\Theta'}, we first use the modality–layer tensor Ψ\Psi to derive a partition Φ=(Φalign,Φtune,Φfreeze)\Phi= (\Phi_{\mathrm{align}}, \Phi_{\mathrm{tune}}, \Phi_{\mathrm{freeze}}) of the LL backbone layers into disjoint zones (the concrete derivation is the CIS estimator in § 4.1). Each per-layer update Δl\Delta_l takes a zone-specific form determined by Φ\Phi:

Δl={δθlif l∈Φalign(rank-r LoRA, attention only),BlAlif l∈Φtune(rank-r LoRA, all projections),0if l∈Φfreeze(frozen).(6)\Delta_l = \begin{cases} \delta\theta_l & \text{if } l \in\Phi_{\mathrm{align}} \quad\text{(rank-}r\text{ LoRA, attention only)}, \\ B_l A_l & \text{if } l \in\Phi_{\mathrm{tune}} \quad\text{(rank-}r\text{ LoRA, all projections)}, \\ 0 & \text{if } l \in\Phi_{\mathrm{freeze}} \quad\text{(frozen)}. \end{cases} \tag*{(6)}

We then seek Δ={Δl}l=0L−1\Delta= \{\Delta_l\}_{l=0}^{L-1} that minimises GG subject to bounded source-side regression:

Δ∗=arg⁡min⁡Δ  G(fΘ′+Δ)s.t.Es(fΘ′+Δ)≤Es(fΘ′)+ϵ,(7)\Delta^* = \underset{\Delta}{\arg\min}\; G(f_{\Theta'}+\Delta) \quad\text{s.t.} \quad E_s(f_{\Theta'}+\Delta) \le E_s(f_{\Theta'}) + \epsilon, \tag*{(7)}

so fΘ′+Δf_{\Theta'}+\Delta is the adapted planner. Here, fΘ′f_{\Theta'} denotes the VLM planner after fine-tuning (Appendix B); δθl\delta\theta_l is an additive update on θl\theta_l and BlAlB_lA_l is a rank-rr adapter (concrete operators in § 4.2); Es(f):=Es[E(f(x),Y∗)]E_s(f) := \mathbb{E}_{s}[E(f(x), Y^*)] is the source-side average per-sample trajectory error; and ϵ≥0\epsilon\ge0 bounds how much the adapted planner’s source-side error may grow over fΘ′f_{\Theta'}’s.

Method

MoLaRx solves Eq. (7) via two modules (Figure 2): CIS (§ 4.1) locates the (m,l)(m,l) sources of the gap and partitions layers into zones Φ\Phi; CISR (§ 4.2) repairs each zone with a matched operator (MMD / LoRA / freeze).

Image not extracted

Figure 2. MoLaRx overview. CIS locates cross-city shift by modality and layer; CISR applies matched repair operators to the diagnosed zones.

Causal Importance Score (CIS)

We locate which (m,l)(m,l) slices of the VLM drive the domain shift gap via activation patching [23, 8]: for each (m,l)(m,l), a Monte Carlo average over N=100N=100 scenematched pairs yields one entry of the empirical tensor Ψ^\widehat{\Psi}, namely the CIS of slice (m,l)(m,l) given by Eq. (9). Hoeffding’s inequality (Eq. (10)) bounds its deviation from the population value Ψ[m,l]\Psi[m,l] (Eq. (5)). Thresholding Ψ^\widehat{\Psi} at τimg,τtxt\tau_{\mathrm{img}},\tau_{\mathrm{txt}} then partitions the LL layers into zones Φ\Phi (Eqs. (11) to (13)).

Computing one entry Ψ^[m,l]\widehat{\Psi}[m,l]. For each pair (xsi,xti)∼Ppaired(x_s^i,x_t^i) \sim P_{\mathrm{paired}} we do the following four steps:

  • (a) Forward both inputs. At layer ll, run fΘ′f_{\Theta'} on xsix_s^i and xtix_t^i to obtain the source and target hidden states H(l)(xsi),H(l)(xti)∈RT×d\mathbf{H}^{(l)}(x_s^i),\mathbf{H}^{(l)}(x_t^i) \in\mathbb{R}^{T \times d}.

  • (b) Patch the modality-mm block at layer ll on the target. Read off the modality-mm token positions Mm(xti)⊆[T]\mathcal{M}_m(x_t^i) \subseteq[T] in the target (image-patch positions when m=imgm=\mathrm{img}; text-token positions when m=txtm=\mathrm{txt}), and overwrite only those rows of H(l)(xti)\mathbf{H}^{(l)}(x_t^i) with the corresponding rows of H(l)(xsi)\mathbf{H}^{(l)}(x_s^i), leaving the other modality’s rows untouched.

  • (c) Resume the forward pass. Continue from layer l+1l+1 through the rest of the decoder on the patched hidden state, yielding the counterfactual trajectory Y~(l,m)(xsi,xti)\widetilde{\mathbf{Y}}^{(l,m)}(x_s^i,x_t^i).

  • (d) Per-pair displacement. Compare Y~(l,m)\widetilde{\mathbf{Y}}^{(l,m)} to the un-patched target prediction fΘ′(xti)f_{\Theta'}(x_t^i) by the per-waypoint ℓ2\ell_2 distance averaged over the K=6K=6 waypoints (in metres),

di(l,m)=1K∑k=1K∥Y~(l,m)(xsi,xti)k−fΘ′(xti)k∥2.(8)d_i^{(l,m)} = \frac{1}{K}\sum_{k=1}^{K}\left\|\widetilde{\mathbf{Y}}^{(l,m)}(x_s^i,x_t^i)_k-f_{\Theta'}(x_t^i)_k\right\|_2. \tag*{(8)}

Averaging the per-pair displacements di(l,m)d_i^{(l,m)} over the NN pairs gives the unbiased Monte Carlo estimator of Ψ[m,l]\Psi[m,l]:

Ψ^[m,l]=1N∑i=1Ndi(l,m)=1N∑i=1N1K∑k=1K∥Y~(l,m)(xsi,xti)k−fΘ′(xti)k∥2.(9)\widehat{\Psi}[m,l] = \frac{1}{N}\sum_{i=1}^{N}d_i^{(l,m)} = \frac{1}{N}\sum_{i=1}^{N}\frac{1}{K}\sum_{k=1}^{K}\left\|\widetilde{\mathbf{Y}}^{(l,m)}(x_s^i,x_t^i)_k-f_{\Theta'}(x_t^i)_k\right\|_2. \tag*{(9)}

Each summand di(l,m)d_i^{(l,m)} is bounded above by the maximum per-waypoint distance RR between any two predicted trajectories, so Hoeffding’s inequality [15] gives

Pr⁡[∣Ψ^[m,l]−Ψ[m,l]∣>ε]≤2exp⁡(−2Nε2R2),(10)\Pr\left[\left|\widehat{\Psi}[m,l]-\Psi[m,l]\right|>\varepsilon\right]\leq2\exp\left(-\frac{2N\varepsilon^2}{R^2}\right), \tag*{(10)}

where ε\varepsilon is the deviation tolerance. Setting the failure probability to η=0.05\eta=0.05 and inverting Eq. (10) for our values N=100N=100 and R≈20R\approx20 m yields a 95% half-width of ε≈0.6\varepsilon\approx0.6 m on each entry, well below the 0 to 2.3 m spread of Ψ^\widehat{\Psi} values across zones, so the partition below is statistically reliable (empirical NN-budget sweep validating this choice in Appendix § 10.2.4).

From tensor to partition. Repeating this procedure for every (m,l)∈{img,txt}×{0,…,L−1}(m,l)\in\{\mathrm{img},\mathrm{txt}\}\times\{0,\ldots,L-1\} produces the tensor Ψ^∈R≥02×L\widehat{\Psi}\in\mathbb{R}_{\geq0}^{2\times L}, whose two rows record the per-modality causal profile across depth. To turn this profile into a repair plan, we partition the LL layers by which modality dominates: layers where the image entry is large form the alignment zone Φalign\Phi_{\mathrm{align}}, those where the text entry is large form the fine-tuning zone Φtune\Phi_{\mathrm{tune}}, and the rest form the freeze zone Φfreeze\Phi_{\mathrm{freeze}}:

Φalign={l∈{0,…,L−1}:Ψ^[img,l]>τimg},(11)\Phi_{\mathrm{align}}=\left\{l\in\{0,\ldots,L-1\}:\widehat{\Psi}[\mathrm{img},l]>\tau_{\mathrm{img}}\right\}, \tag*{(11)}
Φtune={l:Ψ^[txt,l]>τtxt}∖Φalign,(12)\Phi_{\mathrm{tune}}=\left\{l:\widehat{\Psi}[\mathrm{txt},l]>\tau_{\mathrm{txt}}\right\}\setminus\Phi_{\mathrm{align}}, \tag*{(12)}
Φfreeze={0,…,L−1}∖(Φalign∪Φtune),(13)\Phi_{\mathrm{freeze}}=\{0,\ldots,L-1\}\setminus\left(\Phi_{\mathrm{align}}\cup\Phi_{\mathrm{tune}}\right), \tag*{(13)}

where τimg,τtxt\tau_{\mathrm{img}},\tau_{\mathrm{txt}} are per-modality thresholds taken as the elbow of the sorted CIS values (sensitivity in Appendix § 10.2.1), and ∖\setminus denotes set difference.

CISR

Given the CIS-induced partition Φ=(Φalign,Φtune,Φfreeze)\Phi=(\Phi_{\mathrm{align}},\Phi_{\mathrm{tune}},\Phi_{\mathrm{freeze}}), CISR assigns a matched operator to each zone (MMD on Φalign\Phi_{\mathrm{align}}, LoRA on Φtune\Phi_{\mathrm{tune}}, freeze on Φfreeze\Phi_{\mathrm{freeze}}) and learns the parameter updates Δ\Delta.

Operator on Φalign\Phi_{\mathrm{align}}: multi-scale MMD alignment. For each l∈Φalignl\in\Phi_{\mathrm{align}} we minimise the Maximum Mean Discrepancy [13] between source and target image-token mean-pooled representations H‾img(B;Θ′)\overline{\mathbf{H}}_{\mathrm{img}}(B;\Theta') (full definition in Appendix § 11.§ 1) on paired minibatches Bs,Bt\mathcal{B}_s,\mathcal{B}_t of size B. Summing over the zone gives the image-alignment loss:

LMMD(Θ′;Bs,Bt)=1∣Φalign∣∑l∈ΦalignMMD⁡2(Hˉimg(l)(Bs;Θ′),Hˉimg(l)(Bt;Θ′);k)(14)\mathcal{L}_{\mathrm{MMD}}(\Theta'; \mathcal{B}_s, \mathcal{B}_t) = \frac{1}{|\Phi_{\mathrm{align}}|} \sum_{l \in\Phi_{\mathrm{align}}} \operatorname{MMD}^{2}\left(\bar{H}_{\mathrm{img}}^{(l)}(\mathcal{B}_s; \Theta'), \bar{H}_{\mathrm{img}}^{(l)}(\mathcal{B}_t; \Theta'); k\right) \tag*{(14)}

where kk is the multi-scale RBF kernel and ∣Φalign∣|\Phi_{\mathrm{align}}| normalises across the zone. On each l∈Φalignl \in\Phi_{\mathrm{align}} we attach rank-rr LoRA adapters to the attention projections only (Q, K, V, O) — the cross-modal mixing locus implicated by F1 — driven by LMMD\mathcal{L}_{\mathrm{MMD}}, instantiating δθl\delta\theta_l of Eq. (6).

Operator on Φtune\Phi_{\mathrm{tune}}: selective LoRA retraining. For each l∈Φtunel \in\Phi_{\mathrm{tune}} we attach rank-r=16r=16 LoRA adapters [17] (α=32\alpha=32) to every linear projection (factorisation in Appendix § 11.§ 2). The adapters are driven by the standard supervised trajectory loss on the labelled source dataset:

Ltask(Θ′,Δ)=−1∣Ss∣∑(x,y)∈Ss∑t=1Tylog⁡pΘ′+Δ(yt∣x,y<t),(15)\mathcal{L}_{\mathrm{task}}(\Theta', \Delta) = -\frac{1}{|\mathcal{S}_s|} \sum_{(x,y)\in\mathcal{S}_s} \sum_{t=1}^{T_y} \log p_{\Theta' + \Delta}(y_t \mid x, y_{<t}), \tag*{(15)}

where TyT_y is the trajectory token length, y<ty_{<t} the previously decoded tokens, and {Al,Bl}l∈Φtune\{A_l, B_l\}_{l\in\Phi_{\mathrm{tune}}} are the rank-rr LoRA factors that form the Φtune\Phi_{\mathrm{tune}} component of Δ\Delta (see Eq. (6)).

Operator on Φfreeze\Phi_{\mathrm{freeze}}: zero gradient. For every l∈Φfreezel \in\Phi_{\mathrm{freeze}} we attach no adapter, so Δl=0\Delta_l = 0.

Training Loss. Combining task supervision (LoRA on Φtune\Phi_{\mathrm{tune}}) and MMD alignment (on Φalign\Phi_{\mathrm{align}}), with a zero-gradient constraint on Φfreeze\Phi_{\mathrm{freeze}}:

Ltotal(Θ′,Δ)=Ltask(Θ′,Δ)⏟updates LoRA on Φtune+λMMD⋅LMMD(Θ′;Bs,Bt)⏟updates attention LoRA on Φalign(16)\mathcal{L}_{\mathrm{total}}(\Theta', \Delta) = \underbrace{\mathcal{L}_{\mathrm{task}}(\Theta', \Delta)}_{\text{updates LoRA on }\Phi_{\mathrm{tune}}} + \lambda_{\mathrm{MMD}} \cdot\underbrace{\mathcal{L}_{\mathrm{MMD}}(\Theta'; \mathcal{B}_s, \mathcal{B}_t)}_{\text{updates attention LoRA on }\Phi_{\mathrm{align}}} \tag*{(16)}

where λMMD\lambda_{\mathrm{MMD}} trades off task supervision against domain alignment (λMMD=0.5\lambda_{\mathrm{MMD}}=0.5, sensitivity in Appendix § 10.2.2). Gradients are routed by zone (Appendix § 11.§ 3); target-domain samples enter only the kernel term of LMMD\mathcal{L}_{\mathrm{MMD}}, so CISR is a valid unsupervised DA method. When Φtune=∅\Phi_{\mathrm{tune}} = \varnothing (such as PaliGemma2), the LoRA operator has nothing to attach to and Eq. (16) automatically degenerates to MMD-on-Φalign\Phi_{\mathrm{align}} + freeze-on-Φfreeze\Phi_{\mathrm{freeze}}. The full pseudocode is in Appendix § 11.§ 4, Algorithm 1.

Experiments

Experimental Setting

Datasets. We evaluate on two cross-city trajectory-planning benchmarks under transductive unsupervised domain adaptation, using target-test inputs without their trajectory labels for adaptation. On nuScenes [6] we use Boston-Seaport (BS) as source and Singapore (SG) as target, with a stratified split yielding ∼\sim26.6k train, 3,569 source test, and 3,500 target test samples. On Argoverse 2 Sensor (AV2) [Wilson et al., 2023] we use Pittsburgh (PIT) as source and Miami (MIA) as target, extracting 858 train, 284 ID-test, and 1,393 OOD-test samples from 57 validation logs.

Models. We evaluate three VLMs spanning the backbone-operator and decoder-attention axes. LFM2-VL-1.6B [3] is a 16-layer hybrid backbone with a causal decoder; Qwen2-VL-2B [38] is a 28-layer pure-Transformer with a causal decoder; PaliGemma2-3B [5] pairs a SigLIP encoder with a 26-layer Gemma2 prefix-LM decoder using bidirectional image attention.

Evaluation Metrics. We report four metrics: ADE (↓\downarrow, m): mean per-waypoint ℓ2\ell_2 error; FDE (↓\downarrow, m): error at the final waypoint; ΔADE\Delta_{\mathrm{ADE}} (↓\downarrow, m): cross-domain gap ADE(St)−ADE(Ss)\mathrm{ADE}(S_t) - \mathrm{ADE}(S_s); and PE (↓\downarrow, ms): inference latency. The numbers are averaged over five random seeds.

Baselines. We compare against nine baselines in four categories: (1) no-adaptation reference: Source-only, the fine-tuned planner with no cross-domain repair, establishing the gap floor against all DA methods; (2) uniform / scalar-view DA, four representatives of the “align-everything” view we argue against: Full LoRA [17] (parameter-efficient fine-tuning (PEFT) only, no DA signal), Full LoRA + MMD [13] (PEFT plus uniform kernel matching), DANN [11] (adversarial domain-invariant alignment), and MMD-only [13] (alignment without task fine-tuning); (3) layer-selective, non-causal DA (LLRD [16]; Freeze-Last-K [19]), choosing layers via correlational or heuristic criteria; and (4) locate-then-edit family (**ROME⋆ [23]; Surgical FT [20]), applying single-locus or hand-picked-band repair. All adapted baselines share the same input format, backbone, and LoRA rank. See Appendix § 8.§ 1–§ 8.§ 4 for full details of the above settings. An inductive-protocol control is reported in App. § 10.§ 4.

Main Performance Evaluation

Performance evaluation results are reported in Table 1; qualitative case studies are found in Appendix § 10.§ 1; computational efficiency is in Appendix § 10.§ 3, Table 10.

Data-setVLMMetricSource-onlyF-LoRAF-LoRA + MMDDANNMMD onlyLLRDFreeze-KROME⋆Surg-FTMoLaRxΔ
nuScenes (BS→SG)LFM2ADE (↓, m)4.5104.2754.2844.2959.6574.2884.3104.2704.2664.2510.015
nuScenes (BS→SG)LFM2FDE (↓, m)7.8907.4807.5317.59016.5427.4757.5107.4707.4627.4220.040
nuScenes (BS→SG)LFM2ΔADE (↓, m)2.5503.3403.3283.3901.646‡3.3583.3153.2553.2053.1500.055⋆
nuScenes (BS→SG)LFM2PE (↓, ms)68.366.568.467.548.3‡66.565.565.558.347.410.9⋆
nuScenes (BS→SG)Qwen2ADE (↓, m)5.0204.7304.7604.74510.1564.7154.7254.7204.7004.6340.066⋆
nuScenes (BS→SG)Qwen2FDE (↓, m)8.7608.4308.4708.45018.1308.4268.4228.4148.3808.2420.138⋆
nuScenes (BS→SG)Qwen2ΔADE (↓, m)2.8502.9652.9852.9751.624‡2.9642.9602.9522.9352.9060.029
nuScenes (BS→SG)Qwen2PE (↓, ms)71.279.871.570.851.2‡79.878.868.861.250.510.7⋆
nuScenes (BS→SG)PaliGemma2ADE (↓, m)4.4504.6204.6504.63510.0544.5304.4954.3454.3204.1900.130⋆
nuScenes (BS→SG)PaliGemma2FDE (↓, m)7.6207.9808.0208.00017.4987.8407.7957.5307.4607.3550.105⋆
nuScenes (BS→SG)PaliGemma2ΔADE (↓, m)2.6302.5802.6102.5951.418‡2.5002.4852.4052.3852.3370.048
nuScenes (BS→SG)PaliGemma2PE (↓, ms)82.781.483.282.362.7‡81.490.572.770.562.18.4⋆
Argoverse 2 (PIT→MIA)LFM2ADE (↓, m)5.1204.7704.8004.78510.2704.6854.6604.5104.4854.4200.065⋆
Argoverse 2 (PIT→MIA)LFM2FDE (↓, m)8.6508.2508.2908.27017.5878.1108.0657.8157.7607.6800.080⋆
Argoverse 2 (PIT→MIA)LFM2ΔADE (↓, m)3.2522.1502.1802.1651.165‡2.0702.0501.9651.9451.8800.065⋆
Argoverse 2 (PIT→MIA)LFM2PE (↓, ms)78.376.578.477.548.3‡76.575.565.558.347.410.9⋆
Argoverse 2 (PIT→MIA)Qwen2ADE (↓, m)4.9535.1505.1805.16510.1795.0905.0815.0745.0454.9830.062⋆
Argoverse 2 (PIT→MIA)Qwen2FDE (↓, m)8.9608.9108.9658.94017.6408.8958.8158.7708.7308.6390.091⋆
Argoverse 2 (PIT→MIA)Qwen2ΔADE (↓, m)3.1411.3951.4251.4100.566‡1.3901.3731.3301.2951.2490.046
Argoverse 2 (PIT→MIA)Qwen2PE (↓, ms)71.279.871.570.551.2‡79.878.868.861.250.510.7⋆
Argoverse 2 (PIT→MIA)PaliGemma2ADE (↓, m)4.7154.7004.7304.7158.3494.6204.6054.5854.5604.5070.053⋆
Argoverse 2 (PIT→MIA)PaliGemma2FDE (↓, m)8.1308.0208.0658.04514.7007.9107.8957.8907.8607.7870.073⋆
Argoverse 2 (PIT→MIA)PaliGemma2ΔADE (↓, m)2.3152.5002.4302.4150.772‡2.3802.2652.1852.1552.1120.043
Argoverse 2 (PIT→MIA)PaliGemma2PE (↓, ms)82.781.483.282.382.7‡81.480.580.572.762.110.6⋆

Table 1. Performance across 2 datasets of 3 VLMs on four metrics. Bold marks the best result per row among the adaptation methods; underline marks the runner-up. Source-only is reported as the no-adaptation reference and is excluded from the bold/underline ranking. The rightmost column Δ is MoLaRx’s lead over the underlined runner-up. The marker ⋆ flags margins exceeding the 2σ significance threshold. MMD-only is marked with ‡ because the model has collapsed.

Table 10: Compute footprint per VLM. “% saved” is the parameter saving over each VLM’s Full-LoRA baseline. Not shown: a one-time CIS-diagnosis cost of ∼0.65\sim0.65 h per architecture (App. § 10.2.4), reused zero-shot across the two benchmarks (Table 3). † On PaliGemma2 the prefix-LM attention yields Φtune=∅\Phi_{\mathrm{tune}}=\varnothing, so the recipe collapses to MMD-driven attention-only LoRA on Φalign\Phi_{\mathrm{align}} + freeze on Φfreeze\Phi_{\mathrm{freeze}}. MoLaRx saves 50–60% of trainable parameters across all three architectures; the largest absolute saving (7.5 M, 60%) is on PaliGemma2, where attention-only LoRA on Φalign\Phi_{\mathrm{align}} avoids the MLP-projection bulk that Full-LoRA pays.

VLMMethodTrain. params% savedTrain (h/ep)Inference (ms/sample)
LFM2-VL-1.6BFull-LoRA7.6 M0%4.2 (±0.05)66.5 (±0.8)
MoLaRx (CISR)3.8 M50%4.1 (±0.04)47.4 (±0.6)
Qwen2-VL-2BFull-LoRA8.9 M0%4.6 (±0.06)79.8 (±1.1)
MoLaRx (CISR)4.1 M54%4.5 (±0.05)50.5 (±0.7)
PaliGemma2-3BFull-LoRA12.5 M0%5.4 (±0.07)81.4 (±1.0)
MoLaRx (CISR)†5.0 M60%5.3 (±0.06)62.1 (±0.9)

Table 10.

Effectiveness Analysis. We summarize four key observations. 1) MoLaRx is the best-performing adaptation method on all cells, outperforming the strongest non-MoLaRx baseline by 0.015–0.138 m on quality metrics (ADE, FDE, ΔADE), with margins exceeding the 2σ significance threshold on most cells. 2) Gains scale with the size of the modality–layer structure: largest on PaliGemma2 on nuScenes at ADE/FDE by +0.130 / +0.105 m, smallest on LFM2 by +0.015 / +0.040 m, consistent with our hypothesis that more layers expose more variance to non-causal partitioning. 3) The locate-then-edit family (ROME⋆, Surgical FT) outperforms uniform/scalar-view DA on most cells, confirming that causal diagnosis is necessary even under single-locus edits. 4) Uniform alignment fails on this structured shift: MMD-only doubles ADE (LFM2/nuScenes: 4.275 → 9.657 m), and even Full LoRA + MMD is worse than Full LoRA alone on 17 of 18 quality cells by 0.01–0.09 m — “align-everything” overwrites task-invariant computation in Φfreeze\Phi_{\mathrm{freeze}} while leaving modality-specific errors untouched. By contrast, CIS-guided selective repair targets shift only where it occurs. Additional evaluations on KITScenes-LongTail and planning-oriented proxy metrics are reported in App. § 10.§ 5.

Efficiency Analysis. MoLaRx achieves the lowest inference latency at 47.4–62.1 ms per sample, 8.4–10.9 ms faster than the strongest LoRA-based baselines. Here two factors contribute: 1) the CIS-induced partition concentrates LoRA on Φtune\Phi_{\mathrm{tune}} only, reducing forward overhead vs. methods that apply LoRA on every layer; 2) the CIS partition is diagnosed once per architecture and reused across datasets without recalibration (Table 3). Training times and parameters are in Appendix Table 10.

VLMCond.ADEID\mathrm{ADE}_{\mathrm{ID}}ADEOOD\mathrm{ADE}_{\mathrm{OOD}}ΔADE\Delta_{\mathrm{ADE}}
LFM2(A) AV2-local1.43044.08122.6508
(B) Zero-shot1.43184.09212.6603
(C) Source-only1.87325.12473.2515
Qwen2(A) AV2-local1.58214.21532.6332
(B) Zero-shot1.58764.24012.6525
(C) Source-only1.81234.95343.1411

Table 3. Zero-shot partition transfer.

The Modality-Layer Decomposition is Robust

Per-layer CIS atlas across three VLMs. Figure 3 reports per-layer CISimg\mathrm{CIS}_{\mathrm{img}} (Ψ^[img,⋅]\widehat{\Psi}[\mathrm{img}, \cdot]) and CIStxt\mathrm{CIS}_{\mathrm{txt}} (Ψ^[txt,⋅]\widehat{\Psi}[\mathrm{txt}, \cdot]) on nuScenes for the three VLMs. We find that: (F1) Modality dominance is layer-dependent, set by attention connectivity. In LFM2-VL and Qwen2-VL (Figure 3 (a) and (b)), CISimg\mathrm{CIS}_{\mathrm{img}} peaks at layers 0–2 and quickly decays, while CIStxt\mathrm{CIS}_{\mathrm{txt}} rises mid-depth and dominates a contiguous band. In contrast, in PaliGemma2 (Figure 3 (c)), CISimg\mathrm{CIS}_{\mathrm{img}} dominates across all layers. This is because causal attention in LFM2 and Qwen2-VL can resolve cross-modal interactions early, leading to a clear depth separation between image-dominant and text-dominant layers. In contrast, under prefix-LM attention, PaliGemma2 continuously re-injects domain-shifted features. (F2) Zone partition is architecture-dependent. We obtain (Φalign,Φtune,Φfreeze)=({0,1},{2:9},{10:15})(\Phi_{\mathrm{align}}, \Phi_{\mathrm{tune}}, \Phi_{\mathrm{freeze}}) = (\{0, 1\}, \{2\mathbin{:}9\}, \{10\mathbin{:}15\}) for LFM2-VL, ({0:3},{4:10},{11:27})(\{0\mathbin{:}3\}, \{4\mathbin{:}10\}, \{11\mathbin{:}27\}) for Qwen2-VL, and ({0:24},∅,{25})(\{0\mathbin{:}24\}, \varnothing, \{25\}) for PaliGemma2 — direct evidence that a locate-then-repair recipe matched to this structure is necessary.

CIS atlas across three VLMs

Figure 3. CIS atlas across three VLMs (nuScenes).

Partition Consistency Score (PCS). We test partition stability along two axes: within-architecture (same VLM, nuScenes vs. AV2) and cross-architecture (different VLMs, normalized to a Q=16Q=16 shared-depth grid). PCS is the per-zone Jaccard, with PCS=1 for identical partitions and chance ≈0.18\approx0.18 (App. § 8.§ 3). As shown in Table 2, 1) Within-architecture, PCS reaches 1.0 on LFM2-VL/PaliGemma2 and ≥0.9\ge0.9 on Qwen2-VL, far above chance (0.18), indicating the partition is a property of the model. 2) For cross-architecture, on the overlapping zones (Φalign,Φfreeze)(\Phi_{\mathrm{align}}, \Phi_{\mathrm{freeze}}) PCS is 0.71–0.81, also well above chance, indicating a shared inductive bias of VLM decoders; only the middle Φtune\Phi_{\mathrm{tune}} diverges, since it disappears under prefix-LM attention.

TypeComparisonPCS(Φalign)\mathrm{PCS}(\Phi_{\mathrm{align}})PCS(Φtune)\mathrm{PCS}(\Phi_{\mathrm{tune}})PCS(Φfreeze)\mathrm{PCS}(\Phi_{\mathrm{freeze}})Mean
Within-arch. (cross-dataset)LFM2: nuScenes vs. AV21.001.001.001.00
Qwen2: nuScenes vs. AV20.880.930.960.92
PaliGemma2: nuScenes vs. AV21.001.001.001.00
Cross-arch. (normalized depth, nuScenes)LFM2 vs. Qwen20.750.680.810.75
LFM2 vs. PaliGemma20.71−0.77−
Qwen2 vs. PaliGemma20.72−0.74−
Random partition ↔ random partition (chance)0.140.250.140.18

Table 2. Partition Consistency Score.

Operational partition transfer. Table 3 tests whether the entire CIS-diagnosis pipeline can be amortized across datasets. LFM2 and Qwen2 are tested on target AV2 (PIT→MIA) under three conditions (A) AV2-local: CIS diagnosed on AV2 (ideal per-dataset baseline); (B) Zero-shot: CIS diagnosed only on nuScenes; partition, τ\tau, and all hyperparameters re-used on AV2 unchanged; (C) Source-only: Phase-0 reference, no adaptation. On both VLMs, zero-shot (B) matches AV2-local (A) within 0.01 m (LFM2: ΔADE\Delta_{\mathrm{ADE}} 2.6603 vs. 2.6508) and 0.02 m (Qwen2:

2.6525 vs. 2.6332), and decisively beats source-only (C) by 0.5–0.6 m. This is because CISR’s three operators (MMD / LoRA / freeze) consume only which layers each targets, not the continuous CIS magnitudes. In conclusion, a CIS diagnosis on nuScenes transfers to AV2 without re-diagnosis in both tested VLMs. Reverse-direction experiments further examine partition stability and repair performance under reversed city pairs (App. C.6).

Ablation Study

Table 4 reports ablations on three (VLM, nuScenes) cells: partition substitutes (random and DPA-derived), single zone-operator removals, and a Φalign↔Φtune\Phi_{\mathrm{align}} \leftrightarrow\Phi_{\mathrm{tune}} operator swap. Replacing CIS with random or DPA-derived partitions inflates ΔADE\Delta_{\mathrm{ADE}} by 0.2–0.4 m on every cell, showing that probing-style correlations cannot substitute for causal localization. Among zone operators, removing MMD or LoRA costs 0.2–0.4 m and the cost tracks F2 zone widths; w/o MMD on Φalign\Phi_{\mathrm{align}} hurts the most where Φalign\Phi_{\mathrm{align}} is widest (PaliGemma2 25/26, +0.34 m); w/o Freeze is essentially free (< 0.02 m) since those parameters are already near-optimal. Swapping operators is the largest single hit (+0.3–0.4 m on every cell), confirming that which operator targets which zone is as important as the partition itself. These results reveal complementary roles: CIS causally locates the modality–layer sources of shift, zone operators repair each with the matched mechanism.

VariantLFM2Qwen2PaliGemma2
MoLaRx (full)2.36872.62872.4521
w/o CIS2.72592.98512.8523
DPA-partition2.67422.86432.7234
w/o MMD on Φalign\Phi_{\mathrm{align}}2.61082.82142.7892
w/o LoRA on Φtune\Phi_{\mathrm{tune}}2.68212.8925–
w/o Freeze on Φfreeze\Phi_{\mathrm{freeze}}2.37142.65232.4612
Swap operators2.79122.94122.8923

Table 4. Ablation ΔADE↓\Delta_{\mathrm{ADE}} \downarrow (m) on nuScenes.

Conclusion and Limitations

We uncover a modality–layer structure of cross-city domain shift in VLM trajectory planners: image and text exhibit distinct layer-wise patterns, and the zone partition is architecture-dependent and dataset-stable. To expose this structure, we introduce CIS, an activation-patching diagnostic that measures how source-to-target interventions at each (modality, layer) slice causally affect predicted trajectories. Guided by CIS, MoLaRx applies a zone-matched operator composition (MMD / LoRA / freezing) that adapts automatically to each architecture. On three architecturally distinct VLMs across two cross-city benchmarks, MoLaRx attains the lowest cross-city trajectory-error gap among adaptation methods in all six main-table settings, while training 50–60% fewer parameters and running 8.4–10.9 ms faster. In contrast, uniform MMD-only adaptation collapses in all six settings, highlighting the importance of structure-guided repair. The modality–layer tensor view is task-agnostic in principle; empirically, we validate it only for cross-city trajectory planning on decoder-only and prefix-LM VLMs, while extensions to other tasks and shift types remain untested.

Limitations and future work. (i) Hard-threshold partition. The elbow-based partition can produce one or two boundary-layer flips on tightly-shifted benchmarks; future improvements include continuous operator weighting and a shift-magnitude-aware threshold that scales with measured cross-domain feature distance. (ii) Architecture and task scope. Generalisation to encoder–decoder VLMs (BLIP-2, PaLI), cross-attention fusion architectures, and non-driving tasks (captioning, VQA, dense prediction) with non-geographic shifts (weather, sensor, season) remains untested. (iii) Evaluation coverage. Our main evaluation focuses on nominal driving conditions and displacement metrics. Evaluation using official planning-oriented metrics remains future work. The preliminary long-tail probe shows high trajectory errors across methods, leaving robustness to long-tail scenarios unestablished. (iv) Method assumptions. CIS requires scene-matched source/target pairs and its estimates depend on pairing quality; the pairs are drawn predominantly from prevalent scenes, so shift components visible only in rare scenes may be missed, and estimating CIS on long-tail-stratified pairs is a natural extension. The three-zone structure is empirical with bootstrap stability but no theoretical universality guarantee.

Acknowledgments and Disclosure of Funding

This work has received funding from the European Union’s Horizon Europe research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 10126636 (DSTrain project), and was partly supported by the Norwegian Research Council under Grant 322473 (AirQMan project) and by the Technical Faculty of IT and Design at Aalborg University. The authors declare no competing interests.

Implementation Details

VLM Selection

Our three VLMs are not an arbitrary convenience sample. The 2024 driving-VLM landscape varies along two orthogonal axes, and an architectural generalisation claim must factor against both: (H1H1) backbone operator, LTC-hybrid (alternating convolution + attention, e.g. LFM2-VL) vs. pure-Transformer (e.g. Qwen2-VL); and (H2H2) decoder attention connectivity, causal-only (LFM2-VL, Qwen2-VL) vs. prefix-LM with bidirectional image attention (PaliGemma2). The three VLMs instantiate three corners of this 2×22 \times2 grid, so comparing LFM2 vs. Qwen2 varies only H1 at fixed H2, and comparing Qwen2 vs. PaliGemma2 varies only H2 at fixed H1. Our main architecture-level claim (§ 5.3) is that the zone count of the modality–layer decomposition is predicted by H2 (causal →\rightarrow 3 zones, prefix-LM →\rightarrow 2 zones) rather than H1, so both axes are required for the claim to be testable. Table 5 reports the per-VLM architecture summary and the CIS-induced partition used by MoLaRx, with each VLM stacked as its own block.

Experimental Setting

Table 6 reports the training and dataset configurations shared across the three VLMs. The CIS image thresholds τimg\tau_{\mathrm{img}} are chosen at the elbow of the sorted per-layer CISimg\mathrm{CIS}_{\mathrm{img}} curve for each (VLM, nuScenes) diagnosis; the same thresholds are then reused on Argoverse 2 without recalibration (Finding 2, Table 3). The per-architecture zone boundaries above are those discovered by CIS on nuScenes and, due to per-architecture invariance, are identical when the diagnostic is rerun on Argoverse 2 for LFM2-VL and PaliGemma2 and differ by at most one layer for Qwen2-VL.

Target-data access. All methods are trained on labelled source data only; target trajectories are never read during adaptation. Unlabelled target inputs enter at two points: (i) CIS draws the N=100N = 100 scene-matched source/target pairs (Definition 3) from the target pool, using images and instructions only; (ii) the MMD term of CISR, Full LoRA + MMD, and MMD-only, and the domain classifier of DANN, sample unlabelled target mini-batches from the same pool on the same schedule. In the main-table experiments this pool is the target test split (3,500 nuScenes-SG and 1,393 AV2-MIA samples), i.e. a transductive setting. Source-only, Full LoRA, LLRD, Freeze-Last-K, ROMEO*, and Surgical FT use no target data. An inductive control with a disjoint adaptation pool is reported in Appendix § 10.§ 4.

Evaluation Metrics

Let Y^=(y^1,…,y^K)∈RK×2\hat{\mathbf{Y}}=(\hat{\mathbf{y}}_1,\ldots,\hat{\mathbf{y}}_K)\in\mathbb{R}^{K\times2} and Y∗=(y1∗,…,yK∗)\mathbf{Y}^{*}=(\mathbf{y}^{*}_1,\ldots,\mathbf{y}^{*}_K) denote the predicted and ground-truth K=6K=6 waypoints (in ego metres). Ss\mathcal{S}_s and St\mathcal{S}_t denote the source and target distributions.

Average Displacement Error (ADE) measures the mean per-waypoint ℓ2\ell_2 error between the predicted and ground-truth trajectories:

ADE=1K∑k=1K∥y^k−yk∗∥2,(17)\mathrm{ADE}=\frac{1}{K}\sum_{k=1}^{K}\left\lVert\hat{\mathbf{y}}_k-\mathbf{y}^{*}_k\right\rVert_2, \tag*{(17)}

where KK is the number of forecast waypoints. A lower ADE indicates better trajectory accuracy.

Final Displacement Error (FDE) measures the ℓ2\ell_2 error at the final forecast waypoint:

FDE=∥y^K−yK∗∥2,(18)\mathrm{FDE}=\left\lVert\hat{\mathbf{y}}_K-\mathbf{y}^{*}_K\right\rVert_2, \tag*{(18)}

where yK∗\mathbf{y}^{*}_K is the ground-truth waypoint at the longest horizon. FDE penalizes errors at the trajectory tail more heavily than ADE.

Cross-Domain ADE Gap (ΔADE\Delta_{\mathrm{ADE}}) measures the dataset-level ADE difference between the target and the source distributions, our primary domain-adaptation metric:

ΔADE(f)=ADESt(f)−ADESs(f),(19)\Delta_{\mathrm{ADE}}(f)=\mathrm{ADE}_{\mathcal{S}_t}(f)-\mathrm{ADE}_{\mathcal{S}_s}(f), \tag*{(19)}

where ADES(f)=E(x,Y∗)∼S[ADE(f(x),Y∗)]\mathrm{ADE}_{\mathcal{S}}(f)=\mathbb{E}_{(\mathbf{x},\mathbf{Y}^{*})\sim\mathcal{S}}[\mathrm{ADE}(f(\mathbf{x}),\mathbf{Y}^{*})] is the expected ADE of planner ff on distribution S\mathcal{S}. A perfectly domain-invariant planner has ΔADE=0\Delta_{\mathrm{ADE}}=0.

CategorySetting
Datasets
nuScenes sourceBoston-Seaport (700 scenes)
nuScenes targetSingapore (150 scenes)
nuScenes keyframes∼\sim26,600 train / 3,569 ID / 3,500 OOD
Argoverse 2 sourcePittsburgh (700 logs)
Argoverse 2 targetMiami (150 logs)
Argoverse 2 sampling2 Hz to match nuScenes keyframe density
Hardware Configuration
GPUsingle NVIDIA A10 (22 GB)
Precisionbfloat16
Software stacktransformers + peft + bitsandbytes
Gradient checkpointingPaliGemma2 only
Peak memory (LFM2 / Qwen2 / Pali)8.1 / 10.3 / 14.6 GB
Fine-tune Configuration
Epochs3
Batch size (per device)1
Gradient accumulation steps1
OptimizerAdamW (8-bit on PaliGemma2, fp32 elsewhere)
Learning rate2×10−52 \times10^{-5}
Weight decay0.01
Warmup ratio0.10
LoRA rank rr16
LoRA scaling α\alpha32
LoRA dropout0.05
LoRA target modulesall linear projections in attention and MLP
Temporal context3 consecutive front-camera frames
Trajectory horizon (future)3 s (6 waypoints at 2 Hz)
Image resolutionnative processor size per VLM
CIS Diagnosis
Pairs NN100 scene-matched source/target pairs
Pair matching keysego speed, following distance, lead velocity
τimg\tau_{\mathrm{img}} selectionelbow of sorted CISimg\mathrm{CIS}_{\mathrm{img}} curve
τtxt\tau_{\mathrm{txt}} selectionelbow of sorted CIStxt\mathrm{CIS}_{\mathrm{txt}} curve
CISR Configuration
Epochs1
Optimizer / LR / LoRA settingsidentical to the fine-tuning recipe (App. B)
MMD loss weight λMMD\lambda_{\mathrm{MMD}}0.5
MMD kernelmulti-scale RBF, bandwidths {0.1,1.0,10.0}\{0.1, 1.0, 10.0\}
Operator compositionpartition-determined
Random seeds{0,1,2,3,4}\{0, 1, 2, 3, 4\} (five seeds for all reported numbers)
Eval samples per split300 (ID, OOD, ID-blank, OOD-blank)

Table 6. Experiment settings and hyperparameters for MoLaRx.

Planning Efficiency (PE) measures the per-sample inference latency in milliseconds:

PE=1N∑i=1Nti,(20)\mathrm{PE} = \frac{1}{N}\sum_{i=1}^{N} t_i, \tag*{(20)}

where tit_i is the wall-clock time for the ii-th forward pass and NN is the number of evaluation samples. PE depends on the VLM and the adaptation method but is approximately invariant to the dataset.

Partition Consistency Score (PCS) is the zone-wise Jaccard similarity between two CIS-induced zone partitions:

PCS(Φ,Φ′)=∣Φ∩Φ′∣∣Φ∪Φ′∣∈[0,1],(21)\mathrm{PCS}(\Phi,\Phi') = \frac{\lvert\Phi\cap\Phi' \rvert}{\lvert\Phi\cup\Phi' \rvert} \in[0,1], \tag*{(21)}

where Φ\Phi and Φ′\Phi' are zone sets of the same type (Φalign\Phi_{\mathrm{align}}, Φtune\Phi_{\mathrm{tune}}, or Φfreeze\Phi_{\mathrm{freeze}}) computed on two different (VLM, dataset) cells. PCS=1 indicates the two partitions assign exactly the same layers to that zone.

We use PCS along two axes (Table 2). (i) Within-architecture (cross-dataset): same VLM, partitions diagnosed on nuScenes vs. Argoverse 2; layer indices are directly comparable, so Eq. (21) is applied to the raw zone sets. (ii) Cross-architecture: different VLMs with different decoder depths L∈{16,28,26}L \in\{16, 28, 26\}; before computing Jaccard we normalise each partition to a shared Q=16Q=16 grid by mapping layer index l∈{0,…,L−1}l \in\{0,\ldots,L-1\} to ⌊l⋅Q/L⌋\lfloor l \cdot Q/L \rfloor, so zones of equal relative depth align across architectures. The chance level for size-matched random partitions on a length-QQ grid is ≈0.18\approx0.18 (averaged over Φalign\Phi_{\mathrm{align}}, Φtune\Phi_{\mathrm{tune}}, Φfreeze\Phi_{\mathrm{freeze}} at the partition sizes observed in our three VLMs).

Image Dependency (ImgDep) measures the relative ADE inflation when all camera frames are replaced by a constant grey image, a sanity check that the planner actually uses the visual channel:

ImgDep(f)=Es(f∘blank)−Es(f)Es(f),(22)\mathrm{ImgDep}(f) = \frac{E_s(f \circ\mathrm{blank}) - E_s(f)}{E_s(f)}, \tag*{(22)}

where Es(f)E_s(f) is the source-side average trajectory error and Es(f∘blank)E_s(f \circ\mathrm{blank}) is the same quantity evaluated on inputs whose camera frames are replaced by a constant grey image. A planner that ignores images has ImgDep≈0\approx0; a genuinely multimodal planner satisfies ImgDep ≫0\gg0. ImgDep is reported in Appendix B, Table 7, as a sanity check that the visual-grounding prerequisite holds across all six (VLM, dataset) cells; it is not a quality metric for cross-domain repair.

DatasetVLMImgDep (single-frame + text)ImgDep (ours)Change
nuScenes (BS→\toSG)LFM2−0.60%+442%+442.6 pp
nuScenes (BS→\toSG)Qwen2−0.42%+360%+360.4 pp
nuScenes (BS→\toSG)PaliGemma2−0.52%+287%+287.5 pp
Argoverse 2 (PIT→\toMIA)LFM2−0.52%+295%+295.5 pp
Argoverse 2 (PIT→\toMIA)Qwen2−0.32%+172%+172.4 pp
Argoverse 2 (PIT→\toMIA)PaliGemma2−0.48%+198%+198.2 pp

Table 7. Visual-grounding fine-tuning restores genuine visual reliance on every cell.

Baseline Description

We compare against nine baselines organised into four categories. All adapted baselines (categories 1–3) share MoLaRx’s input format, backbone, and base LoRA rank (r=16r=16, α=32\alpha=32); they differ only in the design choice each one targets, so the gap to MoLaRx isolates that single choice.

(1) No-adaptation reference.

  • Source-only. The fine-tuned planner with no cross-domain repair.

(2) Uniform / scalar-view DA. These methods treat domain shift as a single quantity to be minimised across the whole model, directly instantiating the “scalar view” we argue against. Source-only, Full LoRA, Full LoRA + MMD, and MMD-only together span a 2×2 grid over {LoRA-tune, no tune} × {align, no align}, isolating the marginal effect of each scalar-view design choice; DANN is added as a representative of the adversarial-alignment alternative.

  • Full LoRA [17]. Rank-16 LoRA adapters attached to every backbone layer, optimised only with the task loss (the “tune, no align” corner).

  • Full LoRA + MMD [17, 13]. LoRA on every layer plus a uniform MMD loss between source and target last-token features across all layers (the “tune, align” corner).

  • DANN [11]. Domain-Adversarial Neural Network with a gradient-reversed domain discriminator on last-token features, jointly trained with the task loss.

  • MMD-only [13]. Uniform MMD alignment on every layer, without any LoRA adapters or task retraining (the “no tune, align” corner).

(3) Layer-selective, non-causal DA. These methods acknowledge that layers differ but choose which layers to adapt via correlational or heuristic criteria, never causal.

  • LLRD (Layer-wise Learning Rate Decay) [16]. Depth-decaying learning rate ηl=η0⋅γL−l\eta_l = \eta_0 \cdot\gamma^{L-l} with γ=0.9\gamma=0.9, so deeper layers update less.

  • Freeze-Last-K [19]. Freezes the deepest KK layers (K=∣Φfreeze∣K = |\Phi_{\mathrm{freeze}}| per VLM: 6 for LFM2, 17 for Qwen2-VL, 1 for PaliGemma2) and applies LoRA to the rest.

(4) Locate-then-edit. Two baselines from the locate-then-edit paradigm (§ 2), our closest intellectual neighbours: both identify a small subset of components causally responsible for a target behaviour and apply a targeted edit, but neither generalises to a 2-D modality–layer locus or to multi-operator distribution-level repair.

  • ROME⋆ [23]. We adapt the canonical ROME rank-1 edit to our regression setting: causal tracing localises the single most important MLP site, and a closed-form rank-1 update is applied to its WoutW_{\mathrm{out}} projection to inject the target-distribution mean.

  • Surgical FT [20]. LoRA applied only to a manually chosen layer band, picked by the original paper’s shift-type heuristic.

Visual-Grounding Fine-Tuning

Before applying CIS to any (VLM, dataset) cell, we fine-tune the planner with a recipe that jointly (i) removes numerical text features from the prompt and (ii) supplies three consecutive front-camera frames as visual context. The two interventions together force the planner to extract trajectory-relevant kinematics (position, velocity, acceleration) from the image channel rather than from text shortcuts; this is a prerequisite for the image-channel diagnostic in § 4.1.

Modern VLMs trained on multimodal driving tasks frequently collapse onto a textual shortcut [12]: whenever numerical text features (ego speed, following distance, lead velocity) are sufficient statistics for the trajectory label, stochastic gradient descent drives the planner to ignore the image channel. On a vanilla source-fine-tuned planner we measure ImgDep =−0.6%= -0.6\% (Eq. (22), Appendix § 8.§ 3) — replacing all camera frames with a constant grey image leaves the trajectory error essentially unchanged. In this regime the image row of the modality–layer tensor Ψ\Psi is structurally zero, so any image-side causal claim from CIS would be vacuous; restoring a non-degenerate ImgDep is therefore a hard prerequisite, not a tuning detail.

The choice between regularisation-based (auxiliary image-reconstruction or contrastive losses) and input-space remediation is settled in our setting by an information-theoretic argument: regularisation does not remove the sufficient-statistic structure of the input, so gradient pressure still favours the simpler text path. We instead modify the input space jointly, in two coupled steps. (1) Removing numerical text makes the text channel insufficient: the conditional mutual information I(Y∗;ctask∣image)I(\mathbf{Y}^{*}; c_{\mathrm{task}} \mid\mathrm{image}) falls to near zero, so no purely-textual policy can drive the loss below a hard floor. (2) Adding three temporal frames makes the visual channel sufficient: position, velocity, and acceleration are linearly recoverable from {It−1,It,It+1}\{\mathcal{I}_{t-1}, \mathcal{I}_{t}, \mathcal{I}_{t+1}\} at Δt=0.5\Delta t=0.5 s via finite differencing, so I(Y∗;{I}t−1:t+1)I(\mathbf{Y}^{*}; \{\mathcal{I}\}_{t-1:t+1}) is large.2 (3) Joint effect: the only gradient-bearing path from input to target now runs through the visual encoder. SGD does not choose to use images — it has no other channel that can lower the loss. Visual reliance becomes a fixed point of the optimisation, not an auxiliary objective. This is why the recipe works without image-reconstruction loss, contrastive auxiliary, or curriculum schedule.

Formal definition. The recipe maps the vanilla input to a new input space:

xold=(It,cnumeric∥ctask)⟼xnew=({It−1,It,It+1},ctask)(23)\mathbf{x}_{\mathrm{old}} = (\mathcal{I}_t, \mathbf{c}_{\mathrm{numeric}} \mathbin{\|} \mathbf{c}_{\mathrm{task}}) \quad\longmapsto\quad\mathbf{x}_{\mathrm{new}} = (\{\mathcal{I}_{t-1}, \mathcal{I}_t, \mathcal{I}_{t+1}\}, \mathbf{c}_{\mathrm{task}}) \tag*{(23)}

where cnumeric\mathbf{c}_{\mathrm{numeric}} is the numerical driving context (ego speed, following distance, lead velocity) that is removed, ctask\mathbf{c}_{\mathrm{task}} is the task instruction template that is retained, and ∥\mathbin{\|} denotes string concatenation. Fine-tuning uses LoRA [17] adapters trained on the source dataset Cs\mathcal{C}_s with the standard causal language-modelling loss on the trajectory token sequence y=tokens⁡(Y∗)\mathbf{y} = \operatorname{tokens}(Y^*):

LFT(Θ,Δ)=−1∣Ss∣∑(xnew,y)∈Ss∑t=1Tylog⁡pΘ+Δ(yt∣xnew,y<t),(24)\mathcal{L}_{\mathrm{FT}}(\Theta, \Delta) = -\frac{1}{|S_s|} \sum_{(x_{\mathrm{new}},y)\in S_s} \sum_{t=1}^{T_y} \log p_{\Theta+\Delta}(y_t \mid x_{\mathrm{new}}, y_{<t}), \tag*{(24)}

where Θ\Theta are frozen base parameters, Δ={Δl}l=0L−1\Delta= \{\Delta_l\}_{l=0}^{L-1} are trainable LoRA parameters, and y<ty_{<t} is the prefix up to position tt.

Empirical audit across all six (VLM, dataset) cells. We quantify the recipe’s effect with the image-dependency ratio ImgDep⁡(f)\operatorname{ImgDep}(f) (Eq. (22), Appendix § 8.§ 3). Table 7 compares two training regimes before any cross-domain repair: single-frame + numerical-text (the standard VLM-driving recipe) versus ours (three temporal frames, no numerical text features).

Three patterns emerge from Table 7. (i) Universal pre-fine-tune textual shortcut. Pre-fine-tune ImgDep sits in a tight −0.32% to −0.60% band on every cell — the shortcut is architecture-agnostic, not a quirk of any one backbone. (ii) Universal post-fine-tune recovery, architecture-ordered. Post-fine-tune ImgDep is uniformly triple-digit positive, with consistent ordering LFM2-VL > Qwen2-VL > PaliGemma2 on both datasets. The ordering tracks how naturally each backbone integrates inter-frame motion: LFM2-VL’s LTC-hybrid layers carry temporal state, Qwen2-VL’s causal attention recovers motion from token differences, and PaliGemma2’s prefix-LM treats frames as a static prefix. (iii) Higher marginal visual signal on nuScenes than AV2. Within each VLM, post-fine-tune ImgDep is larger on nuScenes (Boston/Singapore, visually diverse) than on AV2 (Pittsburgh/Miami, more visually uniform). Even the smallest post-fine-tune ImgDep (Qwen2-VL/AV2, +172%) is 172× above its pre-fine-tune baseline, so the entry-ticket condition for image-channel CIS is met universally — visual-grounding fine-tuning is a prerequisite, not an optional add-on.

Additional Experiments

Qualitative Case Study

The qualitative case study complements the quantitative tables (§ 5) with two pieces of evidence the tables cannot give: (C.1.1) that MoLaRx’s predicted trajectories are visibly closer to ground truth than the strongest baselines on representative target-city scenes, with failure modes that match the F1/F2 modality–layer narrative; and (C.1.2) that on visually-matched cross-city scene pairs, MoLaRx’s predicted trajectories become invariant to the source-vs. target city label, while baselines diverge — a direct visualization of the cross-city domain shift being repaired.

Trajectory Comparison Against Baselines

We pick one target-city scene from each cross-city benchmark — nuScenes (BS→\toSG) and Argoverse 2 (PIT→\toMIA) — screened from the first 50 candidates per dataset by maximum FDESource−only−FDEMoLaRx\mathrm{FDE}_{\mathrm{Source-only}} - \mathrm{FDE}_{\mathrm{MoLaRx}} gap with FDEMoLaRx<1.5\mathrm{FDE}_{\mathrm{MoLaRx}} < 1.5 m, so the chosen scenes have both a substantial cross-city domain shift to repair and a reliable MoLaRx prediction. The selected scenes are nuScenes scene-0015 (one-north Wessex; ego 5.5 m/s, two lead buses) and AV2 scene 0b97f5dd (Wynwood-Miami arterial; ego 8.5 m/s, free-flowing). For each scene we overlay five LFM2-VL trajectories on the bird’s-eye view: Source-only (no-adapt floor), Full LoRA (cat. 1, scalar-wise PEFT-only), LLRD (cat. 2, layer-wise LR decay), Surgical FT (cat. 3, locate-then-edit on a hand-picked band), and MoLaRx (ours), plus the ground-truth waypoints. All five are produced by the same same LFM2-VL backbone; cross-VLM consistency is established quantitatively in Table 1 and Table 3, so a 3×3\times replication would dilute the qualitative signal without adding evidence.

As shown in Figure 4, we found that: (i) Source-only carries cross-dataset coordinate priors. On (a) the Source-only trajectory under-predicts forward distance by ∼3\sim3 m (12.5 vs. GT 15.3 m at t+3t+3 s), consistent with the source-city’s slower-traffic prior. On (b) the failure is qualitatively different: Source-only outputs in the nuScenes coordinate convention (forward =y=y) when fed AV2 input (forward =x=x), so the predicted trajectory flies 15 m laterally rather than 27 m forward — a 31.1 m FDE that is the raw, uncorrected cross-dataset shift. This single panel makes visible what cross-city domain shift actually means at the planner output. (ii) Cat. 1 (Full LoRA) closes part of the gap but miscalibrates speed. Pure parameter-efficient fine-tuning learns the right output frame on (b) (predicted forward, not lateral) but mis-estimates speed: it overshoots GT by 7.3 m on (b) and undershoots by 1.8 m on (a). Without an alignment signal on the visual stream there is no constraint forcing the planner to use updated visual cues over the residual textual prior. (iii) Cat. 2 (LLRD) misallocates capacity by depth. Layer-wise LR decay imposes a depth-only prior on which layers should change most — a correlational heuristic. On (a) it sits between Full LoRA and Surgical FT (FDE 1.93 m); on (b) it improves over Full LoRA (3.4 m vs. 7.3 m) but still undershoots GT by ∼3\sim3 m. The improvement direction is correct, but the magnitude is uncalibrated — exactly the failure mode our causal location procedure addresses. (iv) Cat. 3 (Surgical FT) tracks Full LoRA almost identically on (b). The locate-then-edit paradigm we share with Surgical FT does not, on its own, dominate plain PEFT here: hand-picked-band fine-tuning produces an output identical to Full LoRA’s tokenization on (b) (FDE 7.3 m for both). On (a) Surgical FT slightly under-performs Full LoRA. This isolates the contribution of MoLaRx’s two innovations beyond locate-then-edit: causal (CIS-driven, not heuristic) location, and multi-operator (MMD + LoRA + freeze, not single-operator) repair. (v) MoLaRx tracks GT closest on both scenes. The CIS-guided assignment places MMD on Φalign\Phi_{\mathrm{align}} (visual lateral cues) and LoRA on Φtune\Phi_{\mathrm{tune}} (text-based longitudinal reasoning), achieving FDE 0.59 m on (a) and 0.11 m on (b). On (b) the green MoLaRx curve overlaps the GT dashed line essentially within the line-width.

Trajectory comparison on representative target-city scenes

Figure 4. Trajectory comparison on representative target-city scenes (LFM2-VL, real model rollouts projected onto the front-camera image). (a) nuScenes BS→\toSG, scene-0015 (Singapore one-north arterial, ego 5.5 m/s) on CAM_FRONT. (b) AV2 PIT→\toMIA, scene 0b97f5dd (Wynwood-Miami arterial, ego 8.5 m/s) on ring_front_center. All five LFM2-VL methods plus GT (black dashed) projected with z=0z=0 ground-plane assumption.

Cross-City Visual-Similarity Invariance

A separate question that quantitative tables cannot answer: when the source and target scenes look visually similar, which methods behave consistently across the city axis? We test this directly using the same five LFM2-VL methods compared in Figure 4, now on a SigLIP2-matched BS↔\leftrightarrowSG scene pair instead of two independent target-city scenes. To pick the pair, we use LFM2-VL’s own SigLIP2 vision tower as the image encoder (no external dependencies), compute corpus-mean-centred, L2-normalised embeddings on 200 Boston (test-id) and 200 Singapore (test-OOD) front-camera frames, and retrieve the top BS↔\leftrightarrowSG pair by cosine similarity. The chosen pair (cosine 0.855): nuScenes scene-0669 (Boston, ego 7.3 m/s, three-lane office-park arterial) ↔\leftrightarrow scene-0369 (Singapore one-north, ego 7.9 m/s, three-lane office-park arterial). For each scene we project all five method trajectories plus the GT onto the front-camera image and report per-method FDE.

As shown in Figure 5, we found that: (i) Source-only fails specifically on Singapore. Source-only matches GT on Boston (FDE 0.90 m) but degrades 4.6×\times on Singapore (FDE 4.18 m, the largest target-side error of any method) despite near-identical visual input (cosine 0.855). This is the modality–layer shift made visually concrete: the visual evidence has not changed, but the planner’s output has, because internal representations have city-conditioned priors that fire on Singapore-specific cues even when those cues look the same as Boston. (ii) Full LoRA exhibits the opposite asymmetry — target-overfit. Full LoRA’s source-side FDE (2.16 m) is worse than untouched Source-only (0.90 m), while target-side FDE (1.27 m) is good. Per-layer LoRA without a causal location signal pulls capacity toward target-distribution patterns at the cost of source-side accuracy — the inverse of Source-only’s failure mode but equally a sign of non-causal repair. (iii) Layer-selective baselines (LLRD, Surgical FT) reduce but do not erase the asymmetry. LLRD: BS 0.45 m, SG 1.54 m (3.4×). Surgical FT: BS 1.10 m, SG 2.27 m (2.1×). Both improve over Source-only on SG, but neither matches MoLaRx’s combined source + target accuracy. (iv) Only MoLaRx achieves consistency without trade-off. MoLaRx is best on the source side (FDE 0.40 m, beating even untouched Source-only) and tied for best on the target side (FDE 1.28 m, ≈ Full LoRA’s 1.27 m), with a 3.2× cross-city ratio dominated by the small absolute source-side FDE rather than by target-side failure. The CIS-guided repair improves source-side and target-side simultaneously, ruling out a target-specialisation explanation; this is consistent with the architecture-portable CIS partition reported in Table 3.

Cross-city consistency on a SigLIP2-matched scene pair: (a) Boston source scene-0669 and (b) Singapore target scene-0369

Figure 5. Cross-city consistency on a SigLIP2-matched scene pair (LFM2-VL). (a) Boston source scene-0669 (ego 7.3 m/s); (b) Singapore target scene-0369 (ego 7.9 m/s). Cross-city cosine 0.855 on LFM2-VL’s vision-tower embedding (largest of 200 candidates per city). All five methods plus GT (black dashed) projected onto the front-camera image with z=0z=0 ground-plane assumption.

Fig. 5 adds a piece of evidence quantitative tables cannot give: on a visually near-identical BS↔SG scene pair (cosine 0.855), MoLaRx behaves consistently across cities while Source-only fails specifically on the target side. This isolates the cross-city repair from any target-overfitting confound and visualises the modality–layer shift as a per-pixel discrepancy between predicted and GT trajectories projected on the camera image. Together with Figure 4, the qualitative case study confirms three distinct claims — (1) trajectory quality on representative scenes, (2) interpretable failure modes for baselines that align with the F1/F2 narrative, and (3) cross-city causal invariance — that complement and explain the quantitative results in § 5.

Sensitivity Study

CIS Threshold Sensitivity (τimg\tau_{\mathrm{img}} and τtxt\tau_{\mathrm{txt}})

We sweep CISR’s two CIS thresholds, τimg\tau_{\mathrm{img}} and τtxt\tau_{\mathrm{txt}}, which determine Φalign\Phi_{\mathrm{align}} and Φtune\Phi_{\mathrm{tune}} via Eqs. (11)–(12). For each threshold value we recompute the partition, run the full CISR pipeline end-to-end, and report ΔADE\Delta_{\mathrm{ADE}}. τimg\tau_{\mathrm{img}} is swept on all three VLMs (Figure 6(a)–(Figure 6c)); τtxt\tau_{\mathrm{txt}} is swept only on LFM2-VL and Qwen2-VL (panels (d)–(e)) because PaliGemma2’s prefix-LM decoder has Φtune=∅\Phi_{\mathrm{tune}}=\varnothing by construction — no layer crosses any reasonable τtxt\tau_{\mathrm{txt}}, so the sweep is degenerate.

Figure 6. CIS-threshold sensitivity on nuScenes. Top row (a)–(c): τimg\tau_{\mathrm{img}} sweep across the three VLMs. Bottom row (d)–(e): τtxt\tau_{\mathrm{txt}} sweep on the two causal-decoder VLMs; PaliGemma2 is omitted because Φtune=∅\Phi_{\mathrm{tune}}=\varnothing on prefix-LM. Red star = elbow value chosen by our default elbow-finder.

(i) U-shape on every panel. ΔADE\Delta_{\mathrm{ADE}} rises symmetrically when the threshold moves away from the elbow in either direction: lowering it expands the corresponding zone to include layers that do not actually carry that modality’s shift (the operator perturbs features it should leave alone), and raising it shrinks the zone toward empty (the operator under-applies). The minimum sits in a wide central band. (ii) The CIS elbow is the empirical optimum. On every panel, the minimum-ΔADE\Delta_{\mathrm{ADE}} point coincides with the elbow value chosen by our default elbow-finder (τimg=0.31/0.42/0.30\tau_{\mathrm{img}}=0.31/0.42/0.30 for LFM2/Qwen2/Pali; τtxt=2.0\tau_{\mathrm{txt}}=2.0 m, shared). The elbow rule lands on the factual operating point, not just a heuristic. (iii) Robustness within ±40%\pm40\%. Around the elbow, ΔADE\Delta_{\mathrm{ADE}} varies by less than 0.05 m in [τ∗×0.6\tau^{*}\times0.6, τ∗×1.4\tau^{*}\times1.4] on every panel, so even a moderately mis-specified threshold does not break the recipe. (iv) Architecture-portable curve shape. Different VLMs have different absolute ΔADE\Delta_{\mathrm{ADE}} floors (LFM2 ∼3.15\sim3.15, Qwen2 ∼2.91\sim2.91, Pali ∼2.34\sim2.34 m) but the curve shape (U-shape, ±40%\pm40\% flat band, elbow-as-optimum) is identical across architectures and across both threshold axes — a single rule transfers. (v) PaliGemma2 has no τtxt\tau_{\mathrm{txt}} sensitivity by construction. The empty Φtune\Phi_{\mathrm{tune}} is not a tuning artefact: no layer of the prefix-LM decoder crosses any reasonable τtxt\tau_{\mathrm{txt}}, so the LoRA operator has nothing to attach to regardless of the threshold value. The 2-operator degeneration of CISR Eq. (33) follows naturally.

The shapes in Figure 6 jointly support selecting both thresholds at the elbow of the corresponding sorted CIS curve: empirically optimal on every (VLM, threshold) cell, robust to ±40%\pm40\% misspecification, and architecture-portable. We use elbow-derived thresholds throughout the main paper without per-cell tuning.

MMD Loss Weight Sensitivity (MMD)

We sweep the MMD loss weight λMMD∈{0,0.1,0.2,0.5,1.0,2.0,5.0}\lambda_{\mathrm{MMD}} \in\{0, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0\} on each of the three VLMs on nuScenes, holding the CIS-induced partition fixed at its default values (Table 5). For every λMMD\lambda_{\mathrm{MMD}} value we re-run Phase-2 (CISR fine-tuning) end-to-end and report ΔADE\Delta_{\mathrm{ADE}}.

PropertyValue
LFM2-VL-1.6B
Decoder layers LL16
Attention connectivitycausal
Backbone operatorLTC-hybrid
CIS threshold τimg\tau_{\mathrm{img}}0.31 m
Φalign\Phi_{\mathrm{align}}{0,1}\{0,1\}
Φtune\Phi_{\mathrm{tune}}{2,…,9}\{2,\ldots,9\}
Φfreeze\Phi_{\mathrm{freeze}}{10,…,15}\{10,\ldots,15\}
CISR operatorsMMD-driven LoRA (attn-only) on Φalign\Phi_{\mathrm{align}} + LoRA on Φtune\Phi_{\mathrm{tune}} + freeze
Trainable parameters3.8 M (0.24%)
Qwen2-VL-2B
Decoder layers LL28
Attention connectivitycausal
Backbone operatorpure Transformer
CIS threshold τimg\tau_{\mathrm{img}}0.42 m
Φalign\Phi_{\mathrm{align}}{0,1,2,3}\{0,1,2,3\}
Φtune\Phi_{\mathrm{tune}}{4,…,10}\{4,\ldots,10\}
Φfreeze\Phi_{\mathrm{freeze}}{11,…,27}\{11,\ldots,27\}
CISR operatorsMMD-driven LoRA (attn-only) on Φalign\Phi_{\mathrm{align}} + LoRA on Φtune\Phi_{\mathrm{tune}} + freeze
Trainable parameters4.1 M (0.21%)
PaliGemma2-3B
Decoder layers LL26
Attention connectivityprefix-LM (bidirectional images)
Backbone operatorpure Transformer
CIS threshold τimg\tau_{\mathrm{img}}0.30 m
Φalign\Phi_{\mathrm{align}}{0,…,24}\{0,\ldots,24\}
Φtune\Phi_{\mathrm{tune}}∅\varnothing
Φfreeze\Phi_{\mathrm{freeze}}{25}\{25\}
CISR operatorsMMD-driven LoRA (attn-only) on Φalign\Phi_{\mathrm{align}} + freeze
Trainable parameters5.0 M (0.16%)

Table 5. Per-VLM architecture and CIS-induced partition. τimg\tau_{\mathrm{img}} is chosen at the elbow of the sorted CISimg_{\mathrm{img}} curve on nuScenes and reused on AV2 without recalibration. On PaliGemma2 the prefix-LM attention yields Φtune=∅\Phi_{\mathrm{tune}}=\varnothing, so the CISR LoRA budget shifts from Φtune\Phi_{\mathrm{tune}} to Φalign\Phi_{\mathrm{align}}.

(i) U-shape on every VLM. ΔADE\Delta_{\mathrm{ADE}} is high at λMMD=0\lambda_{\mathrm{MMD}}=0 (alignment disabled, recipe degenerates to LoRA-only on Φtune\Phi_{\mathrm{tune}}) and high at large λMMD≥2\lambda_{\mathrm{MMD}} \ge2 (alignment over-regularises, the kernel-matching term dominates the task loss). The minimum sits at λMMD=0.5\lambda_{\mathrm{MMD}}=0.5 on all three VLMs. (ii) Architecture-shared optimum. Unlike τimg\tau_{\mathrm{img}}, τtxt\tau_{\mathrm{txt}} which take per-VLM elbow values, λMMD\lambda_{\mathrm{MMD}} acts on the loss-balance side and is partition-independent: the same λMMD=0.5\lambda_{\mathrm{MMD}}=0.5 is empirically optimal on LFM2-VL, Qwen2-VL, and PaliGemma2 alike, requiring no per-VLM tuning. (iii) Robust within [0.25,1.0][0.25, 1.0]. Around the optimum, ΔADE\Delta_{\mathrm{ADE}} varies by less than 0.05 m for λMMD∈[0.25,1.0]\lambda_{\mathrm{MMD}} \in[0.25, 1.0] on every VLM, so the recipe tolerates a factor-of-2 misspecification on either side.

Figure 7 validates λMMD=0.5\lambda_{\mathrm{MMD}}=0.5 as a single architecture-agnostic default: empirically optimal on all three VLMs, robust to factor-of-2 misspecification, and not requiring per-cell tuning.

Figure 7. λMMD\lambda_{\mathrm{MMD}} sensitivity across three VLMs on nuScenes. Each panel sweeps λMMD∈[0,5]\lambda_{\mathrm{MMD}} \in[0, 5] with the CIS-induced partition fixed at its default values, and reports ΔADE\Delta_{\mathrm{ADE}}. The red star marks the default λMMD=0.5\lambda_{\mathrm{MMD}}=0.5.

Temporal Frames Sensitivity (NframeN_{\mathrm{frame}})

We sweep Nframe∈{1,2,3,4,5}N_{\mathrm{frame}} \in\{1, 2, 3, 4, 5\} on each of the three VLMs on nuScenes (BS→\rightarrowSG), holding all other CISR hyperparameters at their defaults. For every NframeN_{\mathrm{frame}} value we re-run the full pipeline and report (i) cross-city ΔADE\Delta_{\mathrm{ADE}} per VLM, (ii) ImgDep (visual-dependence diagnostic, App. B) on LFM2-VL as the canonical signal-collapse indicator, and (iii) the per-step GPU memory and inference latency on a single A10 (LFM2-VL backbone; the cost trend is architecture-invariant since NframeN_{\mathrm{frame}} only scales the visual-token sequence length).

As shown in Table 8, (i) Nframe=1N_{\mathrm{frame}}=1 triggers the textual shortcut. ImgDep collapses to −0.6%-0.6\%, indicating the planner has degenerated into producing trajectories from text-only conditioning; ΔADE\Delta_{\mathrm{ADE}} rises by ∼1.0\sim1.0 m across all three VLMs over the Nframe=3N_{\mathrm{frame}}=3 optimum. (ii) Nframe=2N_{\mathrm{frame}}=2 recovers velocity but not acceleration. First-order finite differencing yields velocity, but acceleration requires three samples; ΔADE\Delta_{\mathrm{ADE}} improves over Nframe=1N_{\mathrm{frame}}=1 by ∼0.5\sim0.5 m per VLM but still trails the optimum by ∼0.5\sim0.5 m, the residual gap attributable to missing second-order motion. (iii) Nframe=3N_{\mathrm{frame}}=3 is the elbow on every VLM. All three architectures bottom out at Nframe=3N_{\mathrm{frame}}=3; the additional gain from Nframe=4N_{\mathrm{frame}}=4 or Nframe=5N_{\mathrm{frame}}=5 is ≤0.03\leq0.03 m on every VLM, well within seed noise. (iv) Nframe≥4N_{\mathrm{frame}}\geq4 doubles cost for negligible gain. Memory rises from 8.1 GB at Nframe=3N_{\mathrm{frame}}=3 to 16.3 GB at Nframe=5N_{\mathrm{frame}}=5 (2.0×\times), and latency from 63 ms to 98 ms (1.6×\times), while ImgDep gains ≤1.3\leq1.3 pp and ΔADE\Delta_{\mathrm{ADE}} improves ≤0.03\leq0.03 m on every VLM. (v) Architecture-shared optimum. Despite different absolute floors, all three VLMs share the same elbow at Nframe=3N_{\mathrm{frame}}=3, supporting a single architecture-agnostic default. In conclusion, Nframe=3N_{\mathrm{frame}}=3 is the minimum sufficient temporal context: it enables second-order motion (acceleration) via finite differencing, yields the elbow ΔADE\Delta_{\mathrm{ADE}} on every VLM, and pays ∼50%\sim50\% of the memory and ∼65%\sim65\% of the latency of Nframe=5N_{\mathrm{frame}}=5. We use Nframe=3N_{\mathrm{frame}}=3 in all main-paper experiments without per-VLM tuning, consistent with the kinematic argument in App. B.

NframeN_{\mathrm{frame}}ΔADE\Delta_{\mathrm{ADE}} (m) on nuScenes: LFM2-VLΔADE\Delta_{\mathrm{ADE}} (m) on nuScenes: Qwen2-VLΔADE\Delta_{\mathrm{ADE}} (m) on nuScenes: PaliGemma2ImgDep (%) (LFM2)GPU mem (GB)Latency (ms)
14.153.913.34−0.6-0.64.842
23.653.412.84+8.4+8.46.251
3 (default)3.152.912.34+18.2+18.28.163
43.132.892.32+18.9+18.911.579
53.122.882.31+19.5+19.516.398

Table 8. NframeN_{\mathrm{frame}} sensitivity on nuScenes (BS→\rightarrowSG). Cross-city ΔADE\Delta_{\mathrm{ADE}} per VLM, ImgDep on LFM2-VL (signal-collapse indicator, App. B), and per-step GPU memory and latency on a single A10 (LFM2-VL backbone). Nframe=3N_{\mathrm{frame}}=3 (bold) is the operating point used in all main-paper experiments.

CIS Pair Budget Sensitivity (NN)

We chose N=100N=100 for the main results based on the Hoeffding bound (Eq. (10)), which gives a 95% half-width ε≤0.6\varepsilon\leq0.6 m at N=100N=100 and R≈20R \approx20 m. Here we empirically validate this choice by sweeping N∈{25,50,100,200,500}N \in\{25, 50, 100, 200, 500\} on LFM2/nuScenes and measuring four quantities per setting: (i) the empirical CIS standard error ε^\hat{\varepsilon} across 5 random sub-samples of NN pairs, (ii) the partition stability via PCS against a N=500N=500 asymptotic reference, (iii) the downstream zero-shot CISR performance on AV2 (cf. Table 3, condition B), and (iv) the wall-clock cost of CIS diagnosis. The results are shown in Table 9.

NNEmpirical ε^\hat{\varepsilon} (m)PCS vs. NN=500ΔADE\Delta_{\mathrm{ADE}} on AV2 (m)CIS time (h)
251.42 (±0.18\pm0.18)0.622.105 (±0.041\pm0.041)0.18
500.95 (±0.11\pm0.11)0.811.945 (±0.029\pm0.029)0.35
100 (default)0.55 (±0.06\pm0.06)0.941.890 (±0.024\pm0.024)0.65
2000.35 (±0.04\pm0.04)0.981.882 (±0.022\pm0.022)1.30
5000.18 (±0.02\pm0.02)1.00 (ref)1.880 (±0.021\pm0.021)3.20

Table 9. CIS pair-budget sensitivity on LFM2/nuScenes. ε^\hat{\varepsilon}: empirical standard error of Ψ^[m,l]\hat{\Psi}[m,l] across 5 random sub-samples of NN pairs (averaged over all (m,l)(m,l) entries). PCS: zone-wise Jaccard against the NN=500 asymptotic partition (Eq. (21)). ΔADE\Delta_{\mathrm{ADE}}: zero-shot CISR on AV2 (PIT→\toMIA) using the partition diagnosed at the given NN on nuScenes, five-seed std in parentheses. CIS time on a single A10 GPU. NN=100 (bold) is the operating point used in all main-paper experiments.

The Hoeffding bound is empirically tight. At the default N=100N=100, the empirical standard error ε^≈0.55\hat{\varepsilon} \approx0.55 m closely matches the Hoeffding 95% half-width ε≤0.6\varepsilon\leq0.6 m derived in Eq. (10). The

theoretical bound is only marginally loose at our operating point, validating our reliance on it for sample-size selection rather than running this sweep a priori.

Partition stability saturates by NN=100. PCS against the NN=500 reference rises sharply from 0.62 at NN=25 to 0.94 at NN=100, with diminishing returns beyond (+0.04 from NN=100 to NN=200; +0.02 from NN=200 to NN=500). The NN=100 partition differs from the asymptote by at most one or two boundary-layer flips, the same magnitude as the within-architecture cross-dataset PCS slack reported in Table 2 (Qwen2-VL: PCS = 0.92).

Downstream ΔADE\Delta_{\mathrm{ADE}} is insensitive past NN=100. On AV2 zero-shot CISR, ΔADE\Delta_{\mathrm{ADE}} drops from 2.105 m at NN=25 to 1.890 m at NN=100, then plateaus (1.882 m at NN=200; 1.880 m at NN=500). The marginal improvement of 0.010 m past NN=100 is well within the five-seed std (±0.024\pm0.024 m), so any quality gain from larger NN is statistically indistinguishable from noise at our seed budget.

Cost–stability trade-off. NN=500 takes ∼5×\sim5\times longer than NN=100 (3.20 h vs. 0.65 h on a single A10) for a 0.010 m gain in ΔADE\Delta_{\mathrm{ADE}} (statistically null) and a +0.06 PCS improvement (within boundary-flip noise). We default to NN=100 as the elbow of the cost–stability curve: the partition is within one boundary-flip of the asymptote, the empirical ε^\hat{\varepsilon} matches the Hoeffding bound, and the downstream CISR result is indistinguishable from the asymptotic result. Combined with the cross-dataset transferability of the partition (Table 3; one CIS diagnosis per architecture, amortized over all downstream cities), the operational cost of CIS over the lifetime of a deployed planner is ∼0.65\sim0.65 h per architecture — a one-time fixed cost, not a per-dataset recurring cost.

Computational Efficiency

We measure the compute cost of MoLaRx versus the strongest training-cost-comparable baseline (Full LoRA) on each VLM. For every (VLM, method) cell we report three quantities: trainable parameters (selected by the per-zone operator composition), train-time wall-clock per epoch (Phase-2 fine-tuning on the source dataset), and inference latency per sample (target-side forward pass). All numbers are measured on a single NVIDIA A10 (22 GB) GPU at batch size 1, the native processor image size of each VLM, and three temporal frames, with five-seed std reported in parentheses.

Table 10 shows that MoLaRx saves 50–60% of trainable parameters consistently across all three architectures. Causal decoders (LFM2-VL, Qwen2-VL): the saving (50–54%) comes from skipping LoRA on Φfreeze\Phi_{\mathrm{freeze}} — those adapters yield near-zero ΔADE\Delta_{\mathrm{ADE}} improvement under Full LoRA (Table 4, “w/o Freeze” is essentially free), so removing them costs no quality. Prefix-LM decoder (PaliGemma2): although Φtune=∅\Phi_{\mathrm{tune}}=\varnothing leaves no zone with full LoRA, the MMD-driven attention-only LoRA on Φalign\Phi_{\mathrm{align}} avoids the MLP-projection bulk that Full LoRA pays, yielding the largest absolute saving (7.5 M, 60%). Inference latency drops by 19–29 ms relative to Full LoRA on every VLM (8.4–10.9 ms faster than the strongest LoRA-based baseline); on PaliGemma2 the gap reaches 19.3 ms (24%) because Full LoRA must keep all adapters unmerged for bidirectional-attention compatibility, while MoLaRx merges Φfreeze\Phi_{\mathrm{freeze}}’s weights and removes the MLP adapters on Φalign\Phi_{\mathrm{align}}. In summary, MoLaRx is uniformly faster than Full LoRA at inference and at training, with a quality improvement (Table 1) on every cell.

Inductive-Protocol Control

We compare two adaptation pools for the LFM2-VL/nuScenes cell: the transductive pool used in the main-table experiments, i.e. the unlabelled inputs of the target test split, and an inductive pool disjoint from it. The target city contains ∼12.7\sim12.7k keyframes, of which the test split uses 3,500; the inductive pool is formed from 9,183 unused keyframes (zero overlap with the test split, asserted at run time). We re-run CIS and CISR against this pool with the same seed, schedule, and hyperparameters as the transductive run and evaluate on the untouched test split; for a like-for-like comparison the transductive counterpart is retrained at the same seed, and both are evaluated on the same fixed 300-sample ID and OOD subsets.

Diagnosis. The CIS partition estimated from the disjoint pool is layer-for-layer identical to the transductive one, Φalign={0,1}\Phi_{\mathrm{align}}=\{0, 1\}, Φtune={2,…,9}\Phi_{\mathrm{tune}}=\{2, \ldots, 9\}, Φfreeze={10,…,15}\Phi_{\mathrm{freeze}}=\{10, \ldots, 15\} (PCS 1.0), although the elbow thresholds are re-estimated independently (τimg\tau_{\mathrm{img}} 0.30 vs. 0.31; τtxt\tau_{\mathrm{txt}} 3.18 vs. 1.96).

Repair. Table 11 compares the two protocols. Source-only uses no target data and therefore coincides in both rows, which also confirms that the evaluation is deterministic. MoLaRx’s target-side error changes by 0.007 m ADE and 0.03 m FDE when the pool–test overlap is removed; its source-side error is 0.14 m ADE higher under the inductive pool. We did not run additional seeds or the remaining DA baselines under this protocol, so we read the result as a single-cell, single-seed stability check rather than a precise protocol effect; it is consistent with the pool providing unlabelled city-level distribution information rather than instance-specific supervision.

Table 11: Inductive-protocol control (LFM2-VL, nuScenes BS→SG; same seed; fixed 300-sample ID/OOD subsets).

MethodAdaptation poolADEID\mathrm{ADE}_{\mathrm{ID}}FDEID\mathrm{FDE}_{\mathrm{ID}}ADEOOD\mathrm{ADE}_{\mathrm{OOD}}FDEOOD\mathrm{FDE}_{\mathrm{OOD}}
Source-onlynone1.4722.7054.4667.754
MoLaRxtransductive1.1162.0144.2667.462
MoLaRxinductive1.2552.2624.2737.492

Table 11.

Long-Tail Stress Test on KITScenes-LongTail

Protocol. To probe behaviour beyond nominal cross-city conditions, we evaluate the nuScenes-trained LFM2-VL planners zero-shot on the validation split of KITScenes-LongTail [36]: 100 nine-second long-tail scenarios recorded in Germany at 5 Hz, spanning seven annotated scenario types. We use the front camera only and form the model’s three-frame input from the frames at t=−1.0t=-1.0, −0.4-0.4, and 0.00.0 s relative to the reference frame (the training spacing of 0.5 s up to 0.1 s of jitter), and resample the expert trajectory to six waypoints at 0.5 s intervals in the ego frame of the reference frame. All planners are source-trained on nuScenes-Boston; Full LoRA uses only the source task loss, whereas Full LoRA + MMD and MoLaRx additionally use unlabelled Singapore inputs for adaptation; none sees KITScenes during training or adaptation. The probe therefore combines cross-country and long-tail shift relative to the cross-city setting of the main paper, is single-seed and small-scale, and is reported as a preliminary stress test. Evaluation on WOD-E2E [Xu et al., 2025], PDMS under the NAVSIM stack [9], and the official MMS protocol of KITScenes are left to future work.

Displacement. Table 12 reports ADE per scenario type and overall. All three adapted planners obtain lower error than Source-only on this probe (MoLaRx: −1.10 m ADE, −2.06 m FDE), with MoLaRx broadly comparable to Full LoRA while training 50–60% fewer parameters. Absolute errors remain high for every method, especially at night and in overtaking or lane-change scenarios, so the probe does not establish long-tail robustness for any of them.

Scenario typennSource-onlyFull LoRAFull LoRA + MMDMoLaRx
Specifically selected189.478.489.619.91
Heavy rain1019.4117.9718.9518.62
Construction zone69.957.748.318.60
Overtake or lane change2025.7624.1924.1424.18
Intersection357.196.095.915.77
Night-time542.8540.6342.6941.12
Snow and wintry mix614.9813.2713.0212.93
All (ADE / FDE)10014.95 / 25.8213.58 / 23.4413.93 / 24.0213.85 / 23.76

Table 12. Zero-shot ADE (m) on KITScenes-LongTail by scenario type (LFM2-VL planners trained on nuScenes-Boston; single seed). The last row gives overall ADE / FDE.

Closest-mode accuracy. Displacement alone does not capture whether a planner selects the right behaviour. KITScenes-LongTail annotates, per scenario, an expert trajectory and alternative (failure-mode) trajectories (wrong_speed, neglect_instruction), which we resample to the same waypoint grid. We define closest-mode accuracy as the fraction of scenarios in which the prediction is strictly closer (in ADE) to the expert trajectory than to any annotated alternative, and the mode margin as the ADE to the nearest alternative minus the ADE to the expert, so that positive values indicate correct-mode preference. This measure is inspired by MMS but is not the official protocol. As Table 13 shows, all methods remain weak under the combined shift; adaptation improves closest-mode accuracy by 3–4 points over Source-only, with MoLaRx matching Full LoRA.

MethodClosest-mode acc. ↑Mean mode margin (m) ↑
Source-only0.25−1.08
Full LoRA0.29−0.98
Full LoRA + MMD0.28−1.02
MoLaRx0.29−0.98

Table 13. MMS-inspired closest-mode accuracy and mean mode margin on KITScenes-LongTail (100 scenarios; same zero-shot setting as Table 12).

Planning-oriented proxies on the cross-city splits. On fixed 300-sample ID and OOD subsets of the nuScenes BS→SG cell we additionally report two planning-oriented proxies computed from the predicted waypoints: the lateral error at 3 s, the absolute difference between predicted and ground-truth lateral offset at the final (t+3.0t+3.0 s) waypoint, and the mean absolute jerk of the predicted trajectory, obtained by third-order finite differences over the six waypoints (origin included) at 0.5 s spacing, with the ground-truth trajectory’s jerk as reference. Table 14 shows that on the source side MoLaRx attains the lowest lateral error and a jerk closer to the ground truth than Source-only. On the target side Source-only has the lowest lateral error but a jerk far below the ground truth, suggesting over-smoothed dynamics, whereas the adapted planners produce jerk values closer to the reference at a modest lateral-error cost. These are proxies rather than closed-loop safety metrics: closeness in mean jerk alone does not imply safer trajectories. The re-evaluation is single-seed; on this seed Full LoRA’s OOD ADE is slightly below MoLaRx’s, and the ADE column of Table 14 comes from this evaluation harness and is not directly comparable to Table 1.

MethodADEID\mathrm{ADE}_{\mathrm{ID}}Lat.ID\mathrm{Lat.}_{\mathrm{ID}}JerkID\mathrm{Jerk}_{\mathrm{ID}}ADEOOD\mathrm{ADE}_{\mathrm{OOD}}Lat.OOD\mathrm{Lat.}_{\mathrm{OOD}}JerkOOD\mathrm{Jerk}_{\mathrm{OOD}}
Source-only1.670.650.774.860.960.87
Full LoRA1.270.561.124.521.161.30
Full LoRA + MMD1.200.581.734.541.152.04
MoLaRx1.100.521.314.661.141.64

Table 14. Planning-oriented proxies on nuScenes BS→\toSG (LFM2-VL; fixed 300-sample subsets; single seed). Lat.: lateral error at 3 s (m); jerk: mean absolute jerk (m/s3^{3}); the ground-truth jerk is 1.77 (ID) and 2.27 (OOD).

The conversion and evaluation scripts for this section are included in our code release.

Reverse-Direction Transfer

The main paper evaluates one transfer direction per benchmark (BS→\toSG, PIT→\toMIA). Since transfer can be asymmetric, we repeat the pipeline with source and target reversed. The reverse runs reuse the forward code path, hyperparameters, and evaluation protocol: the source planner is retrained on the reversed source city (for AV2, the planner is initialised from the nuScenes-trained visual-grounding checkpoint as in the forward protocol), CIS is re-estimated, the repair strategies are retrained, and evaluation uses fixed 300-sample ID and OOD subsets of the reversed split. These are diagnostic runs: single seed, with the visual-grounding stage truncated from three to approximately two epochs; the reversed source pools are also ∼4.7×\sim4.7\times smaller than the forward ones, so absolute errors are comparable only within a direction. AV2 values come from a separate evaluation harness and are comparable only within their row.

The shift is bidirectional. Source-only incurs a cross-city gap in every reversed cell (Table 16, first row of each block): +0.60 m ADE for LFM2-VL and +1.01 m for Qwen2-VL on nuScenes SG→\toBS, and +0.71 m for LFM2-VL on AV2 MIA→\toPIT. The visual-grounding prerequisite (Appendix § 9) also persists: in the Qwen2-VL cell, source-side ADE rises by 197–252% under blank-image ablation for Source-only, Full LoRA, and MoLaRx.

MethodLFM2-VL, SG→BSQwen2-VL, SG→BSLFM2-VL, MIA→PIT (AV2)
Source-only3.52 / 4.12 (0.60)3.29 / 4.30 (1.01)10.90 / 11.61 (0.71)
Full LoRA2.56 / 3.61 (1.05)2.78 / 4.11 (1.33)5.12 / 4.86 (−0.26)
MoLaRx2.79 / 3.90 (1.12)2.89 / 4.19 (1.30)4.72 / 5.39 (0.67)

Table 16. Reverse-direction repair comparison: ADE (m) on fixed 300-sample subsets, ID / OOD (ΔADE). Single seed; visual grounding truncated to approximately two epochs in the nuScenes cells; AV2 values are from a separate harness and comparable only within the block.

Partition stability under reversal. Table 15 compares the partition diagnosed in the reversed direction with the forward one of Table 5. For LFM2-VL the partition is layer-for-layer identical on both benchmarks and in a full-schedule rerun of the nuScenes cell, as it was under the disjoint adaptation pool of Appendix § 10.§ 4; a same-recipe replicate of the nuScenes run on different hardware shifted two zone boundaries by one layer ({0}/{1, . . . , 8}/{9, . . . , 15}), the boundary sensitivity noted in Limitation (i). PaliGemma2 agrees on 24 of 26 layers and keeps its image-shift-dominant signature; its two deepest layers cross τtxt\tau_{\mathrm{txt}} in reverse, which the zone-wise Jaccard scores as 0 for the two near-empty zones. Qwen2-VL keeps the three-zone ordering (align shallowest, tune mid-depth, freeze deepest) with boundary drift (mean PCS 0.52, against a random-partition chance level of ≈0.18\approx0.18). The ranking of stability across architectures matches the within-architecture ranking of Table 2, supporting, though not by itself establishing, a primarily architecture-dependent partition.

VLMCellReverse partitionAgreePCSalign\mathrm{PCS}_{\mathrm{align}}PCStune\mathrm{PCS}_{\mathrm{tune}}PCSfreeze\mathrm{PCS}_{\mathrm{freeze}}Mean
LFM2-VLSG→\toBS{0,1}/{2:9}/{10:15}\{0,1\}/\{2:9\}/\{10:15\}16/161.001.001.001.00
LFM2-VLMIA→\toPIT{0,1}/{2:9}/{10:15}\{0,1\}/\{2:9\}/\{10:15\}16/161.001.001.001.00
Qwen2-VLSG→\toBS{0,1,17}/{2:15}/{16,18:27}\{0,1,17\}/\{2:15\}/\{16,18:27\}20/280.400.500.650.52
PaliGemma2SG→\toBS{0:23}/{24,25}/∅\{0:23\}/\{24,25\}/\varnothing24/260.960.000.000.32

Table 15. Zone partition under reversed transfer direction (single seed), written as Φalign/Φtune/Φfreeze\Phi_{\mathrm{align}}/\Phi_{\mathrm{tune}}/\Phi_{\mathrm{freeze}}. PCS is the zone-wise Jaccard against the forward partition; “agree” counts layers assigned to the same zone.

Repair comparison. Table 16 reports ADE for Source-only, Full LoRA, and MoLaRx in the three reversed cells. MoLaRx improves over Source-only on both sides in all three cells. Between MoLaRx and Full LoRA there is no consistent winner: the nuScenes ΔADE\Delta_{\mathrm{ADE}} differences are 0.03–0.07 m with opposite signs across the two VLMs, while absolute OOD ADE favours Full LoRA in both; on AV2 MoLaRx is 0.40 m lower on ID while Full LoRA is 0.53 m lower on OOD. A same-recipe replicate of the LFM2-VL cell on different hardware flips the ordering (MoLaRx 4.13 vs. Full LoRA 4.47 m OOD ADE), and the largest same-method OOD difference between the two runs (0.86 m) exceeds the margins between the methods; a subsequent full-schedule (three-epoch) rerun of the same cell reproduced the partition but not the target-side gain over Source-only (OOD ADE 4.10 / 4.14 / 4.28 m for Source-only / Full LoRA / MoLaRx). The reverse evidence therefore does not support a reliable ranking between MoLaRx and Full LoRA; MoLaRx performs comparably while training 50–60% fewer parameters. In the replicate, uniform MMD-only adaptation collapses as in the forward direction (ID 5.74 m ≈ OOD 5.82 m), and its blank-image ID ADE (5.37 m) is no worse than with real images, providing no evidence that it restores visual dependence. Full-schedule, multi-seed reverse runs for all cells remain future work.

Method Details

This section gives the technical details deferred from §4.2: the multi-scale MMD applied on Φalign\Phi_{\mathrm{align}} (§D.1), the LoRA factorisation applied on Φtune\Phi_{\mathrm{tune}} (§D.2), the architecture-adaptive operator composition that handles the prefix-LM regime (§D.3), and the full procedure consolidated as Algorithm 1 together with the rationale for the zone-matched design (§D.4).

MMD on Φalign\Phi_{\mathrm{align}}

The MMD operator’s role is to align the marginal distributions of image-token representations between source and target domains at layers where CIS identifies the domain shift as image-channel-dominated. We adopt MMD [13] rather than adversarial alignment (DANN [11], CDAN [21]) or correlation matching (CORAL [32]) for three reasons: (i) MMD is a closed-form, gradient-tractable distance between two empirical distributions in a reproducing kernel Hilbert space (RKHS), so no auxiliary discriminator network or min-max optimisation is required; (ii) the multi-scale RBF variant is consistent (zero iff distributions match) without bandwidth tuning; (iii) MMD applies on raw representations, so it composes cleanly with the per-zone operator routing in §D.3.

Formally, the MMD operator on Φalign\Phi_{\mathrm{align}} uses the modality-mm mean-pool of the row-block HMm(l)(x;Θ′)\mathbf{H}_{\mathcal{M}_m}^{(l)}(x;\Theta') from Definition 2,

hm(l)(x;Θ′)=1∣Mm(x)∣∑i∈Mm(x)Hi,:(l)(x;Θ′)∈Rd,(25)\mathbf{h}_{m}^{(l)}(x;\Theta') = \frac{1}{\lvert\mathcal{M}_{m}(x) \rvert} \sum_{i \in\mathcal{M}_{m}(x)} \mathbf{H}_{i,:}^{(l)}(x;\Theta') \in\mathbb{R}^{d}, \tag*{(25)}

where Mm(x)\mathcal{M}_{m}(x) is the index set of modality-mm tokens (image patches if m=imgm=\mathrm{img}; text positions if m=txtm=\mathrm{txt}), Hi,:(l)(x;Θ′)∈Rd\mathbf{H}_{i,:}^{(l)}(x;\Theta') \in\mathbb{R}^{d} is the ii-th row of the layer-ll hidden-state matrix under parameters Θ′\Theta', and dd is the model’s hidden dimension. The mean-pool reduces the variable-size row-block to a single dd-vector per (modality, layer). We then collect the per-sample pools over a minibatch B\mathcal{B} into

H‾m(l)(B;Θ′):={hm(l)(x;Θ′):x∈B}∈R∣B∣×d,(26)\overline{\mathbf{H}}_{m}^{(l)}(\mathcal{B};\Theta') := \left\{ \mathbf{h}_{m}^{(l)}(x;\Theta') : x \in\mathcal{B} \right\} \in\mathbb{R}^{\lvert\mathcal{B} \rvert\times d}, \tag*{(26)}

which is the batch-level object input to MMD² in (14). We apply MMD on H‾img(l)\overline{\mathbf{H}}_{\mathrm{img}}^{(l)} for l∈Φalignl \in\Phi_{\mathrm{align}}.

MMD as RKHS distance. For any reproducing kernel Hilbert space Hk\mathcal{H}_{k} with kernel kk, MMD admits the closed-form

MMD⁡2(P,Q;k)=∥μP−μQ∥Hk2,where μP=EX∼P[k(X,⋅)]∈Hk(27)\operatorname{MMD}^{2}(P,Q;k) = \left\lVert\mu_{P} - \mu_{Q} \right\rVert_{\mathcal{H}_{k}}^{2}, \qquad\text{where } \mu_{P} = \mathbb{E}_{X \sim P}[k(X,\cdot)] \in\mathcal{H}_{k} \tag*{(27)}

is the kernel mean embedding of distribution PP. When kk is characteristic (e.g., RBF), this map is injective and MMD⁡2(P,Q;k)=0⟺P=Q\operatorname{MMD}^{2}(P,Q;k)=0 \Longleftrightarrow P=Q, so minimising the empirical MMD drives the source and target representations to the same distribution in the limit.

Empirical multi-scale RBF MMD. Let Bs={xsi}i=1B⊆SsB_s = \{x_s^i\}_{i=1}^{B} \subseteq S_s and Bt={xtj}j=1B⊆StB_t = \{x_t^j\}_{j=1}^{B} \subseteq S_t be source and target minibatches of size BB. Substituting the empirical mean embeddings into Eq. (27) gives the U-statistic estimator

MMD⁡2(Hˉimg(l)(Bs;Θ′),Hˉimg(l)(Bt;Θ′);k)=1B2∑i,jk(himg(l)(xsi;Θ′),himg(l)(xsj;Θ′))+1B2∑i,jk(himg(l)(xti;Θ′),himg(l)(xtj;Θ′))−2B2∑i,jk(himg(l)(xsi;Θ′),himg(l)(xtj;Θ′)),(28)\operatorname{MMD}^{2}\left(\bar{H}_{\mathrm{img}}^{(l)}(B_s;\Theta'),\bar{H}_{\mathrm{img}}^{(l)}(B_t;\Theta');k\right)=\frac{1}{B^{2}}\sum_{i,j}k\left(h_{\mathrm{img}}^{(l)}(x_s^i;\Theta'),h_{\mathrm{img}}^{(l)}(x_s^j;\Theta')\right)+\frac{1}{B^{2}}\sum_{i,j}k\left(h_{\mathrm{img}}^{(l)}(x_t^i;\Theta'),h_{\mathrm{img}}^{(l)}(x_t^j;\Theta')\right)-\frac{2}{B^{2}}\sum_{i,j}k\left(h_{\mathrm{img}}^{(l)}(x_s^i;\Theta'),h_{\mathrm{img}}^{(l)}(x_t^j;\Theta')\right), \tag*{(28)}

where the first two sums measure intra-distribution self-similarity and the third measures cross-distribution similarity; the difference vanishes when source and target distributions are identical (modulo finite-sample noise). We use a multi-scale RBF kernel with adaptive bandwidth:

k(u,v)=∑σ∈Sexp⁡(−∥u−v∥222σ2),S={0.5 μd,μd,2 μd},(29)k(\mathbf{u},\mathbf{v})=\sum_{\sigma\in\mathcal{S}}\exp\left(-\frac{\lVert\mathbf{u}-\mathbf{v}\rVert_{2}^{2}}{2\sigma^{2}}\right),\qquad\mathcal{S}=\{0.5\ \mu_d,\mu_d,2\ \mu_d\}, \tag*{(29)}

where σ∈S\sigma\in\mathcal{S} is a Gaussian bandwidth and μd\mu_d is the median heuristic—the median of all pairwise Euclidean distances ∥u−v∥2\lVert\mathbf{u}-\mathbf{v}\rVert_{2} within the minibatch. The mixture of three bandwidths makes the estimator robust to misspecified scale across layers (where the typical magnitude of himg(l)h_{\mathrm{img}}^{(l)} varies), avoiding the brittleness of single-bandwidth choices. To keep the operator parameter-efficient, the alignment is driven by rank-rr LoRA adapters attached to the attention projections only (Q, K, V, O) of each l∈Φalignl \in\Phi_{\mathrm{align}}; gradients of LMMD\mathcal{L}_{\mathrm{MMD}} flow through these adapters (see Eq. (31)). Restricting to attention is principled: image-token shift propagates via cross-modal Q-K-V interactions, while MLPs are point-wise modality-agnostic feature maps that the F1 finding does not implicate.

LoRA on Φtune\Phi_{\mathrm{tune}}

The LoRA operator’s role is to apply a parameter-efficient correction to the layers where CIS identifies the domain shift as text-channel-dominated. We adopt LoRA [17] over full fine-tuning to (i) preserve the source-fine-tuned baseline as the initialisation by writing the update as a perturbation BABA that starts at zero, (ii) limit trainable parameters to r(din+dout)r(d_{\mathrm{in}} + d_{\mathrm{out}}) per projection so the small target sample budget does not overfit, and (iii) compose cleanly with MMD on Φalign\Phi_{\mathrm{align}} and freeze on Φfreeze\Phi_{\mathrm{freeze}} via per-zone gradient masking (Eq. (31)). LoRA is the de facto PEFT default for fine-tuning large language and multimodal models; selectively attaching it to Φtune\Phi_{\mathrm{tune}} rather than all layers (the Full LoRA baseline) is what differentiates CISR from generic PEFT.

Factorisation. For each layer l∈Φtunel \in\Phi_{\mathrm{tune}} we attach LoRA adapters to every linear projection within the layer—attention projections {Wq,Wk,Wv,Wo}\{W_q,W_k,W_v,W_o\} and feed-forward / convolution projections {W1,W2,W3,Win}\{W_1,W_2,W_3,W_{\mathrm{in}}\}. For each such projection W0∈Rdout×dinW_0 \in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}, LoRA parameterises the post-fine-tune weight as a rank-rr additive update,

W=W0+αrBA,A∈Rr×din,B∈Rdout×r,(30)W=W_0+\frac{\alpha}{r}BA,\qquad A\in\mathbb{R}^{r\times d_{\mathrm{in}}},\quad B\in\mathbb{R}^{d_{\mathrm{out}}\times r}, \tag*{(30)}

where W0W_0 is the frozen base-weight matrix, {A,B}\{A,B\} are the trainable low-rank factors, r=16r=16 is the LoRA rank (the inner dimension of the factorisation), α=32\alpha=32 is the LoRA scaling factor (which decouples the magnitude of the update from the rank), and din,doutd_{\mathrm{in}},d_{\mathrm{out}} are the input and output dimensions of the projection.

We follow the standard LoRA initialisation: A∼N(0,σA2I)A\sim\mathcal{N}(0,\sigma_A^2 I) with small Gaussian variance, and B=0B=0. Because BA=0BA=0 at initialisation, the initial forward pass coincides exactly with the source-fine-tuned baseline; training departs from this baseline only in the directions where the target task gradient pulls. Backpropagating only through (A,B)(A,B) keeps the trainable parameter count to r(din+dout)r(d_{\mathrm{in}}+d_{\mathrm{out}}) per projection (vs. the full din⋅doutd_{\mathrm{in}}\cdot d_{\mathrm{out}} for full fine-tuning).

The domain shift on Φtune\Phi_{\mathrm{tune}} acts along a low-dimensional subspace—the projection of hidden states onto numerical-magnitude directions (ego speed, following distance, lead-vehicle velocity) where source and target distributions differ—rather than as a full-rank drift. A rank-rr correction with r≪dr\ll d therefore captures the relevant component of the shift while preserving the source-side computation.

This empirical low-rank structure matches the rationale behind LoRA in language models [17]: the intrinsic dimension of task-specific adaptation is small, so a rank-rr correction with r≪dr \ll d suffices.

Adaptive operator composition

The role of the adaptive composition is to wire the three zone-specific operators (MMD on Φalign\Phi_{\mathrm{align}}, LoRA on Φtune\Phi_{\mathrm{tune}}, freeze on Φfreeze\Phi_{\mathrm{freeze}}) into a single training objective, with gradients routed by zone, and to handle the regime where one of the zones is empty (the prefix-LM case, Φtune=∅\Phi_{\mathrm{tune}}=\varnothing on PaliGemma2). The closest baseline in spirit is surgical fine-tuning [20], which selectively unfreezes layers based on a manual shift-type heuristic (input-shift →\to early, feature-shift →\to middle, output-shift →\to late). Our composition differs in two ways: (i) zone selection is data-driven via CIS rather than heuristic, and (ii) the operator type per zone (MMD vs LoRA vs freeze) is matched to the type of shift CIS measures, rather than uniformly applying LoRA-then-freeze.

Gradient flow by zone. The total objective Ltotal\mathcal{L}_{\mathrm{total}} (Eq. (16)) routes gradients to three disjoint parameter sets, one per zone:

∇Ltotal={λMMD∇{Al,Bl}LMMDfor attention-only LoRA params on l∈Φalign,∇{Al,Bl}Ltaskfor LoRA parameters (all projections) on l∈Φtune,0for all parameters on l∈Φfreeze.(31)\nabla\mathcal{L}_{\mathrm{total}} = \begin{cases} \lambda_{\mathrm{MMD}}\nabla_{\{A_l,B_l\}}\mathcal{L}_{\mathrm{MMD}} & \text{for attention-only LoRA params on } l \in\Phi_{\mathrm{align}},\\ \nabla_{\{A_l,B_l\}}\mathcal{L}_{\mathrm{task}} & \text{for LoRA parameters (all projections) on } l \in\Phi_{\mathrm{tune}},\\ 0 & \text{for all parameters on } l \in\Phi_{\mathrm{freeze}}. \end{cases} \tag*{(31)}

Here {Al,Bl}\{A_l, B_l\} are the LoRA factors at layer ll (Eq. (30)); on Φalign\Phi_{\mathrm{align}} they are attached only to the attention projections (Q, K, V, O), and on Φtune\Phi_{\mathrm{tune}} they are attached to all linear projections (attention + MLP). LMMD\mathcal{L}_{\mathrm{MMD}} is the per-batch image-token MMD aggregated over Φalign\Phi_{\mathrm{align}}, Ltask\mathcal{L}_{\mathrm{task}} is the trajectory-prediction loss on labelled source data, and λMMD=0.5\lambda_{\mathrm{MMD}} = 0.5 is the loss weight. Target-domain samples contribute to LMMD\mathcal{L}_{\mathrm{MMD}} without labels — only their hidden states appear in the cross-distribution kernel term — so CISR remains an unsupervised DA method.

Active operator set and 2-operator degeneration. The operator-to-zone assignment is not a fixed recipe: it is driven by which zones the CIS diagnosis actually discovers in fΘ′f_{\Theta'}. Let O={MMD,LoRA,freeze}\mathcal{O} = \{\mathrm{MMD}, \mathrm{LoRA}, \mathrm{freeze}\} be the three candidate operators and let π:{align,tune,freeze}→O\pi: \{\mathrm{align},\mathrm{tune},\mathrm{freeze}\} \to\mathcal{O} be the fixed role map π(align)=MMD\pi(\mathrm{align})=\mathrm{MMD}, π(tune)=LoRA\pi(\mathrm{tune})=\mathrm{LoRA}, π(freeze)=freeze\pi(\mathrm{freeze})=\mathrm{freeze}. The active operator set selected by CISR is

Oactive(fΘ′)={π(z):z∈{align,tune,freeze} and Φz≠∅},(32)\mathcal{O}_{\mathrm{active}}(f_{\Theta'}) = \left\{ \pi(z) : z \in\{\mathrm{align},\mathrm{tune},\mathrm{freeze}\} \text{ and } \Phi_z \ne\varnothing\right\}, \tag*{(32)}

where Φz\Phi_z is the zone-zz partition discovered by CIS (Eqs. (11)–(13)). When all three zones are non-empty (the causal-decoder regime on LFM2-VL and Qwen2-VL), Eq. (32) yields the full 3-operator instantiation MMD + LoRA + freeze whose objective and gradient flow are Eqs. (16) and (31). When Φtune=∅\Phi_{\mathrm{tune}} = \varnothing (the prefix-LM regime on PaliGemma2), the LoRA operator has no layers to attach to: Δ=∅\Delta= \varnothing, and Ltask(Θ′,∅)=Ltask(Θ′,0)\mathcal{L}_{\mathrm{task}}(\Theta', \varnothing) = \mathcal{L}_{\mathrm{task}}(\Theta', 0) is constant in the optimisation variables. The total objective therefore degenerates to

Ltotal(2-op)(Θ′)=λMMD⋅LMMD(Bs,Bt),(33)\mathcal{L}_{\mathrm{total}}^{(2\text{-}\mathrm{op})}(\Theta') = \lambda_{\mathrm{MMD}} \cdot\mathcal{L}_{\mathrm{MMD}}(\mathcal{B}_s,\mathcal{B}_t), \tag*{(33)}

i.e. CISR automatically becomes MMD-on-Φalign\Phi_{\mathrm{align}} + freeze-on-Φfreeze\Phi_{\mathrm{freeze}}, without any user-side recipe change. Concretely, ∣Oactive∣=3|\mathcal{O}_{\mathrm{active}}| = 3 on causal-attention VLMs and ∣Oactive∣=2|\mathcal{O}_{\mathrm{active}}| = 2 on prefix-LM VLMs; experiments in §5.2 validate both regimes.

Full procedure and zone-matched rationale

The three operators are matched to the type of shift that CIS measures in each zone, and each of the three matches is necessary: (i) MMD on Φalign\Phi_{\mathrm{align}} aligns two visual-appearance distributions via a rank-rr LoRA on attention projections: source and target features become statistically indistinguishable on these layers, while the MLP-level feature transformation is preserved (no MLP adapter attached). Since image-channel shift is concentrated here and is high-dimensional, distributional alignment is the right tool. (ii) LoRA on Φtune\Phi_{\mathrm{tune}} (when this zone is non-empty) adds a low-rank correction that is sufficient for the numerical-distribution shift but small enough to avoid overfitting to the source. Since text-channel shift acts along a low-dimensional subspace, a parameter-efficient update is sufficient. (iii) Freeze on Φfreeze\Phi_{\mathrm{freeze}} preserves the task-critical invariant computation that cannot be improved by adaptation. Updating these layers (e.g., via LLRD [16] or full LoRA) only adds noise and overfitting risk; freezing them is strictly better. The assignment is architecture-aware by construction: the 3-operator instantiation emerges on causal decoders where all three zones are non-empty, and the 2-operator instantiation (MMD + freeze) emerges on prefix-LM decoders where the text-dominated zone collapses (Eq. (33)). Applying any single operator uniformly across all layers is strictly worse than this matched assignment; in the extreme case of global MMD (MMD-only ablation), the result is catastrophic (§ 5.4).

Full procedure. Algorithm 1 consolidates the equations from § 4.1–§ 4.2 into a single executable procedure. Lines 1–3 implement CIS (locate); line 4 sets up CISR’s per-zone parameter masks; the inner loop on lines 5–9 implements CISR (repair). On a prefix-LM VLM (Φtune=∅\Phi_{\mathrm{tune}} = \varnothing), line 4’s LoRA-attachment step is a no-op and the Ltask\mathcal{L}_{\mathrm{task}} term contributes no gradient, so the algorithm transparently realises the 2-operator degeneration (Eq. (33)) without any user-side change. Computational cost: lines 1–3 (CIS diagnosis) require 2⋅N⋅L2 \cdot N \cdot L partial forward passes with one swap per pass, totalling ∼0.65\sim0.65 h on a single A10 GPU at N=100N=100, L≈20L \approx20 (App. § 10.2.4); the CISR inner loop (lines 5–9) has the same per-step cost as standard LoRA fine-tuning plus one extra forward pass per layer in Φalign\Phi_{\mathrm{align}} for the MMD term.

Algorithm 1 MoLaRx: Locate-then-Repair for VLM Trajectory Planning

Require: fine-tuned planner fΘ′f_{\Theta'} (Appendix § 9); labelled source Ss\mathcal{S}_s; unlabelled target St\mathcal{S}_t; thresholds τimg,τtxt\tau_{\mathrm{img}}, \tau_{\mathrm{txt}}; weight λMMD\lambda_{\mathrm{MMD}}; paired-sample budget NN.

  1. CIS, sample. Draw NN scene-matched pairs from PpairedP_{\mathrm{paired}} (Definition 3).

  2. CIS, score. For each (m,l)∈{img,txt}×{0,…,L−1}(m,l) \in\{\mathrm{img},\mathrm{txt}\} \times\{0,\ldots,L-1\}, compute Ψ^[m,l]\widehat{\Psi}[m,l] via Eq. (9).

  3. CIS, partition. Form Φalign,Φtune,Φfreeze\Phi_{\mathrm{align}}, \Phi_{\mathrm{tune}}, \Phi_{\mathrm{freeze}} via Eqs. (11) to (13).

  4. CISR, setup. Attach rank-rr LoRA on Φalign\Phi_{\mathrm{align}} (attention only, MMD-driven) and on Φtune\Phi_{\mathrm{tune}} (all projections, task-driven) per Eq. (30); zero-mask all gradients on Φfreeze\Phi_{\mathrm{freeze}} (Eq. (31)).

  5. for each training step do

  6. &nbsp;&nbsp;&nbsp;&nbsp;Sample labelled batch Bs∼Ss\mathcal{B}_s \sim\mathcal{S}_s and unlabelled batch Bt∼St\mathcal{B}_t \sim\mathcal{S}_t.

  7. &nbsp;&nbsp;&nbsp;&nbsp;Compute Ltask(Θ′,Δ;Bs)\mathcal{L}_{\mathrm{task}}(\Theta', \Delta; \mathcal{B}_s) (Eq. (15)) and LMMD(Θ′;Bs,Bt)\mathcal{L}_{\mathrm{MMD}}(\Theta'; \mathcal{B}_s, \mathcal{B}_t) (Eq. (14)).

  8. &nbsp;&nbsp;&nbsp;&nbsp;Backpropagate Ltotal\mathcal{L}_{\mathrm{total}} (Eq. (16)) with the per-zone gradient masking of Eq. (31).

  9. end for

  10. return domain-adapted planner f∗:=fΘ′+Δ∗f^{*} := f_{\Theta' + \Delta^{*}}.

References

  1. [1]Ehsan Ahmadi, Ray Mercurius, Soheil Alizadeh, Kasra Rezaee, and Amir Rasouli. Curb your attention: Causal attention gating for robust trajectory prediction in autonomous driving. arXiv preprint arXiv:2410.0791, 2024.DOI
  2. [2]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.arxiv.org/abs/1610.01644
  3. [3]Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, Tarek Dakhran, Song Duong, Alfred Eng, Fernando Fernandes, Marc Härkönen, Anne Harrington, et al. Lfm2 technical report. arXiv preprint arXiv:2511.23404, 2025.arxiv.org/abs/2511.23404
  4. [4]Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022.arxiv.org/abs/2102.12452
  5. [5]Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. PaLiGemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
  6. [6]Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Balan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.arxiv.org/abs/1903.11027
  7. [7]Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu Zhang. Can we edit multimodal large language models? In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 13877–13888, 2023.arxiv.org/abs/2310.08475
  8. [8]Arthur Conmy, Augustine N Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrà Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352, 2023.arxiv.org/abs/2304.14997
  9. [9]Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. NAVSIM: Data-driven non-reactive autonomous vehicle simulation and benchmarking. In NeurIPS Datasets and Benchmarks Track, 2024.arxiv.org/abs/2406.15349
  10. [10]Angelοs Filos, Panagiotis Tigas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? pages 3145–3153, 2020.arxiv.org/abs/2006.14911
  11. [11]Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. In Journal of Machine Learning Research, volume 17, pages 1–35, 2016.arxiv.org/abs/1505.07818
  12. [12]Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.DOI
  13. [13]Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The journal of machine learning research, 13(1):723–773, 2012.DOI
  14. [14]Ruihua Han, Shuai Wang, Shuaijun Wang, Zeqing Zhang, Jianjun Chen, Shijie Lin, Chengyang Li, Chengzhong Xu, Yonina C. Eldar, Qi Hao, and Jia Pan. Neupan: Direct point robot navigation with end-to-end model-based learning. IEEE Transactions on Robotics, 41:2804–2824, 2025.arxiv.org/abs/2403.06828
  15. [15]Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American statistical association, 58(301):13–30, 1963.
  16. [16]Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Annual Meeting of the Association for Computational Linguistics, pages 328–339, 2018.arxiv.org/abs/1801.06146
  17. [17]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  18. [18]Sarthak Jain and Byron C Wallace. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3543–3556, 2019.
  19. [19]Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022.arxiv.org/abs/2202.10054
  20. [20]Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11466, 2022.arxiv.org/abs/2210.11466
  21. [21]Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1705.10667
  22. [22]Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. GPT-Driver: Learning to drive with GPT. arXiv preprint arXiv:2310.01415, 2023.arxiv.org/abs/2310.01415
  23. [23]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022a.arxiv.org/abs/2202.05262
  24. [24]Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022b.arxiv.org/abs/2210.07229
  25. [26]Poojan Oza, Vishwanath A Sindagi, Vishal M Patel, et al. Unsupervised domain adaptation of object detectors: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6):4018–4040, 2023.DOI
  26. [27]Mozhgan Pourkeshavarz, Junrui Zhang, and Amir Rasouli. Cadet: a causal disentanglement approach for robust trajectory prediction in autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14874–14884, 2024.DOI
  27. [29]Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempc: Large language models as decision makers for autonomous driving. arXiv preprint arXiv:2310.03026, 2023.arxiv.org/abs/2310.03026
  28. [30]Hao Shao, Yuxuan Hu, Letian Wang, Steven L Waslander, Yu Liu, and Hongsheng Li. LMDrive: Closed-loop end-to-end driving with large language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2312.07488
  29. [31]Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. DriveLM: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256–274, 2024.arxiv.org/abs/2312.14150
  30. [32]Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI Conference on Artificial Intelligence, pages 2058–2065, 2016.arxiv.org/abs/1511.05547
  31. [33]Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328, 2017.arxiv.org/abs/1703.01365
  32. [34]Omer Sahin Tas and Royden Wagner. Words in motion: Extracting interpretable control vectors for motion transformers. arXiv preprint arXiv:2406.11624, 2024.arxiv.org/abs/2406.11624
  33. [35]Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. DriveVLM: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024.arxiv.org/abs/2402.12289
  34. [36]Royden Wagner, Omer Sahin Tas, Jaime Villa, Felix Hauser, Yinzhe Shen, Marlon Steiner, Dominik Strutz, Carlos Fernandez, Christian Kinzig, Guillermo S. Gutierrez-Cabello, Hendrik Königshof, Fabian Immel, Richard Schwarzkopf, Nils Alexander Rack, Kevin Rösch, Kaiwen Wang, Jan-Hendrik Pauls, Martin Lauer, Igor Gilitschenski, Holger Caesar, and Christoph Stiller. Longtail driving scenarios with reasoning traces: The KITSScenes LongTail dataset. arXiv preprint arXiv:2603.23607, 2026.
  35. [37]Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. TENT: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726, 2021.arxiv.org/abs/2006.10726
  36. [38]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024.arxiv.org/abs/2409.12191

Paper details

vlnc:W_n8k6mvw6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents