betaFeedback

WorksComputer Science

Zero-Shot VLA Adaptation with Retrieved Guidance

Abstract

Pre-training large parametric "Vision-Language-Action" (VLA) models on large demonstration datasets has recently yielded promising results for general-purpose manipulation. However, these models still struggle under even small domain shifts relative to the training data, such as changes in camera viewpoint, background, object identity, or robot morphology. Motivated by the success of Retrieval-Augmented Generation (RAG) in language models, we pursue retrieval to boost VLA performance. Specifically, we seek to leverage retrieval from domain-mismatched, "in-the-wild" demonstration datasets like DROID to guide a VLA's behavior in a new target domain. To achieve this, we introduce RAGu, which overcomes robotics-specific challenges for RAG: (1) It incorporates simple geometric calibration to transform source demonstrations into the target setup and reduce cross-domain differences. (2) It uses a coarse-to-fine retrieval strategy that selects, from a large pool of pre-warped candidates, the trajectory most compatible with the base VLA. (3) It guides the VLA's flow-matching inference procedure rather than seeking to retrain the base model or expand its limited context window. We evaluate RAGu across real-world, simulation, sim-to-real transfer, and cross-embodiment humanoid manipulation. We find it improves real-world VLA success rates from 20% to 80% and outperforms the closest-related baselines in sim. Project website: raguvla.github.io

INTRODUCTION

Vision-language-action (VLA) models have emerged in recent years as a key step toward generalist robots. However, their ability to generalize still falls short of foundation models in language, vision, and coding. One reason is that, in robotics, both pretraining datasets and the models trained on them remain relatively small, with the latter constrained by the need for near-real-time execution. As a result, VLAs still struggle to generalize across variations in camera viewpoint, scene background, object identity, and robot embodiment.

For language models, “retrieval-augmented generation (RAG)” [1] is a well-established technique to boost performance, especially of small models. Given a new query, RAG fetches relevant documents from a large database, typically through nearest neighbor search in an appropriate feature space, and places them into the model’s context to guide its performance on the target task. This augments the limited “parametric knowledge” of the small language model with “non-parametric knowledge” in the form of the retrieved examples.

In this paper, we seek to develop a RAG-style mechanism to improve generalization in vision-language-action (VLA) models. We target three key problems in implementing RAG for robotics: (1) Cross-Domain Generalization. Robotic data suffers from severe distribution shifts in camera viewpoints, layouts, and embodiments, unlike the universal token space of LLMs. For a given target scenario, even the best-matched trajectories in large-scale robotics datasets are still likely to exhibit such domain-shifts. Prior robotic retrieval and RAG-style methods therefore restrict retrieval pools to narrow, identical-embodiment datasets or rely on task-specific fine-tuning of policy networks [2], [3], [4], fundamentally limiting their overall utility. (2) Policy-Aware Retrieval. Good retrievals must be policy-aware. Proximity based on simple measures like geometry do not suffice for a robot policy to follow a retrieved trajectory smoothly and stably. (3) Context-Efficient Adaptation. VLAs have limited context windows that do not permit directly concatenating retrieved demonstration trajectories to current observations.

We propose Retrieval-Augmented Guidance (RAGu), a zero-shot, training-free VLA adaptation framework depicted in Figure 1, that handles all three challenges:

Overview of RAGu pipeline

Figure 1. Overview of RAGu pipeline. (a) We filter real-world datasets using semantic keywords for broad task categories with similar kinematic profiles, such as pick-and-place via language instruction labels in the datasets. (b) Task-relevant objects in the target scene are identified using semantic bounding box detection to extract target waypoints W′W', allowing the source trajectories to be smoothly adapted despite visual shifts. (c) Retrieval candidates ii are coarsely pruned to the top-NN using Dynamic Time Warping (DTW)-based matching. We then select the optimal trajectory i⋆i^\star from pruned trajectories i′∈[N]i' \in[N] that minimizes the guidance gradient norm (∥∇A0LCi⋆(Xt)∥2)(\lVert\nabla_{A_0} \mathcal{L}_{C_{i^\star}}(X_t)\rVert_2), retrieving the demonstration best aligned with the VLA’s expected Cartesian trajectory XtX_t. (d) At test time, the chosen reference i⋆i^\star defines an energy field (Lwarp≈∥Xt−i⋆∥2)(\mathcal{L}_{\mathrm{warp}} \approx\lVert X_t-i^\star\rVert_2), steering diffusion denoising for zero-shot adaptation.

  1. Cross-Domain Generalization. It handles domain shifts through explicit injection of geometric and semantic priors based on Cartesian spatial traces in source data.

  1. Policy-Aware Retrieval. It selects source trajectories for guidance based on compatibility with the VLA rather than only geometric proximity, and this selection is scaled efficiently through offline pre-warping and lightweight online scoring.

  1. Context-Efficient Adaptation. Finally, rather than concatenating retrieved examples to the VLA’s input, it constructs a test-time guidance field to steer the VLA’s outputs.

Retrieval and In-Context Learning for VLAs. Recent methods adapt VLA models using retrieval and in-context examples. REGENT [3], RICL [4] and RA-VLA [5] retrieve demonstrations to improve adaptation to new tasks or environments, while STRAP [2] retrieves relevant sub-trajectories to support test-time policy learning. Unlike our training-free framework, these methods all require in-domain trajectories as source demonstrations to support parametric adaptation.

Inference-Time Steering of Generative Policies. Works such as VLS [6], Omniguide [7], and DemoDiffusion [8] frame adaptation as inference-time control by using external energy fields to steer generative policies, similar to our RAGu. However, these methods typically assume human demonstrations or depend on sparse, VLM-induced waypoints, and they lack a mechanism to handle large kinematic and spatial mismatches when the guidance comes from structurally different sources. RAGu specifically targets this domain mismatch.

Cross-Domain Generalization with Trace Spaces. Prior works abstract manipulation into 3D representations [9], [10], [11] or train unified models for cross-domain transfer [12], [13], yet both rely on parametric training. In contrast, RAGu requires no retraining, enabling frozen VLAs to leverage cross-domain trajectories beyond their original distribution.

Background

Inference-Time Steering for VLA Policies. At time-step tt, we model the VLA as a flow-matching policy that maps a noisy action representation A0A_0 to a clean action chunk A1=[at,at+1,⋯ ,at+H−1]A_1=[a_t,a_{t+1},\cdots,a_{t+H-1}] over denoising time τ\tau [7], [14]. During inference time, the model approximately integrates a conditional vector field v(Aτ,o)v(A_\tau,o):

dAτdτ=v(Aτ,o),A0∼N(0,I),Aτ∈RH×d,τ∈[0,1](1)\frac{\mathrm{d}A_\tau}{\mathrm{d}\tau}=v(A_\tau,o),\quad A_0\sim\mathcal{N}(0,I),\quad A_\tau\in\mathbb{R}^{H\times d},\quad\tau\in[0,1] \tag*{(1)}

where oo is the current observation, HH is the horizon length of each action chunk, and dd is robot joint space dimension. To adapt at inference time without fine-tuning, we impose a differentiable constraint yy through an energy Ly(Aτ)=−log⁡p(y∣Aτ)\mathcal{L}_y(A_\tau)=-\log p(y\mid A_\tau) [7], yielding the guided velocity

vguided(Aτ,o)=v(Aτ,o)−γ∇AτLy(Aτ),(2)v_{\mathrm{guided}}(A_\tau,o)=v(A_\tau,o)-\gamma\nabla_{A_\tau}\mathcal{L}_y(A_\tau), \tag*{(2)}

where γ\gamma controls guidance strength. This steers action generation at inference time while leaving the policy parameters unchanged. We let y=[yt,yt+1,…,yt+H−1]∈RH×dy=[y_t,y_{t+1},\ldots,y_{t+H-1}]\in\mathbb{R}^{H\times d} denote the desired trajectory over the action horizon, and Ly(Aτ)=12∥Aτ−y∥22\mathcal{L}_y(A_\tau)=\frac{1}{2}\lVert A_\tau-y\rVert_2^2. Prior works have assumed yy to be provided from human operator [7] or an external planner [6].

In-Domain Trajectory Warping. Recent work such as Tether [10] extract critical 3D waypoints W={w1,…,wT}W=\{w_1,\ldots,w_T\} from source demonstrations and map them to target waypoints W′={w1′,…,wT′}W'=\{w'_1,\ldots,w'_T\} via visual correspondences, to handle novel but within-domain scene configurations. Intermediate actions ata_t between wiw_i and wi+1w_{i+1} are then warped to at′=at+((1−α)Δdi+αΔdi+1)a'_t=a_t+((1-\alpha)\Delta d_i+\alpha\Delta d_{i+1}), based on displacements

Δdi=wi′−wi\Delta d_i=w'_i-w_i and interpolation α∈[0,1]\alpha\in[0,1] to execute inverse kinematics. While direct inverse kinematics for cross-domain warped paths yields unreliable, suboptimal execution, we overcome this by treating the warped paths solely as spatial priors, allowing the VLA’s innate generalization to manage the closed-loop manipulation.

Cross-Domain Retrieval-Augmented Guidance for Zero-Shot VLA Adaptation

As motivated in § 1, using retrieved examples from a large out-of-domain source dataset to assist a pre-trained VLA faces three challenges: cross-domain shifts in viewpoint, geometry, or embodiment; the candidate closest in trajectory space is not necessarily one the policy can follow; and the lack of context window for demonstration tokens. As shown in Figure 1, RAGu first filters semantically relevant demonstrations and warps them into the target scene via semantic correspondences, turning cross-domain demonstrations into usable spatial priors. It then retrieves the guiding trajectory via coarse offline dynamic time-warping (DTW)-based pruning followed by policy-compatible selection at inference time, avoiding control-loop overhead. Finally, it steers the VLA’s flow-matching denoising toward the chosen trajectory through an energy-based guidance field, leaving the policy’s weights and context untouched.

Semantic Task-Category Filtering. To ensure alignment with the target task, we first curate an initial pool of structurally compatible trajectories from large-scale datasets. We achieve this by filtering unstructured real-world datasets [15] using semantic keywords for broad task categories that share similar kinematic profiles. For example, we query demonstrations with keyword "pick" and "place" included for the task, "putting towel on the plate". See Figure 1 (a).

Trajectory Warping Across Different Domains. Unlike in-domain applications, cross-domain environments frequently exhibit significant semantic shifts, often lacking identical object instances or shared categories. To overcome this domain gap, we replace strict in-domain visual matching from prior work [10] with extracting semantic bounding boxes for task-relevant entities in the target scene via Grounding DINO [16], effectively localizing the corresponding target waypoints W′W'. See Figure 1 (b). We define correspondence here not by object similarity, but at the level of functional roles it plays within a task. Once these semantic correspondences are established, the remainder of the trajectory warping process proceeds identically to the formulation described in § 3.

Primary Retrieval: Coarse-grained static filtering via DTW. For each task category, we designate a single reference trajectory by taking an arbitrary valid demonstration—typically the first entry—from the semantically filtered pool. This one-time reference requires no per-episode tuning or meticulous curation. Using this trajectory as a reference, we then apply Dynamic Time Warping (DTW) [17] once for every episode to prune the remaining candidates down to the top-nn trajectories, NN. Consequently, RAGu maintains nn warped candidate trajectories {Wt(i)}i∈[N]\{W_t^{(i)}\}_{i\in[N]} in the target frame, where Wt(i)={wt(i),…,wt+H−1(i)}W_t^{(i)}=\{w_t^{(i)},\ldots,w_{t+H-1}^{(i)}\} denotes the waypoint sequence of candidate ii at time step tt. Note that RAGu does not assume matched domains in the source data: Across all experiments, target scenes differ from retrieved demonstrations in viewpoint, appearance, object identity, and goal location, requiring only shared task categories.

Secondary Retrieval: Fine-grained inference-time policy alignment. At time-step tt, we estimate the VLA’s output action chunk A^τ\hat{A}_{\tau} referred from § 3 with τ=1\tau=1, and map to a Cartesian end-effector path Xt={xt,…,xt+H−1}X_t=\{x_t,\ldots,x_{t+H-1}\} via forward kinematics applied to A^τ\hat{A}_{\tau}. For candidate trajectory Wt(i)W_t^{(i)}, we define

Ci(Xt)=∥Xt−seqMatch⁡(Xt,Wt(i))∥22(3)C_i(X_t)=\left\lVert X_t-\operatorname{seqMatch}(X_t,W_t^{(i)})\right\rVert_2^2 \tag*{(3)}

where seqMatch is a monotone nearest-neighbor projection approach (details on website). We select the trajectory whose cost gradient has the smallest norm:

i∗=arg min⁡i∈N∥∇A0Ci(Xt)∥2(4)i^*=\underset{i\in N}{\operatorname{arg\,min}}\left\lVert\nabla_{A_0}C_i(X_t)\right\rVert_2 \tag*{(4)}

To understand this choice, observe that this gradient norm is an estimate of how much the VLA must be steered away from its original intent to reduce CiC_i. By selecting trajectories with small gradient values, we obtain reference paths that stay close to the VLA’s learned priors over optimal actions. We perform this selection once at the first denoising step of each inference and keep the selected trajectory Wt(i∗)W_t^{(i^*)} fixed for the remaining denoising steps. This may be thought of as the last and “finest” stage of our multi-stage coarse-to-fine retrieval process for implementing “in-the-loop” near-real-time retrieval augmentation for VLAs.

Inference-Time Steering with Action Denoising. Once i∗i^* is selected, we use the corresponding warped trajectory to instantiate the inference-time guidance energy introduced in § 3:

Lwarp(Aτ)=Ci∗(Xt).(5)\mathcal{L}_{\mathrm{warp}}(A_{\tau})=C_{i^*}(X_t). \tag*{(5)}

Substituting Lwarp\mathcal{L}_{\mathrm{warp}} for LY(Aτ)\mathcal{L}_{\mathcal{Y}}(A_{\tau}) into Equation (2) steers the denoising process toward the retrieved trajectory while preserving the VLA prior. Candidate retrieval is performed once at the beginning of inference, and the selected trajectory remains fixed throughout the remaining denoising steps.

Experiments

Based on the three challenges introduced in § 1, our experiments study the following: (1) Context-Efficient Adaptation. How can retrieved demonstrations guide a VLA without consuming its limited context window? (2) Cross-Domain Generalization. Can cross-domain demonstrations guide a frozen VLA in new domains? (3) Policy-Aware Retrieval. Does retrieving dynamically compatible trajectories improve VLA performance?

Experimental Setup

Tasks & Environments. We evaluate four transfer settings using 10 rollouts for real-world experiments and 25 for simulation experiments. These include real-to-real (DROID [15] 7-DoF Franka, 5 tasks), real-to-sim (DROID to RoboLab [18], 9 tasks), sim-to-real (MetaWorld [19] Sawyer to real DROID, multi-stage Banana-to-Plate task), and cross-morphology (DROID to Fourier GR-1 humanoid in RoboCasa [20], 5 tabletop tasks [21]), shown in Figure 2. DROID provides the source demonstrations for all except the sim-to-real setting, where it is publicly available and does not need additional data collection. Refer to Figure 3 and the project website.

RAGu enables zero-shot adaptation across domains and embodiments

Figure 2. RAGu enables zero-shot adaptation across domains and embodiments. RAGu transfers manipulation knowledge across four settings: Real→Real, Real→Sim, Sim→Real with different morphologies with the base VLA’s weights completely frozen without any parameter update.

Real-to-sim cross-embodiment warping examples

Figure 3. Real-to-Sim Cross-Embodiment Warping. Real-world DROID demonstrations are warped to provide structural priors in simulated environments. The background streams visualize the distribution of candidate trajectories retrieved via coarse-grained DTW filtering.

Baselines. Retrieval-based in-context adaptation baseline: We compare against RICL [4], which performs retrieval from exactly the same pool of 100 warped DROID demonstrations as RAGu during test-time, and uses retrieved trajectories as input context for the VLA to infer actions. We use its released checkpoint, previously trained for in-context adaptation, with no test-time weight updates. VLM-based goal steering baseline: We also compare against Omniguide [7] (real world) and VLS [6] (simulation), a steering method that uses a Vision-Language Model to predict static goal object locations and defines an energy function around them. Compared to RAGu, these models replace retrieval and warping procedures with an external VLM, thereby representing approaches that condition on sparse waypoints or goal states rather than continuous trajectories. Base VLA policy baseline: π0.5\pi_{0.5}-DROID [14] (real world, Figure 7) and GR00T-N1.7 (simulation) were used, while both pre-trained and post-trained GR00T-N1.5 open-sourced checkpoints [21] were used for GR-1 humanoid experiments as the frozen base policies. No weight updates occur during test time for all of the models in our experiments.

Hardware configuration for real-world evaluation

Figure 7. Hardware configuration for real-world evaluation. We use the DROID [15] setup: a 7-DoF Franka Emika Panda arm, a Robotiq 2F-85 parallel-jaw gripper, a wrist-mounted ZED Mini RGB-D camera, and two side-mounted ZED 2 stereo cameras.

Metrics, Protocol, and Measurements. Our primary metric is task success rate (Figure 4 and Figure 6; Table I, Table II, and Table IV}) over 10 rollouts per task in the real world and 25 in simulation. To characterize execution beyond binary success, we report three end-effector trajectory metrics (Table III and {xref}Table V) including EE SPARC (spectral arc length {citep}[22]`), a unitless smoothness measure in which higher value means smoother trajectory. We also report inference latency as mean

Real-world success rates across five manipulation tasks

Figure 4. Real-World Success Rates. Comparing RAGu against the base flow-matching policy (π0.5\pi_{0.5}), retrieval-based in-context learning baseline (RICL) and a VLM-based goal steering baseline (Omniguide). We evaluate the models across 5 diverse manipulation tasks: Open Drawer, Open Cabinet, Close Microwave, Towel to Plate, & Wipe Table.

TaskGR00Tπ0.5\pi_{0.5}RICLVLSRAGu
BananaInBowl0401236
MouseOnKeyboard01201624
MustardAboveRaisin404816
Average1.335.331.331225.33

Table I. RoboLab Real-to-Sim Success Rates. Comparing RAGu vs. base VLA policies (π0.5\pi_{0.5}, GR00T-N1.7), RiCL, and VLS. For fair comparison, RiCL and RAGu use the same pool of 100 demonstrations from DROID. VLS uses π0.5\pi_{0.5} as the base model. Also see Tab IV

TaskGR00T-N1.5 (w/o FT) BaseOurs (1 traj)Δ\DeltaOurs (100 traj)Δ\DeltaGR00T-N1.5 (w/ FT) BaseOurs (1 traj)Δ\DeltaOurs (100 traj)Δ\Delta
CuttingboardToPot412+812+848+48+4
PlacematToPlate4860+1272+244872+2472+24
PlateToBowl4444+056+126872+480+12
PlateToPlate5644-1260+42040+2036+16
TrayToPlate1636+2036+204440-448+4
Average33.639.2+5.647.2+13.636.846.4+9.648.8+12

Table II. Cross-embodiment humanoid manipulation with in-context trajectory guidance. We report success rates on RoboCasa tasks using the base GR00T-N1.5 model and its fine-tuned variant. The Δ\Delta columns indicate absolute improvement over the corresponding unguided baseline. Refer to the project website for confidence intervals.

Figure 6. Sim-to-real cross-embodiment transfer. Success rates (%) for each object-transfer setting. We warp 100 expert trajectories from the MetaWorld Sawyer robot (bin-picking-v3) to the real-world DROID setup.

TaskGR00Tπ0.5\pi_{0.5}π0.5\pi_{0.5} + RAGu (1)π0.5\pi_{0.5} + RAGu (10)π0.5\pi_{0.5} + RAGu (100)Δ\Delta
BananaInBowl04282836+32
MouseOnKeyboard01216824+12
MustardAboveRaisin408016+16
WoodSpatulaToBowl404012+12
RubiksCubeOrBanana044524868+24
SauceBottlesCrate04121612+8
TakeMeasuringSpoonOut036324436+0
RubiksCube052206864+12
ToyInBin2860847660+0
Mean4.023.628.432.036.4+12.9

Table IV. Scaling the retrieval pool on Real-to-Sim RoboLab. Task-wise success rates (%, 25 rollouts per task) for GR00T-N1.7, π0.5\pi_{0.5}, and π0.5\pi_{0.5} + RAGu guided by 1, 10, or 100 warped DROID demonstrations. Δ\Delta is the absolute gain of the 100-trajectory pool over π0.5\pi_{0.5}; bold marks the best entry in each row. Expanding the pool consistently helps: mean success rises from 23.6% (unguided) to 28.4% with a single trajectory and 36.4% with 100. See Table VII in the Appendix for the full version.

wall-clock time per episode (Table VI). Figure 5, Figure 3, Figure 7, and Figure 8 provide qualitative visualizations. Detailed values and rollout videos are on the project website. In the rest of this section, we analyze these raw measurements and qualitative results, seeking answers to our key questions.

Execution rollout comparison on the Wipe Table task

Figure 5. Execution rollout comparison on the Wipe Table task. RAGu continuously guides the end-effector along the table surface to execute a successful wipe. In contrast, Omniguide lifts the towel off the surface when transporting it toward the next discrete waypoint, failing to maintain contact with the table. RICL fails. (videos on website)

GR00T-N1.5BaseOurs-1Δ\DeltaOurs-100Δ\Delta
w/o FT34.131.0-3.133.6-0.5
w/ FT26.830.3+3.532.7+5.9

Table VI. Average inference wall-clock cost (s/episode). We measure inference wall-clock cost of 5 tasks in Table II (max time steps/episode 720, control rate 10 Hz, 25 episodes/task, green: faster, red: slower). The horizon, episode count, task set, and simulator configuration are held constant across all four methods.

Failure-mode breakdown in RoboLab

Figure 8. Failure-mode breakdown in RoboLab. Sankey diagram showing the distribution of warping outcomes across successful executions and major failure categories. Demo parsing errors (22.0%) are source-side: the DROID demonstration exhibits end-effector behavior incompatible with the target task, such as multiple or missing gripper open/close events or an incorrect grasp-release order, and is discarded before warping. Scene correspondence errors (10.4%) are target-side: triangulation failure, cross-view inconsistency, or out-of-bounds projection of the detected waypoints. These percentages describe warping outcomes, not task success rates, which are reported in Table VII.

TaskGROOTπ0.5\pi_{0.5}π0.5\pi_{0.5} + Ours (1 traj) SRΔ\Deltaπ0.5\pi_{0.5} + Ours (10 traj) SRΔ\Deltaπ0.5\pi_{0.5} + Ours (30 traj) SRΔ\Deltaπ0.5\pi_{0.5} + Ours (50 traj) SRΔ\Deltaπ0.5\pi_{0.5} + Ours (100 traj) SRΔ\Delta
BananaInBowl0 [0.0, 13.3]4 [0.7, 19.5]28 [14.3, 47.6]+2428 [14.3, 47.6]+2428 [14.3, 47.6]+2424 [11.5, 43.4]+2036 [20.2, 55.5]+32
MouseOnKeyboard0 [0.0, 13.3]4 [4.2, 30.0]8 [2.2, 25.0]-416 [6.4, 34.7]+420 [8.9, 39.1]+824 [11.5, 43.4]+12
MustardAboveRaisin4 [0.7, 19.5]0 [0.0, 13.3]8 [2.2, 25.0]+80 [0.0, 13.3]+00 [0.0, 13.3]+00 [0.0, 13.3]+016 [6.4, 34.7]+16
WoodSpatulaToBowl4 [0.7, 19.5]0 [0.0, 13.3]4 [0.7, 19.5]+40 [0.0, 13.3]+08 [2.2, 25.0]+84 [0.7, 19.5]+412 [4.2, 30.0]+12
RubiksCubeOrBanana0 [0.0, 13.3]44 [26.7, 62.9]52 [33.5, 70.0]+848 [30.0, 66.5]+452 [33.5, 70.0]+856 [37.1, 73.3]+1268 [48.4, 82.8]+24
SauceBottlesCrate0 [0.0, 13.3]4 [0.7, 19.5]8 [2.2, 25.0]+816 [6.4, 34.7]+128 [2.2, 25.0]+424 [11.5, 43.4]+2012 [4.2, 30.0]+8
TakeMeasuringSpoonOut0 [0.0, 13.3]32 [20.2, 55.5]28 [17.2, 51.6]-444 [26.7, 62.9]+832 [17.2, 51.6]-436 [20.2, 55.5]+036 [20.2, 55.5]+0
RubiksCube0 [0.0, 13.3]52 [33.5, 70.0]20 [8.9, 39.1]-3268 [48.4, 82.8]+1664 [44.5, 79.8]+1264 [44.5, 79.8]+1264 [44.5, 79.8]+12
ToyInBin28 [14.3, 47.6]60 [40.7, 76.6]76 [65.3, 93.6]+2476 [56.6, 88.5]+1664 [44.5, 79.8]+464 [44.5, 79.8]+460 [40.7, 76.6]+0
Mean4.0 [2.1, 7.4]23.6 [18.5, 29.5]28.4 [22.9, 34.7]+4.932.0 [26.3, 38.4]+8.430.2 [24.6, 36.5]+6.732.4 [26.7, 38.8]+8.936.4 [30.4, 42.9]+12.9

Table VII. Evaluation on Real-to-Sim RoboLab tasks with different numbers of guidance trajectories. Task-wise success rates (%, 25 rollouts per task) across GROOT-N1.7, π0.5\pi_{0.5}, and π0.5\pi_{0.5} + RAGu guided by 1, 10, 30, 50, or 100 warped DROID demonstrations. Values in brackets are 95% Wilson confidence intervals. Each Δ\Delta column shows the absolute improvement over π0.5\pi_{0.5}. Multi-stage retrieval is required both to cover large domain gaps and to meet real-time constraints. Shrinking the retrieval pool to a single trajectory reduces mean RoboLab success from 36.4% to 28.4%.

Context-Efficient Adaptation

First, we investigate whether retrieved demonstrations can improve a frozen VLA without expanding its context window or updating its weights. We compare RAGu against two dominant test-time baselines: in-context retrieval (prompt concatenation) and VLM goal steering (sparse waypoint conditioning).

RAGu vs. In-Context Retrieval. See Figure 4 and Table I. Using the same pool of 100 warped DROID demonstrations, RICL [4] provides examples as in-context visual tokens. This yields zero-shot failure across all real-world tasks (0%, Figure 4) and 1% in RoboLab (Table I), falling below the unguided π0.5\pi_{0.5} baseline (20% and 5%). In contrast, RAGu achieves 80% and 25%. In-context examples force the VLA to visually resolve domain gaps zero-shot, whereas trajectory guidance circumvents the context window to provide direct spatial priors the policy can immediately follow. Figure 5 shows an execution.

RAGu vs. VLM Goal Steering. Continuing with Figure 4 and Table I, Omniguide [7] (real-world) and VLS [6] (simulation) employ identical inference steering as RAGu, but steer toward sparse VLM goal points rather than dense trajectories. RAGu doubles their success in both settings (80% vs. 40% real-world; 25% vs. 12% RoboLab). This gap is pronounced in contact-rich tasks like Wipe Table (100% vs. 30%), where sparse goals cause Omniguide to break surface contact between waypoints (Figure 5). Dense trajectories preserve approach and contact constraints that discrete goals omit.

Varying Base Model Competence. As shown in Table II, RAGu improves both base and fine-tuned GR00T-N1.5 models (+13.6% and +12.0% with 100 trajectories). Rather than acting as a redundant replacement for parametric training, test-time trajectory guidance provides complementary gains across varying baseline policy capacities.

Cross-Domain Generalization

We evaluate whether demonstrations from mismatched domains can guide a frozen VLA across shifts in viewpoint, scene, object identity, and morphology.

Crossing Sim→Real & Real→Sim Gaps. For real-to-sim, warping real-world DROID demonstrations into RoboLab raises π0.5\pi_{0.5} success from 5.33% to RAGu’s 25.33% (Table I). For sim-to-real, warping 100 MetaWorld Sawyer trajectories to a real Franka setup enables multi-stage Banana→Plate execution (Figure 6). This gain peaks in Cabinet→Shelf, where both π0.5\pi_{0.5} and Omniguide fail (0%) while RAGu achieves 60%. In this out-of-distribution layout, operating on Cartesian traces makes guidance agnostic to whether sources are rendered or recorded.

Embodiment Shifts. We test for morphological shift by using 7-DoF Franka DROID demonstrations to guide GR00T-N1.5 on a 29-DoF Fourier GR-1 humanoid (Table II), along with the results shown previously in Figure 6 from Sawyer to Franka. With 100 candidates, average success rises by +13.6%. This cross-embodiment transfer succeeds because guidance targets Cartesian wrist trajectories via differentiable forward kinematics, avoiding any explicit joint mapping, inverse kinematics, or trajectory replay. Refer to App. VI for implementation details.

Visual and Environmental Diversity. We also highlight that all warped demonstrations are sourced from DROID [15], which natively exhibits diversity in camera viewpoints, backgrounds, and object instances from our target setting.

Policy-Aware Retrieval

We now evaluate design choices within the trajectory selection rule of the secondary retrieval phase.

Secondary Retrieval Candidate Selection Rule. Table III

EE SPARC ↑\uparrowPath (m) ↓\downarrowSpeed (cm/s)
RAGu (1 traj)-9.934.618.20
RAGu (min dist)-11.745.797.09
RAGu (min grad)-10.305.678.37

Table III. Utility of policy-aware secondary retrieval. Trajectory metrics on RoboLab for three tasks in Table I, 25 rollouts per task. We use the same pool of 100 warped DROID demonstrations, while only the rule that picks the guiding trajectory changes. 1 traj uses only a single heuristically selected trajectory. min dist selects arg⁡min⁡iCi(Xt)\arg\min_i C_i(X_t) (Eq. 3), and min grad selects arg⁡min⁡i∥∇A0Ci(Xt)∥2\arg\min_i \lVert\nabla_{A_0} C_i(X_t)\rVert_2 (Eq. 4) from a 100 trajectory pool. Refer to Secondary Retrieval ablation in the project website.

evaluates Secondary Retrieval from a fixed pool of demonstration candidates. Replacing RAGu’s gradient-norm selection (min grad) with nearest-neighbor selection (min dist) causes slower (7.09 vs. 8.37 cm/s) and less smooth (EE SPARC −11.74 vs. −10.30) motion at similar path lengths (5.79 vs. 5.67 m). A plausible explanation is that geometric proximity ignores the policy’s learned priors, so min dist may repeatedly pull the policy away from its intended motion and vice versa, whereas min grad favors references that are more compatible with the policy. In Table IV, while a single reference achieves smoother motion, it drops success to 17.3% compared to using 100 trajectories (25.3%) due to limited scene coverage. min grad retains the coverage of candidate pooling while recovering execution smoothness.

Benefits of Larger Retrieval Pools. Expanding the candidate pool consistently enhances performance. Across nine RoboLab tasks (Table IV), for a base unguided π0.5\pi_{0.5} model averaging 23.6% success, RAGu boosts performance to 28.4% with a single demonstration retrieval pool, and further to 36.4% with 100 demos. Trajectory quality mirrors these gains (Table V): the 100-demo pool produces the smoothest (−9.32) and most direct (5.25 m) paths at comparable speeds, as denser candidate pools provide references that simultaneously match the target scene and align with the policy prior.

EE SPARC ↑\uparrowPath length (m) ↓\downarrowSpeed (cm/s)
π0.5\pi_{0.5}-9.375.939.37
π0.5\pi_{0.5} + RAGu (1 traj)-9.355.378.67
π0.5\pi_{0.5} + RAGu (100 traj)-9.325.258.89

Table V. Trajectory quality metrics across selected tasks in RoboLab. Using 100 trajectories reduces path length and increases smoothness at similar speeds, improving execution efficiency. We use the same selected tasks as Tab. IV. Refer to Trajectory quality in the project website.

Inference Efficiency. Because cross-domain warping is handled offline and the nn candidates are evaluated only once at the first denoising step, selection incurs minimal computational overhead. As shown in Table VI, mean wall-clock time per episode across five RoboCasa tasks remains within 6 s of the unguided baseline over 720-step episodes at 10 Hz. Consequently, the retrieval pool can scale by two orders of magnitude without degrading closed-loop control frequencies.

Conclusion

We presented Retrieval-Augmented Guidance (RAGu), a zero-shot, training-free framework for adapting pretrained VLAs to OOD environments by warping cross-domain trajectories into the target geometric space and using them as guidance during flow-matching denoising. Across real-to-real, real-to-sim, sim-to-real, and arm-to-humanoid settings, RAGu improves frozen VLA policies without any parameter update, including transfer from simulated Sawyer demonstrations to multi-stage real-world Franka manipulation, while larger retrieval pools further improve robustness.

Limitations. RAGu assumes that the source database contains demonstrations that approximately match the target task category. The method is still not invariant to camera placement and occlusion, since semantic correspondence failures degrade 3D warping and corrupt the guidance signal. Finally, guidance can only reshape motion within the VLA’s prior.

Acknowledgements

This project was supported in part by DARPA TIAMAT HR0011249042, NSF SLES 2331783, Penn Engineering Futures Fund, NSF CAREER 2239301, ONR N00014-22-1-2677.

Appendix

Clarification of Confidence Intervals (Figure 4, Figure 6)

Error bars represent ±1\pm1 standard error of the binomial proportion, computed as SE=p(1−p)/n\mathrm{SE} = \sqrt{p(1-p)/n}, where pp is the empirical success rate over nn rollouts (n=10n = 10 per task, and n=50n = 50 for the pooled average) and whiskers are clipped to [0%,100%][0\%, 100\%]. Because the standard error evaluates to zero when p∈{0,1}p \in\{0,1\}, bars at 0% and 100% success do not display whiskers.

Hyperparameter Selection and Guidance Sensitivity

Table IX shows the selected hyperparameters for experiments. The guidance strength parameter γ\gamma balances policy priors with external trajectory constraints. Refer to Omniglide [7] for comprehensive parameter-sweep curves on gradient-based flow steering.

Failure Mode Analysis. For the RoboLab tasks, we estimate a 65.1% warping success rate over 900 total attempts, with the remaining 34.9% attributed to failures in Figure 8. This suggests that demo-side trajectory parsing errors are the dominant source of error, exceeding scene correspondence failures by roughly a factor of two. This indicates that improving source-demo quality control is likely to provide a larger benefit than further tuning the correspondence stage.

Humanoid implementation details GROOT N1.5 outputs

SourceSuccess (%)EE SPARCSpeed (cm/s)
π0.5\pi_{0.5} - 25 rollouts/task14.0−9.04±3.91-9.04 \pm3.9110.3±2.110.3 \pm2.1
π0.5\pi_{0.5} - 10 rollouts/task [18]28.0−8.34±6.67-8.34 \pm6.675.4±1.65.4 \pm1.6

Table VIII. π0.5\pi_{0.5}-DROID sanity check on all 45 RoboLab tasks. When re-evaluating the unguided base policy against the statistics reported in RoboLab [18], we obtain a 14.0% average success rate compared to their reported 28.0%. We believe that the differences in performance may be due to the fact that the original paper evaluated on only 10 rollouts per task, so their reported success rates may be noisy. For fairness, we evaluate all methods in our tables are evaluated under identical conditions, ensuring that relative comparisons remain valid.

DROID (Real)DROID (RoboLab)GR-1 (RoboCasa)
Top-NN8163
Loss weight λ\lambda1.01.01.0
Guidance scale γ\gamma0.2–2.00.2–1.00.4, 0.5

Table IX. RAGu hyperparameters. NN is the number of DTW-pruned trajectories retained; λ\lambda weights the trajectory guidance loss; and γ\gamma is the denoising guidance scale.

normalized GR-1 joint-space action chunks, which are denormalized and mapped to a Cartesian right-palm trajectory by differentiable forward kinematics. The tracked palm point is an 8 cm offset along the negative local zz-axis of the right-hand frame. The predicted trajectory and warped DROID reference are both expressed in the mobilebase0_support and matched sequentially for guidance. We use no inverse kinematics or direct trajectory replay. We add two base-fixed 60° warp cameras at [0.15,±0.05,0.5][0.15, \pm0.05, 0.5] to obtain stable calibrated target-scene views.

References

References

  1. [1]P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020. [2] M. Memmel, J. Berg, B. Chen, A. Gupta, and J. Francis, “Strap: Robot sub-trajectory retrieval for augmented policy learning,” in The Thirteenth International Conference on Learning Representations, 2025.
  2. [3]K. Sridhar, S. Dutta, D. Jayaraman, and I. Lee, “Regent: A retrieval-augmented generalist agent that can act in-context in new environments,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 39 385–39 414.arxiv.org/abs/2412.04759
  3. [4]———, “Adapting in-context adaptability to pre-trained vision-language-action models,” arXiv preprint arXiv:2508.02062, 2025.
  4. [5]S. Jang, M. Jeon, M. Kim, S. Choi, D. Kim, and H. Yu, “Ra-vla: Retrieval-augmented vla for test-time adaptation,” arXiv preprint arXiv:2608.25585, 2026.arxiv.org/abs/2608.25585
  5. [6]S. Liu, I. S. Singh, X. Xu, J. Duan, and R. Krishna, “VIs: Steering pretrained robot policies via vision-language models,” arXiv preprint arXiv:2602.03973, 2026.arxiv.org/abs/2602.03973
  6. [7]Y. Song, L. Le, Y.-H. Park, J. Wang, J. Shi, L. Liu, J. Gu, E. Eaton, D. Jayaraman, and K. Daniilidis, “Omniguide: Universal guidance fields for enhancing generalist robot policies,” arXiv preprint arXiv:2603.10052, 2026.arxiv.org/abs/2603.10052
  7. [8]S. Park, H. Bharadhwaj, and S. Tulsiani, “Demodiffusion: One-shot human imitation using pre-trained diffusion policy,” arXiv preprint arXiv:2506.20668, 2025.arxiv.org/abs/2506.20668
  8. [9]J. Lee, J. Duan, H. Fang, Y. Deng, S. Liu, B. Li, B. Fang, J. Zhang, Y. R. Wang, S. Lee, et al., “Molmoact: Action reasoning models that can reason in space,” arXiv preprint arXiv:2508.07917, 2025.arxiv.org/abs/2508.07917
  9. [10]W. Liang, S. Wang, H.-J. Wang, O. Bastani, Y. J. Ma, and D. Jayaraman, “Tether: Autonomous functional play with correspondence-driven trajectory warping,” arXiv preprint arXiv:2603.03278, 2026.arxiv.org/abs/2603.03278
  10. [11]J. Zhang, M. Memmel, K. Kim, D. Fox, J. Thomason, F. Ramos, E. Biyık, A. Gupta, and A. Li, “Peek: Guiding and minimal image representations for zero-shot generalization of robot manipulation policies,” arXiv preprint arXiv:2509.18282, 2025.arxiv.org/abs/2509.18282
  11. [12]J. Shi, Z. Zhao, T. Wang, I. Pedroza, A. Luo, J. Wang, J. Ma, and D. Jayaraman, “Zeromimic: Distilling robotic manipulation skills from web videos,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 16 939–16 947.arxiv.org/abs/2503.23877
  12. [13]H. Li, L. Sun, Y. Hu, D. Ta, J. Barry, G. Konidaris, and J. Fu, “Novaflow: Zero-shot manipulation via actionable flow from generated videos,” arXiv preprint arXiv:2510.08568, 2025.arxiv.org/abs/2510.08568
  13. [14]14* K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, b. ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky, “π0.5: a vision-language-action model with open-world generalization,” in Proceedings of The 9th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, J. Lim, S. Song, and H.-W. Park, Eds., vol. 305. PMLR, 27–30 Sep 2025, pp. 17–40. [Online]. Available: https://proceedings.mlr.press/v305/black25a.htmlarxiv.org/abs/2504.16054
  14. [15]A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al., “Droid: A large-scale in-the-wild robot manipulation dataset,” arXiv preprint arXiv:2403.12945, 2024.arxiv.org/abs/2403.12945
  15. [16]S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” in European conference on computer vision. Springer, 2024, pp. 38–55.arxiv.org/abs/2303.05499
  16. [17]P. Senin, “Dynamic time warping algorithm review,” Information and Computer Science Department University of Hawaii at Manoa Honolulu, USA, vol. 855, no. 1-23, p. 40, 2008.
  17. [18]J. X. Yang, R. Dagli, A. Zook, H. Hadfield, A. Goyal, S. Birchfield, F. Ramos, and J. Tremblay, “Robolab: A high-fidelity simulation benchmark for analysis of task generalist policies,” arXiv preprint arXiv:2604.09860, 2026.arxiv.org/abs/2604.09860
  18. [19]T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in Conference on robot learning. PMLR, 2020, pp. 1094–1100.arxiv.org/abs/1910.10897
  19. [20]S. Nasiriany, A. Maddakuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu, “Robocasa: Large-scale simulation of everyday tasks for generalist robots,” arXiv preprint arXiv:2406.02523, 2024.arxiv.org/abs/2406.02523
  20. [21]NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. J. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayanan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu, “GR00T n1: An open foundation model for generalist humanoid robots,” 2025. [Online]. Available: https://arxiv.org/abs/2503.14734
  21. [22]S. Balasubramanian, A. Melendez-Calderon, and E. Burdet, “A robust and sensitive metric for quantifying movement smoothness,” IEEE transactions on biomedical engineering, vol. 59, no. 8, pp. 2126–2136, 2011.DOI

Paper details

vlnc:W_j8sa2szt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents