WorksComputer Science
UniLatent: Cross-Embodiment Transfer via Latent Observation Alignment
Abstract
Robot datasets are costly to collect and are usually tied to a specific embodiment, making it difficult to reuse demonstrations across robots with different geometry, appearance, and kinematics. We introduce UniLatent, a cross-embodiment transfer framework that reduces the embodiment-induced observation gap in a shared latent space. UniLatent renders paired source-target robot observations in simulation and aligns a source visual encoder to a frozen target encoder. A policy trained on encoded source demonstrations can then run directly on target observations, with no pixel-level image translation. Across three simulation tasks and two target robot arms, UniLatent reaches up to 93% zero-shot success, exceeding the strongest pixel-translation baseline by over 30 points on average. In real-world xArm-to-Franka transfer, fine-tuning with only 10 target demonstrations per task yields 27/30 successful rollouts. Code, videos and data are available at https://selina2023.github.io/unilatent/.
Introduction
Large-scale and diverse data is a key driver of recent progress in robot learning [1, 2, 3], and recent efforts have produced sizable manipulation datasets [4, 5, 6]. However, existing robot data is highly imbalanced across embodiments, which concentrates on a small set of popular arms and grippers, while many other platforms remain underrepresented. Meanwhile, collecting real-world data is often costly and embodiment-specific, since demonstrations largely depend on the robots’ morphology, end-effector geometry, reachable workspace, appearance, and kinematics. As robot hardware continues to diversify and advance, recollecting the same task for every new embodiment does not scale. This motivates methods that reuse source-robot demonstrations and allow policies to transfer to unseen robots with little or no target data.
A key challenge in cross-embodiment transfer is the embodiment-induced visual gap. In many existing datasets with third-person camera views, the robot often occupies a large portion of the image, and changing the arm or gripper alters not only appearance but also self-occlusion, contact geometry, and the visibility of task-relevant objects, which causes noticeable performance degradation when deploying a vision-based policy on a new robot [7, 8]. Prior work addresses this gap by editing source demonstration frames through masking [9], inpainting [7], or diffusion-based generation [8] to match the target visual domain. While intuitive, pixel-space translation can introduce errors in segmentation, robot geometry, and contact regions, creating systematic discrepancies from true target observations. Moreover, translated frames can create a synthetic visual domain that differs from real target observations, causing policies to overfit to translation artifacts and limiting their ability to generalize across tasks and viewpoints.
We introduce UniLatent, which reduces the embodiment-induced observation gap in latent space rather than pixel space (Figure 1). In simulation, we render large-scale paired observations in which source and target robots reach matched end-effector poses under the same camera and randomized scenes. These pairs train a source encoder to map each source image into the latent space of a frozen, pretrained target encoder, so that matched poses on the two robots receive the same representation. A policy trained on encoded source demonstrations is then deployed on the target robot by encoding its observations with the same frozen target encoder, with no image synthesis at test time.

Figure 1. UniLatent aligns robot observations for cross-embodiment transfer. UniLatent renders paired source–target images and aligns a source encoder to a target encoder, forming a shared latent space that reduces embodiment-induced visual gaps. Top: source and target embeddings increasingly overlap during training. Bottom: cross-decoded source observations become more target-consistent as alignment improves.
Our contributions are two-fold. (i) Paired rendering and observation alignment. We introduce a simulation-based pipeline that generates large-scale paired observations of different robots at matched end-effector poses and uses them to align visual encoders across embodiments. (ii) Latent-space policy learning. We train flow-matching policies on source demonstrations encoded in the aligned latent space, enabling zero-shot transfer to unseen target embodiments in simulation and data-efficient few-shot transfer in the real world. In simulation, UniLatent raises mean target success from 44.5% with the strongest pixel-translation baseline to 76.0%; in the real world, it completes 27/30 rollouts versus 23/30 for RoVI-Aug pretraining and 11/30 for target-only training.
Related Work
Visual adaptation in robotics. Vision-based policies are brittle under visual shifts caused by lighting, texture, backgrounds, and viewpoints. Domain randomization broadens the training distribution by varying rendering factors in simulation [10], while sim-to-real methods learn mappings between visual domains [11, 12, 13]. Recent data augmentation methods further use photometric, semantic, or generative edits on real robot data to improve robustness [14, 15, 16, 17, 18]. These approaches are effective for appearance and viewpoint changes, but most assume that the robot embodiment is fixed. In our setting, the robot itself changes, so the visual shift is tied to kinematics, gripper shape, self-occlusion, and contact cues.
Policy learning across embodiments. Robot-generalist policies train across many robots or condition on embodiment descriptors [19, 20, 21, 22]. Other work studies shared action abstractions, universal actions, or large multi-embodiment datasets [23, 24, 25, 26]. These methods are promising when broad real-robot coverage is available, but such coverage is expensive and uneven across platforms. UniLatent targets a complementary regime: given demonstrations from a source robot and a desired target robot, we learn a target-consistent observation space from simulation-rendered pairs and reuse the source data without requiring a large multi-robot dataset.
Cross-embodiment transfer. A related line studies cross-embodiment imitation from third-person or human video prompts, including skill discovery from human and robot videos [27], imitation under mismatched execution [28], and video-conditioned policy learning [29]. Human-to-robot and human-to-humanoid transfer bridges an even larger embodiment gap, using egocentric human video [30, 31], human-video editing [32], whole-body shadowing [33], or sim-to-real RL from a human demonstration [34]. These works are complementary to our setting: they interpret human or video demonstrations as task specifications or retarget human motion, whereas we study robot-to-robot reuse of source demonstrations for a specified target embodiment.
More recent methods reduce embodiment shift by modifying images. Mirage cross-paints robot pixels [7]; RoVi-Aug synthesizes robot and viewpoint augmentations [8]; and Shadow masks robot regions to avoid robot-specific appearance [9]. While these methods can improve over direct transfer, pixel editing can discard gripper or contact information or introduce geometry that is inconsistent with the real target observation. UniLatent instead aligns encoders and acts in latent space. This removes the need for test-time image translation and lets the target observation encoder define the representation used by the deployed policy. Cross-embodiment and domain-invariant representation learning has also explored alignment objectives in different settings, e.g., inverse RL [35], time-contrastive video representations [36], and adversarial sim-to-real alignment [37]. Closest to ours, Wang et al. [38] align an adapted-domain latent space to a frozen anchor domain without pixel translation, using unpaired adversarial and cycle-consistency objectives. UniLatent instead constructs explicit correspondence by rendering both robots at matched TCP poses, which reduces alignment to supervised feature regression onto a frozen pretrained target encoder.
Problem Statement
We study robot-to-robot cross-embodiment imitation, where demonstrations collected on a source robot are used to learn a policy for a different target robot with minimal or no target-robot demonstrations. Each robot observes the task through RGB images and executes actions through a shared Cartesian end-effector interface. In our setting, the robots are single-arm manipulators with parallel-jaw grippers, but they may differ in geometry, kinematics, gripper appearance, self-occlusion, and robot-object contact cues. We assume access to source demonstrations , where each trajectory is collected on under a fixed third-person camera. The goal is to learn a target policy that solves the same task on . We further assume access to kinematic and visual models of both robots (e.g., URDF or USD assets) in a simulator with inverse kinematics. These models provide paired observations for visual alignment, without requiring target demonstrations or task-success supervision on the target robot. Source demonstrations need not be collected for alignment; alignment consumes only robot models, inverse kinematics, and rendering.
Method: UniLatent
Given source demonstrations collected on , UniLatent learns a latent-space policy that transfers to with few or no target demonstrations. The pipeline has four stages (Figure 2). We first (i) render paired source–target observations in simulation (§ 4.1) and (ii) align a source encoder to a frozen target encoder (§ 4.2). We then (iii) train a latent policy on encoded source demonstrations and (iv) deploy it on through the target encoder (§ 4.3).

Figure 2. Pipeline overview of UniLatent. We render paired (source, target) observations for matched end-effector poses and use them to align encoders by training to match a fixed in a shared latent space. A latent policy is then trained on source demonstrations and deployed on the target robot by encoding target observations with and calling .
Paired Image Generation
We sample manipulation-relevant end-effector poses within the robots’ reachable workspace, solve inverse kinematics for both embodiments, and render both robots from the same camera under the same randomized scene. The source and target images in each pair share the same task-relevant pose but differ in robot geometry and appearance. This produces paired RGB images in IsaacLab/RoboVerse [39, 40] (Figure 3).

Figure 3. Paired image generation. Examples of simulation-rendered paired observations for three robots (top to bottom): Franka, xArm, and Sawyer. For each sampled end-effector pose, we render the robots under randomized backgrounds, lighting, and scene variations while preserving action-level correspondence between paired frames.
The pose generator combines broad workspace sampling with a manipulation prior so that rendered pairs emphasize poses that can occur in demonstrations and avoid unreachable or visually uninformative configurations. We match poses at the tool-center point (TCP), defined as the midpoint between the parallel-jaw fingertips, apply robot-specific TCP offsets, and retain only samples for which inverse kinematics succeeds for both embodiments. Domain randomization varies backgrounds, lighting, object assets, distractors, and small camera perturbations while preserving pairwise action correspondence. These choices make the supervision robust to appearance changes without requiring the renderer to synthesize photorealistic target demonstrations. Our generic pairs (about 100k) use this broad randomized sampling, with roughly 20% of poses drawn from ManiSkill, LIBERO, RLBench, and RoboSuite trajectories that share no tasks with our evaluation suite. Optional task-specific pairs (about 10k) additionally use the evaluation task’s object assets and poses.
Observation Alignment
We instantiate a source encoder and target encoder from DINOv2-base [41]. The target encoder is frozen and defines the deployment representation. For each paired rendering, we train only to match target features:
In practice the loss is applied to the encoder tokens used by the policy, including global and patch-level features. Freezing anchors the latent space to the representation used at deployment and prevents target-side drift: target images are encoded by exactly the model that defines the alignment target. Training only asks each source image to produce the feature of its paired target image, so the policy sees consistent representations when trained on source data and deployed on the target robot.
Latent Policy Training and Deployment
After alignment, each source demonstration is encoded as . A policy conditions on a history and predicts an end-effector action chunk . Actions are represented in a shared Cartesian end-effector space with position, orientation, and a binary gripper command.
We use a conditional flow-matching model [42, 43]: starting from and a demonstrated chunk , it learns the velocity field . At inference, target observations are encoded as . The learned flow is integrated with 10 solver steps, and the resulting pose command is executed by the target robot controller.
Experiments
We design the experiments to answer three questions. (Q1) How large is the embodiment gap when only the action interface is shared? (Q2) Is explicit alignment necessary, or does a single pretrained encoder shared by both robots suffice? (Q3) Under the same alignment, is it better to learn the policy in latent space or on translated pixels? § 5.1 addresses all three questions with zero-shot transfer in simulation, using 100 rollouts per setting. § 5.2 evaluates few-shot real-world transfer from xArm to Franka, including a comparison with the original diffusion-based RoVi-Aug pipeline.
Simulation-Based Experiments
Robots. We use Franka Panda as the source robot and evaluate zero-shot transfer to xArm7 and Sawyer. All three are 7-DoF arms with parallel-jaw grippers, but they differ in link geometry, joint limits, gripper mechanism, and appearance. All robots share a Cartesian TCP-pose action interface with a binary gripper command, mapped to each robot through robot-specific TCP offsets and gripper joint targets (Appendix § 11.§ 2). Source and target share the task, camera, and non-robot scene factors, so the robot embodiment is the only changed factor. Existing datasets such as Open X-Embodiment [4] and DROID [5] cannot support this setting, as each demonstration is recorded on a single robot.
Tasks. We design three simulated tasks in the RoboVerse framework [40] with an Isaac Sim backend (Figure 4). The tasks are designed to test transfer under different manipulation modes: articulated contact, grasp-and-place precision, and nonprehensile pushing. Success is determined from the final object state. Close Box Lid: a box starts with its lid open at , and the robot closes the lid about its hinge; success requires a final hinge angle below . Stack Cup into Bowl: the robot picks up a small cup and places it into a bowl; success requires all predefined cup keypoints to lie inside a bounding box within the bowl. Kick Football: the robot pushes a ball (4 cm radius) into a goal; success requires the ball center to lie inside a bounding box in the goal mouth. Object initial poses are randomized within task-specific ranges for both training and evaluation.

Figure 4. Simulation tasks. Example frames from our simulation experiments. Each row shows representative frames of a Franka demonstration under a fixed third-person camera.
Data and evaluation. For each task, we collect 300 Franka demonstrations in simulation by teleoperating the simulated robot through a DexCap-based interface [44]. We replay each demonstration in the simulator with physics enabled, render ray-traced RGB observations from a fixed third-person camera, and discard replays that fail the success checker. Each method is evaluated with 100 closed-loop rollouts per task and robot, using initial states drawn from random seeds disjoint from training; a rollout fails if the success condition is not met within 500 control steps. For alignment, we render about 100k generic pairs plus 10k task-specific pairs (§ 4.1). Policies use a frozen aligned encoder, a conditional UNet flow-matching head, history , action horizon , and 66 Hz control (Appendix § 10).
Baselines. We compare against three baselines, all trained on the same 300 Franka demonstrations with the same action space and evaluated with the same protocol. (i) Direct Policy Transfer (Q1) trains a standard visuomotor policy, with an MLP visual encoder learned end-to-end on source demonstrations, and deploys it on the target robot without visual adaptation. (ii) DINOv2 w/o alignment (Q2) encodes both source and target observations with one shared frozen DINOv2 encoder and trains the same latent policy as UniLatent, with the same pooling, but without paired-rendering alignment. It tests whether generic pretrained features are already invariant to the robot embodiment. (iii) RoVi-Aug (Q3) follows the RoVi-Aug pipeline [8]: source demonstrations are translated into the target robot’s visual domain, and a pixel-space policy trained on the translated images is deployed on target RGB observations. RoVi-Aug* changes only the translator. Whereas RoVi-Aug synthesizes the target robot with diffusion-based cross-painting, RoVi-Aug* encodes each source image with our aligned source encoder and decodes the features with the pretrained Representation Autoencoder decoder [45], which maps DINOv2 features back to RGB. Because is aligned to the frozen target encoder, the decoded image shows the target robot. We adopt this translator for its translation quality: on our robot pairs, diffusion-based generation distorted robot links and joint structure under challenging poses, and segmentation masking (Shadow [9]) removed gripper–object contact regions, whereas cross-decoding produced the most faithful target-robot images (Appendix § 12.§ 4). RoVi-Aug* thus gives pixel-space transfer a strong translator while sharing UniLatent’s paired renderings, aligned encoder, demonstrations, and action supervision, so the main remaining difference is whether the policy learns from translated pixels or from aligned latent features. We evaluate the original diffusion-based RoVi-Aug in the real-world experiments (§ 5.2).
Results. In Table 1, the Franka column reports in-domain source performance as a reference, and the xArm and Sawyer columns report zero-shot transfer without target demonstrations. For Q1, direct transfer achieves only 0–5% target success despite 80–97% in-domain success, showing that matching DoF and gripper type does not imply a small visual embodiment gap. For Q2, frozen DINOv2 features improve some xArm settings but remain insufficient, reaching at most 16% on Sawyer, so explicit alignment is necessary. For Q3, averaged over the six target task–robot settings, RoVi-Aug* achieves 44.5% success, while UniLatent achieves 76.0%. UniLatent matches RoVi-Aug* on xArm Close Box Lid and improves the other five settings. The largest gains occur on Stack Cup into Bowl: from 15% to 65% on xArm and from 5% to 51% on Sawyer.
| Task & Setup | Close Box Lid | Stack Cup into Bowl | Kick Football | ||||||
| Franka | xArm | Sawyer | Franka | xArm | Sawyer | Franka | xArm | Sawyer | |
| Direct Policy Transfer | 97 | 2 | 5 | 80 | 0 | 0 | 90 | 1 | 1 |
| DINOv2 w/o align. | 97 | 40 | 16 | 89 | 3 | 0 | 93 | 11 | 1 |
| RoVi-Aug* | 97 | 93 | 54 | 89 | 15 | 5 | 93 | 60 | 40 |
| UniLatent (ours) | 97 | 93 | 81 | 89 | 65 | 51 | 93 | 89 | 77 |
Table 1. Table 1: Zero-shot transfer results in simulation. Policies are trained on Franka demonstrations and evaluated on Franka (in-domain) and zero-shot on xArm and Sawyer. We compare direct policy transfer, a shared frozen DINOv2 encoder without alignment, RoVi-Aug* (our adaptation of RoVi-Aug), and the full UniLatent pipeline. Entries report success rates (%) over 100 rollouts.
Latent vs. pixel space. Cross-decoding preserves global layout but can introduce errors around the wrist, gripper opening, and manipulated objects (Figure 5). RoVi-Aug* thus trains on decoded images but executes on real target images, whereas UniLatent trains and executes on features from the same frozen target encoder, avoiding this train–test mismatch.

Figure 5. Pixel-space translation examples. Each row shows source observation , its target pair , decoded source translation used by RoVi-Aug*, and target reconstruction .
Real-World Experiments
Setup. We transfer from a real xArm source robot to a Franka target robot on Close Drawer (push a drawer closed), Close Box Lid (push an open lid down), and Wipe Table (wipe the table with a cloth from right to left, following the RoVi-Aug setting [8]); see Figure 6. We collect 150 teleoperated xArm demonstrations for Close Drawer and 61 for each of the other two tasks at 10 Hz, with a RealSense D435i third-person camera at 640 × 480 whose extrinsics are identical for the two robots, so that, as in simulation, the embodiment is the main changed factor. A single task-agnostic encoder, aligned using only about 100k generic paired renderings, is shared by all real-world tasks. The target encoder is off-the-shelf DINOv2, pretrained on real images and kept frozen, so simulation supervision only trains toward a fixed real-image representation. Since alignment is learned purely in simulation and the physical robots also differ in controllers and calibration, we evaluate few-shot transfer, fine-tuning with Franka demonstrations per task. Each setting is evaluated with 10 rollouts per task from randomized initial object configurations, and a rollout succeeds if the task criterion is met within 25 s.

Figure 6. Real-world tasks. Each row shows representative frames from a teleoperated demonstration collected on xArm under a fixed third-person camera.
Compared settings. Source-only trains and evaluates on xArm as an in-domain upper reference. The remaining three settings are evaluated on Franka and use the same 10 target demonstrations per task; they differ only in how source data is used. Target-only trains from scratch. RoVi-Aug pretrain + few-shot follows the original RoVi-Aug pipeline [8], providing a direct comparison with the published method. Each xArm frame is segmented with Grounded-SAM2 [46, 47], and the xArm is replaced by a Franka at the same end-effector pose, synthesized by a ControlNet-conditioned diffusion model [48] that we train on rendered xArm–Franka pairs, since RoVi-Aug releases no checkpoints for our robot pair (Appendix § 12.§ 5). A pixel-space policy is pretrained on the translated xArm demonstrations and then fine-tuned on the Franka data. Cross-emb pretrain + few-shot pretrains our latent policy on aligned xArm observations and fine-tunes with the same Franka data.
Results. Table 2 reports 27/30 successful target rollouts for UniLatent, compared with 23/30 for RoVi-Aug pretraining and 11/30 for target-only training. RoVi-Aug pretraining improves over target-only training on all three tasks, confirming that source data is useful when bridged to the target domain. Relative to RoVi-Aug, UniLatent improves Close Drawer from 9/10 to 10/10 and Wipe Table from 5/10 to 8/10, while matching 9/10 on Close Box Lid. A single encoder aligned only on generic simulated pairs thus provides a useful initialization for all three real-world tasks at the evaluated 10-demonstration budget. Rollout videos, including failures, are on the project website.
| Setting | Drawer | Box | Wipe |
| Source-only (xArm) | 10/10 | 8/10 | 10/10 |
| Target-only (few-shot, Franka) | 7/10 | 2/10 | 2/10 |
| RoVi-Aug pretrain + few-shot | 9/10 | 9/10 | 5/10 |
| Cross-emb pretrain + few-shot (ours) | 10/10 | 9/10 | 8/10 |
Table 2. Few-shot real-world transfer. Source-only is an in-domain xArm reference. Rows 2–4 are evaluated on Franka and use the same Franka demonstrations. Cross-emb pretrain + few-shot (ours) is UniLatent.
Multi-Source, Viewpoint, and Background Transfer
We examine three extensions on Close Box Lid (Table 3).
| Source robot(s) | # Demos | Success (%) |
| Sawyer | 100 | 64 |
| Franka | 100 | 63 |
| Sawyer | 200 | 84 |
| Franka | 200 | 92 |
| Sawyer + Franka | 200 | 91 |
Table 3. Table 3: Additional transfer analyses on Close Box Lid (success, %). (a) Multi-source transfer to xArm from Sawyer, Franka, or both source robots; demo counts denote total source demonstrations, and all sources align to the same frozen target encoder . (b) Cross-view paired rendering improves deployment on the shifted view v2. (c) Success under matched backgrounds and under source-background randomization, averaged over five held-out simulated scenes.
| Alignment | Deploy view | ||
| v1 | v2 | ||
| Matched view | 97 | 62 | |
| Cross-view | 96 | 89 |
Table 9.
| Training / evaluation backgrounds | Franka (src.) | xArm | Sawyer |
| Matched | 97 | 93 | 81 |
| Randomized source / five unseen scenes | 88.6 | 85.6 | 80.0 |
Table 10.
Multi-source transfer. We align separate Franka and Sawyer source encoders to the same frozen xArm target encoder and train one policy on their combined demonstrations. Using 100 demonstrations from each source achieves 91% success, compared with 63–64% for either 100-demonstration source alone, and is comparable to Franka-only training with 200 demonstrations (92%). Heterogeneous sources can thus be combined without changing deployment.
Viewpoint-shift transfer. We render paired observations from two camera viewpoints (v1, v2) of the same robot and align v1 features to v2. A matched-view policy achieves 97% on v1 but only 62% on v2. Cross-view alignment achieves 96% and 89%, showing that paired rendering also accommodates systematic viewpoint shifts.
Background generalization. We render Franka source demonstrations with randomized backgrounds and evaluate the transferred policies in five held-out simulated scenes. Mean success is 88.6% on Franka, 85.6% on xArm, and 80.0% on Sawyer, compared with 97%, 93%, and 81% under matched backgrounds. These results suggest that visual diversity added on the source side carries over to the target policy through the aligned latent space. This experiment covers simulated background shifts; real-world clutter and lighting changes remain untested.
Analysis
Figure 7 shows embodiment-separated features from Franka, Sawyer, and xArm becoming increasingly overlapping during alignment. Paired-pose coverage also matters: with the regression objective unchanged, manipulation-relevant pose sampling yields 93%/81% success on xArm/Sawyer, versus 60%/32% with unconstrained sampling (Table 7, Appendix § 13). Further analyses of pooling, diffusion versus flow policies, and pixel-space translation are provided in Appendix § 13.

Figure 7. t-SNE of latent embeddings during alignment. Each point denotes a latent embedding of a rendered image; colors indicate robot embodiments, and marker shapes indicate paired rendered samples. Across training (1→6), embeddings from different robots become more aligned in the latent space.
| Pose generator for paired poses | xArm | Sawyer |
| Unconstrained random poses | 60 | 32 |
| Manipulation-relevant pose prior | 93 | 81 |
Table 7. Effect of pose sampling for paired rendering. We train the observation alignment stage with either unconstrained random TCP-pose samples or the proposed manipulation-relevant pose prior, and evaluate zero-shot Close Box Lid transfer from Franka to xArm and Sawyer. Entries report success rates (%) over 100 simulation rollouts per target robot.
Limitations
UniLatent currently targets 7-DoF single-arm robots with parallel-jaw grippers and a shared end-effector action interface. Substantially different morphologies, such as dexterous hands, humanoids, or mobile manipulators, and tasks requiring tool-specific contact would require additional retargeting and contact modeling. Our tasks were designed so that both robots perform the same task in the same scene; applying UniLatent to external source datasets such as Open X-Embodiment remains to be demonstrated. The method also assumes that the target robot can complete source tasks in a comparable workspace; demonstrations outside the target reachable set may need trajectory modification. Finally, our background-shift evaluation is limited to one simulated task, and our real-world experiments use a single physical setup; robustness to real-world clutter, lighting, distractors, and broader object variation remains unestablished.
Conclusions
We present UniLatent, a latent observation alignment framework for cross-embodiment robot learning. By aligning source observations to a frozen target representation with paired simulation renderings, UniLatent deploys source-trained policies on target robots without test-time image translation. Results in simulation and the real world indicate that latent alignment is an effective and data-efficient path to cross-embodiment reuse among arms with similar morphologies. We release code, the rendering pipeline, datasets, and encoder checkpoints to support reproducibility.
Acknowledgments
We thank researchers at UC Berkeley and USC for helpful discussions. Daniel Seita acknowledges generous support from Samsung Research America and NSF grant #2447397.
Appendix Overview
This appendix provides implementation details, experimental protocols, baseline specifications, and additional analyses for UniLatent. It is organized into four parts: (i) implementation details of the UniLatent pipeline, (ii) experimental setup details for simulation and real-world evaluations, (iii) baseline implementations and pixel-space translation analysis, and (iv) additional analyses and ablations that examine key design choices in representation alignment and latent policy learning, including a study of generalization to unseen backgrounds and scenes.
Implementation details (§B). We describe the full UniLatent pipeline in detail. We first explain how paired source-target robot observations are generated in simulation, including manipulation-relevant pose sampling, tool-center-point matching, inverse-kinematics filtering, domain randomization, and camera configuration. We then describe target-anchored observation alignment with DINOv2 encoders, including the encoder architecture, feature normalization, alignment objective, and training recipe. Finally, we provide the latent-space policy implementation, including the end-effector action representation, flow-matching policy parameterization, training procedure, deployment on the target robot, and compute cost.
Experiment setup details (§C). We specify the simulation and real-world evaluation settings used in the main experiments. For simulation, we provide the tabletop environment setup, task definitions, object randomization ranges, state-based success checkers, rollout protocol, robot control parameters, teleoperation interface, and replay-and-filtering pipeline for demonstration collection. For real-world experiments, we describe the source and target robot setups, camera configuration, data-collection procedure, few-shot target demonstration protocol, task definitions, evaluation randomization, success criteria, and rollout termination rules.
Baselines and pixel-space translation analysis (§D). We provide implementation details for the baselines used to evaluate cross-embodiment transfer, including direct policy transfer, a frozen DINOv2 policy without observation alignment, and RoVi-Aug* as a pixel-space translation baseline. We also compare representative pixel-space translation strategies, including diffusion-based robot-to-robot generation, segmentation-based masking, and cross-decoding of aligned features through a pretrained RAE decoder. This analysis characterizes how pixel-level methods preserve or distort robot geometry, gripper state, scene context, and contact regions, and motivates the use of latent-space policy execution in UniLatent.
Additional ablation studies and explorations (§E). We analyze several design choices in UniLatent and study its generalization beyond the default deployment setting. First, we study pooling and feature design for DINOv2 representations, comparing fixed pooling rules and learnable attention pooling under frozen and unfrozen backbone settings. Second, we compare diffusion and flow-matching policies in the aligned latent space under matched architectures and training protocols. Third, we ablate the pose generator used for paired rendering by comparing manipulation-relevant pose sampling with unconstrained pose sampling. Finally, we evaluate transfer under background and scene shifts between source training and target deployment, showing that source-side background randomization is inherited by the transferred target policy through the shared latent space.
UniLatent Implementation Details
Paired Image Generation
Pose sampling. To generate paired rendered observations for alignment training, we first need a distribution over plausible gripper poses that commonly arise in table-top manipulation. We define poses at the tool center point (TCP) instead of the robot’s terminal link, since the TCP is the manipulation-relevant point that should be matched across robots. For our parallel-jaw grippers, we define the TCP as a frame rigidly attached to the gripper whose origin is the midpoint between the two jaw tips1. This choice accounts for the fixed geometric offset between the terminal link frame used by cuRobo’s robot model and the point in space where interaction with objects is centered. It is this manipulation-relevant point, not the terminal link itself, that should be aligned across robots for paired image generation.
We derive a plausible pose distribution by referring to diverse table-top manipulation tasks in ManiSkill [49], LIBERO [50], RLBench [51], and RoboSuite [52]. Concretely, we parameterize each sampled TCP pose by its Cartesian position and its orientation, represented using a tilt angle measured from pointing straight down (along the axis) together with an azimuthal angle around the vertical axis. As a simplification, we assume an independent distribution of each dimension of the pose. Although this assumption is not necessarily true, we found that in practice it yields plausible and diverse poses.
We implement two sampling methods. The first method samples from a parametric distribution derived from analyzing demonstration data. The TCP position coordinates follow independent Gaussian distributions: , , and , with soft clamping to stay within physical workspace limits. The tilt angle follows a truncated exponential distribution with decay constant and maximum angle , which strongly favors gripper orientations pointing mostly downward as commonly observed in manipulation tasks. The azimuthal angle is sampled uniformly from . The sampled TCP pose is then converted to joint angles via inverse kinematics. The gripper state is independently randomized, with 70% probability of being open and 30% closed.
The second method samples directly from the empirical distribution of end-effector poses extracted from demonstrations across diverse tasks in existing datasets. We load all demonstration trajectory files, extract joint poses at regular intervals, compute the corresponding TCP poses via forward kinematics, and then solve IK for the target robot. This approach preserves the natural correlation structure between pose dimensions present in real manipulation trajectories, at the cost of being limited to the support of the demonstration data.
We combine the two sampling methods at a roughly 8:2 proportion, with the majority coming from the parametric distribution and the remainder from the empirical demonstration distribution. Importantly, the tasks we use for downstream experiments are entirely self-collected and do not appear in any of the existing benchmarks (ManiSkill, LIBERO, RLBench, or RoboSuite) from which we derived our pose distribution. This separation ensures that our alignment training allows the robot to learn from reasonable, manipulation-relevant poses without overfitting to any specific task or demonstration dataset, preserving the generalization capability of the learned representation.
Inverse kinematics. To ensure all generated poses are kinematically feasible, we employ cuRobo [53] as our inverse kinematics solver. Given a sampled TCP pose (position and orientation), cuRobo solves for joint angles that achieve this end-effector configuration while respecting the robot’s joint limits. We initialize the IK solver with seed configurations derived from each robot’s default joint positions, which biases solutions toward natural manipulation-ready arm configurations rather than awkward or self-colliding poses. The IK solver processes poses in batches for computational efficiency. Poses for which IK fails to find a valid solution are rejected from the dataset. When retargeting to multiple target robot embodiments, we apply an intersection filter: only frames where IK succeeds for all target robots are retained. This filtering ensures that each retained pose is kinematically reachable by all robot embodiments, so that corresponding configurations can be rendered across robots.
In practice, we observe that xArm has a relatively compact kinematic structure and thus a smaller reachable workspace compared to Franka and Sawyer. Poses with large height values that are feasible on the latter two robots often fail to be retargeted to xArm. We accommodate this limitation in our task design: the tasks we evaluate on (Close Box Lid, Stack Cup into Bowl, Kick Football) involve relatively small objects and operate in a lower workspace region, ensuring fairness across all robot embodiments.
For Sawyer, the IK seed configuration is particularly critical. The Sawyer’s arm is long, slender, and highly articulated, meaning a single end-effector pose can correspond to multiple valid joint configurations. However, some geometrically valid IK solutions place the elbow or forearm in configurations that obstruct the gripper’s intended motion. During paired image generation this is not immediately problematic since we do not require task completion, but during inference, a poorly-seeded initial pose can leave the arm blocking subsequent movements, rendering the task unsolvable. To ensure consistency between training and inference, we initialize the IK solver with each robot’s default joint positions as a reference seed. This biases solutions toward manipulation-ready configurations where the arm is positioned out of the way, ensuring that the joint configurations shown in training images match what the robot will produce at inference time.
Pose matching details. To ensure action equivalence across embodiments, we match poses at TCP, defined as the point between the fingertips, instead of at the end-effector link (the wrist). Since different robots have different gripper lengths, matching wrist poses directly would result in fingertips at different positions in space, which is problematic for manipulation where contact points matter. Instead, we apply a fixed transformation to convert the source robot’s wrist pose to its TCP pose. We then solve IK for the target robot such that its TCP reaches the same pose.
Domain randomization. To reduce overfitting to simulator-specific visual appearances and better match real-world visual variability, we apply extensive domain randomization during paired image generation using the RoboVerse simulation infrastructure.
For scene objects, we augment existing RoboVerse assets with 3D object models from the YCB dataset, converted to USD format for Isaac Sim.
We randomize surface materials for all scene elements: floors are sampled from carpet, wood, stone, concrete, and architectural textures; walls from architectural panels, wall board, masonry, paint, and composite materials; and table surfaces from wood, stone, plastic, ceramic, and metal finishes. Lighting conditions are randomized by varying the intensity (within plausible ranges for indoor scenes), color temperature (3000K–6000K to simulate different light sources from warm tungsten to cool daylight), and position (a main ceiling disk light and four corner sphere lights).
We additionally apply small perturbations to camera pose (±5cm in position, ±2° in orientation) and look-at point to simulate natural variation in camera placement. This randomization encourages the learned representation to capture pose correspondence, not scene-specific visual features.
Camera configurations. We use a consistent camera setup across all robot embodiments to ensure viewpoint equivalence in paired images. The camera is positioned at a fixed location viewing the tabletop workspace from a front-diagonal perspective, with the look-at point centered on the manipulation area. We render images at resolution with camera intrinsics calibrated to match a real Intel RealSense camera (focal length 6.06mm, horizontal aperture 6.4mm), providing a wide field of view that captures the robot arm throughout its workspace. Critically, for each pose tuple, all robots are rendered from the identical camera viewpoint—same extrinsics and intrinsics—so that the only visual difference between paired images is the robot morphology itself, not the observation perspective. Small camera perturbations (±5cm in position, ±2° in orientation) are applied as part of domain randomization to improve robustness, but these perturbations are consistent within each rendered frame, preserving the correspondence between paired observations.
Observation Alignment Learning
Encoder architecture. Both and are instantiated from the DINOv2-base model with register tokens (facebook/dinov2-with-registers-base) [41]. The backbone is a ViT-B/14 architecture with 12 transformer layers, hidden dimension 768, and patch size . Input images are resized to via bicubic interpolation and normalized with the ImageNet channel-wise mean and standard deviation . We extract features from the last hidden state of the DINOv2 backbone. The output token sequence has length 261: 1 CLS token, 4 register tokens, and patch tokens. We strip the register tokens and use both the CLS token, a single 768-dimensional vector, and the 256 patch tokens, each 768-dimensional, for alignment. The patch token sequence of shape is reshaped into a 2D spatial feature map of shape , with denoting the batch size. We disable the learnable affine parameters (scale and bias) of DINOv2’s final LayerNorm, retaining only the non-parametric mean-centering and variance-normalization. Both encoders are initialized from the same pretrained DINOv2 checkpoint; is then frozen and is the only module updated during alignment training.
Alignment objective. We minimize the MSE between corresponding source and target embeddings. Let and denote the CLS token and reshaped patch feature map produced by encoder , where . The alignment loss is
The pipeline uses two distinct types of normalization. First, input RGB images are normalized with the standard ImageNet pixel statistics expected by the pretrained DINOv2 checkpoint. Second, after feature extraction, we standardize the reshaped patch feature map using feature-channel statistics computed from the frozen DINOv2 encoder over ImageNet images:
Here, indexes a DINOv2 feature channel, not an RGB image channel. Thus, and are feature-level statistics and are distinct from the RGB ImageNet pixel statistics used for input preprocessing. The same feature standardization is applied to both source and target patch features before computing the patch-level MSE.
Training recipe. We use the Adam optimizer with learning rate and momentum parameters . The learning rate follows a cosine annealing schedule that decays to a minimum of . We train for 25 epochs with a batch size of 10 on a single A6000 GPU, with validation performed twice per epoch. Training uses float32 precision. During training, we apply camera-quality degradation augmentations to the source images using Albumentations [54]: at each training sample, one augmentation is randomly selected from Gaussian noise, ISO noise, motion blur, JPEG compression, brightness/contrast perturbation, and hue/saturation shift, with an overall application probability of 0.9. The target image is not augmented, since the target encoder is frozen and serves as the fixed reference. This asymmetric augmentation encourages the source encoder to learn features that are robust to realistic imaging variations while preserving a stable target representation.
Time cost and practical considerations. Paired rendering and encoder alignment are offline costs. In our experiments, the alignment dataset contains approximately 100k generic paired renderings, optionally supplemented with 10k task-specific pairs, and the source encoder is trained for 25 alignment epochs on a single NVIDIA A6000 GPU. An encoder aligned on generic pairs can be reused across downstream tasks, as in our real-world experiments, where only the task policy is trained from the corresponding source demonstrations. This contrasts with real-world demonstration collection or task-specific synthetic trajectory generation, which must be repeated for each new task. The offline rendering and alignment cost is therefore amortized across tasks and does not affect deployment, where target observations are encoded directly by the frozen target encoder.
Latent-Space Policy Training and Deployment
Action representation. The policy uses an 8-dimensional action representation in the end-effector space, formulated as . Translation is represented as a 3D position vector in the robot base frame. Rotation is encoded as a unit quaternion . Because quaternions and represent the identical spatial rotation, we canonicalize the training dataset to a consistent hemisphere by enforcing .
The gripper state is represented by a binary scalar , where 0 indicates a closed gripper and 1 indicates an open gripper. To map continuous demonstration data into this binary space, we compute the L2 distance between the recorded gripper joint state and the robot’s canonical fully-open and fully-closed configurations. Because grasped objects physically prevent the gripper from reaching its absolute closed limit, we apply an asymmetric distance threshold that biases the classification toward the closed state. This abstraction enables cross-embodiment transfer by decoupling the action space from robot-specific gripper joint kinematics.
During training, we independently apply min-max normalization to all eight dimensions to scale them to the range . For the -th action dimension , the normalized value is computed as
where and are the minimum and maximum values observed in the training data for that specific dimension. Standard safeguards are applied for degenerate dimensions where . During inference, the network outputs are de-normalized back to their original scales. To ensure a valid rotation before execution on the robot, the predicted quaternion is explicitly re-normalized to unit length, such that .
Flow matching policy. The policy is instantiated as a conditional flow-matching [43, 42] generative model that maps a window of latent observations to an end-effector action chunk. Given a source demonstration on source robot with length , we encode each observation with the aligned source encoder to obtain the latent vector and form a latent conditioning window . In practice, we train on fixed-length action chunks: the supervision target is , and we define with horizon (we use and in our experiments). Concretely, we sample an initial action chunk and construct an interpolation path between and for . We use the linear path , which induces a constant target velocity . The model is parameterized by a conditional UNet backbone (implemented as a 1D UNet over the action-chunk temporal dimension) and predicts a conditional velocity field . Training minimizes the standard flow-matching objective
where the expectation is taken over and . At inference time, action chunks are generated by integrating the learned flow field from to with 10 solver steps; in our implementation we use a midpoint (second-order Runge–Kutta) ODE solver. The UNet uses channel dimensions (256, 512, 1024), kernel size 5, GroupNorm with 8 groups, and a flow-time embedding dimension of 128.
Training details. We train the flow-matching policy using AdamW optimizer with learning rate , , and weight decay . The learning rate follows a cosine schedule with 500 warmup steps. We use a batch size of 256 and train for 200 epochs. For the temporal structure, the policy conditions on a history of the most recent latent observations and predicts action chunks of length . During training, the model conditions on encoded latent observations provided as a global conditioning signal. We apply Exponential Moving Average (EMA) with maximum decay 0.9999 for stable inference. The flow-matching loss is a simple MSE between the predicted and target velocity fields, where all action dimensions (translation, rotation, gripper) are weighted equally. The aligned source encoder is kept frozen throughout policy training.
Deployment details. Simulation. At inference time, the policy operates in a closed-loop manner. Given the current RGB observation , the policy maintains a sliding window of the most recent latent observations. When the action cache is exhausted, the policy encodes the observation history through the frozen target encoder , producing latent features that are passed to the flow-matching U-Net. The flow field is integrated from to using the midpoint ODE solver with 10 integration steps, generating an action chunk of end-effector commands. These 8 actions are then executed sequentially—one per simulation step—before the policy is queried again for a new chunk.
Each predicted action consists of an absolute Cartesian pose (3D position + quaternion rotation) and a binary gripper command, which are converted to joint positions via inverse kinematics (cuRobo) before being sent to the robot-specific low-level controller. To maintain temporal consistency within an action chunk, each IK solution is seeded with the previous solution. If IK fails for a particular target pose, the robot holds its current joint configuration for that timestep. Episodes terminate upon task success or after a maximum of 500 timesteps.
Real world. Consistent with simulation setup, we utilize the identical model architecture for real-world inference. The policy conditions on a sliding window of historical observations to predict an action chunk of timestamps. We deploy the policy at a control frequency of 10 Hz. Episodes are terminated and recorded as a failure if the task is not successfully completed within 250 steps (25 seconds).
Experiment Setup Details
Tasks and Environments
Simulation environment. Our simulation tasks are set in a tabletop manipulation scene. The workspace consists of a 1.8 m 1.8 m table surface at a height of 0.7 m, enclosed within a 10 m 10 m room with walls and ceiling that provide realistic visual backgrounds for rendering. The robot is mounted at a fixed base position on the table, and task objects are placed within the robot’s reachable workspace. All tasks use the NVIDIA PhysX engine for physics simulation. The physics timestep is s, and the control decimation factor is 15, yielding an effective control frequency of approximately 66 Hz.
Task details. For each task, we collect teleoperated demonstrations in simulation. At evaluation time, initial object poses are re-sampled from the task randomization ranges below; we ensure evaluation uses distinct random seeds from training so that the initial poses are not reused. Success is determined by task-specific state-based checkers in the simulator.
Close Box Lid. The box is an articulated object with a single revolute hinge joint. The lid is initialized at an open joint angle of . The box base position is sampled uniformly with m and m. We declare success if the terminal hinge joint angle satisfies .
Stack Cup into Bowl. The bowl mesh is YCB 024_bowl scaled by , and the cup mesh is YCB 065-d_cups scaled by . The bowl mass is set sufficiently high that it remains effectively stationary under incidental contacts during insertion. The bowl position is sampled uniformly in a 14 cm 14 cm square centered at m, and the cup position is sampled uniformly (independently) in a 14 cm 14 cm square centered at m; equivalently, m and m.
The success checker defines a world-axis aligned bounding box of size 20 cm 20 cm 14 cm anchored to the bowl position with a small positive vertical offset (so that the box lies inside the bowl volume). Let be a fixed set of predefined keypoints on the cup mesh in world coordinates at the final timestep. We declare success if for all keypoints. Since is axis-aligned and depends only on the bowl position (not its orientation), this criterion is robust to small pose perturbations.
Algorithm 1 Simulation evaluation (task-agnostic).
Require: Simulator environment $\mathcal{E}$; trained policy $\pi_{\theta}$; evaluation camera cam; task randomizer SampleInit($\cdot$); success predicate Success($\cdot$); rollouts $n_{\mathrm{rollouts}}=100$; maximum timestep $T=500$
Ensure: Success rate $\hat{P} \in [0,1]$
1: $s \gets 0$ {success counter}
2: for $i \gets 1$ to $n_{\mathrm{rollouts}}$ do
3: $\mathcal{E}$.reset()
4: $\xi \gets$ SampleInit()
5: $\mathcal{E}$.set_object_states($\xi$) {initialize object poses}
6: $C \gets \emptyset$ {action cache}
7: for $t \gets 1$ to $T$ do
8: $o_t \gets \mathcal{E}$.render_rgb(cam)
9: if $C = \emptyset$ then
10: $C \gets \pi_{\theta}(o_{t-k+1:t})$ {predict a chunk of $H$ actions}
11: end if
12: $a_t \gets C$.pop()
13: $\mathcal{E}$.step($a_t$) {advance physics}
14: if Success($\mathcal{E}$.state()) then
15: $s \gets s+1$
16: break {terminate rollout early on success}
17: end if
18: end for
19: end for
20: return $\hat{P} \gets s/n_{\mathrm{rollouts}}$
To stabilize cup–bowl contact dynamics, we use task-specific simulation parameters: maximum depenetration velocity 0.5 m/s, contact detection offset 5 mm, and 16 solver position iterations.
Kick Football. The ball is a sphere with radius 4 cm and mass 0.1 kg, with static and dynamic friction coefficients 0.1 and restitution 0.3. The goal is a rigid mesh asset scaled to approximately 30 cm 12 cm 11 cm (widthheightdepth) and assigned a high mass to prevent displacement. The goal position is sampled uniformly in a 14 cm 14 cm square centered at (0.25, 0.18) m and the ball position is sampled uniformly (independently) in a 14 cm 14 cm square centered at (0.12, ) m; equivalently, m and m.
The success checker defines a world-axis-aligned bounding box of size 30 cm 13 cm 12 cm placed inside the goal mouth, and declares success if the final ball center lies within the box. We use the same stabilized contact parameters as in “Stack Cup into Bowl”.
Evaluation protocol. We evaluate each simulation method using the task-agnostic procedure in Alg. 1. Concretely, for each of rollouts we reset the simulator, sample a fresh initial object configuration from the task randomization ranges, and run closed-loop interaction for up to control steps by rendering an RGB observation from a fixed evaluation camera, executing actions from the current action chunk, querying the policy for a new chunk when the action cache is exhausted, and stepping the physics. We terminate rollouts early when the task-specific success predicate becomes true, and report the empirical success rate .
Robots and Control
Robot specifications. In simulation, we use the Franka Emika Panda as the source robot and evaluate zero-shot cross-embodiment transfer for two target robots with distinct kinematics and appearances: the UFactory xArm7 and the Rethink Robotics Sawyer. In real-world experiments, we instead use xArm as the source and Franka as the target. All three platforms are single-arm manipulators with a 7-DoF revolute arm and a parallel-jaw gripper. For Sawyer, we keep the head-pan joint fixed at its default value () throughout all experiments, and only command the 7 arm joints and gripper.
Although the policy operates in a shared end-effector action space (Cartesian pose at the tool center point plus a binary gripper command), these embodiments differ substantially in link geometry, joint limits, gripper mechanics, and visual appearance, creating large embodiment-dependent observation shifts.
Joint control. All robots use position-mode joint drives with PD control in simulation. We use stiffness/damping (80, 4) for Franka arm joints and (800, 40) for both xArm7 and Sawyer arm joints. All gripper joints use stiffness 1000 and damping 100 to achieve stable opening/closing under contact. All gains are specified in simulator joint-drive units.
TCP offsets and gripper joint values. To instantiate the TCP offset matching described in § B.1, we place the TCP at a fixed translation along the local axis of each robot’s default end-effector link. This offset is a pure translation with no rotation, and the values below are the offset distances from the named link frame to the TCP.
Franka: 10.3 cm from
panda_hand.xArm7: 15.0 cm from
link7.Sawyer: 14.1 cm from
right_gripper_base.
The binary gripper command predicted by the policy corresponds to the gripper dimension of the action vector defined in § B.3. At inference time, we expand this binary command into robot-specific low-level gripper joint targets. These targets are joint positions rather than end-effector poses; each robot has a different gripper mechanism and a different number of actuated joints, so the mapping is robot-specific. The joint targets for open and closed states for each robot are as follows.
Franka: for closed, for open.
xArm7: for closed, for open.
Sawyer: for closed, for open.
During cross-embodiment transfer, the binary gripper command predicted by the source-trained policy is applied on the target robot using the target robot’s open/close mapping above.
Data Collection
Teleoperation system. Source demonstrations are collected by a human operator using a teleoperation interface based on DexCap [44]. The system uses an HTC Vive tracker mounted on the operator’s hand to capture arm motion, and a Rokoko smart glove to capture gripper open/close commands. The Vive tracker streams 6-DOF relative poses at 60 Hz. At each control step, the tracker’s relative displacement and rotation since a calibration pose are composed with the robot’s initial end-effector pose to compute the target end-effector pose , where is the robot’s end-effector pose at the start of the episode and is the Vive tracker’s relative pose transformed into the robot’s coordinate frame via a fixed rotation mapping. An inverse kinematics solver then converts the target end-effector pose to joint commands, which are executed by the simulator. This compositional formulation avoids integration drift that would accumulate with a velocity-based interface. The operator records episodes interactively, with keyboard shortcuts to start recording, stop, save or discard individual episodes, and reset the environment to a new random configuration. Each recorded trajectory stores the initial environment state (object and robot poses), the sequence of joint-space actions, the corresponding end-effector targets (position and quaternion), and optionally the full simulator state at each timestep.
Replay, rendering, and filtering. After collection, each trajectory is replayed in the full physics simulator to produce the final observation–action training pairs. During replay, the joint-space actions from the recorded trajectory are executed with physics enabled, and the environment state is advanced by the simulator at each step. The simulator renders RGB observations from a fixed third-person camera in ray-tracing mode, and we record the corresponding end-effector actions (Cartesian poses and gripper commands). Domain randomization is applied once at the beginning of each replay episode, so all frames within a single demonstration share the same randomized appearance. We then validate each replayed trajectory using the task-specific success checker: trajectories that fail to achieve the task criterion are discarded. Only successfully completed demonstrations are retained for policy training. The resulting filtered dataset of observation–action pairs serves as training data for the policy. For each simulation task, we collect and retain successful demonstrations following this pipeline.
Real-world data collection. For real-world experiments, demonstrations are collected via teleoperation directly on the source xArm robot at a control frequency of 10 Hz, with episode durations ranging from 7 to 15 seconds. Visual observations are captured using a fixed Intel RealSense D435i camera at a resolution of pixels. The camera extrinsics are kept identical between the source (xArm) and target (Franka) setups to ensure consistent visual input across embodiments. In total, we collect 150 demonstrations for Close Drawer and 61 demonstrations each for Close Box Lid and Wipe Table on the source robot.
Few-shot target demonstrations and real-world evaluation. For each real-world task, we collect 10 target demonstrations on the Franka robot using the same camera viewpoint and control frequency as the source xArm setup. These demonstrations are used for the target-only few-shot baseline and for fine-tuning the source-pretrained policies. During evaluation, each method is tested for 10 rollouts per task on the Franka robot with randomized initial object configurations. A rollout is counted as successful if the task-specific completion criterion is satisfied within 25 seconds; otherwise it is recorded as a failure. For Close Drawer, success requires the drawer to be fully closed; for Close Box Lid, the lid must be closed; and for Wipe Table, the target region must be wiped. All methods use the same target demonstrations, evaluation initializations, and camera setup.
Baseline Implementation and Analysis
We first discuss the problem scope in the context of broader cross-embodiment transfer and representation-learning literature, and clarify our problem setting where we isolate the robot-induced visual gap under a shared Cartesian end-effector interface. We then describe two direct visual transfer baselines in the main paper: Direct Policy Transfer, which deploys a source-trained visual policy on the target robot without adaptation, and DINOv2 without observation alignment, which uses a shared frozen visual encoder for both source and target observations. Next, we detail pixel-space translation baselines that explicitly transform source observations into the target visual domain, including diffusion-based robot-to-robot generation, segmentation-masking, and cross-decoding of aligned features through a pretrained RAE decoder. We further provide qualitative analyses of the translated observations, highlighting artifacts in robot geometry, gripper state, scene context, and contact regions. Finally, we elaborate on the implementation of the real-world RoVi-Aug preprocessing pipeline.
Problem Scope and Baseline Rationale
Cross-embodiment transfer aims to reuse policies or demonstrations collected on one robot for another robot with different hardware. Prior work approaches this goal from several directions. One line of work focuses on the action and control interface, for example by conditioning policies on hardware descriptions, defining shared or universal action spaces, training across many embodiments, or adapting demonstrations across morphologies through trajectory optimization [22, 23, 24, 25, 26]. These approaches reduce kinematic or action-space mismatch, but often rely on strong state/action abstractions, broad multi-robot training data, or task-specific optimization. In third-person RGB manipulation, however, action alignment alone does not remove the observation-level embodiment gap: even under a shared Cartesian end-effector interface, the same command can produce different robot silhouettes, self-occlusions, gripper appearances, and contact-region evidence that a vision policy must interpret.
A second line of work addresses the visual embodiment gap directly through pixel-space editing or augmentation, such as cross-painting, robot generation, viewpoint augmentation, or segmentation masking [7, 8, 9]. These methods are closely related to our setting because they explicitly reduce the visual shift induced by changing the robot. However, pixel-space methods require the downstream policy to consume edited or generated observations. As a result, segmentation errors, missing robot pixels, distorted gripper geometry, or inconsistent contact regions can directly affect policy learning and closed-loop execution.
A third line of work learns shared representations from videos or domains, including time-contrastive learning, cross-embodiment inverse reinforcement learning, cross-embodiment skill discovery, video-conditioned imitation, retrieval-based execution alignment, and domain-invariant sim-to-real adaptation [36, 35, 27, 28, 29, 37]. These methods learn useful invariances across viewpoints, embodiments, or simulation and real domains, but their supervision is usually indirect, relying on temporal contrast, cycle consistency, retrieval, adversarial alignment, or video-level task correspondence. They often require target-domain or multi-domain videos, and the learned representation can be sensitive to data coverage and correspondence assumptions. Their downstream use is also typically narrower than direct robot-to-robot imitation: TCN focuses on viewpoint-invariant representations for imitation or reward learning [36]; XIRL learns a cross-embodiment task-progress embedding used as a dense RL reward [35]; and RHyME retrieves and composes unpaired human and robot videos for video-conditioned imitation [28]. In contrast, our setting directly reuses source robot observation-action trajectories and learns explicit source-to-target observation alignment from paired robot renderings at matched TCP poses.
UniLatent focuses on the visual observation gap without synthesizing target-domain RGB demonstrations. After generating paired renderings of source and target robots at the same TCP pose, UniLatent trains the source encoder to match the embedding produced by a frozen target encoder. This maps each source image to the feature that the target encoder assigns to the corresponding target-robot observation, anchoring source demonstrations in the same latent space used at target deployment. The paired-rendering supervision is also lightweight: it requires robot models, rendering, and inverse kinematics for pose-matched observations, but does not require collecting task-specific target trajectories or training a photorealistic image translator.
We instantiate this problem with tabletop manipulation tasks, 7-DoF arms, parallel-jaw grippers, and a shared Cartesian end-effector action interface. This controlled setting keeps the action interface simple and isolates the observation-level embodiment gap. In the default experiments, the source and target share the same task, camera, and non-robot scene factors; the changed factor is the visible robot embodiment. Thus, the evaluation targets embodiment transfer, not general background or layout robustness: the goal is to preserve source-policy competence when robot body, kinematics, self-occlusions, and contact geometry change. On top of this controlled setting, UniLatent can still benefit from source-side diversity. Since each source encoder is aligned to the same frozen target encoder, demonstrations from multiple source robots can be aggregated in the shared target-anchored latent space. Similarly, when source demonstrations include randomized backgrounds, the transferred target policy can inherit that visual diversity through the aligned latent representation.
This scope motivates the baselines used in our experiments. Direct Policy Transfer tests whether a source-trained visuomotor policy can bridge the embodiment gap using only the shared action interface. DINOV2 without observation alignment tests whether a generic frozen visual representation is already invariant enough to support cross-embodiment policy transfer. Pixel-space translation methods, including RoVi-Aug, Shadow, and RoVi-Aug, address the visual gap by transforming RGB observations instead of aligning representations; we compare their translations qualitatively in §D.4. RoVi-Aug is the strongest controlled pixel-space comparison because it uses the same paired-rendering signal as UniLatent, but decodes aligned source features back to RGB and trains a pixel-space policy. We therefore use RoVi-Aug{sup}*` as the representative pixel-space translation baseline in the simulation-based experiments, where the comparison isolates pixel-space execution from latent-space policy learning. For the real-world few-shot comparison, we separately evaluate the original diffusion-based RoVi-Aug pipeline.
Direct Visual Transfer Baselines
Direct policy transfer. Direct Policy Transfer trains a standard source-domain visuomotor policy on source-robot demonstrations and evaluates the same trained policy on the target robot without any adaptation. In simulation, the policy is trained on Franka demonstrations and directly deployed on xArm or Sawyer observations. The policy uses the shared Cartesian end-effector action interface used throughout our experiments, so the baseline isolates the effect of the embodiment-induced visual shift when the action representation and low-level control interface are already shared. The visual encoder and policy are trained end-to-end from source demonstrations using the same action representation and evaluation protocol as the other policies, except that no pretrained DINOv2 encoder, paired-rendering alignment, or target-domain adaptation is used.
DINOv2 without observation alignment. This baseline replaces the aligned encoder pair in UniLatent with a single frozen DINOv2 encoder shared by both source and target observations. It does not use paired rendered observations for representation learning, and no source encoder is adapted toward a target-anchored feature space. During policy training, each source demonstration frame is encoded as
and the resulting features are pooled and used to train the same flow-matching latent policy as in UniLatent. At deployment, target observations are encoded by the same frozen encoder,
and passed to the trained policy. This baseline tests whether general-purpose DINOv2 features are already sufficiently invariant to robot geometry, appearance, self-occlusion, and gripper-object contact differences. Compared with UniLatent, the policy architecture, action representation, pooling design, and rollout protocol are kept fixed; the only removed component is target-anchored observation alignment.
Pixel-Space Translation Baselines
All three translation methods operate on RGB observations, but differ in how they construct target-domain images. RoVi-Aug synthesizes the target robot with an image-to-image diffusion model; Shadow{sup}* reimplementation removes the source robot pixels using segmentation masks; and *RoVi-Aug*{sup}* decodes aligned source features into the target visual domain. We describe each implementation below.
RoVi-Aug: diffusion-based robot-to-robot generation. We reproduce RoVi-Aug [8] in the xArm (source) Franka (target) setting. RoVi-Aug translates source observations to the target embodiment using a conditional image-to-image diffusion model. For robot segmentation, we use Grounded-SAM2 [47, 46]. For robot-to-robot generation, we follow RoVi-Aug and train an image-to-image diffusion model that takes an xArm observation as input and outputs a Franka observation under the same end-effector pose. Since the original paper does not release checkpoints or datasets for the xArm-to-Franka setting, we render paired source-target images using PyBullet [55] and train the diffusion model on these paired renderings. Concretely, we fine-tune a pretrained Stable Diffusion XL (SDXL) [56] with ControlNet [48] conditioning, and apply the released RoVi-Aug cross-painting pipeline to obtain translated observations for downstream pixel-space policy learning.

Figure 8. Qualitative comparison of pixel-space translation baselines for cross-embodiment transfer. Source is the input xArm observation; Target is the ground-truth Franka observation under the same end-effector pose. RoVi-Aug, Shadow, and RoVi-Aug show diffusion-based image generation, segmentation-masking, and cross-decoding translations, respectively.
Shadow: segmentation-masking translation. Since Shadow [9] is not open source, we re-implement its segmentation-masking procedure as faithfully as possible based on the description in the original paper, and denote our reimplementation as Shadow. We first estimate a segmentation mask for the source robot using Grounded-SAM2 and then remove the robot region to obtain a robot-free observation for policy learning. This method aims to reduce embodiment appearance shift by discarding robot pixels. However, masking can also remove robot-specific visual cues that are important for manipulation, such as gripper pose, self-occlusion, and local contact geometry near the manipulated object.
RoVi-Aug: cross-decoding translation. We design RoVi-Aug as a controlled pixel-space translation baseline that uses the same paired-rendering alignment pipeline as UniLatent, but executes the downstream policy in RGB space instead of latent space. After learning aligned encoders, we translate a source observation by encoding it with the aligned source encoder and decoding the resulting feature with a decoder :
The decoder is the pretrained Representation Autoencoder (RAE) [45] decoder, used off the shelf without fine-tuning, which maps DINOv2 latents back to RGB images. Because is trained to reproduce the target encoder’s features, decoding yields an image in the target visual domain. We then train a standard pixel-space policy on the translated source demonstrations using the same end-effector action supervision as the original source demonstrations. At evaluation, the policy directly consumes target-robot RGB observations. Thus, RoVi-Aug provides a controlled comparison of pixel-space and latent-space policy learning: it uses the same paired-rendering information as UniLatent, but trains the policy on translated pixels instead of aligned latent features and deploys it on raw target RGB observations.
Qualitative Analysis of Pixel-Space Translation Methods
Figure 8 compares the three translation strategies on held-out source-robot observations. Each method takes an xArm observation as input and aims to produce a Franka observation under the same end-effector pose.
The diffusion-based RoVi-Aug baseline often produces visually plausible target-robot images, but the generated robot geometry can be unstable under challenging poses, heavy occlusions, or large kinematic gaps. We observe distorted links, inconsistent joint structure, and occasional gaps between robot parts. These artifacts are problematic for policy learning because the policy is trained directly on the translated RGB observations.
Shadow avoids hallucinating a target robot by removing source-robot pixels, but this makes it brittle in interaction-heavy frames. In particular, masking can erase visual evidence near the gripper and object, including contact cues that are essential for manipulation. As a result, Shadow reduces embodiment appearance shift at the cost of discarding task-relevant robot-object interaction information.
RoVi-Aug produces the most stable translations among the compared pixel-space methods in our setting. Because the source observation is first mapped into a target-aligned latent space before decoding, RoVi-Aug better preserves scene context and produces fewer structural artifacts around the robot. It also most closely matches the ground-truth target-robot observations among the compared baselines. We therefore adopt RoVi-Aug as the representative pixel-space translation baseline in our simulation experiments. Since RoVi-Aug and UniLatent use the same paired-rendering alignment signal, their comparison evaluates the practical difference between training a policy on translated RGB observations and training it directly in the aligned latent space.
Real-World RoVi-Aug Preprocessing
For the RoVi-Aug pretrain + few-shot setting in the real-world experiments of the main paper (§ 5.2), source xArm demonstration frames are translated into the Franka visual domain before pixel-space policy pretraining. After collecting source demonstrations, we run Grounded-SAM2 on each frame to obtain a source-robot mask. We then apply the RoVi-Aug cross-painting procedure following the original paper: first, the masked source-robot region is removed to form a robot-free canvas; second, the robot-to-robot generation model described in §D.3 (SDXL with ControlNet conditioning [8, 56, 48]) synthesizes the target robot in the same scene context; third, the generated target robot is composited back into the frame to obtain a translated observation . We use the original diffusion-based RoVi-Aug pipeline for this real-world baseline to compare against the published pixel-space translation method.
This preprocessing is computationally expensive. The robot-to-robot generation step takes approximately 10 seconds per frame. Since our real-world demonstrations are recorded at 10 Hz, translating a 10-second trajectory with 100 frames takes roughly 1000 seconds, or about 17 minutes. This cost is incurred for the source demonstration frames used by the pixel-space translation baseline. In contrast, UniLatent does not synthesize target-domain RGB demonstrations: source observations are encoded into the target-aligned latent space for policy learning, and target observations are encoded directly by the frozen target encoder at deployment.
Discussion. The few-shot real-world comparison in the main paper includes a diffusion-based RoVi-Aug pretrain + few-shot setting. RoVi-Aug pretraining provides a useful transfer signal, improving over training from scratch on the target robot on all three tasks (23/30 versus 11/30 in total). However, UniLatent matches or exceeds it on all three tasks, with the largest margin on Wipe Table, the most contact-heavy task. One possible explanation is that translation artifacts and inconsistent robot geometry introduce discrepancies between the translated training images and real target observations. UniLatent avoids this pixel-space mismatch by training and deploying the policy in the aligned latent space.
Additional Results and Analyses
Pooling and Feature Design in Latent Policy Training
Motivation and scope. Our approach uses a pretrained DINOv2 backbone that produces patch tokens and a global [CLS] token . In the cross-embodiment setting, we perform observation alignment by optimizing a source encoder to match a frozen target encoder instantiated from pretrained DINOv2 weights (Eq. (1) in the main paper). Independently of the alignment objective, a key practical design choice for downstream policy learning is how to pool token-level representations into a compact embedding. To isolate the effect of pooling, we evaluate different pooling strategies in a single-robot (Franka), simulation-based setting, where no embodiment shift is present. Throughout this section, “frozen encoder” refers to freezing the DINOv2 backbone; any pooling module on top (when used) is treated as part of the policy and trained jointly with the policy network.
Pooling strategies. We construct the policy input embedding from DINOv2 tokens using the following pooling operators:
Mean: .
CLS: .
Mean+CLS: .
Attn+CLS: , where and is a learnable pooling token trained jointly with the policy.
Experimental design. We evaluate pooling strategies in simulation on the Franka robot across three tasks, described in § C.1. All settings use the same latent policy architecture, training hyperparameters, and evaluation procedure as in the main paper. We vary only the pooling operator used to construct the policy input embedding . We report task success rates over 100 rollouts per task. We consider two training regimes: a frozen-backbone setting, and an end-to-end setting with an unfrozen backbone.
Results and analyses. Table 4 reports results with a frozen DINOv2 backbone, where only the policy is trained (including the learnable pooling token in Attn+CLS). Attn+CLS substantially outperforms fixed pooling across all tasks, indicating that a lightweight, trainable pooling module is important when the visual backbone is kept fixed. Based on this analysis, we use Attn+CLS in all experiments in the main paper.
| Pooling | Close Box Lid | Stack Cup into Bowl | Kick Football |
| Mean | 10 | 3 | 31 |
| CLS token | 6 | 0 | 5 |
| Mean+CLS | 15 | 3 | 29 |
| Attn+CLS | 97 | 89 | 93 |
Table 4. Pooling ablation in a single-robot setting (Franka, simulation) with frozen DINOv2 backbone. We report task success rates (%).
We additionally train policies end-to-end with the DINOv2 backbone unfrozen. Although Uni-Latent keeps the DINOv2 backbone frozen during policy training to preserve the target-anchored representation, this experiment examines whether freezing the backbone limits single-robot performance. As shown in Table 5, end-to-end training with simple pooling (Mean or Mean+CLS) achieves high success rates. In comparison, in the frozen-backbone regime, Attn+CLS reaches similarly strong performance among the pooling variants (Table 4), suggesting that freezing the backbone can still yield high performance when paired with an appropriate pooling module. Interestingly, when the backbone is unfrozen, Attn+CLS performs poorly relative to Mean/Mean+CLS, suggesting that higher-capacity pooling can interact unfavorably with end-to-end representation learning and destabilize optimization. We therefore retain the frozen-backbone design for cross-embodiment transfer and leave end-to-end fine-tuning to future work.
| Pooling | Close Box Lid | Stack Cup into Bowl | Kick Football |
| Mean | 95 | 92 | 88 |
| CLS token | 80 | 69 | 79 |
| Mean+CLS | 95 | 91 | 89 |
| Attn+CLS | 15 | 2 | 18 |
Table 5. Pooling ablation in a single-robot setting (Franka, simulation) with unfrozen DINOv2 backbone trained end-to-end with the policy. We report task success rates (%).
Policy Types: Diffusion vs. Flow Matching
Motivation and scope. Our policy framework is applicable to different generative policy types operating in the aligned latent space. The main paper uses a conditional flow-matching policy as the default instantiation. Here, we implement a DDPM-style diffusion policy [57] and compare it against flow matching under a matched training and evaluation design. We hold the aligned encoders fixed and vary only the generative formulation, thereby isolating its effect on in-domain performance and cross-embodiment transfer.
DDPM variant. The DDPM variant replaces the flow-matching objective with a denoising diffusion objective over action chunks. Given a demonstrated action chunk and Gaussian noise , we sample a diffusion step with and construct a noisy action chunk , where follows a cosine noise schedule. The conditional UNet predicts the noise residual and is trained with
At inference, action chunks are generated by iterating the reverse diffusion process for denoising steps (one network forward pass per step).
In contrast, the flow-matching policy integrates the learned ODE for 10 solver steps using a second-order Midpoint method; each solver step evaluates the velocity network twice (at the start and midpoint), resulting in 20 network forward passes in total. All other components are identical between the two variants: the same conditional UNet backbone, the same frozen aligned encoder with Attn+CLS pooling (§E.1), observation history , action horizon , end-effector action representation, optimizer, batch size, and training schedule.
Results and analyses. Table 6 compares diffusion (DDPM) and flow matching on three simulation tasks (§C.1), reporting both in-domain performance on the source robot (Franka) and crossembodiment transfer to xArm (), with 100 rollouts per setting. Flow matching consistently outperforms DDPM across tasks and evaluation settings, improving the overall average success from 77.0% to 87.7% (averaged over three tasks and two robots). Both policy types exhibit consistent task-level trends: success is highest on Close Box Lid and Kick Football, while Stack Cup into Bowl shows the largest transfer gap, reflecting its higher contact sensitivity. These results indicate that UniLatent is compatible with different generative policy types in the aligned latent space, while flow matching is a stronger instantiation in our setting. In addition to higher success, flow matching requires fewer network forward passes at inference (20 vs. 100 NFEs), yielding substantially faster action generation. We therefore adopt flow matching as the default policy throughout the paper.
| Close Box Lid | Stack Cup into Bowl | Kick Football | Avg. | ||||
| Policy | Franka | xArm | Franka | xArm | Franka | xArm | |
| Diffusion (DDPM) | 89 | 85 | 79 | 49 | 84 | 76 | 77.0 |
| Flow Matching | 97 | 93 | 89 | 65 | 93 | 89 | 87.7 |
Table 6. Policy type comparison in the aligned latent space. Both policies use the same aligned encoders, UNet backbone, training data, and hyperparameters; only the generative formulation differs. Franka is the source robot (in-domain); xArm is the transfer target (Franka→xArm). We report success rates (%) over 100 rollouts.
Effect of Manipulation-Relevant Pose Sampling
We ablate the pose distribution used to generate paired renderings for observation alignment. The proposed sampler draws manipulation-relevant TCP poses using the mixture described in §B.1, combining broad workspace sampling with empirical demonstration-pose samples. As a baseline, we use unconstrained random TCP-pose sampling, which samples feasible TCP poses broadly without biasing them toward task-plausible tabletop manipulation configurations. The downstream policy-training and evaluation protocol is kept fixed. We evaluate zero-shot transfer on Close Box Lid from Franka to xArm and Sawyer, using 100 simulation rollouts per target robot.
Table 7 shows that manipulation-relevant pose sampling substantially improves transfer performance on both target embodiments. Compared with unconstrained random TCP-pose sampling, the manipulation-relevant prior improves success from 60% to 93% on xArm and from 32% to 81% on Sawyer. For reference, the corresponding in-domain Franka policy reaches 97% success on Close Box Lid; thus, manipulation-relevant pose sampling nearly recovers source-level performance on xArm and substantially narrows the transfer gap on Sawyer. These results indicate that paired rendering is most effective when alignment pairs cover robot configurations likely to appear during manipulation.
Transfer under Simulated Background Shifts
Motivation. Cross-embodiment transfer primarily targets the embodiment gap between the source and target robots. At deployment, however, the target robot is often placed in scenes whose background and appearance differ from those seen during source data collection. We therefore examine whether UniLatent remains effective when the scene changes between training and deployment, and whether source-side visual diversity can be inherited by the transferred target policy.
Experimental design. We study the Close Box Lid task with Franka as the source robot and evaluate transfer to xArm and Sawyer. We compare two settings that share the same alignment and latent-policy pipeline and differ only in the scene distribution. In the matched-background setting, training and evaluation scenes are drawn from the same background distribution, following the default protocol used elsewhere in the paper. In the randomized-background setting, source demonstrations are rendered with randomized backgrounds, and the transferred policy is evaluated in five held-out scenes that are not seen during training; we report the mean success rate across these scenes.

Figure 9. Qualitative comparison of pose samplers for paired rendering. Unconstrained random pose sampling frequently produces implausible robot configurations, including twisted joints, collisions, and out-of-view grippers. In contrast, the manipulation-relevant pose prior produces task-plausible tabletop configurations with diverse gripper states and orientations, providing more useful paired observations for alignment training.
Results and discussion. Table 8 shows that UniLatent retains most of its performance under unseen backgrounds. Relative to the matched-background setting, success on the five held-out scenes drops only modestly for the source Franka (97% 88.6%) and for xArm (93% 85.6%), and is essentially unchanged for Sawyer (81% 80.0%). The aligned latent policy therefore retains most of its performance under the tested held-out simulated backgrounds, suggesting that background randomization applied on the source side carries over to the target embodiments through the shared latent space.
| Franka (source) | xArm | Sawyer | |
| Randomized background | 88.6 | 85.6 | 80.0 |
| Matched background | 97 | 93 | 81 |
Table 8. Background generalization via image randomization. Success rates (%) on the simulated Close Box Lid task, with Franka as the source robot and xArm and Sawyer as transfer targets. The first row trains with randomized source backgrounds and tests in 5 unseen scenes; the second uses matched train–test backgrounds.
More broadly, observation alignment addresses embodiment differences but does not guarantee robustness to unseen visual conditions. These results suggest that source-side visual diversity can support generalization after cross-embodiment transfer, although its effectiveness depends on the coverage of the source training distribution. This is convenient in practice, as the source-side data can be diversified (e.g., via background randomization) and remains compatible with complementary techniques such as domain randomization, co-training, and fine-tuning with additional synthetic or real trajectories.
References
- [1]Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, and Mac Schwager. Foundation models in robotics: Applications, challenges, and the future. arXiv preprint arXiv:2312.07843, 2023.
- [2]Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Shibo Zhao, Yu Quan Chong, Chen Wang, Katia Sycara, Matthew Johnson-Roberson, Dhruv Batra, Xiaolong Wang, Sebastian Scherer, Zsolt Kira, Fei Xia, and Yonatan Bisk. Toward general-purpose robots via foundation models: A survey and meta-analysis. arXiv preprint arXiv:2312.08782, 2023.arxiv.org/abs/2312.08782
- [3]Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuanfei Wang, Shaoyang Guo, Tianrui Guan, Ka Nam Lui, Zhiquan Qi, Yitao Liang, Yuanpei Chen, and Yaodong Yang. A survey on vision-language-action models: An action tokenization perspective. arXiv preprint arXiv:2507.01925, 2025.arxiv.org/abs/2507.01925
- [4]Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddakuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Thompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi “Jim” Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingting Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Muhammad Zubair Irshad, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick ”Tree” Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart’in-Mart’in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Vitor Guizilini, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
- [5]Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiacheng Hu, Muhammad Zubair Irshad, Donovan Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukkuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O’Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2403.12945
- [6]Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Xindong He, Xu Huang, et al. AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025.arxiv.org/abs/2503.06669
- [7]Lawrence Yunliang Chen, Kush Hari, Karthik Dharmarajan, Chenfeng Xu, Quan Vuong, and Ken Goldberg. Mirage: Cross-embodiment zero-shot policy transfer with cross-painting. In Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2402.19249
- [8]Lawrence Yunliang Chen, Chenfeng Xu, Karthik Dharmarajan, Muhammad Zubair Irshad, Richard Cheng, Kurt Keutzer, Masayoshi Tomizuka, Quan Vuong, and Ken Goldberg. RoVi-Aug: Robot and viewpoint augmentation for cross-embodiment robot learning. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2409.03403
- [9]Marion Lepert, Ria Doshi, and Jeannette Bohg. Shadow: Leveraging segmentation masks for cross-embodiment policy transfer. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2503.00774
- [10]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.arxiv.org/abs/1703.06907
- [11]Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Su-vir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Arm-strong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Vitor Guizilini, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Yinghao Zhu, Xinyang Geng, Yiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), 2024. Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12627–12637, 2019.
- [12]Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, and Yunfei Bai. RetinaGAN: An object-aware approach to sim-to-real transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 10920–10926. IEEE, 2021.arxiv.org/abs/2011.03148
- [13]Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. RL-CycleGAN: Reinforcement learning aware simulation-to-real. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11157–11166, 2020.arxiv.org/abs/2006.09001
- [14]Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. RoboAgent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In IEEE International Conference on Robotics and Automation (ICRA), 2024.arxiv.org/abs/2309.01918
- [15]Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia. Scaling robot learning with semantically imagined experience. In Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2302.11550
- [16]Stephen Tian, Blake Wolfe, Kyle Sargent, Katherine Liu, Sergey Zakharov, Vitor Guizilini, and Jiajun Wu. View-invariant policy learning via zero-shot novel view synthesis. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2409.03685
- [17]Zoey Chen, Zhao Mandi, Homanga Bharadhwaj, Mohit Sharma, Shuran Song, Abhishek Gupta, and Vikash Kumar. Semantically controllable augmentations for generalizable robot learning. In International Journal of Robotics Research (IJRR), 2024.arxiv.org/abs/2409.00951
- [18]Chengbo Yuan, Suraj Joshi, Shaoting Zhu, Hang Su, Hang Zhao, and Yang Gao. RoboEngine: Plug-and-play robot data augmentation with semantic robot segmentation and background generation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025.arxiv.org/abs/2503.18738
- [19]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π₀: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025.DOI
- [20]Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Jorssa Sadhich, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2405.12213
- [21]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π₀.₅: a vision-language-action model with open-world generalization. In Conference on Robot Learning (CoRL), 2025.arxiv.org/abs/2504.16054
- [22]Tianhao Chen, Adithyavairavan Murali, Abhishek Gupta, and Jitendra Malik. Hardware conditioned policies for multi-robot transfer learning. In Neural Information Processing Systems (NeurIPS), 2018.arxiv.org/abs/1811.09864
- [23]Jonathan Yang, Catherine Glossop, Arjun Bhorkar, Dhruv Shah, Quan Vuong, Chelsea Finn, Dorsa Sadigh, and Sergey Levine. Pushing the limits of cross-embodiment learning for manipulation and navigation. In Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2402.19432
- [24]Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2408.11812
- [25]Gautam Salhotra, I-Chun Arthur Liu, and Gourav Sukhatme. Learning robot manipulation from cross-morphology demonstration. In Conference on Robot Learning (CoRL), 2023.arxiv.org/abs/2304.03833
- [26]Jinliang Zheng, Jianxiong Li, Dongxiu Liu, Yinan Zheng, Zhihao Wang, Zhonghong Ou, Yu Liu, Jingjing Liu, Ya-Qin Zhang, and Xianyuan Zhan. Universal actions for enhanced embodied foundation models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2501.10105
- [27]Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. XSkill: Cross embodiment skill discovery. In Conference on Robot Learning (CoRL), 2023.arxiv.org/abs/2307.09955
- [28]Kushal Kedia, Pritwish Dan, Angela Chao, Maximus Adrian Pace, and Sanjiban Choudhury. One-shot imitation under mismatched execution. In IEEE International Conference on Robotics and Automation (ICRA), 2025.arxiv.org/abs/2409.06615
- [29]Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Panang R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, Igor Gilitschenski, Yonatan Bisk, and Debidatta Dwibedi. Vid2Robot: End-to-end video-conditioned policy learning with cross-attention transformers. In Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2403.12943
- [30]Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. EgoMimic: Scaling imitation learning via egocentric video. In IEEE International Conference on Robotics and Automation (ICRA), 2025.arxiv.org/abs/2410.24221
- [31]Simar Kareer, Karl Pertsch, James Darpinian, Judy Hoffman, Danfei Xu, Sergey Levine, Chelsea Finn, and Suraj Nair. Emergence of human to robot transfer in vision-language-action models, 2025. URL https://arxiv.org/abs/2512.22414.
- [32]Marion Lepert, Jiaying Fang, and Jeannette Bohg. Phantom: Training robots without robots using only human videos. In Conference on Robot Learning (CoRL), 2025.arxiv.org/abs/2503.00779
- [33]Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. HumanPlus: Humanoid shadowing and imitation from humans. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2406.10454
- [34]Tyler Ga Wei Lum, Olivia Y. Lee, C. Karen Liu, and Jeannette Bohg. Crossing the human-robot embodiment gap with sim-to-real RL using one human demonstration. In Conference on Robot Learning (CoRL), 2025.arxiv.org/abs/2504.12609
- [35]Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Thompson, Jeannette Bohg, and Debidatta Dwibedi. XIRL: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning (CoRL), 2021.arxiv.org/abs/2106.03911
- [36]Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), 2018.arxiv.org/abs/1704.06888
- [37]Ajay Kumar Tanwani. DIRL: Domain-invariant representation learning for sim-to-real transfer. In Conference on Robot Learning (CoRL), 2020.arxiv.org/abs/2011.07589
- [38]Tianyu Wang, Dwait Bhatt, Xiaolong Wang, and Nikolay Atanasov. Cross-embodiment robot manipulation skill transfer using latent space alignment, 2024.arxiv.org/abs/2406.01968
- [39]Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, Lukasz Wawrzyniak, Milad Rakhsha, Alain Denzler, Eric Heiden, Ales Borovicka, Ossama Ahmed, Iretiayo Akinola, Abrar Anwar, Mark T. Carlson, Ji Yuan Feng, Animesh Garg, Renato Gasoto, Lionel Gulich, Yijie Guo, M. Gussert, Alex Hansen, Mihir Kulkarni, Chenran Li, Wei Liu, Viktor Makoviychuk, Grzegorz Malczyk, Hammad Mazhar, Masoud Moghani, Adithyavairavan Murali, Michael Noseworthy, Alexander Poddubny, Nathan Ratliff, Welf Rehberg, Clemens Schwartke, Ritvik Singh, James Latham Smith, Bingjie Tang, Ruchik Thaker, Matthew Trepte, Karl Van Wyk, Fangzhou Yu, Alex Millane, Vikram Ramasamy, Remo Steiner, Sangeeta Subramanian, Clemens Volk, CY Chen, Neel Jawale, Ashwin Varghese Kuruttukulam, Michael A. Lin, Ajay Mandlekar, Karsten Patzwaldt, John Welsh, Huihua Zhao, Fatima Anes, Jean-Francois Lafleche, Nicolas Moënne-Loccoz, Soowan Park, Rob Stepinski, Dirk Van Gelder, Chris Ameyor, Jan Carius, Junyung Chang, Anka He Chen, Pablo de Heras Ciechomski, Gilles Daviet, Mohammad Mohajerani, Julia von Muralt, Viktor Reutsky, Michael Sauter, Simon Schirm, Eric L. Shi, Pierre Terdiman, Kenny Vilella, Tobias Widmer, Gordon Yeoman, Tiffany Chen, Sergey Grizan, Cathy Li, Lotus Li, Connor Smith, Rafael Wiltz, Kostas Alexis, Yan Chang, David Chu, Linxi ”Jim” Fan, Farbod Farshidian, Ankur Handa, Spencer Huang, Marco Hutter, Yashraj Narang, Soha Pouya, Shiwei Sheng, Yuke Zhu, Miles Macklin, Adam Moravanszky, Philipp Reist, Yunrong Guo, David Hoeller, and Gavriel State. Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025.arxiv.org/abs/2511.04831
- [40]Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, Chaoyi Xu, Haozhe Chen, Yuxi Qian, Yiran Geng, Jiangeng Mao, Weikang Wan, Mingtong Zhang, Jiangran Lyu, Siheng Zhao, Jiazhang Zhang, Jialiang Zhang, Chengyang Zhao, Haoran Lu, Yufei Ding, Ran Gong, Yuran Wang, Yuxuan Kuang, Ruihai Wu, Baoxiong Jia, Carlo Sferrazza, Hao Dong, Siyuan Huang, Yue Wang, Jitendra Malik, and Pieter Abbeel. Robo-Verse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning. In Robotics: Science and Systems (RSS), 2025.arxiv.org/abs/2504.18904
- [41]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeddin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024.arxiv.org/abs/2304.07193
- [42]Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577, 2022.arxiv.org/abs/2209.14577
- [43]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.02747
- [44]Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C. Karen Liu. DexCap: Scalable and portable mocap data collection system for dexterous manipulation. In Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2403.07788
- [45]Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders, 2025.arxiv.org/abs/2510.11690
- [46]Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded SAM: Assembling open-world models for diverse visual tasks, 2024.arxiv.org/abs/2401.14159
- [47]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos, 2024. URL https://arxiv.org/abs/2408.00714.
- [48]Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2302.05543
- [49]Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. ManiSkill: Generalizable manipulation skill benchmark with large-scale demonstrations, 2021. URL https://arxiv.org/abs/2107.14483.
- [50]Bo Liu, Yifeng Zhu, Chonkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning, 2023. URL https://arxiv.org/abs/2306.03310.
- [51]Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. RLBench: The robot learning benchmark & learning environment, 2019. URL https://arxiv.org/abs/1909.12271.
- [52]Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Kevin Lin, Abhiram Maddakuri, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning, 2025. URL https://arxiv.org/abs/2009.12293.
- [53]Balakumar Sundaralingam, Siva Kumar Sastry Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, Nathan Ratliff, and Dieter Fox. cuRobo: Parallelized collision-free minimum-jerk robot motion generation, 2023. URL https://arxiv.org/abs/2310.17274.
- [54]Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin. Albumentations: Fast and flexible image augmentations. Information, 11(2):125, 2020. doi:10.3390/info11020125.DOI
- [55]Erwin Coumans and Yunfei Bai. PyBullet, a Python module for physics simulation for games, robotics and machine learning, 2016–2020.
- [56]Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2307.01952
- [57]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023.