betaFeedback

WorksComputer Science

Render the Robot, Generate the World: One Video World Model Across Embodiments

Abstract

Predicting the consequences of an agent's actions requires modeling both the body's motion and the environment's response. Most video world models attempt to learn both from numerical actions, even though geometry and kinematics already specify the intended motion. We introduce Doppel, an embodiment-agnostic video world model conditioned on camera-aligned renderings of future body configurations and numerical action trajectories. By explicitly providing the intended motion, Doppel can focus on predicting object motion and interaction-driven changes in the environment. Rendered motion also provides a common visual interface across different embodiments and action spaces. To train on robot datasets without camera extrinsics, we recover camera poses and align rendered trajectories with recorded observations. We jointly train a single shared video backbone and conditioning adapter on eight datasets spanning eight robot and human embodiments, from single- and bimanual robot arms to a dexterous humanoid and human hands and bodies, using about 600 hours of video without embodiment labels. Across three robot-arm datasets, Doppel improves PSNR over action-only baselines by 1.00 dB on held-out trajectories and 1.20 dB under out-of-distribution evaluation, while predicting more consistent object motion. Training on all eight embodiments (Doppel-8) improves robot predictions over the three-robot model (Doppel-3) on four of six evaluation sets. Project page: https://doppelwm.github.io/.

Introduction

“Civilization advances by extending the number of important operations which we can perform without thinking about them.”

Alfred North Whitehead, An Introduction to Mathematics (1911)

When we reach for a cup, we have a sense of where our arm is and can anticipate how it will move. Proprioceptive feedback and internal predictions of movement help us estimate the position of our limbs, even when we cannot see them [1]. Knowing our own motion, however, does not fully determine what happens to the cup: it may slide, tip over, or stay in our grasp, depending on how we make contact. In everyday physical interactions, we intuitively combine knowledge of our own body with predictions of how the surrounding world will respond. We build on this factorization by using known robot kinematics and camera extrinsics to render the intended robot motion, allowing the world model to focus on modeling the environment’s response.

Doppel shared world model: manipulator, human, and humanoid inputs with future renders, predicting future scenes using future embodiment renders across robots and humans

Figure 1. Doppel is a surprisingly simple approach for incorporating future embodiment renderings and improving robot-object interactions in action-conditioned video world models.

For a robot, its geometry and kinematics provide an explicit description of its body. We study how to incorporate this knowledge in video world models, which predict future observations from an initial image and a proposed action trajectory. Recent methods adapt pretrained video generators to this task by conditioning them on numerical action vectors (for example robot joint poses) [2]. The model must then learn both how the actions move the robot in the image and how that motion affects the environment. However, much of the first mapping can already be computed from the robot’s geometry, joint configurations, and camera parameters.

Our key idea is to render the robot along the proposed action trajectory and condition the video model on these future frames. We use forward kinematics to compute link poses from joint configurations and the robot URDF, and a learned differentiable rendering model to generate camera-aligned robot renderings from the observed camera viewpoint. The resulting frames show where the arm and gripper would be if the robot followed the provided action trajectory. In the spirit of Whitehead’s observation, we use this known structure to simplify what must be learned. With the intended robot motion provided explicitly through visual renderings, the video model can focus on predicting its consequences in the environment: whether a grasp would remain stable, how far would a pushed object move, and which way would a cup rotate.

We introduce Doppel, an embodiment-agnostic video world model that uses this conditioning. Doppel combines future robot renderings with the numerical action trajectory. Like a doppelgänger, the rendered robot is a faithful double of the real one; Doppel lets this double show where the robot will move, so the model can focus on how the world responds. Renderings describe the robot’s geometry and motion in the image, while the numerical inputs retain information that may be visually ambiguous. Expressing actions this way also enables learning across robots with different joint structures and control spaces. Although Franka, UR5e, and WidowX move differently, their renderings describe the intended motion in a common image space. We therefore train a shared conditioning adapter and video backbone across these embodiments, using robot-specific geometry and normalization only to construct the inputs. This yields a single set of model weights that can represent multiple robot embodiments, avoiding the need to train and maintain a separate world model for each robot.

Realizing this representation across large-scale robot datasets requires accurate alignment between the rendered robot and the observation camera. This alignment depends on the camera pose relative to the robot base, which is often missing from manipulation datasets. We recover these missing extrinsics by optimizing the differentiable renderer to align rendered robot motion with the observations. This allows us to construct camera-aligned conditioning videos across datasets with different robots and camera setups.

Finally, because our formulation explicitly separates the robot’s intended motion from its effects on the environment, we evaluate these components separately. We measure prediction quality over robot and moving-object regions, allowing us to distinguish accurate generation of the robot’s motion from accurate prediction of the resulting environment dynamics. We train Doppel jointly on Droid (Franka), RoboMind (UR5e), and Bridge (WidowX), a model we call Doppel-3, and evaluate generalization to unseen trajectories and environments. Because the conditioning only needs a rendering of the body, the same model extends beyond robot arms: we also train it on eight embodiments (Doppel-8), adding a bimanual robot (ABC), a humanoid with dexterous hands (T-Rex), and human hands and bodies (GigaHands, OakInk2, Ego-Exo4D) rendered from fitted body and hand meshes. Our contributions are:

  • Embodiment-agnostic conditioning. We introduce Doppel, a shared video world model conditioned on future robot renderings and numerical actions. This representation describes robot motion in a common image space across embodiments (§ 4).

  • Joint multi-embodiment learning. We train one set of weights across three robot arms, outperforming action-only experts, and achieve higher average PSNR and SCS than a shared action-only baseline with 63.8% fewer training draws. Scaling the same model to eight embodiments (Doppel-8), including a humanoid and human hands and bodies, with no embodiment label, further improves robot predictions on Droid and RoboMind (§ 7).

  • Interaction-focused evaluation. We evaluate robot motion and object dynamics separately using region-specific metrics. Our model improves visual fidelity and motion consistency over action-only and skeleton-conditioned baselines.

  • A camera-calibrated corpus. We release recovered camera extrinsics for over 45k demonstrations across WidowX, UR5e, and bimanual YAM, along with calibration code for additional datasets and embodiments (§ 4.1).

Related Work

Video world models. World models predict how an environment evolves under an agent’s actions and use these predictions for planning and policy learning [3, 4, 5]. In robotics, action-conditioned video prediction has benefits in model-predictive control [6, 7] and policy improvement [8, 9]. Recent large-scale models extend this idea to interactive simulators learned from video [10, 11, 12]. Since pretrained video generators [13, 14, 15] provide strong visual priors, they can be adapted into action-conditioned world models [16], used to synthesize demonstrations from which actions are inferred for policy learning [17, 18], or used as learned simulators for policy evaluation [19, 20].

Most robot world models condition on numerical actions per frame [21], with multi-view memory [20], or through timestep modulation [2]; others jointly model video and actions [22]. These models must learn how actions translate into image-space robot motion, although kinematics and camera geometry largely determine this mapping. Recent work introduces geometry through skeletons [23], 3D robot point flow [24], occupancy-derived visual priors [25], depth- and normal-aware 4D representations [26], embodiment masks injected through a ControlNet branch [27], projected end-effector pose maps with delta-action embeddings [28], robot pointmaps with joint RGB and pointmap generation [29], egocentric hand-mesh and static-scene renderings [30], and rendered robot motion for demonstration synthesis [31, 32]. Unlike these, we render the full articulated body along the trajectory and pair it with numerical actions; ORV [25] and EnerVerse-AC [28] also keep numerical actions, but alongside occupancy renderings or end-effector maps rather than the body itself. Render tokens attend to action tokens, and the result modulates every transformer block per token, in one model shared across robots, humans, and a humanoid without an embodiment label. We can use joint-space trajectories from a policy’s action chunks or an off-the-shelf planner.

Multi-embodiment learning. Pooling data across robots lets policies share experience across embodiments [33, 34, 35, 36], typically through embodiment-specific tokenizers or action heads [37, 38]. Other work closes the embodiment gap in image space by swapping one embodiment for another, at test time or in the training data [39, 40, 41], or by replacing both with a shared segmentation mask [42]. Motion can also be expressed visually, as trajectory sketches [43], object flow [44], point trajectories [45, 46], projected skeletons [23, 47], or shared particle representations for dynamics [48].

We bring both joint space and image-space embodiment representations together in video world models. Differentiable rendering [49] places each robot’s motion in a common camera-aligned space, showing shape, gripper geometry, and self-occlusion, while numerical actions retain joint and gripper details that are ambiguous in the image. This lets us train one backbone and adapter jointly across robots, keeping robot-specific assets and normalization and padding actions to a common size. Many offline datasets lack camera extrinsics, so we recover them before training for joint training across embodiments.

Setting and Preliminaries

We learn a single visual forward-dynamics model across robots with different geometries and kinematics. Let e∈Ee \in\mathcal{E} denote an embodiment with a ded_e-dimensional configuration space. For each embodiment, we use episodes of time-aligned RGB observations o0:T−1\mathbf{o}_{0:T-1} and configuration trajectories a0:T−1ea^e_{0:T-1}, where ate∈Rdea^e_t \in\mathbb{R}^{d_e} contains the joint positions and gripper state. We also assume access to a URDF Ue\mathcal{U}_e describing the robot’s geometry and kinematics.

Given an initial image o0\mathbf{o}_0 and a planned configuration trajectory a1:T−1ea^e_{1:T-1}, we predict future images showing how the robot moves and how the environment responds:

pθ(o1:T−1∣o0,a1:T−1e,Ue),(1)p_\theta(\mathbf{o}_{1:T-1} \mid\mathbf{o}_0, a^e_{1:T-1}, \mathcal{U}_e), \tag*{(1)}

where we share θ\theta across embodiments and model uncertainty in physical properties and object states not visible in the initial image. Since joint vectors differ in size and meaning across robots, we need a common action representation. We evaluate unseen trajectories in familiar environments and shifts to completely new objects, scenes, and environments.

Latent video flow matching. We build on a pretrained image-to-video rectified-flow transformer [15]. We use a frozen causal VAE to encode a clip of TT RGB frames into a latent tensor z∈RC×Tz×Hz×Wz\mathbf{z} \in\mathbb{R}^{C \times T_z \times H_z \times W_z}, where TzT_z, HzH_z, and WzW_z are the compressed temporal and spatial dimensions. Given Gaussian noise ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), we interpolate between the clean latent and noise at flow timestep τ∈[0,1]\tau\in[0,1]:

Doppel Architecture

Figure 2. Doppel Architecture. We encode camera-aligned robot renderings with a frozen VAE and fuse their features with numerical actions. We use the resulting representation to modulate the video backbone, which predicts future frames from the initial observation.

zτ=(1−τ)z+τϵ,(2)z_\tau= (1-\tau)z + \tau\epsilon, \tag*{(2)}

where τ\tau is distinct from video time. We train the model to predict the target velocity ϵ−z\epsilon-z [50].

To condition on the initial observation, we replace the first temporal slice of zτz_\tau with its clean value. We compute the loss over the remaining slices using a mask M∈{0,1}1×Tz×1×1M \in\{0,1\}^{1 \times T_z \times1 \times1}, with M0=0M_0=0 and Mj=1M_j=1 for j>0j>0:

Lflow=Eτ,ϵ[∥M⊙(fθ(zτ,τ,c)−(ϵ−z))∥22],(3)\mathcal{L}_{\mathrm{flow}} = \mathbb{E}_{\tau,\epsilon}\left[\left\|M \odot\left(f_\theta(z_\tau,\tau,c)-(\epsilon-z)\right)\right\|_2^2\right], \tag*{(3)}

where cc is the conditioning provided to the transformer. We pass an empty string to the text encoder throughout training and evaluation. Our predictions therefore depend on the initial image and the robot conditioning described below.

Differentiable robot rendering. We use Dr. Robot [49] to render the robot at a given configuration. For each embodiment ee, we use a set of 3D Gaussians Ge0\mathcal{G}_e^0 to represent the robot in a canonical pose. Given a configuration atea_t^e, we compute the joint transformations through forward kinematics and deform the Gaussians using an implicit linear blend skinning function DeD_e. We then apply the pose-conditioned appearance model XeX_e, which adjusts their orientation, scale, opacity, and spherical-harmonic coefficients:

Ge(ate)=Xe(De(Ge0,ate),ate).(4)\mathcal{G}_e(a_t^e) = X_e\left(D_e(\mathcal{G}_e^0,a_t^e),a_t^e\right). \tag*{(4)}

We render the posed Gaussians with a differentiable rasterizer using the camera intrinsics KiK_i and extrinsics CiC_i:

rte=πKi,Ci(Ge(ate)),(5)r_t^e = \pi_{K_i,C_i}\left(\mathcal{G}_e(a_t^e)\right), \tag*{(5)}

where rte∈RH×W×3r_t^e \in\mathbb{R}^{H \times W \times3} is the robot image for episode ii at video time tt. We repeat this for the configuration trajectory a0:T−1ea_{0:T-1}^e to obtain a rendering sequence r0:T−1er_{0:T-1}^e aligned with the observation camera. The renderer is differentiable with respect to both robot configurations and camera parameters. We use its camera gradients to recover missing extrinsics.

Doppel

Our key idea is to provide the robot’s intended motion explicitly so the video model can focus on predicting its effects on the environment. We first recover the camera pose relative to the robot base. We then render the robot along the supplied configuration trajectory from this viewpoint. DOPPEL uses our render-enriched action conditioning adapter to combine these renderings with the numerical joint and gripper trajectories. The renderings show where the robot moves in the image, while the numerical inputs retain configuration information that may be difficult to see. We use this representation to condition DOPPEL, a single image-to-video model shared across embodiments.

Camera calibration preprocessing

We need to render the robot from the same viewpoint as the recorded observations. However, the source datasets have missing or inconsistent camera extrinsics. We recover these extrinsics using the calibration procedure of Jangir et al. [51]. We optimize a six-degree-of-freedom camera-to-robot transform by aligning rendered robot trajectories with the recorded video using RGB, optical flow, and DINOv2 features [52]. We keep the robot renderer fixed and optimize only the camera pose using analysis by synthesis approach.

We calibrate each fixed camera once using a recorded RGB trajectory, the corresponding robot configuration trajectory, the robot’s URDF, and the camera intrinsics. We reuse the same differentiable renderer that produces our conditioning videos and recover extrinsics without calibration targets. The recovered camera-to-robot transform is then reused for all demonstrations captured with that camera and remains fixed unless the camera viewpoint changes. We replace the original procedure’s optical-flow estimator with RAFT [53] and optimize multiple camera-pose initializations in parallel. We use a coarse pose search for the initialization. After optimization, we select the pose with the lowest alignment loss and keep it fixed while training with DOPPEL. This procedure removes the need for manual camera calibration by automatically recovering missing camera extrinsics during dataset preprocessing at scale. Qualitative results of the camera calibration are shown in Figure 20. See App. § 9.9.

Figure 20. Overlays of camera calibration results.

Rendering-Enriched Dense Action Representation

We combine robot renderings and numerical actions to produce one conditioning vector for each video token. Renderings describe the robot’s motion in image space, while numerical actions provide joint and gripper information hidden by occlusion or ambiguous from a single view.

Render and action encoding. We encode the rendering sequence r0:T−1r_{0:T-1} with the same frozen causal VAE used for the target video. The encoded renderings have the same latent resolution as the target video. Most of each rendering is empty background, so we center each latent channel by subtracting its mean across frames and spatial locations. We then use a small 3D convolutional encoder to produce features on the backbone’s token grid. We add separate positional embeddings for time, height, and width, and use self-attention to let tokens exchange information across the rendering sequence. We normalize the configuration trajectory a0:T−1ea^{\mathrm{e}}_{0:T-1} using the training statistics for each robot. We pad these vectors so all robots use inputs of the same size, although the joint represented by each coordinate can differ across robots. We apply a shared MLP to each configuration and add a learned embedding for its time index.

Render-action fusion. We let the render tokens attend to the action sequence. In this cross-attention step, render tokens supply the queries and action tokens supply the keys and values. Each output token keeps its location on the video grid and also contains information about the robot’s configuration. We denote the complete render–action adapter by gϕg_{\phi}:

G=gϕ(r0:T−1,a0:T−1e)∈RN×d,(6)G = g_{\phi}\left(\mathbf{r}_{0:T-1}, \mathbf{a}^{e}_{0:T-1}\right) \in\mathbb{R}^{N \times d}, \tag*{(6)}

where NN is the number of video tokens and dd is the backbone width. We write the conditioning vector at video token jj as gjg_j.

Per-token timestep modulation. We follow Bagchi et al. [2] and condition the video model through its flow-timestep embedding. We use a separate conditioning vector for each video token so the model receives information about robot motion at different positions and times. For token jj, we add gjg_j to the embedding e(s)∈Rde(s) \in\mathbb{R}^{d} of flow timestep ss:

e~j(s)=e(s)+gj.(7)\widetilde{e}_j(s) = e(s) + g_j. \tag*{(7)}

We pass e~j(s)\widetilde{e}_j(s) through the backbone’s existing projection to obtain the shift, scale, and gate parameters for the attention and feed-forward sublayers in every transformer block. We retain the pretrained attention and feed-forward architecture and give the adapter details and tensor dimensions in Appendix § 9.§ 4.

Sharing across embodiments. We train a single render–action adapter and video backbone across all embodiments. Each robot uses its own frozen renderer and action-normalization statistics. We share all trainable parameters and use no embodiment tokens or robot-specific heads. The renderer supplies the geometry and kinematics of each robot, and we use the same world model to learn how its motion changes the scene.

Experimental Setup

Datasets and splits

We train jointly on DROID (Franka), BRIDGE (WidowX), and RoboMind (UR5e) [54, 55, 56]. Franka and UR5e use Robotiq 2F-85 grippers. We sample datasets uniformly and balance environment categories, using short contiguous and temporally subsampled full-span clips. We evaluate held-out trajectories in familiar environments and OOD shifts to a new lab (DROID), a new kitchen (BRIDGE), and unseen objects and tasks (RoboMind). We call this model DOPPEL-3. For DOPPEL-8, we add ABC [57] (bimanual YAM), T-Rex [58] (Vega-1 humanoid with Sharpa hands), GigaHands [59] (human hands), OakInk2 [60] (human upper body), and Ego-Exo4D [61] (human body). The two new robots are rendered from their kinematic models; people are rendered from fitted MANO [62], SMPL-X [63], and SMPL-H [62] meshes. Their test splits hold out tasks (ABC, OakInk2, Ego-Exo4D), activities (GigaHands), or wall backgrounds (T-Rex). All actions are zero-padded to a common 159-dimensional vector and enter one shared action encoder without an embodiment label, and we sample each dataset in proportion to its training clips. All evaluated embodiments appear in training. We give corpus sizes in Table 1, construction details in Appendix § 9.§ 3, and examples in Appendix § 9.§ 7.

DatasetEmbodimentTrain episodesTrain clipsVal. clipsTest clipsTest distribution shift
DROIDFranka Panda1672683511200200Unseen lab
BRIDGEWidowX1238761935200200Unseen kitchen
RoboMindUR5e + 2F-851480174005200200Unseen objects
ABCBimanual YAM9115274523200200Unseen tasks
T-RexVega-1 + Sharpa hands478458120200200Unseen backgrounds
Ego-Exo4DHuman body (SMPL-H)2041213292200200Unseen tasks
GigaHandsHuman hands (MANO)849131218200200Unseen activities
OakInk2Human upper body (SMPL-X)4036998200200Unseen tasks
Three robot datasets—43914219451600600—
All eight datasets—6874880360216001600—

Table 1. Table 1: Training corpus. Validation clips come from held-out trajectories in seen settings; test clips hold out the listed shift.

Baselines. We compare Doppel against both an action-only [2] and geometric-conditioned baselines [23]. The action-only baseline follows EgoWM-style [2] timestep-modulated action conditioning using the same Wan 2.1 backbone and corresponding dataset’s training corpus. We compare single dataset experts with our multi-embodiment Doppel at approximately matched training exposure per dataset. In separate multi embodiment comparison, both models use same training recipe. OSCAR [23] serves as a geometry-conditioned system-level baseline; we fine-tune one model per embodiment on the corresponding training demonstrations using OSCAR’s temporal windowing and evaluate on the same splits. Because OSCAR uses a different backbone and training formulation, we retain its native architecture while matching the evaluation resolution and removing text prompts for a fairer comparison. Full training configurations for all baselines are provided in Appendix § 9.§ 5.

Evaluation metrics. We report whole-frame PSNR, SSIM [64], and LPIPS [65], along with object SCS [2]. We compute SCS as the overlap of SAM2 masks propagated from the same initial frame through predicted and reference videos [66]. We also measure PSNR separately over the robot arm, moving objects, and SCS over the arm and moving objects. We exclude the conditioning frame from all metrics and give definitions in Appendix § 9.8.

Implementation details. We use 8 H100 GPUs for single-dataset training and 32 GPUs for the multi-dataset runs at a global batch size of 256: Doppel-3 trains for 49.5k steps and Doppel-8 for 189k steps. We give complete training recipes in Appendix Tables 6–9.

Results and Analysis

Prediction across embodiments. We improve whole-frame prediction quality over action-only conditioning and fine-tuned OSCAR [23] on all three robot datasets, across validation and OOD splits (Table 2). With the same training data, adding the robot rendering to the action input improves PSNR over action-only conditioning by 0.2–1.5 dB. Our shared models outperform separately trained action-only experts, and Doppel-8 has the best PSNR, SSIM, and LPIPS on Droid and RoboMIND in both splits. Compared with Doppel-3 on the same clips, adding the five other datasets raises PSNR by 0.52 and 0.21 dB on Droid validation and OOD and by 0.24 and 0.22 dB on RoboMIND, leaves Bridge validation unchanged, and lowers Bridge OOD by 0.24 dB (Appendix Table 10). With every model at ~51 passes over each robot dataset, Doppel-8 has the highest PSNR on five of the six robot evaluation sets and trails Doppel-3 by 0.05 dB on Bridge OOD (Appendix Table 12. On Bridge OOD, both multi-dataset models peak before the end of training and then decline while Bridge validation keeps improving, which suggests that late training specializes to the seen kitchens. Table 3 reports Doppel-8 on the five added datasets, and Figure 4 shows held-out test examples. We show predictions of Doppel-3 in held-out environments and with unseen objects in Figure 3.

Figure 3. We show predictions with DOPPEL on Franka, WidowX, and UR5e arms in out-of-domain scenes with unseen objects (top to bottom). Each row shows the initial image, future robot renderings used as conditioning, and our predicted frames. We use the same model across all three embodiments. Our model preserves embodiment-consistent motion while predicting plausible environment responses under substantial distribution shift.

Figure 4. We show predictions with Doppel-8 on held-out test clips from the five added datasets: a bimanual YAM robot (ABC), a humanoid with dexterous hands (T-Rex), a human body (Ego-Exo4D), human hands (GigaHands), and a human upper body (OakInk2), from top to bottom. Each row shows the initial image, future embodiment renderings used as conditioning, and our predicted frames. We use the same model across all five embodiments, with no embodiment label.

DatasetMethodInputTrained onValidation: unseen trajectories PSNR ↑Validation: unseen trajectories SSIM ↑Validation: unseen trajectories LPIPS ↓Validation: unseen trajectories SCS ↑OOD test PSNR ↑OOD test SSIM ↑OOD test LPIPS ↓OOD test SCS ↑
DROIDAction-onlyActionsThis dataset22.8880.8410.0780.92720.0690.7790.1310.873
OSCAR (fine-tuned)SkeletonThis dataset20.4270.7510.1350.90318.6190.7050.1740.854
DOPPELRender + actionsThis dataset24.3570.8640.0650.94021.2760.8020.1130.902
DOPPEL-3Render + actions3 robot datasets24.2970.8620.0660.94021.2500.8030.1110.899
DOPPEL-8Render + actionsAll 8 datasets24.8190.8700.0620.94321.4600.8060.1080.903
BRIDGEAction-onlyActionsThis dataset26.2900.8710.0500.94620.5840.7550.1310.866
OSCAR (fine-tuned)SkeletonThis dataset22.5590.7710.1170.91419.4870.6330.2110.854
DOPPELRender + actionsThis dataset26.6180.8750.0480.94921.7220.7710.1110.901
DOPPEL-3Render + actions3 robot datasets26.5510.8730.0490.94922.4740.7860.0970.905
DOPPEL-8Render + actionsAll 8 datasets26.5440.8730.0490.94822.2380.7840.0990.899
RoboMindAction-onlyActionsThis dataset30.4740.9230.0380.94030.0980.9180.0390.932
OSCAR (fine-tuned)SkeletonThis dataset27.4560.8820.0790.91427.6610.8780.0760.920
DOPPELRender + actionsThis dataset31.1040.9280.0370.94230.2900.9190.0390.928
DOPPEL-3Render + actions3 robot datasets31.0650.9280.0370.94130.4420.9200.0380.935
DOPPEL-8Render + actionsAll 8 datasets31.3020.9300.0360.94230.6590.9210.0380.936

Table 2. Table 2: Prediction quality on the three robot datasets. Input is what the model is conditioned on besides the first frame; Trained on is its training data. All our models are evaluated at their final checkpoint and saw a similar amount of each robot dataset during training. Best in bold, second-best underlined; ties share emphasis. 200 clips per split; every method is scored on the same clips.

DatasetValidation: unseen trajectoriesTest: held-out shift
PSNR ↑SSIM ↑LPIPS ↓SCS ↑PSNR ↑SSIM ↑LPIPS ↓SCS ↑
ABC24.6530.8500.0960.92025.2170.8610.0890.927
T-Rex28.3270.8710.0500.96128.3460.8670.0510.956
Ego-Exo4D27.9350.8950.0450.96627.6110.8920.0440.965
GigaHands25.6220.8850.0530.94125.7840.8970.0490.944
OakInk229.0140.9180.0360.97029.0900.9210.0330.970

Table 3. Doppel-8, evaluated on the five datasets added beyond the three robot-arm datasets: a bimanual robot (ABC), a humanoid with dexterous hands (T-Rex), human bodies (Ego-Exo4D), human hands (GigaHands), and human upper bodies (OakInk2). These are the same weights as the last row of each block in Table 2. Validation holds out trajectories; the test split holds out the shift listed in Table 1.

DatasetMethodInputTrained onValidation: unseen trajectories PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowSCS ↑\uparrowOOD test PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowSCS ↑\uparrow
DroidAction-onlyActionsThis dataset22.8470.8410.0790.92720.0470.7800.1310.873
DoppelRender + actionsThis dataset24.3220.8640.0650.94021.2780.8020.1130.903
Doppel-3Render + actions3 robot datasets24.2970.8620.0660.94021.2500.8030.1110.899
Doppel-8Render + actionsAll 8 datasets24.7490.8690.0630.94321.4210.8050.1090.901
BridgeAction-onlyActionsThis dataset25.6250.8640.0540.94120.6650.7560.1290.869
DoppelRender + actionsThis dataset26.3110.8710.0500.94721.8240.7730.1090.905
Doppel-3Render + actions3 robot datasets26.3890.8710.0500.94722.3480.7840.0990.905
Doppel-8Render + actionsAll 8 datasets26.5210.8730.0490.94822.2950.7850.0990.900
RoboMindAction-onlyActionsThis dataset29.9920.9190.0400.93829.8050.9150.0400.930
DoppelRender + actionsThis dataset31.0070.9280.0370.94230.3870.9200.0380.932
Doppel-3Render + actions3 robot datasets31.0130.9270.0380.94030.5300.9200.0370.939
Doppel-8Render + actionsAll 8 datasets31.2210.9290.0370.94230.6620.9210.0380.936

Table 12. Prediction quality on the three robot datasets at matched training exposure: every model is evaluated at its saved checkpoint closest to 51 passes over each robot dataset.

What joint multi-embodiment training contributes. The five added datasets contain no Franka, UR5e, or WidowX data, yet they improve robot prediction. The model is never told which body it sees: the rendering identifies the embodiment, and the shared action encoder receives zero-padded vectors whose slots mean different joints for different bodies. Overall, Doppel-8 improves PSNR over Doppel-3 on four of the six robot evaluation sets (Table 2; Appendix Table 10), so training on other embodiments helps robot prediction. The exception is Bridge OOD, where Doppel-8 is 0.24 dB below Doppel-3. We suspect that the models begin to overfit to Bridge, the smallest of the three robot datasets, and specialize to its seen kitchens: Bridge OOD PSNR peaks early and then drops while Bridge validation PSNR keeps rising. In our evaluation, Doppel-3 peaks on Bridge OOD at 22.94 dB after ∼\sim10 passes over Bridge and ends at 22.47 dB, while its validation PSNR rises from 24.40 to 26.55 dB; Doppel-8 peaks at 22.57 dB after ∼\sim30 passes and ends at 22.24 dB, while its validation PSNR rises from 26.14 to 26.54 dB.

Predicting object motion. We also improve predictions of the robot and moving objects, measured separately in Figure 6. On the Bridge OOD split, Doppel-8 raises moving-object PSNR from 21.50 to 24.85 dB and SCS from 0.778 to 0.831 over the action-only baseline. We show more consistent object motion on unseen objects in Figure 5. We can see that the baseline moves the arm but leaves the object on the table, while we predict the lift. We provide further examples of trajectory generalization in Figure 21.

Figure 5. Qualitative comparison across trajectory and OOD generalization settings. Compared to the baseline we observe that Doppel generates much plausible robot-object interactions as intended by the provided action trajectory. Additional qualitative results are available on our project website (Appendix § 9.§ 1) and a trajectory generalization illustration in Figure 21.

Figure 6. Prediction quality by image region (robot and moving objects) on the three robot datasets, for every model at its final checkpoint, on the same clips as Table 2. Values are in Appendix Table 13.

Ground truth, action-only (DROID), and Doppel trajectory-generalization comparison

Figure 21. Additional trajectory-generalization example on DROID. We compare DOPPEL with the action-only baseline on a held-out trajectory from a familiar environment.

Training efficiency. We outperform separate action-only experts at comparable numbers of training draws per dataset (Figure 7). We also compare with a shared action-only model, matching the batch size and optimization schedule and counting total draws across datasets (Figure 9). Averaged across datasets, we obtain higher validation and OOD PSNR and SCS after 1.216M draws than this baseline obtains after 3.356M draws, using 63.8% fewer draws. These counts include repeated examples. We provide additional quantitative results in Appendix § 9.10.

Figure 7. Training efficiency of Doppel versus single-dataset action-only experts. Doppel achieves higher validation and OOD prediction quality at comparable training draws.

Figure 9. Training efficiency of Doppel-3 versus a three-robot action-only baseline. Doppel achieves higher validation and OOD prediction quality throughout 3.36M training draws.

Long-horizon rollouts. We chain 3.2 s predictions to roll out held-out test episodes for 25.6 s from a single real frame: each window starts from the model’s own last predicted frame and receives that window’s rendering and actions, so after the first frame the model never observes the real video. Figure 8 shows two long horizon rollouts.

Figure 8. Long-horizon rollouts of Doppel-8 on held-out test episodes from OakInk2 (top) and Ego-Exo4D (bottom). Each episode is predicted for 25.6 s from its first frame alone by chaining 3.2 s windows on the model’s own predictions. Each row shows the initial image, the embodiment renderings given as conditioning, and our predictions at 8.6, 17.0, and 25.6 s.

Discussion, Limitations, and Conclusion

We presented Doppel, a shared video world model that uses known robot geometry to predict interactions across embodiments. Doppel combines numerical actions with camera-aligned future robot renderings. This provides the intended robot motion explicitly, reducing what the model must learn about how joint configurations translate into image-space motion. Across Droid, Bridge, and RoboMIND, we improve whole-frame prediction over action-only and skeleton-conditioned baselines. Our region-specific evaluation also shows better predictions of moving-object appearance and motion, suggesting that explicit robot motion helps the model learn the environment’s response. We observe these gains on held-out trajectories and OOD interactions with new objects and scenes. We retain them with one model trained across all three embodiments and outperform the shared action-only baseline with fewer training draws. Extending training to eight embodiments, including human hands and bodies, further improves robot predictions on DROID and RoboMIND, suggesting that rendered bodies let a world model learn from data well beyond a single robot. This supports using geometric action representations to share dynamics learning across robots with different bodies and configuration spaces.

We require accurate robot geometry, a URDF, and camera calibration to construct the renderings. Although we recover missing extrinsics in offline datasets, unknown or rapidly changing viewpoints remain a limitation. Camera-aware or camera-invariant representations could reduce this dependence. We also condition on configuration trajectories; handling imperfect geometry or lower-level control commands would require modeling uncertainty in robot motion and execution. Our evaluation covers embodiments seen during training and one camera per trajectory. We leave transfer to unseen embodiments, multi-view prediction, and longer-horizon prediction to future work. Combining pretrained video models with learned 3D representations may further reduce the need for explicit pixel-level alignment.

Appendix

Website and Video

Additional qualitative results are available on our project webpage, including extended video comparisons across all eight datasets, long-horizon rollouts, examples of trajectory and environment generalization, and side-by-side comparisons with our baselines. We encourage readers to view the webpage for the full video results, which are difficult to represent comprehensively in static paper figures.

https://doppelwm.github.io/

Rendering Pipeline

We render the robot along the supplied joint and gripper trajectory using an articulated Gaussian robot model and the target camera calibration. The renderer is fixed during world-model training. RGB observations, robot configurations, and rendered frames are constructed using the same sampled source-frame indices, ensuring temporal alignment across all conditioning signals. When constructing training windows, we additionally verify that cached renderings correspond to the same temporal sampling grid.

The robot-only renderings encode the embodiment’s geometry and motion directly in image coordinates; they contain no future scene or object information. Figure 10 and Figure 11 show representative alignment between the rendered robot trajectories and the corresponding RGB observations across all three datasets.

Alignment between observed and rendered robot trajectories

Figure 10. Alignment between the observed and rendered robot trajectories. The top row shows frames from the real RGB trajectory, and the bottom row shows the corresponding target-view robot renderings. The renderings express the commanded embodiment motion in the same image coordinates as the observations, providing the dense visual component of Doppel.

Figure 11. Alignment between the observed and rendered trajectories for the five added embodiments: a bimanual YAM robot (ABC), a humanoid with dexterous hands (T-Rex), a human body (Ego-Exo4D), human hands (GigaHands), and a human upper body (OakInk2). For each dataset, the top row shows the real RGB frames and the bottom row the corresponding camera-aligned renderings.

For each embodiment, we use the differentiable robot-rendering framework of Liu et al. [49]. Starting from the robot’s URDF, we train an articulated Gaussian model using the original renderer architecture and training procedure provided by the authors. We train a separate renderer for each robot embodiment and keep its parameters frozen when training Doppel.

Additional embodiments. The bimanual YAM arms in ABC are rendered with a DrRobot model trained for the YAM in the same way. The Vega-1 humanoid and its Sharpa hands in T-Rex are rendered from the robot’s URDF meshes with the dataset’s released head-camera calibration. People are rendered from fitted body and hand meshes: MANO hands for GigaHands [62], SMPL-X for OakInk2 [63], and SMPL-H for Ego-Exo4D, using the meshes of Maddukuri and Pavlakos [67]; each mesh is rasterized through the dataset’s camera model, including lens distortion. All renderings use a black background and contain only the embodiment.

Dataset Construction and Evaluation Splits

Training data. We construct training clips from demonstrations in Droid, RoboMind, and Bridge. Within each dataset, we balance sampling across environment categories. From each demonstration, we extract up to five clips: four contiguous windows capturing local interaction dynamics and one full-span window covering the entire demonstration at a coarser temporal resolution. Demonstrations that are too short contribute fewer clips. Table 1 reports the resulting numbers of demonstrations and training clips. During three-robot training, we sample uniformly across the three datasets, such that each embodiment contributes approximately one third of the optimization batches.

Additional embodiments. For ABC, T-Rex, GigaHands, OakInk2, and Ego-Exo4D, we cut each video into back-to-back 3.2 s windows at 5 fps, plus equidistant segment clips and one whole-video clip; for Ego-Exo4D we use two exocentric cameras per take. Whole-video clips of ABC and Ego-Exo4D are evaluated but not used for training. Validation holds out a random 5% of episodes. The test splits hold out: 19 of ABC’s tasks; two of T-Rex’s twelve wall backdrops (400 of 5,464 episodes), never seen in training or validation; one task per Ego-Exo4D activity (Mikan layup, chain cleaning, latte making, advanced salsa, CPR, piano scales, a bouldering route, and penalty kicks); five GigaHands activities (tea, stone, laptop, phone, crochet); and 51 OakInk2 tasks across its four scenes. Table 4 gives the amount of video per split.

DatasetTrainVal.TestTotal
Droid98.95.411.6115.9
Bridge26.41.43.231.1
RoboMind62.13.26.872.1
ABC242.612.731.4286.7
T-Rex44.22.43.950.5
Ego-Exo4D88.35.110.4103.8
GigaHands25.41.22.128.6
OakInk25.21.21.57.9
All eight593.132.670.8696.6

Table 4. Hours of video per dataset and split, from the full length of each source episode; each episode counts once, and Ego-Exo4D is counted per take (training uses up to two exocentric cameras per take, 176.3 h of camera video).

Doppel-8 samples each dataset in proportion to its training clips, which gives about 60 passes over every dataset.

Evaluation splits. We evaluate generalization along two distinct axes. The validation split measures trajectory generalization: for each embodiment, we reserve 200 clips from held-out demonstrations collected in environments that remain represented in the training set. These comprise 100 contiguous-window clips and 100 full-span clips. This split tests whether the model can predict previously unseen robot trajectories within familiar environment distributions.

The test split measures environment generalization under three complementary distribution shifts. For DROID, we hold out an entire laboratory; no scene, camera, operator, or task from this laboratory appears in the training set. For BRIDGE, whose demonstrations are collected in toy-kitchen environments, we hold out one complete kitchen environment. For RoboMind, we hold out four object-centric tasks and evaluate interactions involving object instances absent from the training set. These splits therefore test generalization to a completely new laboratory, an unseen instance of a semantically familiar environment, and unseen objects and tasks, respectively.

Each dataset contributes 200 test clips, evenly divided between 100 contiguous-window clips and 100 full-span clips. We report validation and test results separately because predicting a novel trajectory in a familiar environment and predicting interaction dynamics under an environmental distribution shift capture different generalization capabilities.

Adapter architecture details

This section provides the architectural details of the render–action adapter gϕg_{\phi} introduced in § 4.2. The adapter contains about 30.5M parameters and is shared across all embodiments.

Render normalization. Robot-only renderings are sparse, with most spatial locations containing empty background. We first encode the rendering sequence r0:T−1r_{0:T-1} using the same frozen causal VAE of Wan 2.1:

zrend=Evae(r0:T−1).(8)z^{\mathrm{rend}} = E_{\mathrm{vae}}(r_{0:T-1}). \tag*{(8)}

Before applying the trainable render encoder, we subtract the per-sample, per-channel spatiotemporal mean:

zˉc,t,h,wrend=zc,t,h,wrend−1TzHzWz∑t′,h′,w′zc,t′,h′,w′rend.(9)\bar{z}^{\mathrm{rend}}_{c,t,h,w} = z^{\mathrm{rend}}_{c,t,h,w} - \frac{1}{T_z H_z W_z}\sum_{t',h',w'} z^{\mathrm{rend}}_{c,t',h',w'}. \tag*{(9)}

This prevents the empty-backgrounds from making the renders tokens very similar. If this mean subtractions is skipped the adapter can collapse to zero weights. We compute this operation in fp32, since the robot signal can be small relative to the shared latent response of the empty rendering background.

Render encoder. The centered render latent is processed by three 3D convolutions with GELU activations:

  1. Conv3D(16, 256, 1×3×31 \times3 \times3)

  1. Conv3D(256, 512, 3×3×33 \times3 \times3)

  1. Conv3D(512, 512, 1×3×31 \times3 \times3)

The output grid is flattened into 2025 tokens of width 512. We then add factorized learned spatiotemporal positional embeddings,

pt,h,w=pttime+phheight+pwwidth,(10)p_{t,h,w} = p^{\mathrm{time}}_{t} + p^{\mathrm{height}}_{h} + p^{\mathrm{width}}_{w}, \tag*{(10)}

using temporal, height, and width embedding tables of maximum sizes 32, 32, and 64, respectively. Two pre-normalization self-attention blocks contextualize the render tokens. Each block uses eight attention heads, an FFN expansion factor of four, and residual connections around both the attention and FFN sublayers. This global mixing allows information from the spatially sparse robot region to reach tokens outside the convolutional encoder’s local receptive field.

Action encoder. Each frame is associated with a configuration vector: up to seven joint values and one gripper value for the robot arms, and up to 159 values for the other embodiments. We zero-pad every vector to 159 dimensions. We normalize each embodiment using its own training-set statistics and zero-pad configurations with fewer degrees of freedom to the shared 159-dimensional layout. Padding provides a common input shape but does not establish correspondence between the joint dimensions of different embodiments.

A shared framewise MLP maps each action vector to width 512:

htact=W2 GELU⁡(W1ate)+ptact.(11)h^{\mathrm{act}}_{t} = W_{2}\,\operatorname{GELU}(W_{1}a^{e}_{t}) + p^{\mathrm{act}}_{t}. \tag*{(11)}

where W1:R159→R512W_{1} : \mathbb{R}^{159} \to\mathbb{R}^{512}, W2:R512→R512W_{2} : \mathbb{R}^{512} \to\mathbb{R}^{512}, and ptactp^{\mathrm{act}}_{t} is selected from a learned temporal position table supporting up to 128 action frames.

Render–action fusion. We fuse the encoded streams using two pre-normalization cross-attention blocks. The 2025 render tokens serve as queries, while the framewise action tokens serve as keys and values:

Hfused=CrossAttn⁡(Hrend,Hact,Hact).(12)H_{\mathrm{fused}} = \operatorname{CrossAttn}(H_{\mathrm{rend}}, H_{\mathrm{act}}, H_{\mathrm{act}}). \tag*{(12)}

Each block uses eight attention heads, an FFN expansion factor of four, and residual connections around both sublayers. A final projection maps each fused token to the Wan hidden dimension:

G=Linear⁡2048→5120(GELU⁡(Linear⁡512→2048(LayerNorm⁡(Hfused)))).(13)G = \operatorname{Linear}_{2048\to5120}\left(\operatorname{GELU}\left(\operatorname{Linear}_{512\to2048}\left(\operatorname{LayerNorm}(H_{\mathrm{fused}})\right)\right)\right). \tag*{(13)}

The resulting tensor has shape G∈R2025×5120G \in\mathbb{R}^{2025 \times5120}. Neither the output projection nor its preceding layers are zero-initialized.

Backbone injection. Let gj∈R5120g_j \in\mathbb{R}^{5120} denote the adapter output for video token jj, and let e(s)e(s) denote the embedded flow timestep. We form the token-specific timestep embedding

e~j(s)=e(s)+gj.(14)\widetilde{e}_j(s) = e(s) + g_j. \tag*{(14)}

The backbone’s existing time_proj maps this embedding to six modulation vectors per token: the shift, scale, and gate applied to the attention and FFN sublayers. The same token-specific embedding is used in all 40 transformer blocks and determines the shift and scale of the final output normalization. Thus, rendering-enriched actions enters exclusively through the pretrained timestep-modulation pathway and does not modify the backbone’s attention or FFN architecture.

During training, we set the adapter contribution GG to zero with probability 0.1, while retaining the initial-frame conditioning. This conditioning dropout regularizes the adapter and provides an action-unconditioned branch for evaluation controls.

Baseline Re-implementation

Action-only baseline. Our action-only baseline follows the conditioning mechanism of EgoWM [2]. Starting from the same Wan 2.1 image-to-video backbone used by Doppel, we remove the rendering encoder and render–action fusion modules entirely. The numerical robot configuration at each timestep is normalized using the same per-embodiment statistics as Doppel, projected with an MLP followed by a 1D temporal convolution, and injected through the backbone’s timestep-modulation pathway. Thus, the baseline receives the same numerical robot trajectory as Doppel but no camera-aligned robot renderings.

All remaining components are kept identical to the corresponding Doppel configuration, including backbone initialization, training clips, clip construction, optimization schedule, conditioning dropout, and training budget. All models are evaluated on exactly the same validation and test splits. The complete single-embodiment training recipe is given in Table 6. This baseline therefore provides a controlled comparison between numerical action-only conditioning and Doppel conditioning.

BackboneWan 2.1 I2V 14B (16.4B parameters). We train all 40 transformer blocks together with the output and timestep projections, yielding 16.3B trainable backbone parameters.
AdaptersRender–action adapter: 30.5M parameters; action-only adapter: 4.2M parameters.
DataOne dataset per model: see Table 1.
Input17 frames at 240 × 432. Contiguous windows use 0.2 s temporal spacing, while full-span windows cover the complete demonstration. The text prompt is always empty.
OptimizerAdamW [75] with weight decay 10−310^{-3} and gradient clipping at 1.0. Training uses bf16, fp32 master weights, FSDP, and activation checkpointing.
Learning rates5×10−65 \times10^{-6} for the backbone and 2×10−52 \times10^{-5} for the conditioning adapter.
ScheduleCosine decay with 500 warm-up steps and a minimum learning-rate ratio of 0.05.
Batch size8 examples per GPU on 8 H100-80GB GPUs, giving a global batch size of 64.
Weight EMADecay of 0.9975, updated every 10 steps after a 500-step warm-up.
Conditioning dropoutProbability 0.1 per training step.
ObjectiveLatent flow matching [50].

Table 6. Training configuration for the single-embodiment models.

OSCAR. For OSCAR [23], we retain the released architecture and its 2D kinematic-skeleton conditioning rather than modifying its conditioning formulation. We initialize from the released OSCAR checkpoint and fine-tune a separate model for each of Droid, Bridge, and RoboMind using the corresponding training demonstrations and selected start positions, with OSCAR’s 81-frame temporal windows. We do not report zero-shot results on Bridge or RoboMind, since these datasets are not part of OSCAR’s original training corpus.

To align OSCAR more closely with our evaluation setting, we fine-tune it at the same 432×240432 \times240 resolution used by Doppel and use an empty text prompt throughout training and evaluation. We preserve its native skeleton-conditioning mechanism and otherwise follow the fine-tuning configuration summarized in Table 5. All models are evaluated on the same validation and test clips. Because OSCAR differs from Doppel in backbone, pretraining data, model size, conditioning representation, and optimization history, we do not attempt to equalize architecture-level compute and treat OSCAR as a system-level baseline rather than a controlled ablation of Doppel.

Released OSCAROSCAR (Fine-tuned)
InitializationCosmos-Predict2.5 [68], 2BReleased OSCAR checkpoint
Conditioning2D skeleton + text2D skeleton + empty text prompt
Training data180657 episodes: 94830 robot episodes (RH20T [69], InternData-A1 [70], DROID [54] 21904, AgiBot-Beta [71], AIROA-MoMa [72]) and 85827 human episodes (EgoDex [73], EPIC-Kitchens [74])Our training split (Table 1), using 81-frame windows at the selected start positions, with overlapping windows permitted
Clip length81 frames, native frame rate81 frames, native frame rate
Resolution640 × 480432 × 240
OptimizerAdamWAdamW, β=(0.9,0.99)\beta= (0.9, 0.99), weight decay 10−310^{-3}
Learning rate3×10−53 \times10^{-5}3×10−53 \times10^{-5} with 500-step warmup and cosine decay to 3×10−63 \times10^{-6}
Batch size1616
Training steps15k robot-only steps, followed by robot + human training10000
Hardware1×1 \times GH2008×8 \times A100
Sampling35 steps, guidance scale 635 steps, guidance scale 6

Table 5. OSCAR configurations used in our experiments. The fine-tuned models are initialized from the released OSCAR checkpoint [23].

Dataset Visualizations

We show representative RGB observations and corresponding camera-aligned robot renderings from Droid, Bridge, and RoboMind in Figure 12, Figure 13, and Figure 14, respectively, and from the five added datasets in Figure 15–Figure 19. These examples illustrate the diversity of embodiments, environments, and viewpoints across the eight datasets.

Figure 12. Representative examples from DROID with corresponding camera-aligned robot renderings.

Representative examples from Bridge with corresponding camera-aligned robot renderings, showing Demo 1, Demo 2, and Demo 3 across Frames 0 (initial), 4, 8, 12, and 16

Figure 13. Representative examples from Bridge with corresponding camera-aligned robot renderings.

Representative examples from RoboMind with corresponding camera-aligned robot renderings

Figure 14. Representative examples from RoboMind with corresponding camera-aligned robot renderings.

Representative examples from ABC (bimanual YAM) with corresponding camera-aligned renderings

Figure 15. Representative examples from ABC (bimanual YAM) with corresponding camera-aligned renderings.

Figure 19. Representative examples from OakInk2 (human upper body) with corresponding camera-aligned renderings.

Implementation Details

We provide the training configurations for all our models in Table 6, Table 7 and Table 8. To ensure that the multi-embodiment model observes a comparable number of draws per dataset as the corresponding single-dataset action-only and baseline runs, we use multi-node training with a larger global batch size. Unless otherwise noted, the multi-embodiment models use the same architecture and optimization settings as the single-embodiment models.

DataJoint training on DROID, ROBO MIND, and BRIDGE. Datasets are sampled with equal probability, and clips are sampled uniformly within each dataset.
Hardware8 H100-80GB GPUs on one node, with one FSDP rank per GPU.
Batch size8 examples per GPU, giving a global batch size of 64.
Learning ratesBackbone: 5×10−65 \times10^{-6}; adapter: 2×10−52 \times10^{-5}.
ScheduleCosine decay with 500 warm-up steps and a minimum learning-rate ratio of 0.05.
EMADecay of 0.9975, updated every 10 steps after a 500-step warm-up.

Table 7. Training configuration for the single-node multi-embodiment model. Settings not listed are identical to Table 6.

DataJoint training on DROID, ROBO MIND, and BRIDGE. Datasets are sampled with equal probability, and clips are sampled uniformly within each dataset.
Hardware32 H100-80GB GPUs across four nodes, with one FSDP rank per GPU.
Batch size8 examples per GPU, giving a global batch size of 256.
Learning ratesBackbone: 1×10−51 \times10^{-5}; adapter: 4×10−54 \times10^{-5}.
ScheduleCosine decay with 250 warm-up steps and a minimum learning-rate ratio of 0.05.
Weight EMADecay of 0.995, updated every 10 steps after a 250-step warm-up.

Table 8. Training configuration for the four-node multi-embodiment render–action model. Settings not listed are identical to Table 6.

Evaluation Metrics

We evaluate generated videos against the reference videos at 240×432240 \times432 resolution. All metrics exclude the conditioning frame and are computed over the sixteen predicted frames. Unless otherwise specified, scores are first averaged over time within each clip and then averaged across clips with equal weight. Undefined scores are omitted.

Whole-frame visual fidelity. We report peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [64], and learned perceptual image patch similarity (LPIPS). PSNR measures pixel-level reconstruction fidelity, SSIM measures local agreement in luminance, contrast, and structure, and LPIPS measures perceptual discrepancy using pretrained AlexNet features. Each metric is computed independently for every predicted frame and averaged over the prediction horizon. Higher PSNR and SSIM are better, while lower LPIPS is better.

DataJoint training on the eight datasets of Table 1 (803,602 training clips, 593 hours of training video, Table 4). Datasets are sampled in proportion to their training clips, and clips uniformly within each dataset.
ActionsEvery action vector is zero-padded to 159 dimensions and passed to one shared action encoder; there is no embodiment label or embodiment-specific parameter.
Hardware32 GPUs across four nodes, one FSDP rank per GPU: H100-80GB.

Table 9. Training configuration for DOPPEL-8. Settings not listed are identical to Table 8.

Region-specific reconstruction. To separate robot reconstruction from changes in the surrounding scene, we compute PSNR over the robot and moving non-robot objects. We initialize the robot mask from the first rendered frame and obtain non-robot object masks using SAM 2 on the first reference frame. These masks are propagated through the reference video to define the evaluation regions.

A non-robot object is classified as moving when its maximum centroid displacement, normalized by the square root of its initial mask area, is at least 0.25. We compute PSNRarm\mathrm{PSNR}_{\mathrm{arm}} over the reference robot mask and PSNRobj\mathrm{PSNR}_{\mathrm{obj}} over moving non-robot object masks. For object metrics, scores are first averaged over time for each object and then averaged equally across objects.

Structural consistency. Structural consistency score (SCS) measures agreement between predicted and reference region trajectories. Starting from identical first-frame masks, SAM 2 independently propagates each region through the generated and reference videos. We compute the intersection-over-union between corresponding masks at each predicted frame and average over time and regions. Higher SCS indicates greater consistency in region position and shape.

In the whole-frame results table, SCS is averaged over all non-robot objects. In the region-specific results, SCSarm\mathrm{SCS}_{\mathrm{arm}} evaluates the robot and SCSobj\mathrm{SCS}_{\mathrm{obj}} evaluates moving non-robot objects. The training-budget comparisons use SCS averaged over all tracked non-robot objects, as in the Table 2.

DatasetSplitDoppel-3Doppel-8Δ\Delta PSNRBetterΔ\Delta armΔ\Delta moving obj.
DroidVal.24.3024.82+0.52 ±\pm 0.0392%+0.70+0.69
OOD21.2521.46+0.21 ±\pm 0.0373%+0.34+0.13
BridgeVal.26.5526.54−0.01 ±\pm 0.0657%+0.09+0.12
OOD22.4722.24−0.24 ±\pm 0.0538%−0.01−0.08
RoboMindVal.31.0731.30+0.24 ±\pm 0.0475%+0.42+0.14
OOD30.4430.66+0.22 ±\pm 0.0472%+0.34+0.16

Table 10. Doppel-3 versus Doppel-8, paired on the same 200 clips per set (final checkpoints). Δ\Delta is Doppel-8 minus Doppel-3 with its standard error; Better is the share of clips where Doppel-8 has higher PSNR; arm and moving-object Δ\Delta use the region metrics of Appendix § 9.8.

Camera extrinsics dataset

We release the camera extrinsics and intrinsics we recover for 45,345 demonstrations: Bridge 14,824, RoboMind 17,588, and ABC 12,933. The other five datasets ship their own cameras, which we use and do not re-release; for GigaHands we refine the released cameras per recording session. Droid extrinsics and ZED intrinsics are used as provided by Khazatsky et al. [54]. RoboMind and Bridge ship no extrinsics, so we treat each camera as fixed within a recording session and recover one pose per session with the procedure of § 4.1: 49 poses for RoboMind and 56 for Bridge. Intrinsics are the RealSense D435i values for RoboMind [56] and MoGe-estimated [76] focal lengths for Bridge. We show sample camera-calibration overlays in Figure 20. ABC’s top-down cameras also ship without extrinsics and move between recordings, so we calibrate each ABC video separately by fitting the camera pose to segmentation masks of the two YAM arms and to photometric agreement with the rendering. We fit 13,518 videos, of which 12,933 (95.7%) pass our alignment check.

Results

Additional Qualitative Results

We provide additional qualitative results across all our experiments. Please see A.1 for the complete set of results. We also include an example of trajectory generalization, comparing Doppel with the baseline in Figure 21.

From three to eight embodiments

Table 10 compares Doppel-3 with Doppel-8 on the same clips. Both models see a similar amount of each robot dataset (Table 2); Doppel-8 additionally trains on the five other datasets. Table 12 repeats the comparison of Table 2 at matched training exposure, with every model at its saved checkpoint closest to 51 passes over each robot dataset. Table 11 follows the same comparison over training.

DatasetSplitDoppel-3 passesΔ\Delta PSNR (Doppel-8 −- Doppel-3), ≈\approx30 passes51 passesfinal
DroidVal.24.8+0.70+0.45+0.52
OOD24.8+0.51+0.17+0.21
BridgeVal.33.4+0.25+0.13−0.01-0.01
OOD33.4−0.10-0.10−0.05-0.05−0.24-0.24
RoboMindVal.27.9+0.52+0.21+0.24
OOD27.9+0.25+0.13+0.22

Table 11. PSNR of Doppel-8 minus Doppel-3, on the same 200 clips per set, at three points in training.

Numerical results for the training curves

We report the numerical values underlying both training-curve comparisons. Each entry averages 200 held-out clips for the corresponding dataset and split. PSNR excludes the conditioning frame and averages over the 16 predicted frames before averaging across clips; SCS averages all tracked non-robot objects. All values are reported to three decimal places. Training draws include repeated examples. The two comparisons use different budget conventions, explicitly indicated in the tables. The uncertainty bars in the figures use 5,000 source-episode bootstrap resamples; the tables below report the corresponding point estimates.

Single-dataset experts versus Doppel-3. Tables Table 14 and Table 15 give the four-checkpoint curves for the separately trained action-only experts and Doppel-3 (batch 256). The training budgets are measured per dataset. The expert rows combine the corresponding single-dataset model for each dataset.

MethodDraws/dataset (M)Droid ValDroid OODBridge ValBridge OODRoboMind ValRoboMind OOD
Action-only expert∼0.5\sim0.519.19418.05822.13621.11925.19925.848
Doppel-3∼0.5\sim0.520.81519.50524.25322.31428.56028.634
Action-only expert∼1.0\sim1.019.58318.11023.37421.27926.44927.016
Doppel-3∼1.0\sim1.022.76820.54524.93522.90729.38629.589
Action-only expert∼2.0\sim2.020.96318.98224.80520.55227.50128.039
Doppel-3∼2.0\sim2.023.66420.96525.80922.47230.38730.172
Action-only expert∼3.4\sim3.422.22819.76725.86920.55529.629†29.629†
Doppel-3∼3.4\sim3.424.22021.14726.45822.43830.96430.436

Table 14. PSNR at every checkpoint in the expert-versus-shared-model scaling comparison. Action-only experts are trained separately on each dataset; Doppel is trained jointly on all three.

MethodDraws/dataset (M)DROID ValDROID OODBRIDGE ValBRIDGE OODROBOMIND ValROBOMIND OOD
Action-only expert∼0.5\sim0.50.8680.8260.9010.8750.8970.899
DOPPEL-3∼0.5\sim0.50.9140.8760.9270.9170.9270.926
Action-only expert∼1.0\sim1.00.8770.8260.9160.8830.9110.909
DOPPEL-3∼1.0\sim1.00.9260.8880.9360.9150.9330.931
Action-only expert∼2.0\sim2.00.9050.8520.9330.8680.9190.919
DOPPEL-3∼2.0\sim2.00.9330.8960.9420.9070.9370.932
Action-only expert∼3.4\sim3.40.9210.8690.9430.8630.9370.929
DOPPEL-3∼3.4\sim3.40.9400.8970.9480.9040.9400.937

Table 15. SCS at every checkpoint in the expert-versus-shared-model scaling comparison. Action-only experts are trained separately on each dataset; DOPPEL is trained jointly on all three.

Shared action-only versus Doppel-3 (batch 64). Tables Table 16 and Table 17 give the three-checkpoint curves for the matched batch-64 shared models. Here, training budgets count total draws across all three datasets. The two models use the same training exposure and optimization schedule at each checkpoint. This Doppel-3 (batch 64) run differs from the batch-256 Doppel-3 run in the expert comparison above.

MethodTotal draws (M)Droid ValDroid OODBridge ValBridge OODRoboMind ValRoboMind OOD
Action-only (multi)∼1.2\sim1.219.12618.02921.70320.76224.52024.958
Doppel-3 (batch 64)∼1.2\sim1.222.07020.25124.09822.78128.50429.082
Action-only (multi)∼2.4\sim2.420.38118.76223.19621.72226.57526.965
Doppel-3 (batch 64)∼2.4\sim2.422.83220.69724.96823.15629.38229.670
Action-only (multi)∼3.4\sim3.420.95919.03624.07021.87327.56428.200
Doppel-3 (batch 64)∼3.4\sim3.423.18820.86225.43122.72629.93729.818

Table 16. PSNR at every checkpoint in the matched shared-model scaling comparison. Both models are trained jointly on all three datasets.

MethodTotal draws (M)Droid ValDroid OODBridge ValBridge OODRoboMind ValRoboMind OOD
Action-only (multi)∼1.2\sim1.20.8640.8240.8920.8590.8860.886
Doppel-3 (batch 64)∼1.2\sim1.20.9190.8830.9270.9140.9270.932
Action-only (multi)∼2.4\sim2.40.8930.8390.9140.8860.9100.910
Doppel-3 (batch 64)∼2.4\sim2.40.9270.8920.9360.9190.9300.934
Action-only (multi)∼3.4\sim3.40.9010.8460.9250.8900.9200.916
Doppel-3 (batch 64)∼3.4\sim3.40.9300.8960.9400.9120.9330.930

Table 17. SCS at every checkpoint in the matched shared-model scaling comparison. Both models are trained jointly on all three datasets.

DatasetMethodInputTrained onValidation: unseen trajectoriesOOD test
PSNRarm_{\mathrm{arm}}PSNRobj_{\mathrm{obj}}SCSarm_{\mathrm{arm}}SCSobj_{\mathrm{obj}}PSNRarm_{\mathrm{arm}}PSNRobj_{\mathrm{obj}}SCSarm_{\mathrm{arm}}SCSobj_{\mathrm{obj}}
DroidAction-onlyActionsThis dataset18.7427.260.8080.83917.6424.670.7120.746
OSCAR (fine-tuned)SkeletonThis dataset16.5422.920.7780.80116.0522.080.7120.728
DoppelRender + actionsThis dataset20.88<u>28.63</u>0.8430.867<u>19.33</u>26.500.7730.805
Doppel-3Render + actions3 robot datasets<u>20.89</u>28.56<u>0.846</u><u>0.869</u>19.2326.41<u>0.777</u><u>0.800</u>
Doppel-8Render + actionsAll 8 datasets21.5929.250.8530.87619.5726.540.7950.805
BridgeAction-onlyActionsThis dataset21.1527.650.7760.89417.7421.500.5910.778
OSCAR (fine-tuned)SkeletonThis dataset18.2522.780.7020.84017.5020.620.5910.788
DoppelRender + actionsThis dataset<u>21.52</u>28.040.7870.90118.7223.64<u>0.689</u><u>0.838</u>
Doppel-3Render + actions3 robot datasets21.4927.84<u>0.791</u><u>0.900</u>19.0524.930.6920.843
Doppel-8Render + actionsAll 8 datasets21.57<u>27.96</u>0.7930.901<u>19.04</u><u>24.85</u>0.6850.831
RoboMindAction-onlyActionsThis dataset25.6429.160.9090.86625.0328.700.8960.826
OSCAR (fine-tuned)SkeletonThis dataset22.9624.960.8820.80922.8925.340.8760.798
DoppelRender + actionsThis dataset26.71<u>29.53</u>0.916<u>0.869</u>25.1228.780.9020.826
Doppel-3Render + actions3 robot datasets<u>26.81</u>29.46<u>0.917</u><u>0.869</u><u>25.30</u><u>28.91</u><u>0.903</u>0.844
Doppel-8Render + actionsAll 8 datasets27.2329.600.9190.87325.6429.070.905<u>0.841</u>

Table 13. Prediction quality by image region on the three robot datasets, for the same models, checkpoints and clips as Table 2. Arm is the robot; Obj. is the objects that move during the clip (non-robot regions segmented in the first frame with SAM 2 and tracked through the clip). PSNR is computed inside each region and averaged over the 16 predicted frames; SCS is structural consistency of the same regions. Best in bold, second-best underlined; ties at displayed precision share emphasis. Higher is better for all metrics. 200 clips per split, the same clips for every method.

Figure 16. Representative examples from T-Rex (humanoid with dexterous hands) with corresponding camera-aligned renderings.

Figure 17. Representative examples from Ego-Exo4D (human body) with corresponding camera-aligned renderings.

Figure 18. Representative examples from GigaHands (human hands) with corresponding camera-aligned renderings.

References

  1. [1]Daniel M. Wolpert, Zoubin Ghahramani, and Michael I. Jordan. An internal model for sensorimotor integration. Science, 269(5232):1880–1882, 1995. doi: 10.1126/science.7569931.DOI
  2. [2]Anurag Bagchi, Zhipeng Bao, Homanga Bharadhwaj, Yu-Xiong Wang, Pavel Tokmakov, and Martial Hebert. Walk through paintings: Egocentric world models from Internet priors. In European Conference on Computer Vision (ECCV), pages 472–489, 2026. doi: 10.1007/978-3-032-37098-3_26.DOI
  3. [3]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, pages 2450–2462, 2018.arxiv.org/abs/1809.01999
  4. [4]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640(8059):647–653, 2025. doi: 10.1038/s41586-025-08744-2.DOI
  5. [5]Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Day-Dreamer: World models for physical robot learning. In Conference on Robot Learning (CoRL), volume 205, pages 2226–2240, 2022.
  6. [6]Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In IEEE International Conference on Robotics and Automation (ICRA), pages 2786–2793, 2017. doi: 10.1109/ICRA.2017.7989324.DOI
  7. [7]Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual Foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018.arxiv.org/abs/1812.00568
  8. [8]Yanjiang Guo, Tony Lee, Lucy Xiaoyang Shi, Jianyu Chen, Percy Liang, and Chelsea Finn. VLAW: Iterative co-improvement of vision-language-action policy and world model. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 of Proceedings of Machine Learning Research, pages 37998–38011, 2026.arxiv.org/abs/2602.12063
  9. [9]Junwon Seo, Sushant Veer, Ran Tian, Wenhao Ding, Apoorva Sharma, Karen Leung, Edward Schmerling, Marco Pavone, and Andrea Bajcsy. StressDream: Steering video world models for robust policy evaluation and improvement. In Conference on Robot Learning (CoRL), 2026.arxiv.org/abs/2606.00267
  10. [10]Sherry Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Thompson, Leslie Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In International Conference on Learning Representations (ICLR), pages 45210–45234, 2024.arxiv.org/abs/2310.06114
  11. [11]Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative interactive environments. In International Conference on Machine Learning (ICML), volume 235, pages 4603–4623, 2024.arxiv.org/abs/2402.15391
  12. [12]NVIDIA, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiajiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixé, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung-Yi Lin, Huan Ling, Ming-Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, Despoina Paschalidou, Zeeshan Patel, Lindsey Pavano, Morteza Ramezani, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang, Heng Wang, Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, and Artur Zolkowski. Cosmos world foundation model platform for physical AI. arXiv preprint arXiv:2501.03575, 2025.
  13. [13]Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.arxiv.org/abs/2311.15127
  14. [14]Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenying Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. CogVideoX: Text-to-video diffusion models with an expert transformer. In International Conference on Learning Representations (ICLR), pages 83048–83077, 2025.arxiv.org/abs/2408.06072
  15. [15]Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yuton Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.arxiv.org/abs/2503.20314
  16. [16]Siqiao Huang, Jialong Wu, Qixing Zhou, Shangchen Miao, and Mingsheng Long. Vid2World: Crafting video diffusion models to interactive world models. In International Conference on Learning Representations (ICLR), pages 10763–10806, 2026.arxiv.org/abs/2505.14357
  17. [17]Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 9156–9172, 2023.arxiv.org/abs/2302.00111
  18. [18]Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjork, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loïc Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xiaohui Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming-Yu Liu, Luke Zettlemoyer, Dieter Fox, Jan Kautz, Scott Reed, Yuke Zhu, and Linxi Fan. DreamGen: Unlocking generalization in robot learning through video world models. In Conference on Robot Learning (CoRL), volume 305, pages 5170–5194, 2025.arxiv.org/abs/2505.12705
  19. [19]Julian Quevedo, Ansh Kumar Sharma, Yixiang Sun, Varad Suryavanshi, Percy Liang, and Sherry Yang. WorldGym: World model as an environment for policy evaluation. In International Conference on Learning Representations (ICLR), pages 78932–78957, 2026.arxiv.org/abs/2506.00613
  20. [20]Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-World: A controllable generative world model for robot manipulation. In International Conference on Learning Representations (ICLR), pages 6121–6138, 2026.arxiv.org/abs/2510.10125
  21. [21]Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. IRASim: A fine-grained world model for robot manipulation. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 9834–9844, 2025. doi: 10.1109/ICCV51701.2025.00917.DOI
  22. [22]Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In Robotics: Science and Systems (RSS), 2025. doi: 10.15607/RSS.2025.XXI.015.DOI
  23. [23]Zhuoyuan Wu and Jun Gao. OSCAR: Omni-embodiment action-conditioned world model for robotics. arXiv preprint arXiv:2606.04463, 2026.arxiv.org/abs/2606.04463
  24. [24]Wenlong Huang, Yu-Wei Chao, Arsalan Mousavian, Ming-Yu Liu, Dieter Fox, Kaichun Mo, and Li Fei-Fei. PointWorld: Scaling 3D world models for in-the-wild robotic manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20765–20779, 2026.arxiv.org/abs/2601.03782
  25. [25]Xiuyu Yang, Bohan Li, Shaocong Xu, Nan Wang, Chongjie Ye, Zhaoxi Chen, Minghan Qin, Yikang Ding, Zheng Zhu, Xin Jin, Hang Zhao, and Hao Zhao. ORV: 4D occupancy-centric robot video generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1053–1066, 2026.arxiv.org/abs/2506.03079
  26. [26]Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. TesserAct: Learning 4D embodied world models. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 5337–5347, 2025. doi: 10.1109/ICCV51701.2025.00507.DOI
  27. [27]Yixiang Chen, Peiyan Li, Jiabing Yang, Keji He, Xiangnan Wu, Yuan Xu, Kai Wang, Jing Liu, Nianfeng Liu, Yan Huang, and Liang Wang. BridgeV2W: Bridging video generation models to embodied world models via embodiment masks. arXiv preprint arXiv:2602.03793, 2026.arxiv.org/abs/2602.03793
  28. [28]Yuxin Jiang, Shengcong Chen, Siyuan Huang, Liliang Chen, Pengfei Zhou, Yue Liao, Xindong He, Chiming Liu, Hongsheng Li, Maoqing Yao, and Guanghui Ren. EnerVerse-AC: Envisioning embodied environments with action condition. arXiv preprint arXiv:2505.09723, 2025.arxiv.org/abs/2505.09723
  29. [29]Mutian Xu, Tianbao Zhang, Tianqi Liu, Zhaoxi Chen, Xiaoguang Han, and Ziwei Liu. Kinema4D: Kinematic 4D world modeling for spatiotemporal embodied simulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 4495–4500, 2026.arxiv.org/abs/2603.16669
  30. [30]Byungjun Kim, Taeksoo Kim, Junyoung Lee, and Hanbyul Joo. Dexterous world models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 29663–29673, 2026.arxiv.org/abs/2512.17907
  31. [31]Junjie Ye, Rong Xue, Basile Van Hoorick, Pavel Tokmakov, Muhammad Zubair Irshad, Yue Wang, and Vitor Guizilini. AnchorDream: Repurposing video diffusion for embodiment-aware robot data synthesis. In IEEE International Conference on Robotics and Automation (ICRA), pages 6137–6144, 2026. doi: 10.1109/ICRA57385.2026.11697505.
  32. [32]Junjie Ye, Rong Xue, Basile Van Hoorick, Runhao Li, Harshitsha Rajaprakash, Pavel Tokmakov, Muhammad Zubair Irshad, Vitor Guizilini, and Yue Wang. RoboDream: Compositional world models for scalable robot data synthesis. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026.arxiv.org/abs/2606.02577
  33. [33]Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyeok Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi Jim Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martín-Martín, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhou Xu, Zichen Jeff Cui, Zichen Zhang, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903, 2024. doi: 10.1109/ICRA57147.2024.10611477.
  34. [34]Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Robotics: Science and Systems (RSS), 2024. doi: 10.15607/RSS.2024.XX.090.DOI
  35. [35]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning (CoRL), volume 270, pages 2679–2713, 2024.arxiv.org/abs/2406.09246
  36. [36]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π₀: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025. doi: 10.15607/RSS.2025.XXI.010.DOI
  37. [37]Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 124420–124450, 2024.arxiv.org/abs/2409.20537
  38. [38]Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation. In Conference on Robot Learning (CoRL), volume 270, pages 496–512, 2024.arxiv.org/abs/2408.11812
  39. [39]Lawrence Yunliang Chen, Kush Hari, Karthik Dharmarajan, Chenfeng Xu, Quan Vuong, and Ken Goldberg. Mirage: Cross-embodiment zero-shot policy transfer with cross-painting. In Robotics: Science and Systems (RSS), 2024. doi: 10.15607/RSS.2024.XX.069.DOI
  40. [40]Lawrence Yunliang Chen, Chenfeng Xu, Karthik Dharmarajan, Muhammad Zubair Irshad, Richard Cheng, Kurt Keutzer, Masayoshi Tomizuka, Quan Vuong, and Ken Goldberg. RoVi-Aug: Robot and viewpoint augmentation for cross-embodiment robot learning. In Conference on Robot Learning (CoRL), volume 270, pages 209–233, 2024.arxiv.org/abs/2409.03403
  41. [41]Marion Lepert, Jiaying Fang, and Jeannette Bohg. Phantom: Training robots without robots using only human videos. In Conference on Robot Learning (CoRL), volume 305, pages 4545–4565, 2025.arxiv.org/abs/2503.00779
  42. [42]Marion Lepert, Ria Doshi, and Jeannette Bohg. SHADOW: Leveraging segmentation masks for cross-embodiment policy transfer. In Conference on Robot Learning (CoRL), volume 270, pages 3536–3550, 2024.arxiv.org/abs/2503.00774
  43. [43]Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, and Ted Xiao. RT-Trajectory: Robotic task generalization via hindsight trajectory sketches. In International Conference on Learning Representations (ICLR), pages 44918–44941, 2024.arxiv.org/abs/2311.01977
  44. [44]Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. In Conference on Robot Learning (CoRL), volume 270, pages 2475–2499, 2024.arxiv.org/abs/2407.15208
  45. [45]Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. DragAnything: Motion control for anything using entity representation. In European Conference on Computer Vision (ECCV), pages 331–348, 2024. doi: 10.1007/978-3-031-72670-5_19.DOI
  46. [46]Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Yusuf Aytar, Michael Rubinstein, Chen Sun, Oliver Wang, Andrew Owens, and Deqing Sun. Motion prompting: Controlling video generation with motion trajectories. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–12, 2025. doi: 10.1109/CVPR52734.2025.00010.DOI
  47. [47]Yuang Wang, Chao Wen, Haoyu Guo, Sida Peng, Minghan Qin, Hujun Bao, Xiaowei Zhou, and Ruizhen Hu. Precise action-to-video generation through visual action prompts. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 12713–12724, 2025. doi: 10.1109/ICCV51701.2025.01181.DOI
  48. [48]Zihao He, Bo Ai, Tongzhou Mu, Yulin Liu, Weikang Wan, Jiawei Fu, Yilun Du, Henrik I. Christensen, and Hao Su. Scaling cross-embodiment world models for dexterous manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026.arxiv.org/abs/2511.01177
  49. [49]Ruoshi Liu, Alper Canberk, Shuran Song, and Carl Vondrick. Differentiable robot rendering. In Conference on Robot Learning (CoRL), volume 270, pages 117–129, 2024.arxiv.org/abs/2410.13851
  50. [50]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.02747
  51. [51]Yash Jangir, Yidi Zhang, Pang-Chi Lo, Kashu Yamazaki, Chenyu Zhang, Kuan-Hsun Tu, Tsung-Wei Ke, Lei Ke, Yonatan Bisk, and Katerina Fragkiadaki. RobotArena ∞: Scalable robot benchmarking via real-to-sim translation. In International Conference on Learning Representations (ICLR), pages 25017–25043, 2026.arxiv.org/abs/2510.23571
  52. [52]Maxime Ouqab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
  53. [53]Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV), pages 402–419, 2020. doi: 10.1007/978-3-030-58536-5_24.DOI
  54. [54]Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O’Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems (RSS), 2024. doi: 10.15607/RSS.2024.XX.120.DOI
  55. [55]Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. BridgeData V2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), volume 229, pages 1723–1736, 2023.arxiv.org/abs/2308.12952
  56. [56]Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, Shichao Fan, Xinhua Wang, Fei Liao, Zhen Zhao, Guangyu Li, Zhao Jin, Lecheng Wang, Jilei Mao, Ning Liu, Pei Ren, Qiang Zhang, Yaoxu Lyu, Mengzhen Liu, Jingyang He, Yulin Luo, Zeyu Gao, Chenxuan Li, Chenyang Gu, Yankai Fu, Di Wu, Xingyu Wang, Sixiang Chen, Zhenyu Wang, Pengju An, Siyuan Qian, Shanghang Zhang, and Jian Tang. RoboMIND: Benchmark on multi-embodiment intelligence normative data for robot manipulation. In Robotics: Science and Systems (RSS), 2025. doi: 10.15607/RSS.2025.XXI.152.DOI
  57. [57]Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, Xi Chen, Rocky Duan, Phillip Isola, Jitendra Malik, Fred Shentu, Guanya Shi, Philipp Wu, and Angjoo Kanazawa. Scalable behavior cloning with open data, training, and evaluation. In Conference on Robot Learning (CoRL), 2026.arxiv.org/abs/2606.27375
  58. [58]Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, Ryan Punamiya, Mengda Xu, Yuqi Xie, Yunfan Jiang, Letian Fu, Konstantinos Kallidromitis, Matteo Gioia, Junyi Zhang, Jiaxin Ge, Haiwen Feng, Fabio Galasso, Wei Zhan, David M. Chan, Yutong Bai, Roei Herzig, Jiahui Lei, Li Fei-Fei, Ken Goldberg, Jitendra Malik, Pieter Abbeel, Yuke Zhu, Danfei Xu, Linxi Fan, and Trevor Darrell. T-Rex: Tactile-reactive dexterous manipulation. In Conference on Robot Learning (CoRL), 2026.arxiv.org/abs/2606.17055
  59. [59]Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. GigaHands: A massive annotated dataset of bimanual hand activities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17461–17474, 2025.arxiv.org/abs/2412.04244
  60. [60]Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. OakInk2: A dataset of bimanual hands-object manipulation in complex task completion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 445–456, 2024.arxiv.org/abs/2403.19417
  61. [61]Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Tri-antafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abraham Gebreselassie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Zhecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Gedas Bertasius, Dima Damen, Jakob Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C.V. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, and Michael Wray. Ego-Exo4D: Understanding skilled human activity from first- and third-person perspectives. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19383–19400, 2024.
  62. [62]Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6):245:1–245:17, 2017.DOI
  63. [63]Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10967–10977, 2019.arxiv.org/abs/1904.05866
  64. [64]Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
  65. [65]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586–595, 2018. doi: 10.1109/CVPR.2018.00068.DOI
  66. [66]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In International Conference on Learning Representations (ICLR), pages 28085–28128, 2025.arxiv.org/abs/2408.00714
  67. [67]Abhiram Maddukuri and Georgios Pavlakos. Ego-Exo4D human meshes dataset: 4D human motion reconstruction for ego-exo captures. arXiv preprint arXiv:2609.30187, 2026.
  68. [68]NVIDIA, Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, Prithvijit Chattopadhyay, Mike Chen, Yongxin Chen, Yu Chen, Shuai Cheng, Yin Cui, Jenna Diamond, Yifan Ding, Jiaojiao Fan, Linxi Fan, Liang Feng, Francesco Ferroni, Sanja Fidler, Xiao Fu, Ruiyuan Gao, Yunhao Ge, Jinwei Gu, Aryaman Gupta, Siddharth Gururani, Imad El Hanafi, Ali Hassani, Zekun Hao, Jacob Huffman, Joel Jang, Pooya Jannaty, Jan Kautz, Grace Lam, Xuan Li, Zhaoshuo Li, Maosheng Liao, Chen-Hsuan Lin, Tsung-Yi Lin, Yen-Chen Lin, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Seungjun Nah, Yashraj Narang, Abhijeet Panaskar, Lindsey Pavao, Trung Pham, Morteza Ramezanali, Fitsum Reda, Scott Reed, Xuanchi Ren, Haonan Shao, Yue Shen, Stella Shi, Shuran Song, Bartosz Stefania k, Shangkun Sun, Shitao Tang, Sameena Tasmeen, Lyne Tch apmi, Wei-Cheng Tseng, Jibin Varghese, Andrew Z. Wang, Hao Wang, Haoxiang Wang, Heng Wang, Ting-Chun Wang, Fangyin Wei, Jiashu Xu, Dinghao Yang, Xiaodong Yang, Haotian Ye, Seonghyeon Ye, Xiaohui Zeng, Jing Zhang, Qinsheng Zhang, Kaiwen Zheng, Andrew Zhu, and Yuke Zhu. World simulation with video foundation models for physical AI. arXiv preprint arXiv:2511.00062, 2025.
  69. [69]Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. RH20T: A comprehensive robotic dataset for learning diverse skills in one-shot. In IEEE International Conference on Robotics and Automation (ICRA), pages 653–660, 2024.arxiv.org/abs/2307.00595
  70. [70]Yang Tian, Yuyin Yang, Yiman Xie, Zetao Cai, Xu Shi, Ning Gao, Hangxu Liu, Xuekun Jiang, Zherui Qiu, Feng Yuan, Yaping Li, Ping Wang, Junhao Cai, Jia Zeng, Hao Dong, and Jiangmiao Pang. InternData-A1: Pioneering high-fidelity synthetic data for pre-training generalist policy. arXiv preprint arXiv:2511.16651, 2025.arxiv.org/abs/2511.16651
  71. [71]AgiBot World Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xi uqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Ch iming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Cheng en Xie, Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang, Shukai Yang, Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu. AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3549–3556, 2025.arxiv.org/abs/2503.06669
  72. [72]Ryosuke Takanami, Petr Khrapchenkov, Shu Morikuni, Jumpei Arima, Yuta Takaba, Shunsuke Maeda, Takuya Okubo, Genki Sano, Satoshi Sekioka, Aoi Kadoya, Motonari Kambara, Naoya Nishiura, Haruto Suzuki, Takanori Yoshimoto, Koya Sakamoto, Shinnosuke Ono, Hu Yang, Daichi Yashima, Aoi Horo, Tomohiro Motoda, Kensuke Chiyoma, Hiroshi Ito, Koki Fukuda, Akihito Goto, Kazumi Morinaga, Yuya Ikeda, Riko Kawada, Masaki Yoshikawa, Norio Kosuge, Yuki Noguchi, Kei Ota, Tatsuya Matsushima, Yusuke Iwasawa, Yutaka Matsuo, and Tetsuya Ogata. AIRoA MoMa dataset: A large-scale hierarchical dataset for mobile manipulation. arXiv preprint arXiv:2509.25032, 2025.
  73. [73]Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivaprapuru, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations (ICLR), pages 4218–4237, 2026.arxiv.org/abs/2505.11709
  74. [74]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal of Computer Vision, 130(1):33–55, 2022.DOI
  75. [75]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
  76. [76]Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. MoGe: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5261–5271, 2025. doi: 10.1109/CVPR52734.2025.00496.DOI

Paper details

vlnc:W_jssbpemu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents