DreamX: Video Diffusion Model is a Universal Latent Action Learner

Project page: https://cvlab-kaist.github.io/DreamX

Introduction

Generalist robot policies, including vision-language-action (VLA) models [19, 2, 15] and world action models (WAMs) [20, 38], benefit from diverse robot demonstrations, but collecting such data at scale remains costly. Human egocentric videos offer a promising alternative as a scalable source of diverse manipulation behaviors, and recent works leverage these videos for policy learning [21, 16]. However, action supervision remains tied to the embodiment that produces it. Human videos lack robot action labels, and even robot demonstrations use different control spaces across embodiments. The same behavior can therefore correspond to different low-level actions, making action labels difficult to share directly across embodiments. Overcoming this mismatch calls for a common action representation that preserves shared motion and effects across embodiments.

Latent action models (LAMs) offer a promising route toward learning such a representation from video. Although humans and robots have different bodies and control spaces, their actions can all be observed as changes in a visual scene. LAMs typically use dedicated encoders to compress these changes into compact representations, called latent actions [37, 8, 5], where these encoders are trained using a decoder that reconstructs future frames solely from the current frame and latent actions. However, reconstruction accuracy alone does not ensure that latent actions isolate the underlying action. The learned representations can also encode domain-specific visual information, such as viewpoint and scene appearance, whenever it helps reconstruction [24]. Capturing action-relevant structure shared across embodiments while separating it from domain-specific visual factors therefore remains a challenge.

Pretrained video diffusion models provide a promising foundation for addressing this challenge. First, through large-scale video generation, they learn rich spatiotemporal representations that capture motion, correspondence, geometry, and semantic structure [22, 34, 30]. Such representations encode how scenes, objects, and agents evolve over time, making them a natural source of action-relevant dynamics. Second, these models possess a generative prior that enables future-frame synthesis conditioned on an initial frame and an additional signal. With scene appearance supplied by the initial frame, additional conditioning can guide how the scene evolves. Together, these priors motivate extracting latent actions directly from pretrained spatiotemporal representations and supervising their learning with a pretrained generative decoder.

We introduce DreamX, a framework for learning universal latent actions across embodiments using a pretrained video diffusion model. We pass learnable query tokens alongside the video tokens through the model’s spatiotemporal attention, allowing the queries to gather action-relevant information across frames. The queries condense frame-to-frame changes represented by 1,200 frame tokens into just 4 latent-action tokens, yielding a $300 \times$ reduction in token count. These latent actions replace text conditioning in the same model to reconstruct future frames from the initial frame. With scene appearance supplied by the initial frame and visual detail by the pretrained generative prior, the reconstruction objective encourages latent actions to capture the motion needed to explain the observed future.

Trained on heterogeneous human and robot videos, DreamX achieves state-of-the-art performance among latent action models on LARYBench [23] and enables any-to-any video transfer and human-to-robot action transfer without paired training data. As a downstream application, we also show that the learned latent actions can serve as prediction targets for an observation-conditioned robot policy, without requiring future-video generation during execution. Together, these results suggest that DreamX learns a shared, transferable latent action space that supports motion transfer across embodiments and can also serve as a basis for downstream policy learning.

In summary, our contributions are as follows:

• We introduce DreamX, a framework that uses the spatiotemporal and generative priors of a pretrained video diffusion model to learn universal latent actions. The resulting representations achieve state-of-the-art performance among latent action models on LARYBench.

• We demonstrate that the learned latent space supports any-to-any video transfer and human-to-robot action transfer without paired source–target training data. We further show the use of compact latent-action codes as prediction targets for downstream policy learning.

Related Works

Latent action models. Latent action models (LAMs) learn action representations from visual transitions, enabling action-unlabeled videos to supervise downstream control. LAPO [29] and Genie [3] established the paradigm, recovering discrete latent actions through an inverse–forward dynamics bottleneck over adjacent frames. LAPA [37] further scales this recipe for vision-language-action pretraining, and Moto [8] tokenizes visual motion for manipulation. Later work incorporates additional cues: UniVLA [5] grounds latent actions with semantic visual features, villa-X [7] integrates language and robot-state dynamics, and CoMo [36] and DiLA [39] separate motion from visual content. Across these variants, the action space is learned from scratch on a local visual transition, often a single frame pair. Such supervision offers no view of how motion persists beyond one step, so the encoder entangles action with appearance and yields representations tied to their training embodiments.

Emergent capabilities of video generative models. Video generative models acquire spatiotemporal structure by modeling how scenes evolve over time. Their internal representations support correspondence, point tracking, optical flow, and geometric understanding without task-specific supervision [22, 32, 31, 14], and text-conditioned models further exhibit semantic grounding over objects, behaviors, and interactions [34]. Rather than encoding physical laws explicitly, these representations capture motion, geometry, and semantics jointly: which entities are present, where they are, and how they move over time. This is precisely the composition a latent action must express, which is a structured change of a specific entity in a specific place, and it is acquired from web-scale video rather than from a single frame pair.

Video generative priors for latent action learning. Recent work connects these two lines. VPP [13] conditions a manipulation policy on the predictive visual representations inside a fine-tuned video diffusion model, but still learns the policy from action labels. AdaWorld [10] learns latent actions for adaptable video prediction, and VideoWorld2 [28] pairs a pretrained video diffusion model for generation with a separate latent dynamics model. In both, the video model acts as a generative prior supplying pixels or rollouts, while the action representation is produced by an external module trained from scratch on frame-level transitions. Thus, the latent action encoder does not directly read the video diffusion backbone’s internal representations. We instead read latent actions directly from the model’s rich spatiotemporal representation, making maximal use of video pretraining and requiring no per-embodiment action encoder.

Method

Given a video and instruction, DreamX uses learnable queries to extract compact latent actions from a pretrained video diffusion backbone’s spatiotemporal features. Future-frame reconstruction conditioned on these latent actions and the initial frame provides generative supervision (Sec. 3.2). The learned representation enables video and action transfer with a target initial frame or robot state (Sec. 3.3). For control, the DreamX policy predicts compact codes and low-level actions without generating future video (Sec. 3.4).

Preliminaries: Video Diffusion Models

We build on Cosmos-Predict2.5 [1], a pretrained video diffusion transformer. A causal video autoencoder compresses a clip into $N$ latent frames, $V = (V_0, \ldots, V_{N-1})$, where $V_0$ encodes only the initial frame. These frames are patchified into spatial tokens and jointly modeled through spatiotemporal attention. Through video pretraining, the model learns spatiotemporal priors that capture spatial context and temporal relationships across the clip. Conditioned on an initial frame and an additional signal such as text, collectively denoted by $c$, the model learns future-video generation through flow matching. Given clean video latents $x$ and noise $\epsilon\sim\mathcal{N}(0, I)$, we form $x_{\tau} = (1 - \tau)x + \tau\epsilon$, with $\tau\in[0, 1]$, and minimize

where $v_{\theta}$ predicts the flow velocity.

By optimizing this objective over large-scale video data, the model acquires a generative prior over scene appearance and plausible dynamics. When the initial frame supplies the scene’s appearance, additional conditioning steers how the scene evolves. We exploit this property by using latent actions to steer future-frame reconstruction, whose objective supervises what the latent actions encode.

Learning Universal Latent Actions from Video Diffusion Priors

DreamX adapts a pretrained video diffusion backbone for two complementary roles: encoding observed dynamics into latent actions and supervising them through future-frame reconstruction to encourage separation of motion from visual appearance. Building on Sec. 3.1, we introduce learnable queries that read out the video representations to summarize observed dynamics as latent actions, while the decoder reconstructs the future from these latent actions and the initial frame. Leveraging a video diffusion backbone for both the encoder and decoder allows its generative prior to account for visual appearance, encouraging the latent action tokens to focus on motion.

Latent Action Encoding from Spatiotemporal Dynamics. DreamX uses pretrained video model itself to encode latent actions. As shown in Figure 2(a), we append learnable query tokens $\mathbf{q}$ to the video token sequence and process them jointly through the backbone’s spatiotemporal attention. We assign a group of $K$ query tokens $\mathbf{q}_g$ to each latent frame $V_g$. Once contextualized by the video, their output features form the latent-action group $Q_g$. All $N$ groups are extracted in a single forward,

where $\mathcal{E}_{\phi}(\cdot)$ is the encoder and $\ell$ is the language instruction supplied through the backbone’s original cross-attention. As shown in Figure 2(b), query-to-video attention is frame-causal, so $Q_g$ directly attends to $V_{0:g}$, while query-to-query attention is unrestricted. This lets the queries encode temporally grounded motion from the pretrained dynamics, with query-layout settings in Appendix § 9.§ 2.

Generative Decoding for Action-Centric Latent Learning. Encoding from video alone may retain scene- and embodiment-specific visual cues. We therefore use the same pretrained video backbone as the encoder to supervise latent actions through future-frame reconstruction, conditioned on $V_0$ and $Q_{0:N-1}$ without text. As $V_0$ provides scene appearance and the pretrained generative prior captures plausible evolution, $Q$ is encouraged to encode the motion needed to explain the future, rather than visual cues already available to the decoder.

As shown in Figure 2(a), the decoder $\mathcal{D}_{\theta}(\cdot)$ generates future frames conditioned on $Q_{0:N-1}$, concatenated with the video tokens along the temporal sequence. When action labels are available, it also takes the proprioceptive state $p_0$ and decodes action chunks as an auxiliary task:

Under the decoder attention mask in Figure 2(b), latent-action queries serve solely as conditioning signals and do not attend to any tokens. The video and action streams attend to those queries without attending to each other. Specifically, video frame $V_t$ attends to the query prefix $Q_{0:t}$, while action chunk $A_m$ attends to $Q_{0:m}$.

Training Objective. We train both the encoder and the decoder end-to-end with the standard flow-matching objective applied to video and action generation as Eq. (4), with $\lambda= 0$ for video clips without action labels:

Motion Transfer via Latent Actions

Building upon the rich generative prior from the video diffusion backbone, DreamX learns latent actions that capture motion transferable across embodiments, even though cross-embodiment transfer is not an explicit training objective. This capability emerges from the pretraining described in the previous section and requires no additional training. The auxiliary robot action branch further encourages the latent actions to encode action-relevant cues and further enables motion extracted from a source video to be decoded into continuous actions executable by a target embodiment.

Given a source video, we extract its latent actions using the frozen DreamX encoder, $Q^{\mathrm{src}} = \mathcal{E}_{\phi}(q; V^{\mathrm{src}}, \ell^{\mathrm{src}})$, which captures the motion to be transferred. As shown in Figure 3(a), we reuse the same $Q^{\mathrm{src}}$ in two complementary ways. For video transfer, we combine it with a target initial frame $V_{0}^{\mathrm{tgt}}$ and use the frozen DreamX decoder $\mathcal{D}_{\theta}$ to generate a target video that reproduces the source motion under the target scene and embodiment. For action transfer, we instead combine $Q^{\mathrm{src}}$ with the target robot state $p_{0}^{\mathrm{tgt}}$ and decode the same motion into executable low-level actions,

Thus, the shared latent action realizes the same underlying motion either as a generated video in a new domain or as executable actions on a target robot, without paired source–target datasets.

Policy Learning with Compact Latent Actions

As a downstream application, we examine whether the latent actions learned by DreamX can serve as prediction targets for an observation-conditioned robot policy. Whereas motion transfer uses latent actions extracted from a source demonstration, the policy predicts them from the current observation and instruction. To this end, we repurpose the same pretrained video backbone used by DreamX to predict latent actions in place of future video ([18]). Although $Q$ is compact in token count, each token retains the full backbone width, making direct diffusion high-dimensional ([6]). We therefore compress $Q$ into a low-dimensional code $Z$ first, which serves as the policy’s prediction target.

Latent Action Compression. Following Gao et al. [11], we learn a variational bottleneck that compresses each latent-action token from backbone width $D$ to $d \ll D$ while preserving the token layout, as shown in Figure 3(b). With the DreamX encoder frozen, we train the bottleneck using feature reconstruction and KL regularization. After training, we use its posterior mean to obtain deterministic compact codes $Z$ for policy learning. The formulation and implementation details are provided in Appendix § 9.§ 3.

Policy Learning on Compact Latent Action Space. With the learned bottleneck frozen, the compact codes serve as prediction targets for the DreamX policy $\pi(\cdot)$. As shown in Figure 3(b), the policy generates these codes from the current frame $V_0$ and instruction $\ell$, while a separate DiT action head predicts low-level actions conditioned on the generator’s latent-action features, the same observation and instruction, and the state $p_0$:

Action tokens attend to latent-action features, but not vice versa. This separation allows human videos to supervise latent-code prediction without robot action labels, while robot demonstrations additionally supervise low-level action prediction. We train the policy with

where both terms follow Eq. (1), applied to compact codes $Z$ and low-level actions $A$ respectively, with $\lambda= 0$ for human videos without action labels. Further details are provided in Appendix § 9.§ 4.

Experimental Results

We evaluate whether DreamX learns universal latent actions by examining how well they capture motion (§ 4.1) and whether they can transfer across scenes and embodiments (§ 4.2–§ 4.3). We further assess their utility for learning effective and efficient policies from human and robot demonstrations (§ 4.4), followed by ablations and analysis of the learned representation (§ 4.5–§ 4.6).

Implementation details. We instantiate DreamX with Cosmos-Predict2.5 2B and pretrain on robot demonstrations from MolmoAct2-BimanualYAM [9], Bridge [33], DROID [17], and AgiBot [4], together with human videos from EgoDex [12] and EgoVerse [26]. Videos are processed at 480 × 640 resolution with variable-length clips of up to 29 frames. We assign four query tokens to each VAE latent frame. We adapt the encoder and decoder using separate rank-16 LoRA modules for 78,000 training steps. Further training details are provided in Appendix § 9.§ 2.

Representation Quality

We examine how well DreamX’s latent actions capture motion on LARYBench [23]. Regression asks whether the representation retains enough fine-grained motion to recover end-effector trajectories, and classification whether it carries enough semantic information to distinguish different motions. Table 1 compares DreamX with latent action models that encode either frame pairs or video clips. DreamX achieves the best performance across all benchmarks. Compared with the strongest baseline, DiLA, DreamX reduces regression MSE from 0.38 to 0.17 and increases classification accuracy from 62.25% to 69.67%. These results support leveraging video models as a source of action-relevant representations: reading latent actions directly from their pretrained priors yields representations that capture spatial, temporal, and semantic structure across embodiments. FAE-32, our 32-channel compressed representation (§ 3.4), also outperforms all baselines, indicating that even the compact codes retain substantial motion and semantic information. Further evaluation details are provided in Appendix § 10.§ 4.

Method	CALVIN	VLABench	RoboCOIN	AgiBotWorld-Beta	Avg.	Atomic robot	Composite human	Composite robot	Avg.
Latent actions from 2 frames
LAPA	0.96	0.95	0.96	1.00	0.97	22.27	14.61	23.64	20.17
UniVLA	0.82	0.74	0.94	0.97	0.87	18.62	19.08	18.56	18.75
villa-X	0.86	0.72	0.94	0.97	0.87	15.00	17.80	29.90	20.90
LAPA-MAGVIT2	0.59	0.36	0.80	0.83	0.65	33.12	59.70	29.53	40.78
LAPA-SigLIP2	0.57	0.30	0.86	0.88	0.65	46.83	54.74	29.44	43.67
LAPA-DINOv2	0.55	0.26	0.85	0.86	0.63	58.04	55.86	34.19	49.36
LAPA-DINOv3	0.50	0.25	0.82	0.84	0.60	56.27	64.19	27.04	49.17
Latent actions from video
MoTo [8]	0.43	0.13	0.54	0.68	0.44	41.57	55.11	56.34	51.01
Adaworld [10]	0.64	0.26	0.94	0.94	0.69	31.58	49.8	51.48	44.29
VideoWorld2 [27]	0.72	0.41	0.82	0.85	0.70	14.50	30.24	37.36	27.37
DiLA [39]	0.43	0.09	0.51	0.52	0.38	58.51	65.47	62.76	62.25
DreamX (Ours)	0.15	0.05	0.26	0.22	0.17	68.03	72.72	68.26	69.67
DreamX (Ours) + FAE-32	<u>0.23</u>	<u>0.07</u>	<u>0.31</u>	<u>0.26</u>	<u>0.22</u>	<u>65.90</u>	<u>69.49</u>	<u>66.06</u>	<u>67.15</u>

Video Transfer

We evaluate whether DreamX learns universal latent actions by transferring source motion across scenes and embodiments while preserving the appearance specified by a target initial frame. We examine any-to-any video transfer and quantitatively evaluate zero-shot human-to-robot transfer. Zero-shot evaluation details and additional transfer examples are provided in Appendices B.5 and D.1.

Any-to-any video transfer. Figure 4 shows motion transfer across environments for the same robot, between different robot embodiments, and from humans to robots, with a DAVIS example extending beyond manipulation. DreamX reproduces the source motion while preserving the target scene and embodiment. These results support the universality of the learned latent actions across embodiments and environments, extending to general-domain videos.

Zero-shot human-to-robot transfer. We evaluate zero-shot human-to-robot transfer on HumanAndRobot (Xie et al. [35]), comparing with AdaWorld (Gao et al. [10]) and VideoWorld 2 (Ren et al. [28]), both of which use diffusion-based video decoders. Paired robot recordings serve only as evaluation references; no paired human–robot data is used for training. DreamX outperforms both baselines on every metric in Table 2 and follows the source human motion while preserving the target robot and scene (Figure 5). In contrast,

Method	$\mathrm{FVD}\downarrow$	$\mathrm{LPIPS}\downarrow$	$\mathrm{PSNR}\uparrow$	$\mathrm{SSIM}\uparrow$
AdaWorld	170.19	0.344	22.95	0.846
VideoWorld2	241.92	0.299	22.53	0.881
Ours	57.44	0.179	26.52	0.917

AdaWorld exhibits visual degradation in this out-of-distribution setting, while VideoWorld 2 occasionally introduces a human hand despite its appearance-disentangling design. These results suggest that DreamX better separates transferable motion from appearance, enabling more consistent human-to-robot transfer.

Action Transfer

We evaluate action transfer to examine whether our latent actions form a universal representation space that connects similar motions across embodiments. Following § 3.3, we extract latent actions from a human video and decode them into low-level actions for the target robot. Execution is open-loop without visual observations: only the latent actions specify the motion, while proprioception provides the robot’s initial state. Notably, this transfer is learned without paired human–robot training data, so the two embodiments are connected through the shared latent action space. Figure 6 shows that the robot reproduces the stroke structure of the letter “N,” preserving the demonstrated motion beyond a coarse task outcome. In single-arm pushing, it follows the forward-then-left sequence, retaining the direction change and temporal order. Bimanual pushing further shows corresponding movements of both arms, suggesting that the shared latent representation retains arm-specific motion and transfers it to the corresponding robot arms. Evaluation protocols and additional examples are provided in Appendices § 10.§ 2 and § 12.§ 2, respectively.

Policy Evaluation

We conduct a real-world drawer-closing study to examine whether DreamX’s learned latent actions can support downstream policy learning. Given the instruction close the drawer, the robot must close the upper drawer, the lower drawer, or both sequentially. We compare DreamX with Mimic-Video ([25]), both trained using human videos and robot demonstrations. During rollout, Mimic-Video generates future video, while DreamX generates compact latent actions alongside low-level actions.

Table 3 reports an average subgoal success rate of 96.7% for DreamX, averaging reaching and pushing completion. within each configuration and weighting the three configurations equally. Pushing completion rates are 80%, 100%, and 100% for the upper, lower, and both-open configurations, respectively. We also report an inference latency of 1.47 s per policy call for DreamX. These results demonstrate the feasibility of using the learned latent actions as prediction targets for a robot policy in the evaluated drawer-closing setting. See Appendix § 10.§ 3, C.1, and D.3 for additional evaluation details and results.

Method	Upper	Lower	Both	Avg.↑	Latency↓ (ms)
Mimic-Video	50.0	50.0	50.0	50.0	28,226
DreamX	90.0	100.0	100.0	96.7	1,474

Ablation Study

We examine how auxiliary action supervision affects DreamX’s latent actions, using the same subsets of MolmoAct2, EgoVerse, and EgoDex to limit computational cost. Table 4 shows that action supervision slightly improves regression MSE from 0.24 to 0.23, while classification accuracy slightly decreases from 65.3% to 64.8%. The comparable performance suggests that DreamX captures motion and action semantics from video alone, without relying on action labels.

Supervision	Reg. avg. ↓	Cls. avg. ↑
RGB only	0.24	65.3
RGB + action	0.23	64.8

Analysis

Sensitivity to latent-action perturbations. We examine whether latent actions guide the motion reconstructed by the decoder. Keeping the initial frame and sampling noise fixed, we perturb the latent actions or replace them with zeros. In Figure 7, the original latent actions reconstruct the grasping and lifting motion, whereas perturbation disrupts object manipulation and zeroing largely suppresses motion. The surrounding scene remains largely unchanged, supporting the role of latent actions in specifying motion beyond appearance. Quantitative results are provided in Appendix § 11.§ 3.

What latent actions attend to. To examine where DreamX reads motion information, Figure 8 visualizes query-to-video attention during latent-action encoding of human and robot demonstrations. Attention highlights the hands and grippers involved in object interactions, suggesting that the queries read action-relevant regions across embodiments. Additional visualizations are provided in Appendix § 12.§ 4.

Conclusion

We present DreamX, a framework that leverages pretrained video diffusion priors to learn universal latent actions. Its spatiotemporal representations provide action-relevant dynamics, while its generative prior supervises latent-action learning through future-frame reconstruction. DreamX achieves state-of-the-art performance among latent action models on LARYBench and enables any-to-any video transfer and human-to-robot action transfer without paired training data. Predicting these compact latent actions also supports effective and efficient policy learning from human and robot demonstrations. Together, these results highlight pretrained video models as a foundation for learning reusable action representations across embodiments.

AI Use Statement

We used generative AI tools to assist with manuscript drafting and editing for clarity and readability, code editing during implementation, and discussions during the research process. Research ideas, methodological choices, experimental designs, and interpretation of results were formulated, evaluated, and ultimately determined by the authors. All AI-assisted text and code were reviewed by the authors, and code changes were tested for correctness. We take responsibility for the final content of this work, including text, claims, code, and artifacts produced with the aid of generative AI.

Ethics Statement

All datasets used in this work are publicly available, including the human egocentric and robot demonstration datasets listed in Appendix § 9.§ 1. Human videos may contain identifiable people or surroundings, so any redistributed examples should follow the source datasets’ usage terms and privacy requirements. Video-transfer outputs are synthetic and should be identified as such. Cross-embodiment transfer and robot policies may fail outside the evaluated settings and require safety validation before deployment.

Reproducibility Statement

§ 3 describes the model, training objectives, transfer procedure, and policy. Appendix § 9 documents the data sources and implementation details, while Appendix § 10 specifies the benchmark, transfer, and real-world policy evaluation protocols. Supplementary videos illustrate video-transfer and action-transfer results; policy rollouts are provided in Appendix § 12.§ 3.

This appendix provides implementation details, evaluation protocols, additional results and analysis.

Implementation Details

Training Data

We pretrain DreamX on 2.19k hours of human and robot video spanning six datasets (Figure 9). Human demonstrations contribute 469 hours from EgoDex ([12]) and EgoVerse ([26]), complementing robot demonstrations from Bridge ([33]), DROID ([17]), AgiBot ([4]), and MolmoAct-BimanualYAM ([9]).

DreamX Pretraining

We initialize DreamX from Cosmos-Predict2.5 2B and train at $480 \times640$ resolution for 78,000 steps. Training clips contain 5–29 frames in increments of four, corresponding to 2–8 VAE latent frames. We assign four query tokens to each latent frame, yielding up to 32 query tokens per clip. The encoder and decoder use separate rank-16 LoRA modules, while the pretrained backbone weights remain frozen. Learning rates are $10^{-4}$ for the LoRA parameters and query embeddings and $5 \times10^{-5}$ for the remaining trainable parameters, with 500 warm-up steps. Video and action losses receive equal weights for action-labeled clips; clips without action labels use only the video loss.

Latent-Action Compression

We train a variational bottleneck on latent actions $Q$ extracted by the frozen DreamX encoder. Following ([11]), the bottleneck compresses each token from backbone width $D$ to $d \ll D$ while preserving the token layout.

Formulation. The encoder $f_{\psi}$ parameterizes a factorized Gaussian posterior $q_{\psi}(Z \mid Q)$ through

During bottleneck training, we sample compact codes using

and reconstruct the original latent actions as $\hat{Q} = g_{\psi}(Z)$.

Only the bottleneck encoder $f_{\psi}$ and decoder $g_{\psi}$ are optimized, using

The reconstruction term encourages the compact codes to retain the information in $Q$, while the KL term regularizes their posterior toward a standard Gaussian. We compute the reconstruction loss in a fixed normalized feature space.

Architecture and Training. The bottleneck encoder uses a single self-attention block to produce a 32-dimensional code for each 2,048-dimensional latent-action token. A six-layer transformer decoder reconstructs the original features. The token layout is preserved: for a 32-token input, the bottleneck maps a $32 \times2{,}048$ representation to $32 \times32$ compact codes.

We train the bottleneck for 50,000 steps with a learning rate of $10^{-4}$, 500 warm-up steps, and a KL weight of $\beta= 0.01$. After training, we freeze the bottleneck and use the posterior mean,

to obtain deterministic compact codes for downstream policy learning and evaluation.

Policy Training and Inference

To convert the pretrained video model into a latent-action generator, we train it to predict compact latent-action codes conditioned on the current observation and language instruction. We use approximately one-third of the DreamX pretraining data and optimize rank-128 LoRA modules in self-attention, cross-attention, and feed-forward layers, together with trainable input and output projections. The generator is trained for 100,000 steps on two NVIDIA H200 GPUs with a global batch size of 60, a learning rate of $10^{-4}$, and 1,000 warm-up steps. The Action DiT is not separately pretrained; it is introduced during task-specific policy learning and trained jointly with the generator. Task-specific training settings are provided in Appendix § 10.§ 3.

Experimental Setup and Evaluation Protocols

Real-World Robot Setup

We conduct both action-transfer and policy experiments on a bimanual YAM robot. The action space consists of 14-dimensional absolute commands, comprising six joint positions and one gripper command per arm. We leverage one Intel RealSense D405 camera for observation, which is mounted on top view.

Action Transfer Details

We extract latent actions from human demonstrations using the frozen DreamX encoder and decode them into robot commands. The resulting trajectory is executed open-loop, without visual feedback, conditioned on the human latent actions and the robot’s initial proprioceptive state.

We evaluate three types of motion transfer. Writing tests whether latent actions preserve fine-grained spatial trajectories and stroke order; a marker attached to the right gripper reproduces the demonstrated shape. Single-arm pushing tests whether the representation captures changes in motion direction and their temporal order, requiring the robot to follow a forward-then-left sequence. Bimanual manipulation tests whether latent actions retain arm-specific motion and transfer it to the corresponding robot arms. Additional examples are provided in Section D.2.

Policy Evaluation Details

Task and observations. We evaluate three drawer-closing configurations: only the upper drawer open, only the lower drawer open, and both drawers open, as illustrated in Figure 11. All configurations use the instruction close the drawer. When both drawers are open, the robot must close the upper drawer before the lower drawer. A fixed top-view RGB camera observes the workspace.

Training. We compare DreamX with Mimic-Video {(citep}(Pai et al., 2025)}, using human videos and robot demonstrations for task-specific learning. DreamX starts from the pretrained latent-action generator, while Mimic-Video starts from a video backbone pretrained on MolmoAct. Each method then incorporates an action head for task-specific learning, and both are trained for 30,000 steps. For DreamX, human videos supervise latent-action prediction, while robot demonstrations additionally supervise low-level actions. During rollout, DreamX generates compact latent actions alongside low-level actions, whereas Mimic-Video generates future video.

Evaluation. We evaluate reaching and pushing as two subgoals: reaching measures whether the robot approaches the drawer for interaction, and pushing measures whether it completes the required closure. The reported subgoal success rate averages the two completion rates within each configuration, and the overall average weights the three configurations equally. We additionally report inference latency per policy call. Separate reaching and pushing results are provided in Section C.1. We collected 45 trajectories to train the policy and evaluated its success rate over 10 trials in each of three scenarios.

LARYBench Evaluation Details

Evaluation protocol. We freeze all representation models and train a common attentive probe for action regression and classification. Each input token is projected to 768 dimensions through a linear layer, layer normalization, and GELU. For representations spanning multiple transitions, learned temporal embeddings identify the transition positions before the temporal and token dimensions are flattened. A learned-query attentive pooler aggregates the tokens into a single feature for the regression or classification head.

Feature extraction. The regression benchmark originally provides only two frames per sample. Using the accompanying metadata, we recover the corresponding video sequences from the source datasets, including the intervening frames, for video-based models.

We extract each model’s native representation without fine-tuning and preserve its temporal and token structure. Pair-based models encode consecutive frame pairs, while video-based models encode clips using their supported temporal windows. Features are precomputed for probe training.

Zero-Shot Human-to-Robot Transfer Details

We evaluate on 1,008 clips spanning 21 tasks from HumanAndRobot [35], selecting 48 clips per task. Each method receives a human source video and the initial frame of its paired robot recording. The remaining robot frames serve only as evaluation references; no paired human–robot data is used for training. We compare DreamX with AdaWorld [10] and VideoWorld 2 [28] using their native generation horizons.

For a common evaluation horizon, we retain the first 20 frames of each output and exclude the conditioning frame, yielding 19 generated frames per clip. We report FVD using a Kinetics-400-pretrained I3D network, VGG-based LPIPS, PSNR, and SSIM against the corresponding robot reference videos.

Additional Quantitative Results

Policy Evaluation

Subgoal completion Table 5 reports reaching and pushing completion separately. Averaging each pair gives the per-configuration SR in Table 3: 50.0% for Mimic-Video in all configurations, and 90.0%, 100.0%, and 100.0% for DreamX.

Method	Upper Reach	Upper Push	Lower Reach	Lower Push	Both Reach	Both Push
Mimic-Video	80	20	100	0	100	0
DreamX	100	80	100	100	100	100

Policy learning from human demonstrations We additionally train a policy using only human videos for the upper-only drawer-closing configuration, while retaining robot demonstrations for the lower-only and both-open configurations. Human videos supervise latent-action prediction without low-level action labels. As shown in Table 6, the policy achieves a 100% subgoal success rate in upper drawer task even if trained only with human dataset.

Method	Upper	Lower	Both	$SR^\uparrow$
DreamX	100/100	100/100	100/100	100

Ablation Study

Latent-action conditioning on CALVIN. We compare the DreamX policy with and without allowing the action head to attend to latent-action features. In the variant without latent actions, we block the action head’s attention to these features. Table 7 reports the percentage of evaluation sequences completing at least $k$ consecutive tasks, for $k = 1,\ldots,5$, and the average completed sequence length. Conditioning on latent actions improves success rates at all five sequence lengths and increases the average length from 2.378 to 2.643. These results suggest that latent-action information provides useful guidance for robot action prediction in the DreamX policy.

Policy	$\ge1$	$\ge2$	$\ge3$	$\ge4$	$\ge5$	Avg. Len.$\uparrow$
DreamX policy w/o latent actions	83.4	61.7	42.5	29.8	20.4	2.378
DreamX policy	85.7	67.0	49.5	36.4	25.7	2.643

Representation Analysis

Motion structure across embodiments

We examine whether latent actions exhibit common directional structure across robot embodiments using 2,362 RoboCOIN clips spanning ten robot types and six end-effector translation directions.

We examine whether latent actions exhibit common directional structure across robot embodiments using 2,362 RoboCOIN clips spanning ten robot types and six end-effector translation directions. Figure 12 visualizes 60 direction-conditioned latent-action centroids using t-SNE. Centroids for the same direction exhibit similar organization across robot types. This aggregate analysis supports shared directional structure across embodiments.

Effect of latent-action conditioning

We quantify how changes to latent actions affect decoded motion and appearance. For latent actions $Q \in\mathbb{R}^{N_q \times D}$, we perturb the decoder’s conditioning as

where $s_d$ is the channel-wise standard deviation estimated from training features. We evaluate $\alpha\in\{0,0.1,0.3,1.0\}$ and a zero-query control on eight selected high-motion clips, with two decoder seeds per clip. Within each comparison, we fix the initial frame, decoder conditioning schedule, and perturbation draw, rescaling only the perturbation magnitude. The stronger qualitative perturbation in Figure 7 uses $\alpha= 2$.

Table 8 reports mean changes relative to the unperturbed reconstruction, and Figure 13 shows their distributions across clips and decoder seeds. We compute Farneback optical flow at $160 \times120$ and measure mean flow-vector deviation and motion magnitude relative to the unperturbed output. RGB differences use intensities in $[0,1]$ and exclude the conditioning frame. A fixed static-region mask selects pixels below the 40th percentile of temporal RGB variation in the reference video, followed by $5 \times5$ erosion.

Conditioning	Motion ratio	Flow deviation	RGB L1	Static RGB L1
Original	1.000	0.000	0.0000	0.0000
Noise ($\alpha= 0.1$)	1.000	0.141	0.0054	0.0030
Noise ($\alpha= 0.3$)	0.995	0.211	0.0077	0.0034
Noise ($\alpha= 1.0$)	0.946	0.481	0.0215	0.0077
Zero queries	0.456	0.866	0.0758	0.0227

Increasing perturbation strength produces larger changes in decoded motion, while static-region appearance changes remain smaller than whole-image changes. Zeroing the latent actions reduces the mean per-sample motion ratio to 0.456. Together with the qualitative examples, these measurements support the role of latent actions in steering motion while the decoder retains much of the scene appearance.

Additional Qualitative Results and Analysis

Any-to-Any Video Transfer

In-domain Transfer

Figure 14 provides additional any-to-any video transfer examples across embodiments and environments, complementing Figure 4 in the main paper.

Additional Zero-Shot Human-to-Robot Transfer

Figure 15 provides additional zero-shot human-to-robot video transfer comparisons on HumanAndRobot, complementing Figure 5. All methods use the evaluation protocol in Section B.5.

Human-to-Robot Action Transfer

Figure 16 provides additional human-to-robot action transfer examples for writing and single-arm manipulation, complementing Figure 6. The experimental setup is described in Section B.2.

Policy Rollouts

Figure 17 provides illustrative real-world drawer-closing rollouts for the upper-only, lower-only, and both-open configurations.

Query-to-Video Attention

Figure 18 provides additional query-to-video attention visualizations across human and robot demonstration sequences, complementing Figure 8. We average query-to-video attention over all heads, the four query slots within each temporal group, and the final seven transformer blocks.