betaFeedback

WorksComputer Science

Future Representation Distillation for Imagination-free World Action Models

Abstract

World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.

TL;DR: DreamFree scales imagination-free WAMs through future representation distillation that effectively leverages the predictive priors of pretrained video diffusion models.

Website https://harryjo97.github.io/dreamfree/

Code https://github.com/harryjo97/dreamfree

Introduction

Pretrained video diffusion models offer a promising foundation for robot control [35, 15, 9]. Through large-scale video pretraining, these models learn to predict how physical environments evolve over time, providing predictive knowledge that can guide robot actions. World Action Models (WAMs; Ye et al. [28]) use this knowledge by adapting video models to predict visual futures and actions, demonstrating strong manipulation performance and generalization to new tasks and environments [15, 28].

A common approach in WAMs is to use this predictive knowledge of pretrained video diffusion models through future video prediction, either by jointly predicting visual futures and actions [15, 28] or by decoding actions from predicted visual futures [10, 21]. These methods leverage the pretrained models’ knowledge through their native video prediction process. However, iterative video denoising incurs substantial inference cost, and errors in the predicted visual futures can lead to inaccurate actions. Moreover, it remains unclear whether action prediction requires the same level of visual detail as video generation.

A central question is how to make better use of the predictive knowledge in pretrained video models for action prediction. The success of video-generating WAMs demonstrates the value of this knowledge, while leaving open how it should be represented to support control. This motivates investigating whether predictive information can be made directly available in the visual features used for action prediction.

We hypothesize that this predictive knowledge can be effectively captured in visual representations to support action prediction, without iterative video denoising. Our key idea is to distill the video model’s predictive knowledge into the visual representations computed from the current observation. We train a visual encoder to anticipate visual futures from the observation by using the representations extracted from a pretrained video diffusion model. Our analysis shows that distillation enables the encoder to effectively capture future visual information from the current observation and make its features more useful for action prediction. The resulting predictive representations enable action prediction without generating future videos.

Based on this finding, we introduce DreamFree, a WAM that repurposes a pretrained video diffusion backbone as a predictive visual encoder through representation distillation. We train the model in two stages, separating predictive representation learning from action learning. In the first stage, a frozen video diffusion model extracts representations from both current and future frames of a video demonstration, and the visual encoder learns to predict these representations from the current observation. The visual encoder thus learns to anticipate scene evolution in feature space, making the video model’s predictive knowledge available in a single forward pass without future video generation. This stage requires no action labels, allowing the visual encoder to learn from any in-the-wild video demonstrations. In the second stage, we train an action module on robot demonstrations to map these visual features to executable actions, with the visual encoder either frozen or jointly fine-tuned.

We evaluate DʀᴇᴀᴍFʀᴇᴇ on diverse manipulation tasks in both simulation and real-world environments [12], demonstrating stronger performance compared to state-of-the-art VLA and WAM baselines [23, 28]. In real-world evaluations, DʀᴇᴀᴍFʀᴇᴇ succeeds on tasks involving object deformation or successive rearrangements where video-generating WAM fails (Figure 1). These successes suggest a benefit of using predictive visual representations for control without relying on potentially inaccurate video predictions. Experiments show that using a larger video backbone and additional pretraining video data improve success, demonstrating DʀᴇᴀᴍFʀᴇᴇ can translate stronger video priors and additional video data into better robot control without additional action annotation. We additionally show task transfer from video demonstrations, including videos from a different embodiment, showing that DʀᴇᴀᴍFʀᴇᴇ can learn new tasks from videos without collecting action annotations for those tasks.

Figure 1. Overview. DreamFree distills predictive knowledge from a frozen video diffusion model into a predictive visual encoder, enabling action prediction from the current observation without generating future videos. In real-world evaluations, DreamFree achieves higher success than DreamZero and π0.5\pi_{0.5} with latency comparable to that of π0.5\pi_{0.5} and a 17.8× inference speedup over DreamZero. DreamFree generalizes across diverse real-world manipulation tasks.

In addition to strong performance, DʀᴇᴀᴍFʀᴇᴇ offers a practical advantage for real-time control. During deployment, the policy obtains predictive visual representations in a single encoder pass and uses them to generate an action chunk without iterative video denoising. DʀᴇᴀᴍFʀᴇᴇ using a 14B video backbone achieves real-time closed-loop control at 11Hz, achieving a 17.8×17.8 \times inference speedup over a video-generating WAM with the same video backbone size and latency comparable to that of a VLA (Figure 1, middle).

We highlight our main contributions below:

  • We introduce DʀᴇᴀᴍFʀᴇᴇ, a WAM that repurposes a pretrained video diffusion backbone as a predictive visual encoder by distilling future information from a frozen video diffusion model.

  • We outperform state-of-the-art VLA and WAM baselines in real-world manipulation without task-specific adaptation, while achieving real-time closed-loop control at 11Hz.

  • We show that DʀᴇᴀᴍFʀᴇᴇ translates stronger video priors and additional video data into better robot control.

  • We demonstrate that DʀᴇᴀᴍFʀᴇᴇ can learn new tasks from video demonstrations, including videos from a different embodiment, without action annotations for the new tasks.

Related work

Vision-language-action policies Vision-Language-Action (VLA) policies adapt pretrained vision-language models to map visual observations and language instructions directly to robot actions [23, 6, 36, 5, 13]. While their backbones provide semantic and linguistic priors, pretraining on static image-text pairs gives them limited knowledge of the spatiotemporal dynamics important for robot control.

Video model-based robot policies Video model-based robot policies build on pretrained video diffusion models by coupling future visual prediction with action prediction. Some decode actions from video representations through inverse dynamics [10, 21, 24], while others jointly denoise future video and action sequences [15, 16, 4, 18, 28]. These World Action Models (WAMs) transfer spatiotemporal video priors to control, achieving strong action-data efficiency and generalization. However, generating future video at deployment introduces substantial inference cost, and errors in the predicted video can lead to inaccurate action prediction. Recent works retain video prediction during training while enabling action prediction without future video generation at deployment [30, 27], yet how faithfully their representations preserve the video model’s predictive knowledge remains unclear.

Predictive visual representation learning Predictive visual learning methods train encoders to capture scene structure and dynamics by predicting representations of unobserved visual content [2]. Related predictive supervision is used in VLA policies, where future images or visual features serve as auxiliary training targets alongside action supervision [26, 33, 32]. For example, FLARE [34] aligns future tokens in the action expert with vision-language embeddings of future observations. Our approach learns predictive visual representations from the current observation by distilling features from a frozen video diffusion model into a visual encoder.

DREAMFREE

As shown in Figure 2, DREAMFREE consists of a predictive visual encoder followed by an action module. The training stage of DREAMFREE is separated into two stages. First, the predictive visual encoder is trained to predict the representations from the frozen video diffusion model (§ 3.1). The frozen video diffusion model extracts representations from both current and future frames of a video demonstration, while the visual encoder conditions on the current observation only. Second, the action module is trained to predict actions by extracting action-relevant information from the predictive visual encoder through a readout module (§ 3.2). During inference, only a single forward pass through the predictive visual encoder is required to extract visual features relevant for action prediction, enabling real-time closed-loop control up to 11Hz.

Figure 2. Model architecture of DREAMFREE. Training is separated into two stages: (a) In the first stage, a pretrained video diffusion backbone is repurposed as a visual encoder through video generative training with a structured attention mask and distillation of predictive representations from the frozen video diffusion model. (b) In the second stage, the action readout module and action decoder are trained to predict actions by extracting action-relevant information from the encoder.

Learning a predictive visual encoder

Predictive visual encoder We initialize the encoder from a pretrained video diffusion backbone and use the hidden states of its current-frame latent tokens as the visual representation. We apply a structured mask to the backbone’s bidirectional video self-attention (Figure 2(a) left), restricting current-frame tokens to attend only to each other within self-attention. This ensures that the current-frame hidden states depend only on the current observation and instruction. During training, the backbone receives a clean current frame and noisy future frames, as shown in Figure 2(a).

Future representation distillation The main idea is to distill the representation of a frozen video diffusion model into the visual encoder that observes only the current observation and instruction. Motivated by REPA [29], we introduce future representation distillation, which aligns the projection of the encoder’s current frame representation with a video diffusion model representation as shown in Figure 2(a).

Formally, let uθu_\theta be the predictive visual encoder, gg be the pretrained video diffusion model, and pψp_\psi be the trainable projector. Consider a clean video x⋆x_\star with the first frame x⋆0x_\star^0 and language instruction ℓ\ell, and let hih_i be the ii-th layer hidden representation of uθ(x⋆0,ℓ)u_\theta(x_\star^0,\ell). We use gg as the teacher and take its hidden representation at layer jj, computed from a perturbed latent of x⋆x_\star, as the distillation target. We use a lightweight Transformer decoder for the projector pψp_\psi that maps the current-frame representation hih_i to the spatiotemporal token layout and feature dimension of the teacher’s full-video representation. Note that gg and pψp_\psi are used only during training and discarded at deployment.

Future representation distillation is achieved by maximizing the similarity between the current-only encoder representation and the teacher representation as follows:

Ldistill(θ,ψ):=Ex⋆,ℓd[−sim⁡(pψ(hi),gj(xtd,td,ℓ))],(1)\mathcal{L}_{\mathrm{distill}}(\theta,\psi) := \mathbb{E}_{x_\star,\ell_d}\left[-\operatorname{sim}\left(p_\psi(h_i),g_j(x_{t_d},t_d,\ell)\right)\right], \tag*{(1)}

where ϵd\epsilon_d is Gaussian noise, td∈[0,1]t_d \in[0,1] is a fixed timestep controlling the teacher input’s noise level, xtd=(1−td)x⋆+tdϵdx_{t_d}=(1-t_d)x_\star+t_d\epsilon_d, and sim⁡(⋅,⋅)\operatorname{sim}(\cdot,\cdot) is a pre-defined similarity function.

To preserve the predictive capability inherited from the pretrained video backbone, we also use the original generative flow matching objective as a regularization:

Lgen(θ):=Ex∗,ϵ,t[∥uθ(xt,t,ℓ)−(ϵ−x∗)∥22],(2)\mathcal{L}_{\mathrm{gen}}(\theta) := \mathbb{E}_{\mathbf{x}_*,\boldsymbol{\epsilon},t}\left[\left\|\mathbf{u}_{\theta}(\mathbf{x}_t,t,\ell)-(\boldsymbol{\epsilon}-\mathbf{x}_*)\right\|_2^2\right], \tag*{(2)}

where uθ\mathbf{u}_{\theta} is conditioned on clean current-frame latent tokens with timestep 0, while the future-frame latent tokens in xt\mathbf{x}_t are perturbed with Gaussian noise ϵ\boldsymbol{\epsilon} at timestep tt.

The encoder is trained with both the native generative loss and the distillation loss as follows:

L(θ,ψ):=Lgen(θ)+Ldistill(θ,ψ).(3)\mathcal{L}(\theta,\psi) := \mathcal{L}_{\mathrm{gen}}(\theta)+\mathcal{L}_{\mathrm{distill}}(\theta,\psi). \tag*{(3)}

Learning actions from visual representations

After training the predictive visual encoder, we use action-labeled robot demonstrations to train the action module, which consists of a readout module, state and action encoders, and an action decoder. Given the observation and instruction, the visual encoder produces the predictive representation that serves as a visual prior for the action module. Then the readout module extracts action-relevant features from the representation, and the action module conditions on the action-relevant features together with the instruction and proprioception to predict the action chunk, as illustrated in Figure 2(b).

Action readout module Given observation oo and instruction ℓ\ell, we extract the hidden representation hk\mathbf{h}_k from the visual encoder, where kk denotes the layer used for action readout. We transform these visual tokens into a fixed number of action-facing tokens using a latent-query readout inspired by Perceiver [11]. The trainable readout rω\mathbf{r}_{\omega} updates learned queries through self-attention and cross-attention, with hk\mathbf{h}_k serving as keys and values. The resulting tokens c=rω(hk)\mathbf{c}=\mathbf{r}_{\omega}(\mathbf{h}_k) provide the action decoder with action-relevant predictive visual representations.

Action decoder Conditioned on the action-facing tokens c\mathbf{c}, instruction ℓ\ell, proprioceptive state q\mathbf{q}, and optional context s\mathbf{s}, the action decoder predicts the action chunk. We instantiate the action decoder uϕ\mathbf{u}_{\phi} with a cross-attention-conditioned video Diffusion Transformer (DiT) [22].

Given an action chunk a∗\mathbf{a}_*, the action decoder is trained with a flow matching objective:

Lact:=Ea∗,ϵ,t[∥uϕ(at,t∣c,q,s,ℓ)−(ϵ−a∗)∥22],(4)\mathcal{L}_{\mathrm{act}} := \mathbb{E}_{\mathbf{a}_*,\boldsymbol{\epsilon},t}\left[\left\|\mathbf{u}_{\phi}(\mathbf{a}_t,t\mid\mathbf{c},\mathbf{q},\mathbf{s},\ell)-(\boldsymbol{\epsilon}-\mathbf{a}_*)\right\|_2^2\right], \tag*{(4)}

where ϵ\boldsymbol{\epsilon} is Gaussian noise, t∈[0,1]t\in[0,1], and at=(1−t)a∗+tϵ\mathbf{a}_t=(1-t)\mathbf{a}_*+t\boldsymbol{\epsilon}.

Inference Without Iterative Video Denoising During inference, the predictive visual encoder processes the observation and instruction in a single forward pass. The resulting predictive visual representation is transformed by the readout into action-facing tokens, and the action decoder conditions on these to generate an action chunk using NN denoising steps. This enables real-time closed-loop control up to 11Hz even for a 14B video backbone.

Experiments

We validate the performance of DreamFree in both simulation and real-world environments. In particular, we investigate the following questions:

  1. (Q1) Does distillation make visual features more useful for action prediction?

  2. (Q2) Do representations learned by DreamFree translate into strong manipulation performance?

  3. (Q3) Can DreamFree translate stronger video priors and additional video data into better control?

  4. (Q4) How effectively and efficiently does DreamFree perform real-world manipulation?

  5. (Q5) Can DreamFree learn new tasks from video demonstrations without action annotations?

Experimental setup

Implementation details We use pretrained Wan2.2-5B and Wan2.2-A14B high-noise expert [25] as the video backbone, including the video DiT, text encoder, and video VAE. The action decoder is a 1B DiT. For all experiments, we set the action horizon to 32. Following Fast-WAM [30], we temporally downsample the videos by 4×4\times, producing 9 video frames per chunk. DreamFree is trained on each dataset without cross-embodiment pretraining. We provide more details in Appendix § 8.

Benchmarks We evaluate our method on two simulation benchmarks and one real-world evaluation setup. For LIBERO [17], we follow the standard benchmarking protocol, using four task suites (Spatial, Object, Goal, Long). For RoboCasa-Kitchen [19], we follow the 24-task benchmark protocol of Kim et al. [15] and train on their released dataset, which contains 50 demos per task. The tasks span object placement and manipulation of kitchen fixtures and appliances. For real-world evaluation, we use the DROID [12] setup, evaluating on 10 tasks selected from the pretraining distribution spanning diverse capabilities, including push, cover, press, fold, pour, and composite skills (see Appendix § 10.§ 2). We measure the task progress and success rate using 5 rollouts per task. We provide task descriptions and evaluation details in Appendix § 10.§ 2.

Baselines On simulation benchmarks, we compare DreamFree against state-of-the-art VLAs [6, 23, 14, 5, 34] and WAMs [4, 16, 30, 15]. For real-world DROID evaluation, we compare with state-of-the-art VLA π0.5\pi_{0.5} and WAM DreamZero [28] with their publicly released checkpoints, namely π0.5\pi_{0.5}-DROID and DreamZero-DROID, following their inference setup.

Q1: Representation analysis

We examine whether future representation distillation makes the visual encoder’s representation more useful for action prediction. We compare features from three models given only the current observation: the pretrained video diffusion model (VDM), a variant of the DreamFree visual encoder trained without distillation, and the visual encoder of DreamFree. As a reference with direct access to future observations, we also evaluate VDM representations computed from both the current frame and ground-truth future frames. The VDM and encoders use the 14B video backbone and are conditioned on the language instructions. Each encoder is trained separately on RoboCasa-Kitchen and DROID datasets. For each representation and input setting, we train a separate probe while keeping the feature extractor frozen. We report normalized mean squared error in Figure 3.

Figure 3. Representation analysis. We evaluate visual representations by measuring how accurately action sequences (Action probing) and future-frame VAE latents (Future probing) can be decoded from them. Future representation distillation makes the encoder’s features more useful for action prediction and improves their ability to capture future visual information from the current observation.

Action probing To assess the usefulness of the features for control, we train a lightweight attention-based probe to predict action sequences from each representation, using paired robot demonstration videos and actions. As an additional reference, we also probe VAE latents of both the current and ground-truth future frames to compare the encoder representations with direct encodings of future observations. Figure 3 shows that the DreamFree encoder achieves lower action prediction error than the VDM using only the current observation and the encoder variant trained without distillation. These results show that distillation makes the encoder’s features more useful for action prediction.

Despite using only the current observation, DREAMFREE encoder achieves lower action prediction error than VAE latents of current and future frames, and approaches the accuracy of the VDM using the future frames. This indicates that the DREAMFREE encoder learns features that support action prediction nearly as accurately as directly observing the future. We additionally probe the features produced by the action readout module of DREAMFREE. These features yield even lower prediction error, demonstrating that the readout effectively extracts action-relevant information from the visual representations.

Future probing To better understand the improvement in action decodability from distillation, we examine whether distillation improves the encoder’s ability to capture future visual information from the current observation. We train a spatially shared linear probe to reconstruct the VAE latent patches of ground-truth future frames. Figure 3 shows that DREAMFREE encoder achieves lower reconstruction error than the VDM using only the current observation and the encoder variant trained without distillation. Furthermore, its reconstruction error approaches that of the VDM given future frames on RoboCasa-Kitchen, while a gap remains on DROID. These results show that our encoder effectively captures future visual information from the current observation.

Q2 & Q3: Performance and scaling on simulation benchmarks

We first assess in-distribution task execution of DREAMFREE in the LIBERO benchmark compared to VLAs and WAMs. In Table 1, DREAMFREE achieves performance comparable to state-of-the-art baselines even without cross-embodiment pretraining, supporting the effectiveness of DREAMFREE’s predictive visual representations for in-distribution manipulation.

MethodAvg. SR
VLAs
π0\pi_0 [6]94.1
π0.5\pi_{0.5} [23]96.9
OpenVLA-OFT [14]97.1
WAMs
Motus [4]97.7
LingBot-VA [16]98.5
Fast-WAM [30]97.6
DREAMFREE (Ours)98.6

Table 1. Average success rate (%) across the four LIBERO task suites.

We next evaluate DREAMFREE on the RoboCasa-Kitchen benchmark to assess multi-task manipulation under object and scene variation, with limited demonstration data. Table 2 shows that DREAMFREE with a 14B video backbone achieves the highest average success rate of 71.7%, demonstrating that our predictive visual representations can translate pretrained video knowledge into strong manipulation performance without generating future videos. Fast-WAM also predicts actions without generating future videos at inference, but underperforms the other baselines with only 44.4% success, suggesting that using video priors without future video generation remains challenging in this setting.

MethodAvg. SR
π0\pi_0 [6]62.5
GR00T-N1.5 [5]64.1
FLARE [34]66.4
Cosmos Policy [15]67.1
Fast-WAM [30]44.4
Fast-WAM 14B [30]44.7
DREAMFREE (Ours)
5B video backbone58.5
14B video backbone71.7
5B video backbone + extra videos68.8
14B video backbone + extra videos74.4

Table 2. Average success rate (%) across 24 RoboCasa-Kitchen tasks.

Moreover, we investigate whether DREAMFREE benefits from a larger video backbone and more pretraining video data. Fast-WAM shows little improvement when its backbone is scaled from 5B to 14B, suggesting limited benefit from stronger video priors. In contrast, scaling our video backbone from 5B to 14B improves success rate by 13.2 pp, with particularly large gains on pick-and-place tasks. These gains suggest that stronger video priors provide more useful representations for coordinating grasping, transport, and placement. We also use videos from the atomic-task subset of RoboCasa365 [20] showing robot demonstrations in simulated kitchen environments. Training the visual encoder with these videos, without their action labels, further improves success. Together, these results show that DREAMFREE can translate stronger video priors and additional video data into better robot control, without additional action annotations.

Selected frames from successful rollouts on four DROID evaluation tasks

Figure 5. Examples of real-world task executions. We show selected frames from successful rollouts of DreamFree on four DROID evaluation tasks.

To assess how each visual encoder training objective contributes to performance, we ablate the generative objective Lgen\mathcal{L}_{\mathrm{gen}} and distillation objective Ldistill\mathcal{L}_{\mathrm{distill}}. As shown in Table 3, removing the distillation objective Ldistill\mathcal{L}_{\mathrm{distill}} leads to a significant drop in the success rate for both datasets. Removing the generative objective Lgen\mathcal{L}_{\mathrm{gen}} also degrades the performance, although the drops are smaller compared to removing the distillation objective. These results indicate that both objectives contribute to learning predictive visual representations, with distillation playing a particularly important role in translating pretrained video knowledge into strong manipulation performance. This is consistent with the representation analysis in § 4.2, where removing distillation reduces both action and future decodability.

MethodLIBERORoboCasa
Ours98.671.7
w/o Lgen\mathcal{L}_{\mathrm{gen}}98.055.6
w/o Ldistill\mathcal{L}_{\mathrm{distill}}94.647.2

Table 3. Ablation of visual encoder training objectives.

Q4: Real-world performance and efficiency

Real-world manipulation To assess real-world manipulation and generalization, we evaluate the models out of the box on 10 task types represented in the pretraining data, using new objects or scene arrangements. Figure 4 reports the task progress and success rates. DreamFree achieves the highest average success rate at 64%, compared with 56% for DreamZero and 16% for π0.5\pi_{0.5}, with successful executions spanning a broader range of task types than either baseline. DreamFree also achieves the highest average task progress at 83%, compared with 71.8% for DreamZero and 33.5% for π0.5\pi_{0.5}.

Figure 4. Real-world evaluation on DROID. We evaluate policies out of the box on 10 tasks with five trials per task. Task descriptions and the progress scoring criteria are provided in Appendix § 10.§ 2.

The gains over DreamZero are notable in tasks involving deformable object manipulation or successive rearrangements, such as cover strawberry with towel, hang hat on tripod, and put bolts in bowl. DreamZero fails in all trials for each of these tasks, with inaccurate video predictions observed in most failures. Qualitatively, DreamZero often persists with unsuccessful actions after an initial error,

Q5: Task transfer from video demonstrations

whereas DreamFree often adjusts its actions and recovers. Together with the analysis in § 4.2, these observations suggest that DreamFree benefits from capturing predictive information directly in its visual representations, enabling successful control without relying on potentially inaccurate future video predictions. These results demonstrate that DreamFree achieves effective real-world control and generalizes zero-shot to new objects or scene arrangements within seen tasks. We describe the failure cases of DreamFree in Appendix § 11.§ 3.

Real-world bimanual manipulation We further examine the applicability of DreamFree to another real-world embodiment using a bimanual YAM setup. We train DreamFree on a filtered 100-hour subset of ABC-130k [1] without additional cross-embodiment training. We deploy the resulting policy out of the box without task adaptation, evaluating tasks represented in the training set but using new scenes, objects, and arrangements. Figure 6 presents successful executions of bimanual manipulation tasks, including pick-and-place, pen uncapping, cloth folding, and dish-rack loading. These qualitative results provide evidence of zero-shot generalization to new scenes and illustrate that our approach can support precise, coordinated two-arm manipulation on a physical robot.

Examples of real-world bimanual manipulation task executions

Figure 6. Examples of real-world bimanual manipulation task executions. We show selected frames from successful rollouts of DreamFree.

Latency In Figure 7 (left), we measure the mean inference latency per action chunk on NVIDIA H200 GPUs. Each policy is measured in its deployed configuration. DreamFree performs real-time closed-loop control at 11Hz, yielding a 17.8× speedup over DreamZero while achieving a higher average success rate on the real-world tasks. Compared with π0.5\pi_{0.5}, DreamFree achieves 4× the average success rate with 1.1× the latency.

Latency and completion time comparison

Figure 7. Latency and completion time comparison.

Figure 7 (right) further compares the task completion times for placing different numbers of blocks into a bowl. DreamFree achieves completion times comparable to those of π0.5\pi_{0.5} on simpler settings (2 or 4 blocks), while completing the task much faster in the harder setting (6 blocks). These results highlight the efficiency of using predictive visual representations for action generation without explicit video generation.

Q5: Task transfer from video demonstrations

We investigate transfer to new tasks using video demonstrations without action annotations. We consider two settings that differ in when target task videos are introduced. In the first setting, the target task videos are introduced during visual encoder pretraining, and we transfer from videos of either the same embodiment or a different embodiment. In the second, the target task videos are excluded from initial policy training and introduced for fine-tuning the policy. In both settings, we use only videos of the new tasks without action annotations.

Atomic-to-composite task transfer from video pretraining We first investigate whether DreamFree can learn new composite tasks from videos, using action labels only from atomic tasks. Atomic tasks involve individual skills while composite tasks combine several of these skills sequentially. We compare a base DreamFree trained on atomic tasks in RoboCasa-Kitchen with two variants that additionally use videos of composite tasks for visual encoder training. One variant uses only Lgen\mathcal{L}_{\mathrm{gen}} without distillation, while the other uses both Lgen\mathcal{L}_{\mathrm{gen}} and Ldistill\mathcal{L}_{\mathrm{distill}}. All models use the same action labels from atomic tasks, and the composite tasks are absent from the action-labeled training data.

As shown in Figure 8 (left), the base model fails on all tasks, and adding the task videos with distillation raises the average success from 0% to 21.7%, with at least one successful trial on each task. Using the same videos without distillation yields 0% average success. These results demonstrate that DreamFree can acquire new task knowledge from videos and transfer it to action execution without additional action annotations, with distillation playing an important role in enabling this transfer. Figure 9(a) illustrates a successful composite task rollout by DreamFree, completing four subgoals in sequence.

Figure 8. Task transfer from video demonstrations. We report success rates and progress over 20 episodes per task. Task descriptions are provided in Appendix § 10.§ 3.

Figure 9. Examples of task execution after transfer from videos. We show selected frames from successful rollouts of DreamFree.

Cross-embodiment task transfer from video pretraining We next investigate whether DreamFree can learn new tasks with task videos from a different embodiment. We use RoboTwin [7] with ALOHA-AgileX as the target robot, and compare a base DreamFree trained on its atomic tasks with a variant whose encoder also learns from composite task videos recorded with a dual-arm Franka Panda setup. Both models use the same ALOHA-AgileX action labels, and the composite tasks are absent from the action-labeled training.

Figure 8 (middle) shows that the base model succeeds only in bowl stacking, with 6.7% average success across the new tasks, and adding task videos from a different embodiment improves the average success to 30%. This demonstrates that DreamFree can transfer task knowledge from different embodiments’ videos to action execution. Figure 9(b) illustrates a successful task rollout by DREAMFREE after learning from videos of a different embodiment, without access to action-labeled demonstrations of that task.

Task adaptation after policy training Finally, we investigate whether target task knowledge can be added after policy training. We start from DREAMFREE trained on atomic tasks in RoboTwin, with target tasks excluded from the training set. We then fine-tune only the visual encoder separately for each task using 10 video demonstrations, keeping the action module frozen. Adaptation takes approximately 10 minutes per task on a single H200 GPU. We compare demonstrations containing multiple camera views with demonstrations containing only the head camera view.

Figure 8 (right) shows that both single-view and multi-view video fine-tuning improve success rates on both tasks, with larger gains from multi-view videos. This shows that task knowledge acquired through video-only fine-tuning of the encoder can transfer to action execution without updating the action module, even with single-camera demonstrations. Figure 9 illustrates a successful task rollout with the fine-tuned policy.

We find that not all tasks can be learned through video demonstrations. For example, a policy unable to open cabinets also failed to learn to place an object inside a cabinet through video-only fine-tuning. This suggests that learning new tasks from videos may be limited by the policy’s ability to execute the required atomic skills. We discuss these failure cases in Appendix § 11.§ 3.

Conclusion

We presented DREAMFREE, a WAM that repurposes pretrained video diffusion models as predictive visual encoders through future representation distillation. By distilling video priors from the intermediate representations of a frozen video diffusion model with access to future frames, DREAMFREE learns to predict future visual representations that guide action prediction without generating future videos. Experiments demonstrate strong manipulation performance in both simulation and the real world, with a significant inference speedup over video-generating WAM. DREAMFREE benefits from a larger video backbone and more video data, and enables task transfer from video demonstrations. These findings highlight predictive representation learning as an effective way to translate pretrained video knowledge into robot control and expand robot capabilities through video-based learning.

Acknowledgments

This work was supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019-II190075, Artificial Intelligence Graduate School Program (KAIST)). National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00256259), and the “Advanced GPU Utilization Support Program” funded by the Government of the Republic of Korea (MSIT). We thank Shusheng Yang and NYU VisionX members for providing the paper template.

Appendix

Model architecture details

Video backbone adaptation

We fine-tune the pretrained video diffusion backbone [25] on robot videos to adapt it to the robotics domain and our multi-view packing format. Due to computational constraints, we do not use additional external video data in this stage. The adapted checkpoint initializes both the teacher and student of the same architecture. During visual-encoder training, the teacher remains frozen with bidirectional video attention and text cross-attention on all frames. The student uses the modifications below. The VAE and text encoder remain frozen throughout adaptation and visual-encoder training.

Visual encoder

Video DiT Following Fast-WAM [30], we prevent current observation tokens from attending to future latent tokens. To encourage the current observation representation to capture planning-related information, we restrict each future latent frame to attend only to the current observation frame and itself, and disable text cross-attention for future latent tokens. Spatial attention is preserved within each frame.

Unlike Wan2.2-TI2V-5B, Wan2.2-I2V-A14B uses a separate image-conditioning mechanism, where a sequence consisting of the first image followed by zero-valued frames is VAE-encoded. To adopt the same first-frame conditioning strategy as TI2V-5B, we replace the first frame of the noisy video-latent stream with its clean latent and assign it timestep zero.

Projector To match the teacher’s spatiotemporal token sequence length, the projector expands NN current observation tokens to FNFN tokens, where FF is the number of latent frames in the teacher’s future representation. The student and teacher use the same spatial token grid. The student tokens are replicated FF times and augmented with learned frame embeddings, then fed into a two-layer decoder as queries. Each decoder layer applies self-attention, cross-attention to the original student tokens, and a 1024 →\to 4096 →\to 1024 GELU MLP, with pre-normalization and residual connections in each sublayer.

Student and teacher features are extracted after complete Transformer blocks ii and jj, respectively. To supervise future positions only, we exclude current observation tokens from both projector outputs and teacher targets. The loss is the mean cosine distance over the remaining token pairs after non-affine feature-wise LayerNorm (ϵ=10−6\epsilon= 10^{-6}).

Action readout module

During action-module training, the visual encoder is frozen. The readout and conditioning layers are trained from scratch. The readout receives current observation tokens from encoder block kk, independently selected from distillation block ii, without passing through the alignment projector. To obtain a fixed-length visual context for the action decoder, we use 64 learned queries updated by two blocks. Each decoder layer applies self-attention, cross-attention to the projected visual tokens, and a 1024 →\to 4096 →\to 1024 GELU MLP. All sublayers use pre-normalization and residual connections. Output LayerNorm is followed by a separate 1024 →\to 1024 conditioning projection and affine LayerNorm. These visual tokens are appended to the projected instruction tokens as decoder cross-attention context.

State and action encoders

Separate linear projections map the normalized proprioceptive state and each noisy action to the transformer’s embedding dimension. The state contributes one prefix token, replaced with a learned mask token with probability 0.2 during training.

Algorithm 1 DREAMFREE training

Require: Data Dv\mathcal{D}_v, Da\mathcal{D}_a; fixed teacher noise tdt_d

  1. Adapt video backbone on Dv\mathcal{D}_v; initialize gg and uθu_\theta.

  2. Freeze gg; initialize projector pψp_\psi.

  3. Stage 1: Visual encoder

  4. while training do

  5. Sample (x∗,ℓ)∼Dv(x_*, \ell) \sim\mathcal{D}_v and t,ϵ,ϵdt, \epsilon, \epsilon_d.

  6. xt←(1−t)x∗+tϵx_t \leftarrow(1-t)x_* + t\epsilon; xt0←x0x^0_t \leftarrow x^0.

  7. (v^,hi)←uθ(xt,t,ℓ)(\hat{v}, h_i) \leftarrow u_\theta(x_t, t, \ell). {hi\{h_i: current frame}\}

  8. xtd←(1−td)x∗+tdϵdx_{t_d} \leftarrow(1-t_d)x_* + t_d\epsilon_d.

  9. y←[gj(xtd,td,ℓ)]+y \leftarrow[g_j(x_{t_d}, t_d, \ell)]^+. {+\{+: future tokens only}\}

  10. Lgen←w(t)MSE⁡(v^,(ϵ−x∗)+)\mathcal{L}_{\mathrm{gen}} \leftarrow w(t)\operatorname{MSE}(\hat{v}, (\epsilon-x_*)^+).

  11. Ldistill←−sim⁡([pψ(hi)]+,y)\mathcal{L}_{\mathrm{distill}} \leftarrow-\operatorname{sim}([p_\psi(h_i)]^+, y).

  12. Update θ,ψ\theta,\psi with Lgen+Ldistill\mathcal{L}_{\mathrm{gen}}+\mathcal{L}_{\mathrm{distill}}.

  13. end while

  14. Stage 2: Action module

  15. Freeze uθu_\theta; initialize the action module.

  16. while training do

  17. Sample (x0,ℓ,q,s,a∗)∼Da(x^0, \ell, q, s, a_*) \sim\mathcal{D}_a and t,ϵt,\epsilon.

  18. hk←[uθ(x0,0,ℓ)]kh_k \leftarrow[u_\theta(x^0,0,\ell)]_k.

  19. c←rω(LN⁡(hk))c \leftarrow r_\omega(\operatorname{LN}(h_k)). {Action readout}

  20. at←(1−t)a∗+tϵa_t \leftarrow(1-t)a_*+t\epsilon.

  21. v^a←uϕ(at,t∣c,q,s,ℓ)\hat{v}_a \leftarrow u_\phi(a_t,t\mid c,q,s,\ell).

  22. Lact←w(t)MSE⁡(v^a,ϵ−a∗)\mathcal{L}_{\mathrm{act}} \leftarrow w(t)\operatorname{MSE}(\hat{v}_a,\epsilon-a_*).

  23. Update the action module with Lact\mathcal{L}_{\mathrm{act}}.

  24. end while

Algorithm 2 DREAMFREE inference

Require: Instruction ℓ\ell, steps NN, execution length RR

  1. {tn}n=0N←Schedule⁡(N)\{t_n\}_{n=0}^{N} \leftarrow\operatorname{Schedule}(N).

  2. Initialize context s←∅s \leftarrow\varnothing.

  3. while task not done do

  4. (o,q)←Observe⁡(); x0←VAE⁡(o)(o,q) \leftarrow\operatorname{Observe}();\ x^0 \leftarrow\operatorname{VAE}(o).

  5. hk←[uθ(x0,0,ℓ)]kh_k \leftarrow[u_\theta(x^0,0,\ell)]_k. {One single pass}

  6. c←rω(LN⁡(hk))c \leftarrow r_\omega(\operatorname{LN}(h_k)).

  7. Sample at0∼N(0,I)a_{t_0} \sim\mathcal{N}(0,I).

  8. for n=0,…,N−1n=0,\ldots,N-1 do

  9. v^n←uϕ(atn,tn∣c,q,s,ℓ)\hat{v}_n \leftarrow u_\phi(a_{t_n},t_n\mid c,q,s,\ell).

  10. atn+1←atn+(tn+1−tn)v^na_{t_{n+1}} \leftarrow a_{t_n}+(t_{n+1}-t_n)\hat{v}_n.

  11. end for

  12. Postprocess atNa_{t_N}; execute its first RR actions.

  13. if history enabled then {DROID}

  14. Update action history in ss.

  15. end if

  16. end while

Action decoder

The decoder uses 30 DiT blocks, hidden width 1024, feed-forward width 4096, and 24 attention heads of dimension 128, totaling approximately 1B parameters. Each block contains bidirectional self-attention over state/history/action tokens and cross-attention to instruction and visual context from the action readout module. Self-attention uses one-dimensional rotary position embeddings and padding masks. A 256-dimensional sinusoidal timestep embedding and an MLP provide block modulation. A final linear layer predicts the flow-matching velocity in action space, and prefix outputs are discarded, and loss is computed only at valid future-action positions.

For DROID, we include a history of the previous 8 actions to address unintended gripper reopening during closed-loop execution. History uses the same linear encoder as noisy actions. The input order is current state, action history, then noisy future actions. Missing history entries are masked, and the entire history is independently dropped with probability 0.2 during training. State and history remain noise-free, with a fixed timestep corresponding to σ=10−3\sigma=10^{-3}.

Training and inference algorithms

Algorithms 1 and 2 summarize training and synchronous closed-loop inference. We write x∗x_* for VAE-encoded video latents, with superscripts 0 and + selecting current observation and future positions. Video-model calls also include the Wan-A14B image-conditioning channels when applicable. sg denotes stop-gradient, LN is non-affine token-wise LayerNorm, and sim is mean cosine similarity after this normalization. States and actions are normalized. ss denotes optional action history, used only for DROID.

Here, ttrain=5r/(1+4r)t_{\mathrm{train}}=5r/(1+4r) with r∼U(0,1)r\sim\mathcal{U}(0,1), and w(t)w(t) is the native scheduler’s timestep weight. Losses are averaged over the batch. MSEm_{\mathrm{m}} excludes padded action positions. The implemented cosine loss 1−sim⁡1-\operatorname{sim} differs from −sim⁡-\operatorname{sim} only by an additive constant. Teacher conditioning keeps the first frame clean for 5B, and Wan-A14B retains a noised first frame in the video stream and separate clean image conditioning.

Implementation details

Baselines

For simulation benchmarks, we use reported baseline scores except for Fast-WAM on RoboCasa-Kitchen, which we train and evaluate ourselves.

Fast-WAM [30] For RoboCasa-Kitchen, we train Fast-WAM 5B using the official implementation with learning rate 10−410^{-4} for 20 epochs and global batch size 128. The 14B baseline uses the Wan2.2-A14B high-noise expert with global batch size 32. We modify the conditioning on the current observation frame to match that of our method.

DreamFree

We use AdamW with a learning rate of 10−410^{-4} for both training stages. For LIBERO and Robocasa models, we use rank-128 LoRA to train the visual encoders initialized from the video diffusion models. For the 14B DROID model, we fully train the weights without using LoRA. Input resolutions are 224×448224 \times448 for LIBERO, 352×352352 \times352 for RoboCasa, and 352×640352 \times640 for DROID. We set the teacher noise level to td=0.7t_d = 0.7 for simulation benchmarks and 0.8 for DROID.

The layer indices used for our distillation loss and action readout are fixed for each backbone size across all datasets and tasks. For the 5B backbone, we extract student features for distillation at i=25i = 25 and features for action readout at k=19k = 19. For the 14B backbone, we use i=j=k=31i = j = k = 31. All indices are zero-based, and features are extracted after the corresponding complete Transformer block.

For LIBERO, we use global batch sizes of 128 and 64 in stages 1 and 2, respectively. Stage 1 runs for 40 epochs, and stage 2 for 10 epochs. For RoboCasa 5B, we use global batch sizes of 64 and 256 in stages 1 and 2, respectively, for four and 20 epochs. For RoboCasa 14B, we use global batch sizes of 32 and 512 in stages 1 and 2, respectively. Stage 1 runs for 10 epochs on RoboCasa-only videos or two epochs with additional videos, and stage 2 runs for 20 epochs. For DROID, we fully train the visual encoder in stage 1 without LoRA. We use global batch sizes of 64 and 4,096 in stages 1 and 2, respectively, for 6 and 16 epochs. Training across both stages took approximately one week on 16 NVIDIA H200 GPUs.

For ABC-130k [1], we train our model on a subset of approximately 100 hours from the full 3,500-hour dataset due to computational constraints. The subset is selected to cover diverse tasks and operators. We use the 14B backbone with an input resolution of 432×384432 \times384. For the bimanual setup, the overhead view occupies the upper two-thirds of the input, while the left and right wrist views are arranged side by side in the lower third. We follow the DROID 14B optimization settings, including global batch sizes of 64 and 4,096 for stages 1 and 2, respectively.

Inference The visual encoder runs once at timestep zero, without video denoising or classifier-free guidance (CFG). Actions are initialized from Gaussian noise and sampled with 10 Euler steps for LIBERO and RoboCasa, and 16 steps for DROID, using a flow-matching schedule with shift 5. Action sampling also omits CFG, requiring one conditional decoder evaluation per step. From each 32-action chunk, we execute 10 actions in LIBERO, 32 in RoboCasa, and 24 in DROID before replanning.

Experimental details

Simulation benchmarks

LIBERO We use the same multi-view frame packing setup as in the official Fast-WAM implementation. We report the average success rate across the four LIBERO suites, evaluated over a total of 2,000 trials covering 40 tasks with different random seeds. The baseline results are taken from Yuan et al. [30].

Robocasa-Kitchen We use the Cosmos Policy [15] dataset with 50 human-teleoperated demonstrations per task. We train and evaluate DreamFree and Fast-WAM on this benchmark and use reported scores for the remaining baselines. Following DreamZero [28], we pack three camera views into a single frame, with the wrist view stretched horizontally by 2× above two side-by-side external views. Each of the 24 tasks is evaluated over 50 trials. The baseline results are taken from Kim et al. [15].

For our additional-video experiments, we use a filtered and converted version of the RoboCasa365 human-teleoperation atomic-task dataset. The source dataset covers 65 atomic tasks spanning ten skill families. Each atomic task involves one basic skill executed over multiple timesteps, whereas composite tasks combine several skills. Demonstrations use a Panda arm mounted on an Omron mobile base, with synchronized left, right, and wrist camera videos. We use the additional videos for backbone adaptation and visual-encoder training without action supervision. Action-module training uses only the Cosmos Policy demonstrations.

Real-world experiments

Tasks and evaluation We evaluate on 10 tasks from the pretrained distribution, performing 5 rollouts per task, for a total of 50 evaluation rollouts for each model. Table 4 describes the ten evaluation tasks, and Table 5 describes their progress scores. Following Barreiro et al. [3], we apply an image overlay to the initial scene to reduce variance.

TaskInstructionSetup and objective
Stand bottle upright“Stand the bottle upright.”A bottle lies on its side. Stand it on its base and release it.
Fold glove“Fold the glove.”A glove lies unfolded. Make a regular fold that turns back more than one-third of its original length and remains after release.
Cover strawberry with towel“Use the towel to cover the strawberry.”A toy strawberry and grape lie on opposite sides of a towel. Lift the towel and release it over the strawberry, fully covering it while leaving the grape uncovered.
Put red marker in blue cup“Put the red marker inside the blue cup.”Red and black markers lie beside blue and green cups. Release the red marker inside the upright blue cup; the black marker and green cup are distractors.
Move cup from bowl to cup“Pick up the cup from the bowl and put it in the other cup.”One cup sits inside a bowl, with another cup and a distractor can nearby. Remove the source cup and release it inside the other cup.
Press pump“Press the pump.”An upright pump bottle stands beside a cup holding a toothbrush. Depress the pump head while keeping the bottle upright.
Put bolts in bowl“Put the bolts in the bowl.”Three bolts lie separately beside a bowl. Place and release all three inside the bowl.
Pour cup into pan“Pour the contents of the cup into the pan.”A source cup and receiving pan sit on the table. Transfer the cup’s contents into the pan.
Hang hat on tripod“Hang the hat on the tripod.”A hat lies beside an upright tabletop tripod. Place and release the hat so that the tripod supports it.
Put ring around pole“Put the ring around the pole.”A ring lies beside an upright pole on a base. Place and release the ring with the pole passing through its hole.

Table 4. Real-world evaluation tasks on DROID.

TaskScoring criteria
Stand bottle upright0.5: bottle axis exceeds 45° above horizontal; upright is 90°. 1: bottle stands on its base after release.
Fold glove0: no fold. 0.5: a nonzero regular fold turns back at most one-third of the original glove length, or an irregular fold occurs, regardless of folded length. 1: a regular fold turns back more than one-third of the original length and remains after release.
Cover strawberry with towelRequires lifting, then partial coverage, then full coverage. 1/3: towel lifted. 2/3: any partial strawberry coverage after lifting, including slight coverage. 1: strawberry fully covered after towel release, with the grape uncovered. Coverage without first lifting does not qualify.
Put red marker in blue cup0.25: red marker picked up. 0.5: red marker contacts the blue cup. 0.75: marker enters the cup, but the cup falls or tips over. 1: marker released inside the upright blue cup. Picking up the black marker before picking up the red marker makes the entire rollout score 0, even after later completion; incidental contact does not trigger this penalty.
Move cup from bowl to cup1/3: correct source cup picked up. 2/3: source cup contacts the receiving cup without successful insertion. 1: source cup inserted and released inside the receiving cup.
Press pump0.5: contact with the movable pump head. 1: pump head depressed while the bottle remains upright. Spring return after release is allowed.
Put bolts in bowl1/3 per bolt placed and released inside the bowl. Retain intermediate credit for the highest count reached; 1 requires all three bolts to remain inside at the end.
Pour cup into pan0.5: source cup picked up. 1: its contents transferred into the pan.
Hang hat on tripod1/3: hat picked up. 2/3: hat contacts the tripod. 1: hat remains supported by the tripod after release.
Put ring around pole0.5: ring picked up. 1: ring placed and released with the pole passing through its hole.

Table 5. Task-progress rubrics for real-world evaluation tasks on DROID.

Latency Although single-pass visual encoding without iterative video denoising or CFG already makes DreamFree faster than DreamZero, we further reduce latency through engineering optimizations. These include VAE encoding of only the current frame, fused attention kernels, and fused QKV/KV projections. Visual and action Transformer blocks are compiled with torch.compile, and CUDA graphs replay VAE encoding, visual feature extraction, and the action sampling loop to reduce launch overhead. Instruction embeddings are cached across queries, while action-conditioning keys and values are computed once per chunk and reused across denoising steps.

We measure latency on a single NVIDIA H200 after startup warm-up, including client preprocessing, server policy execution, and client postprocessing, and excluding communication and robot execution.

Regarding task completion time, we use three rollouts per policy and block-count setting. We estimate each rollout’s duration by adding nominal robot execution time to its total policy computation time. Communication and setup overhead are excluded.

Task transfer from video demonstrations

We provide task descriptions in Table 6. In the first two experiments, target task videos are included during visual encoder pretraining, before training the action module on atomic tasks. In the third experiment, target task demonstrations are excluded from initial policy training, and target task videos are used only for subsequent adaptation. In all three experiments, the target task demonstrations contain no action annotations.

SettingTaskDescription and ordered subgoals
Atomic-to-compositeArrange vegetablesTransfer the first vegetable from the sink to the counter → transfer the second vegetable from the sink to the counter.
Atomic-to-compositeMicrowave thawingOpen the microwave door → transfer the object from the counter into the microwave → close the door → turn on the microwave.
Atomic-to-compositeRestock pantryTransfer the first object from the counter into the cabinet → transfer the second object from the counter into the cabinet.
Cross embodimentPlace can in basketPlace the can inside the basket → lift the basket. Can placement alone constitutes partial completion.
Cross embodimentPlace cans in boxPlace the first can inside the plastic box → place the second can inside the box.
Cross embodimentStack three bowlsEstablish the base bowl → stack the second bowl on the base → stack the third bowl on the stack.

Table 6. Task descriptions and ordered subgoals for task transfer from video demonstrations. Subgoals are evaluated in the listed order.

Atomic-to-composite task transfer from video pretraining We include videos of three target RoboCasa composite tasks from the same embodiment during visual encoder pretraining. We merge the existing LoRA weights into the encoder, freeze the merged weights, and train fresh rank-128 adapters and the alignment projector on the target task videos at a learning rate of 10−410^{-4}. After encoder pretraining, we freeze the encoder and train the action module using only action-labeled demonstrations of atomic tasks. Across all three settings, the instruction remains fixed throughout each rollout, without intermediate subgoal prompts. Progress is the fraction of subgoals achieved in order.

Cross-embodiment task transfer from video pretraining We include videos of three target composite tasks recorded with dual-arm Franka Panda during visual encoder pretraining. We use 50 video demonstrations per task, totaling 150 demonstrations. We then freeze the encoder and train the action module using only ALOHA-AgileX atomic-task demonstrations. We evaluate the three target tasks on ALOHA-AgileX, testing whether task knowledge learned from videos of a different embodiment transfers to action execution.

Task adaptation after policy training We start from a policy trained on RoboTwin atomic tasks, with target task demonstrations excluded from both visual encoder pretraining and action-module training. We then fine-tune only the visual encoder separately for each target task using 10 video demonstrations, while keeping the trained action module frozen. We compare fine-tuning with multiple camera views and with only the head camera: basket placement uses 60 and 40 iterations, respectively, and bowl stacking uses 100 and 80. Both variants are evaluated in RoboTwin using all three camera views.

Additional results

LIBERO per-suite results

Table 7 reports per-suite success rates for DreamFree on LIBERO.

MethodSpatialObjectGoalLongAverage
LingBot-VA [16]98.599.697.298.598.5
Fast-WAM [30]98.2100.097.095.297.6
DreamFree (Ours)99.099.898.896.698.6

Table 7. LIBERO success rates (%) of DreamFree across the four suites.

Robocasa kitchen per-task results

Table 8 reports per-task success rates for the 14B model trained with additional video data.

TaskSR (%)TaskSR (%)
CloseDoubleDoor84PnPCounterToSink72
CloseDrawer98PnPCounterToStove74
CloseSingleDoor98PnPMicrowaveToCounter58
CoffeePressButton96PnPSinkToCounter76
CoffeeServeMug72PnPStoveToCounter72
CoffeeSetupMug38TurnOffMicrowave90
OpenDoubleDoor96TurnOffSinkFaucet86
OpenDrawer88TurnOffStove18
OpenSingleDoor90TurnOnMicrowave94
PnPCabToCounter56TurnOnSinkFaucet76
PnPCounterToCab62TurnOnStove52
PnPCounterToMicrowave50TurnSinkSpout90
Mean over 24 tasks74.4

Table 8. RoboCasa-Kitchen success rates (%) for DREAMFREE with the 14B backbone and additional video data. Each task is evaluated over 50 trials.

Failure case analysis

Real-world manipulation Although DreamFree improves over DreamZero on tasks involving deformable objects and successive rearrangements, pump pressing remains a failure case. The policy contacts the pump head in all five trials but completes the task in only one. We hypothesize that this reflects difficulty generalizing the motion required to actuate a new object, even when the target is reached. Whether broader training coverage of such interactions improves this generalization remains a direction for future work.

Task transfer The cabinet insertion task requires opening the cabinet before placing an object inside. The base policy did not reliably perform the opening motion, and video-only fine-tuning did not enable it to open. This failure suggests that transferring task knowledge from videos may depend on the availability of related action skills in the base policy. With the action module frozen, adapting the visual encoder alone may be insufficient to acquire a new motor skill. We leave it to future work to test whether scaling pretraining with more diverse action demonstrations alleviates this limitation.

Further discussions

Connection to prior works

Comparison with Fast-WAM [30] Fast-WAM uses a pretrained video diffusion backbone for direct action prediction, retaining video co-training but omitting future-video denoising at deployment. The key difference is that DREAMFREE directly supervises the policy features with future representation distillation in addition to the video generative objective. In our RoboCasa-Kitchen experiments (Table 2), Fast-WAM substantially trails the video-generating WAM baseline and shows little improvement from a 5B to a 14B backbone (44.4% to 44.7%). These results suggest that generative supervision through shared weights alone does not ensure that a larger VDM provides more useful predictive features for control. DREAMFREE supports learning and transfer from videos without action labels, while achieving performance and generalization competitive with video-generating WAMs at substantially lower inference latency.

Knowledge Insulation [8] Freezing the visual encoder in stage 2 prevents gradients from the action DiT from updating the video backbone. Knowledge Insulation and LAP [31] similarly block gradients from the action expert while training VLM representations through FAST-token prediction and language-action prediction, respectively. Our method applies a similar separation to a VDM, using the native video generative objective and future representation distillation to shape the visual features. Their concurrent training also suggests an alternative to our two-stage procedure: jointly training the encoder and action module with their respective objectives, while blocking action-module gradients from the encoder. We leave this variant to future work, as we have not evaluated it due to computational constraints.

Limitations

Although DREAMFREE uses video-model priors to predict actions directly like VLAs, achieving lower latency than video-generating WAMs, several limitations remain.

  • Scaling with embodied pretraining. We demonstrate that a 14B video backbone trained on DROID without embodied pretraining can outperform existing VLA and WAM baselines. Although the RoboCasa results in Table 2 suggest that additional video data could further improve performance,

we did not evaluate this extension on DROID due to computational constraints. Scaling to larger backbones, datasets, and embodiments remains an important direction for future work.

  • Training complexity. Our pipeline separates backbone adaptation, predictive representation learning, and action learning. Simplifying these stages and exploring joint training of the visual encoder and action module remain directions for future work.

References

  1. [1]Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, et al. Scalable behavior cloning with open data, training, and evaluation. arXiv preprint arXiv:2606.27375, 2026.arxiv.org/abs/2606.27375
  2. [2]Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.
  3. [3]Jose A. Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, Naveen Kuppuswamy, Kuan-Hui Lee, Katherine Liu, Dale McConachie, Ian McMahon, Haruki Nishimura, Calder Phillips-Grafflin, Charles Richter, Paarth Shah, Krishnan Srinivasan, Blake Wulfe, Chen Xu, Mengchao Zhang, Alex Alspach, Maya Angeles, Kushal Arora, Vitor Campagnolo Guizilini, Alejandro M. Castro, Dian Chen, Ting-Sheng Chu, Sam Creasey, Sean Curtis, Richard Denitto, Emma Dixon, Eric Dusel, Matthew Ferreira, Aimee Goncalves, Grant Gould, Damrong Guoy, Swati Gupta, Xuchen Han, Kyle Hatch, Brendan Hathaway, Allison Henry, Hillel Hochsztein, Phoebe Horgan, Shun Iwase, Donovan Jackson, Siddharth Karamcheti, Sedrick Keh, Joseph Masterjohn, Masayuki Masuda, Jean Mercat, Patrick Tree Miller, Paul Mitiguy, Tony Nguyen, Jeremy Nimmer, Yuki Noguchi, Reko Ong, Aykut Özgun Önol, Owen Pfannenshtiel, Richard Poyner, Leticia Priebe Rocha, Gordon Richardson, Christopher Rodriguez, Derick Seale, Michael A. Sherman, Mariah Smith-Jones, David Tago, Pavel Tokmakov, Matthew Tran, Basile Van Hoorick, Igor Vasiljevic, Sergey Zakharov, Mark Zolotas, Rares Ambrus, Kerri Fetzer-Borelli, Benjamin Burchfiel, Hadas Kress-Gazit, Siyuan Feng, Stacie Ford, and Russ Tedrake. A careful examination of large behavior models for multitask dexterous manipulation. Sci. Robotics, 11(113), 2026.DOI
  4. [4]Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, et al. Motus: A unified latent action world model. In IEEE Conference on Computer Vision and Pattern Recognition, 2026.arxiv.org/abs/2512.13030
  5. [5]Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.arxiv.org/abs/2503.14734
  6. [6]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. In Proceedings of Robotics: Science and Systems, 2025.DOI
  7. [7]Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan ang Gao, Kaixuan Wang, Zhixuan Liang, Yusen Qin, Xiaokang Yang, Ping Luo, and Yao Mu. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. In International Conference on Machine Learning, 2026.arxiv.org/abs/2506.18088
  8. [8]Danny Driess, Jost Tobias Springenberg, Brian Ichter, Lili Yu, Adrian Li-Bell, Karl Pertsch, Allen Z. Ren, Homer Walke, Quan Vuong, Lucy Xiaoyang Shi, and Sergey Levine. Knowledge insulating vision-language-action models: Train fast, run fast, generalize better. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.23705
  9. [9]Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, et al. Dreamdojo: A generalist robot world model from large-scale human videos. arXiv preprint arXiv:2602.06949, 2026.arxiv.org/abs/2602.06949
  10. [10]Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. In International Conference on Machine Learning, 2025.arxiv.org/abs/2412.14803
  11. [11]Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, 2021.arxiv.org/abs/2103.03206
  12. [12]Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024.arxiv.org/abs/2403.12945
  13. [13]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning, 2024.arxiv.org/abs/2406.09246
  14. [14]Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025.arxiv.org/abs/2502.19645
  15. [15]Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, and Jinwei Gu. Cosmos policy: Fine-tuning video models for visuomotor control and planning. In International Conference on Learning Representations, 2026.arxiv.org/abs/2601.16163
  16. [16]Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot control. arXiv preprint arXiv:2601.21998, 2026.arxiv.org/abs/2601.21998
  17. [17]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. In Neural Information Processing Systems Datasets and Benchmarks Track, 2023.arxiv.org/abs/2306.03310
  18. [18]Teli Ma, Jia Zheng, Zifan Wang, Chunli Jiang, Andy Cui, Junwei Liang, and Shuo Yang. Dit4dit: Jointly modeling video dynamics and actions for generalizable robot control. arXiv preprint arXiv:2603.10448, 2026.arxiv.org/abs/2603.10448
  19. [19]Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523, 2024.arxiv.org/abs/2406.02523
  20. [20]Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri, and Yuke Zhu. Robocasa365: A large-scale simulation framework for training and benchmarking generalist robots. In International Conference on Learning Representations, 2026.arxiv.org/abs/2603.04356
  21. [21]Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, and Elvis Nava. mimic-video: Video-action models for generalizable robot control beyond vlas. arXiv preprint arXiv:2512.15692, 2025.arxiv.org/abs/2512.15692
  22. [22]William S. Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE International Conference on Computer Vision, 2023.arxiv.org/abs/2212.09748
  23. [23]Physical Intelligence. π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.arxiv.org/abs/2504.16054
  24. [24]Rhoda AI. Causal video models are data-efficient robot policy learners. Rhoda AI Blog, 2026. URL https://www.rhoda.ai/research/direct-video-action.
  25. [25]Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.arxiv.org/abs/2503.20314
  26. [26]Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations, 2024.arxiv.org/abs/2312.13139
  27. [27]Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Hongtao Li, Jie Li, Jindi Lv, Jingyu Liu, et al. Gigaworld-policy: An efficient action-centered world-action model. arXiv preprint arXiv:2603.17240, 2026.arxiv.org/abs/2603.17240
  28. [28]Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chunung Zhu, Jiannan Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026.arxiv.org/abs/2602.15922
  29. [29]Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.06940
  30. [30]Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026.arxiv.org/abs/2603.16666
  31. [31]Lihan Zha, Asher J Hancock, Mingtong Zhang, Tenny Yin, Yixuan Huang, Dhruv Shah, Allen Z Ren, and Anirudha Majumdar. Lap: Language-action pre-training enables zero-shot cross-embodiment transfer. arXiv preprint arXiv:2602.10556, 2026.arxiv.org/abs/2602.10556
  32. [32]Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, Xinqiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, He Wang, Zhizheng Zhang, et al. Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2507.04447
  33. [33]Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In IEEE Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2503.22020
  34. [34]Ruijie Zheng, Jing Wang, Scott Reed, Johan Bjorck, Yu Fang, Fengyuan Hu, Joel Jang, Kaushil Kundalia, Zongyu Lin, Loic Magne, et al. Flare: Robot learning with implicit world modeling. arXiv preprint arXiv:2505.15659, 2025.arxiv.org/abs/2505.15659
  35. [35]Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets. In Proceedings of Robotics: Science and Systems, 2025.arxiv.org/abs/2504.02792
  36. [36]Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, 2023.arxiv.org/abs/2307.15818

Paper details

vlnc:W_rjuspb23
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents