WorksComputer Science
W2A: Distilling World Priors into Action Models for Real-Time Robotic Control
Abstract
World Action Models (WAMs) have recently shown great promise in robotic control, benefiting from the future imagination capability of large-scale pretrained video models. However, most existing WAMs require explicitly imagining future states at test time, incurring substantial computational overhead and latency that make them unsuitable for real-time robotic control. In this paper, we propose W2A, a two-stage distillation framework for real-time robotic manipulation that distills intermediate denoising representations from a video diffusion model into an action model during training, while entirely discarding the video model at test time. We hypothesize that although video latent representations are high-dimensional, a compressed low-dimensional subspace is sufficient to provide effective supervision for action generation. This makes it possible to transfer such information effectively into an action model. Extensive experiments in both simulation and real-world settings demonstrate competitive success rates while significantly improving inference efficiency. On LIBERO, W2A achieves a 95.9% average success rate. More importantly, W2A achieves about 21x faster inference than the representative WAM baseline.
Introduction
Building general robotic policies is a central goal on the path toward artificial general intelligence. Built upon internet-scale pretrained Vision-Language Models (VLMs), Vision-Language-Action models (VLAs) have emerged as a promising paradigm for robotic control, offering strong semantic grounding and generalization [1, 2, 3, 4, 5]. However, because VLAs are largely trained on static image-text pretraining, they still have limited understanding of physical dynamics and procedures, both of which are crucial for long-horizon and contact-rich robotic manipulation.
To address this limitation, recent works [6, 7, 8, 9, 10] have begun to integrate video-based world modeling into robot policy learning, giving rise to World Action Models (WAMs). By leveraging future visual prediction or latent representations, WAMs incorporate richer temporal and physical structure. Existing WAMs can be broadly categorized into two classes: (1) video-conditioned action modeling and (2) joint modeling of future states and actions. The former injects predicted future observations or latent representations as conditions into an action expert for action generation [6, 7]. The latter, in contrast, jointly models future observations and actions during training, generating both within a unified framework [11, 12]. However, most existing WAMs require explicitly imagining future states at inference time, incurring substantial computational overhead and inference latency that make them unsuitable for real-time robotic control.
In this paper, we refer to world priors as the latent knowledge of video models about how scenes evolve over time. We focus on two questions: (1) what world priors in the video models used by WAMs are truly important for action generation, and whether they can be captured by a low-dimensional latent subspace? (2) Can such essential world priors be distilled into an action model, thereby eliminating the additional inference overhead and enabling real-time robotic control? Our key insight is that the latent representations of video models contain much information that is not directly useful for action generation, such as static scene content and task-irrelevant visual details, while only a small subset of features truly captures action-relevant dynamics. If this critical knowledge can be distilled into an action model, the video model can be removed at inference time.
Motivated by this insight, we propose W2A, a two-stage distillation framework that transfers world priors from video models into action models for real-time robotic control. Specifically, in the first stage of training, we adapt a pretrained language-conditioned video diffusion transformer (DiT) [13] to robotic video data. We then apply principal component analysis (PCA) [14] to the latent representations of the video DiT, projecting them into a lower-dimensional latent space. In the second stage, we train an action DiT from scratch as our action model. For action DiT training, in addition to a flow matching objective [15], we introduce a Q-Former [16] to read out features from the action DiT. These queried features are then aligned with the projected video latents through a similarity-based distillation loss. At test time, our framework removes the video DiT entirely, retaining only the action DiT. We believe that although video latents and action latents are not directly causal variables, they share the same action-conditioned dynamics: one captures the resulting world evolution, while the other must encode the control patterns that induce it. This shared structure makes distilling world priors from video models into action models both practically and conceptually plausible. Figure 1 shows the differences between W2A and the existing WAMs.
Figure 1. Overview of W2A compared with mainstream world action models. While existing WAMs usually keep a video branch at test time, W2A distills world priors from a video DiT into an action DiT during training and removes the video DiT entirely at test time. This yields competitive performance in simulation and real-world manipulation while substantially improving inference efficiency.
Our experimental results suggest that world priors useful for action generation can be effectively distilled into action models. This allows W2A to preserve the benefits of WAMs and improve the performance of action models, while avoiding the deployment overhead of video models.
Related Work
Vision-Language-Action Models. Vision-Language-Action models extend large VLMs from semantic understanding to embodied control, and have become a major direction for generalist robotic policies [17, 18, 19, 20, 21, 22, 23, 24, 25]. Early milestones such as RT-1 [26] and RT-2 [27] established the paradigm of mapping visual observations and language instructions to robot actions, while demonstrating that large-scale internet pretraining can significantly improve semantic grounding and task generalization in manipulation. Subsequent work shifted from task-specific instantiations toward more general and scalable policy learning. In particular, Octo [28] explored large-scale multi-task robot pretraining, and OpenVLA [1] further advanced this line by providing an open and reproducible VLA framework for general-purpose manipulation. More recent robot foundation models, including [3], [29], and GR00T N1 [4], continue to push the frontier by scaling data diversity, improving cross-embodiment transfer, and strengthening real-world adaptability [19]. Despite these advances, VLAs are still largely rooted in static image-text pretraining, which limits their ability to reason about how actions change the world over time.
World Action Models and Video Generation for Policy Learning. Recent progress in video generation has motivated new training paradigms that bring pretrained video world models into robotic control [10, 30, 31, 32, 33, 34, 35, 36, 37]. An early representative work is Video Prediction Policy (VPP) [6], which showed that video diffusion models pretrained for future prediction can provide predictive visual representations that substantially improve downstream policy learning. Mimic-video [7] retains the two-stage recipe of first adapting a video model and then training an action decoder, further demonstrating that video models can serve as strong spatio-temporal backbones for robot action generation. A different line pushes this integration further by reducing or removing the gap between world modeling and policy learning. Cosmos Policy [12] directly adapts a pretrained video model into a robot policy with minimal architectural changes, jointly modeling actions, future states, and values within the latent diffusion process. More recent works such as LingBot-VA [38], DreamZero [39], and GigaWorld-Policy [40] further strengthen this trend by unifying future-state prediction and action generation within autoregressive or joint-denoising formulations. However, most WAMs still retain a video backbone or world-modeling branch at test time, introducing additional computational overhead and inference latency for real-time control. W2A addresses this issue by using the video model only as a training-time teacher, distilling its world priors into an action model and discarding the video model entirely during deployment.
Preliminaries
Problem Statement
We consider language-conditioned robotic manipulation from demonstrations. Let denote the current observation, the task description, and an action chunk of horizon . A standard action policy directly predicts
Many existing WAMs introduce an additional video model. Let denote imagined future visual states generated by the video model. Their action generation can be written as
where and denote the probability distributions of video generation and action generation.
In contrast, our formulation entirely removes the video model at test time. During training, we use a teacher video model to provide intermediate latent representations , which serve as distillation targets for an action model. The deployed policy still follows the direct form , while its hidden representations are shaped by video-based world priors during training.
Flow Matching
We adopt the Flow Matching framework [15] to train the models. Let denote a Gaussian noise sample and let denote a target data sample. Following the flow matching parameterization, we define the interpolation path
Under this convention, corresponds to pure noise and corresponds to the data sample.
The action DiT is trained with a direct flow matching objective. Specifically, a conditional vector field is learned to predict the target velocity
by minimizing
At test time, samples are generated by integrating the learned vector field from to :
which in practice can be implemented with a backward Euler update
where . Moreover, the denoising time (or equivalently, the noise level) can be flexibly specified, a property that is directly exploited in our method to extract intermediate video latents at selected denoising stages as distillation targets.
Method
W2A is a two-stage distillation framework that distills world priors from a video DiT into an action DiT for real-time robotic control. In Stage I, we adapt a pretrained language-conditioned video DiT to robotic video data and compress its latent representations into a lower-dimensional space. In Stage II, we train an action DiT from scratch as our action model and augment its standard action modeling objective with a latent distillation objective, which aligns the action latents with the compressed video latents through a lightweight Q-Former bottleneck. At test time, the video DiT is entirely discarded, and only the action DiT is retained to generate actions conditioned on recent observations and task descriptions. The overall pipeline is illustrated in Figure 2.
Figure 2. The W2A training and inference framework.
Stage I: Video Domain Adaptation and PCA-Based Latent Compression
Stage I aims to obtain a compact video representation from a pretrained video DiT. We instantiate the video DiT as Cosmos-Predict2 [41], a large-scale language-conditioned video model, and adapt it to robotic video data via low-rank adaptation (LoRA) [42]. This reduces the domain gap between generic internet videos and robot interaction data, making the resulting intermediate latents more aligned with manipulation-specific dynamics and task-relevant interaction patterns.
Given the current observation and task description , the adapted video DiT produces latent representations along its denoising trajectory. We extract video latents from an intermediate denoising stage and layer of the video DiT. Concretely, for each sample we extract an intermediate teacher video latent , where denotes the number of temporal tokens, and denote the latent spatial resolution, and denotes the feature dimension. Let be the latent vector at temporal index and spatial location . We then fit PCA over all and obtain the empirical mean together with the top- principal directions . Each teacher video latent vector is then projected into the low-dimensional subspace as
Applying this projection to every spatio-temporal location yields the compressed . This reduces only the feature dimension, while preserving its spatio-temporal structure. The resulting provides compact video latents that are subsequently used as the distillation target in Stage II. The choices of the PCA dimension and the teacher latent extraction depth, together with their implementation settings, are discussed in Appendix § 9 and Appendix § 10.§ 1. To illustrate what information is preserved after PCA projection, we provide a visualization and discussion of raw and PCA-projected video latents in Appendix § 11.
Stage II: Action DiT Training and Latent Distillation with Q-Former
In Stage II, the action DiT is trained with a joint objective that combines the flow matching loss and a distillation loss from compact video latents. Similar to Stage I, we extract an action latent from an intermediate layer of the action DiT. Besides, we introduce a lightweight Q-Former [16] bottleneck with a set of learnable queries to extract from the features for alignment with . Formally, the queried student representation is
where denotes the Q-Former parameters and denotes the learnable queries.
To preserve the spatio-temporal structure of the teacher video latent, we organize the Q-Former outputs to match the layout of . Following the temporal compression ratio of the Cosmos-Predict2 VAE, we partition into temporal groups, each corresponding to one video temporal token in . For the -th group, a set of learnable spatial query tokens aggregates information from the action tokens in that group via cross-attention, producing an aligned action representation corresponding to the spatial grid of the compressed teacher video latent. Applying this operation across all groups yields , whose spatio-temporal layout matches . Details are provided in Appendix § 10.§ 4.
We use a cosine distillation loss to align the queried action latent with the compressed video latent:
where is the total number of aligned spatio-temporal positions, is the sampled flow time for the -th training sample, and is the Gaussian center corresponding to the flow time at which the video latents are extracted. The weighting term
modulates the distillation strength over action flow time, assigning larger weights to action latents whose flow time is closer to .
The final training objective is
where is the flow matching objective in § 3.2, and controls the strength of distillation.
Inference for Real-Time Control
At test time, W2A retains only the action DiT, so action generation no longer depends on any explicit video generation. This yields a fully video-free deployment graph and eliminates the additional computational overhead and inference latency introduced by the video model.
Given the current observation and task description , the action DiT predicts a future action chunk of length . Rather than executing the entire predicted sequence, the policy executes only the first actions, where , and then replans based on the updated observations.
To improve temporal continuity across replanning cycles, we adopt a warmstart mechanism [43] that reuses the unexecuted suffix of the previous action chunk as a reference for the next prediction. Rather than sampling from pure Gaussian noise, this cached trajectory is perturbed to an intermediate flow time and refined under the latest observation and task condition. This enables W2A to maintain short-horizon consistency while adapting to new observations. Overall, W2A achieves real-time control through video-free deployment, action chunking, and warmstarted replanning.
Experiments
We conduct comprehensive experiments in simulation and the real world to answer the following questions: (1) Does W2A improve policy performance? (2) What information is distilled from the video model into the policy? (3) How do key design choices contribute to the performance?
Experimental Setup
Simulated Tasks. We evaluate W2A on the simulated benchmarks LIBERO [44] and CALVIN [45], both of which are language-conditioned manipulation benchmarks. For LIBERO, we focus on four task suites: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, each containing 10 distinct tasks. For CALVIN, we focus on the challenging ABCD setting, where the policy is trained in environments A, B, and C, and evaluated in the unseen environment D. We follow the official evaluation protocol, evaluate with three random seeds, and report the average success rate.
Real-world Tasks. We conduct real-world experiments using a Franka Research 3 robot equipped with a front-view RealSense D435i camera and a wrist-view RealSense D405 camera. We consider three task types with increasing difficulty: (1) pick-and-place (PnP), (2) cube-stacking (Cube), and (3) long-horizon (Long). Each task type contains four subtasks. In addition, we design three challenging tasks for more practical real-world scenarios: Towel folding, Table clearing, and USB insertion. For training, we collect 50 teleoperated demonstrations for each subtask using an Omega 7 device, resulting in 750 real-world demonstrations in total. During evaluation, each subtask is tested with 30 consecutive rollouts, and we report the average success rate for each task type. Figure 3 shows several experimental scenarios.

Figure 3. Evaluation tasks
Main Results
Simulated Tasks. Table 1 and Table 2 report the results on LIBERO and CALVIN. On LIBERO, W2A achieves 95.9% average success rate and consistently outperforms other scratch-trained baselines. Moreover, it remains competitive with large finetuned foundation models, and even achieves the best result on LIBERO-Long (93.6%), surpassing . This suggests that distilled world priors are particularly helpful for long-horizon manipulation. On the CALVIN ABCD benchmark, W2A also demonstrates strong long-horizon instruction-following performance in unseen environments. Our method achieves an average task completion length of 4.30, remaining highly competitive with prior video-based policy learning methods. In particular, W2A attains the best success rates for completing the first two and all five consecutive subtasks, reaching 91.7% and 77.1%, respectively. Detailed per-seed results and standard deviation analysis are summarized in Appendix § 15.
Table 1: Results on LIBERO. “finetuned” denotes models trained on large-scale external robot action data and then finetuned on the target benchmark; “scratch” denotes models trained with robot action data only from the target evaluation benchmark, without extra robot action data. Bold indicates the best result overall, and <u>underline</u> indicates the best result among “scratch” models. Ext. RA denotes external robot-action data (: yes; : no).
| Method | Ext. RA | Spatial (%) | Object (%) | Goal (%) | Long (%) | Average (%) |
| Diffusion Policy (scratch) [46] | 78.3 | 92.5 | 68.3 | 50.5 | 72.4 | |
| Octo (finetuned) | 78.9 | 85.7 | 84.6 | 51.1 | 75.1 | |
| OpenVLA (finetuned) | 84.7 | 88.4 | 79.2 | 53.7 | 76.5 | |
| (finetuned) | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 | |
| mimic-video (scratch) | 94.2 | 96.8 | 90.6 | 84.9 | 91.6 | |
| DreamZero-5B (scratch) [39] | 90.4 | 94.8 | 87.2 | 82.2 | 88.7 | |
| W2A (ours, scratch) | <u>96.3</u> | 98.8 | <u>95.0</u> | 93.6 | 95.9 |
Table 1.
Table 2: Results on CALVIN ABCD in success rates and average task completion length.
| Method | Splits | 1/5 | 2/5 | 3/5 | 4/5 | 5/5 | Avg. Len |
| Diffusion Policy | ABCD | 40.2 | 12.3 | 2.6 | 0.8 | 0.0 | 0.56 |
| UniPi | ABCD | 56.0 | 16.0 | 8.0 | 8.0 | 4.0 | 0.92 |
| OpenVLA | ABCD | 91.3 | 77.8 | 62.0 | 52.1 | 43.5 | 3.27 |
| GR-1 | ABCD | 85.4 | 71.2 | 59.6 | 49.7 | 40.1 | 3.06 |
| VPP | ABCD | 96.5 | 90.9 | 86.6 | 82.0 | 76.9 | 4.33 |
| W2A | ABCD | 95.3 | 91.7 | 85.5 | 80.2 | 77.1 | 4.30 |
Table 2.
Real-world Tasks. Table 3 reports the average success rates on real-world tasks. W2A consistently outperforms both Diffusion Policy and OpenVLA across all three task types, achieving 78.3%, 65.0%, and 54.2% success rates on pick-and-place, cube-stacking, and long-horizon tasks, respectively. The advantage becomes especially evident as task difficulty increases: On the three additional practical tasks, W2A achieves 76.7%, 63.3%, and 50.0% success rates on Towel folding, Table clearing, and USB insertion, respectively, compared with 50.0%, 26.7%, and 20.0% for OpenVLA and 23.3%, 6.7%, and 0.0% for Diffusion Policy. These results suggest that the distilled video priors help the action model capture more transferable scene dynamics and object interaction patterns, improving its robustness to variations in object instances, spatial configurations, and task goals. Figure 4 shows representative real-world task executions, covering pick-and-place, cube-stacking, long-horizon manipulation, and the three practical tasks. Detailed success rates for all subtasks and additional real-world task demonstrations are provided in Appendix § 16.
| Method | PnP | Cube | Long | Towel folding | Table clearing | USB insertion |
| Diffusion Policy | 29.2 | 10.0 | 2.5 | 23.3 | 6.7 | 0.0 |
| OpenVLA | 50.8 | 35.0 | 18.3 | 50.0 | 26.7 | 20.0 |
| W2A | 78.3 | 65.0 | 54.2 | 76.7 | 63.3 | 50.0 |
Table 3. Results on real-world tasks (% success rate).
Figure 4. Representative real-world task cases.
In-depth Analysis
What is actually distilled? We load the Q-Former as a probe at test time, visualize the similarity between the queried action latent and the teacher video latent with a heatmap, and compare it with an action DiT baseline trained without distillation. Figure 5 shows that the distilled action latent aligns more consistently with task-relevant regions in the teacher video latent (such as the gripper and manipulated objects), while background regions are largely ignored during distillation. Notably, the highlighted regions also correspond well to task-relevant areas in the future images, suggesting that the action DiT captures future-aware world priors rather than merely reflecting the current observation. In contrast, the heatmaps read out from the baseline do not exhibit consistent similarity patterns, indicating that W2A truly transfers latent knowledge from the video DiT into the action DiT, rather than merely allowing the Q-Former to fit the video latent. We provide additional experiments on mask-based task-relevant region alignment and linear probing in Appendix § 12 to further investigate the information distilled into the action latents.
Figure 5. Similarity heatmap between action latent and teacher video latent.
What role does PCA play? Figure 7 compares raw video latents and PCA-projected video latents. The PCA-projected latent preserves clearer scene layout and salient patterns in task-relevant regions, such as the gripper and target objects. In contrast, the raw latent visualization is noisier and less structured. This suggests that PCA preserves the dominant information in the teacher latent while suppressing redundant variations, providing a compact teacher representation for distillation. A more complete visualization and detailed discussion are provided in Appendix § 11.
Figure 7. RGB visualizations of PCA-projected and raw teacher video latents across five temporal tokens, alongside the input image. PCA-projected latents show clearer spatial structure.
Inference efficiency. A key goal of W2A is to retain the benefit of video-based world priors without incurring the overhead of an explicit video model at test time. To verify this, we compare inference latency across representative baselines on the same hardware platform, a compute node equipped with a single NVIDIA A100 40GB GPU. As shown in Figure 6, W2A achieves the lowest average inference latency among all compared methods, requiring only 55 ms per action prediction. In contrast, mimic-video, a WAM that retains explicit future imagination, incurs about higher latency. Moreover, with a lightweight text encoder, W2A requires only about 6 GB of GPU memory at deployment, which is an advantage in compute-constrained settings.
Figure 6. Inference latency comparison on the same hardware platform.
Ablations
We conduct ablation studies on two key design choices in W2A: the distillation signal and the PCA-based teacher compression. To this end, we compare W2A with two simplified variants: w/o distillation, which removes the distillation objective and trains the action DiT only with the flow matching objective, and w/o PCA, which directly uses the original teacher latent without feature compression. As shown in Table 4, both variants perform worse than the full model. Removing distillation leads to a clear performance drop, confirming that the compact teacher video latents provide useful supervision beyond standard action modeling. Meanwhile, directly distilling the uncompressed teacher latent also degrades performance, suggesting that the raw teacher features are overly high-dimensional and noisy for effective transfer. These results show that both the proposed distillation design and PCA-based compression are important for transferring control-relevant world priors into the policy. More ablation results are provided in Appendix § 9.
| Method | Spatial | Object | Goal | Long | Avg. |
| w/o distill. | 85.9 (-10.4) | 87.7 (-11.1) | 83.6 (-11.4) | 80.5 (-13.1) | 84.4 (-11.5) |
| w/o PCA | 87.0 (-9.3) | 92.3 (-6.5) | 90.2 (-4.8) | 84.8 (-8.8) | 88.6 (-7.3) |
Table 4. Ablation results on LIBERO, shown as SR drops (%) relative to W2A.
Conclusion
In this paper, we present W2A, a two-stage distillation framework that transfers world priors from a video DiT into an action DiT for real-time robotic control. By compressing intermediate video latents and aligning them with action latents during training, W2A preserves useful video-based world priors while removing the video model entirely at test time. Experiments on LIBERO, CALVIN, and real-world tasks show that W2A achieves strong performance together with substantially lower inference latency. These results suggest that world priors learned by video models can be effectively distilled into action models for efficient real-time robot control.
Limitations and Future Work
Despite its strong efficiency-performance trade-off, W2A has several limitations. First, by distilling video-based world priors into an action model and removing the video model at test time, W2A gives up the additional model capacity and explicit future imagination ability of the video backbone. While crucial for real-time deployment, this design may limit robustness and generalization under large distribution shifts, unseen object dynamics, or tasks requiring longer-horizon adaptation beyond the distilled priors.
Second, the framework compresses teacher latents into a fixed low-dimensional subspace before transfer to the student action model. This may not fully preserve task-relevant or higher-order structure in the original video latent space, limiting complex planning or broader generalization.
A promising direction for future work is to explore more expressive teacher compression and distillation strategies that preserve richer world-model information while maintaining efficient deployment. We will also study hybrid designs that retain action-only inference while selectively invoking stronger world-model reasoning when needed, and extend the framework to more diverse embodiments, longer-horizon tasks, and open-world real-robot settings.
Acknowledgments
This work was supported by National Natural Science Foundation of China 62673160, Shenzhen Fundamental Research ZDCY20250901095402003, Guangdong Provincial Natural Science Foundation 2026A1515011727 and the Open Fund of Innovation Center for Control Actuators.
Supplementary ablation experiments
To further analyze the design choices in W2A, we conduct several supplementary ablation studies.
PCA K-sweep
We study the effect of the PCA compression dimension on distillation. Figure 8 shows the cumulative explained variance ratio as a function of the PCA dimension. The curve increases rapidly at low dimensions and gradually saturates, indicating that a large portion of the variance in the teacher video latent is concentrated in a relatively compact subspace. In our experiments, we choose , which corresponds to retaining approximately 99% of the cumulative explained variance.
Figure 8. PCA cumulative explained variance curve of the teacher video latent.
We then evaluate how different PCA dimensions affect downstream policy performance on LIBERO. Figure 9 reports the success rate under different values of . When is too small, the compressed teacher latent may discard useful control-relevant information, leading to weaker distillation. Increasing improves performance by preserving more information from the video latent. However, using an overly large does not necessarily bring further gains, as the retained dimensions may also include redundant or task-irrelevant visual variations, making the distillation target noisier for the action DiT. Overall, the results suggest that an intermediate compression dimension provides the best trade-off between preserving teacher information and filtering out irrelevant latent variations. This supports our choice of , which preserves most of the teacher latent variance while keeping the distillation target compact.
Figure 9. Results of PCA -sweep experiment.
Teacher Latent Depth Selection
Teacher latent depth is an important factor in determining the quality of the distillation target. Different depths of the video DiT may encode different types of information: shallow representations tend to preserve more low-level visual details, while deeper representations may become more tied to the final video reconstruction objective. For action distillation, the teacher latent should retain useful spatio-temporal structure and task-relevant world priors, rather than being either too low-level or overly generation-specific.
To analyze this effect, we extract teacher video latents from different relative depths of the video DiT while keeping the denoising time fixed. All other settings, including PCA compression, spatio-temporal alignment, and action DiT training, are kept unchanged. Figure 10 reports the LIBERO success rate under different latent extraction depths.
Figure 10. Effect of teacher latent extraction depth on LIBERO success rate.
The results show a clear non-monotonic trend. Using very shallow teacher latents leads to limited improvement, suggesting that early representations are not sufficiently informative for action-level distillation. Performance improves as the extraction depth increases and reaches the best result at the 50% depth. However, further increasing the depth degrades performance, with the 90% depth performing the worst. This indicates that overly deep video latents may be less suitable as distillation targets, possibly because they become more specialized toward video reconstruction rather than control-relevant structure. Based on this result, we use the intermediate teacher latent at 50% depth as the default distillation target in W2A.
What Teacher Signal Matters?
Structured vs. Unstructured Teacher Signals
We study the role of the teacher video latent by comparing our full model with two variants: Shuffled Teacher, which destroys the spatio-temporal ordering of teacher latents, and Random Teacher, which replaces teacher latents with random targets of the same dimensionality. As shown in Table 5, Shuffled Teacher already leads to a substantial average drop of 17.7%, with degradation across all four suites. This indicates that the gain of W2A depends not only on the presence of an auxiliary teacher signal, but also on preserving the original spatio-temporal organization of the teacher latent. In other words, the student does benefit from structured correspondence to the teacher video representation, rather than merely matching a bag of teacher features. At the same time, Random Teacher causes an even larger average drop of 26.7%, showing that the improvement is not explained by an additional regularization loss alone, but relies on meaningful non-random structure encoded in the video teacher. Overall, these results suggest that W2A benefits from both the informative content and the spatio-temporal organization of the teacher latent, with coherent teacher structure being particularly important for effective transfer.
| Suite | w/o distill. | Shuffled Teacher | Random Teacher | Token0 Teacher | Current Feature | Future Feature |
| Spatial | 85.9 (-10.4) | 80.5 (-15.8) | 72.5 (-23.8) | 88.9 (-7.4) | 83.3 (-13.0) | 89.8 (-6.5) |
| Object | 87.7 (-11.1) | 81.6 (-17.2) | 79.8 (-19.0) | 91.0 (-7.8) | 86.6 (-12.2) | 92.5 (-6.3) |
| Goal | 83.6 (-11.4) | 77.7 (-17.3) | 65.2 (-29.8) | 84.4 (-10.6) | 81.7 (-13.3) | 86.4 (-8.6) |
| Long | 80.5 (-13.1) | 73.1 (-20.5) | 59.3 (-34.3) | 82.3 (-11.3) | 77.1 (-16.5) | 82.7 (-10.9) |
| Avg. | 84.4 (-11.5) | 78.2 (-17.7) | 69.2 (-26.7) | 86.7 (-9.2) | 82.2 (-13.7) | 87.8 (-8.1) |
Table 5. Ablation results on teacher signals on LIBERO, shown as success rate and performance drops (%) relative to W2A.
Static, Future, and Video-DiT Teacher Latent
We further study whether the gain of W2A comes specifically from future-aware video teacher latents, rather than from static visual features or merely introducing an auxiliary teacher loss. To this end, we compare the default teacher source in W2A with three alternatives. Token0 Teacher uses only the token corresponding to the current visual anchor from the video DiT, providing a primarily static teacher signal with limited future-aware dynamics. Current Feature replaces the video teacher with ResNet features extracted from the current image, testing whether static visual representations alone are sufficient. Future Feature uses ResNet features extracted from future frames, testing whether access to future visual information alone can account for the gain without relying on video-DiT latents.
For fair comparison, all variants use the same student architecture, Q-Former readout, cosine distillation loss, and distillation weight, while only changing the teacher source. All teacher signals are projected to the same dimensionality and organized into the same temporal-spatial layout as the default teacher latent. Static teacher latents are repeated across temporal groups, while future-frame features are grouped using the same temporal compression setting as the default teacher. As shown in Table 5, both static-feature teachers and future-frame teachers perform better than the unstructured variants, but all remain below the full model. In particular, Token0 Teacher causes an average drop of 9.2%, while Current Feature leads to a larger average drop of 13.7%, showing that static teacher signals alone are insufficient to explain the gain of W2A. Future Feature is the strongest among the three additional variants, but still underperforms W2A by 8.1% on average. This indicates that access to future visual information is helpful, yet does not fully match the benefit of the video-DiT teacher latent. These results suggest that the gain of W2A is not explained by static visual cues or auxiliary supervision alone, but is more closely related to the future-aware latent structure provided by the video DiT.
Effect of the Q-Former
To assess whether the gain comes from the query-based readout itself or simply from introducing a learnable projector, we replace the Q-Former with two MLP-based projectors while keeping the teacher target and distillation loss unchanged. The first is a Global MLP, which flattens the entire action latent and directly predicts the aligned teacher latent grid. The second is a more structured Grouped MLP, which preserves the same temporal grouping as our full model: for each temporal group, a shared MLP maps the flattened action tokens to the corresponding teacher latent grid. Both baselines produce the same output layout as the Q-Former model and are trained with the same cosine distillation loss.
As shown in Table 6, both MLP-based projectors underperform the Q-Former. The Global MLP performs the worst, suggesting that treating latent distillation as a monolithic regression problem is insufficient. Preserving coarse temporal grouping with a Grouped MLP improves performance, but still falls short of the Q-Former. This indicates that the gain of W2A does not come merely from adding a learnable projection head; rather, the query-based structured readout is more effective for extracting teacher-compatible features from action latents.
| Suite | Global MLP | Grouped MLP |
| Spatial | 83.7 (-12.6) | 89.5 (-6.8) |
| Object | 85.1 (-13.7) | 90.4 (-8.4) |
| Goal | 82.2 (-12.8) | 87.2 (-7.8) |
| Long | 79.8 (-13.8) | 83.8 (-9.8) |
| Avg. | 82.7 (-13.2) | 87.7 (-8.2) |
Table 6. Ablation results about Q-Former on LIBERO, shown as performance drops (%) relative to W2A.
Implementation Details
Model Structure
Table 7 summarizes the main architecture of W2A. The model consists of an observation encoder, an action DiT policy, and a training-only spatio-temporal alignment module for latent distillation. At inference time, the teacher video DiT and the alignment module are removed.
| Component | Configuration |
| RGB observations | Front-view and wrist-view images |
| Image resolution | |
| Vision encoder | ResNet-18 [47] |
| Observation history length | 2 |
| Action horizon | 16 |
| Executed horizon | 8 |
| Language embedding | Precomputed T5 embedding [48] |
| Action DiT hidden size | 768 |
| Action DiT depth | 10 transformer blocks |
| Attention heads | 8 |
| MLP ratio | 4.0 |
| Condition injection | Cross-attention |
| Flow-time injection | AdaLN |
| Q-Former hidden dimension | 128 |
| Q-Former layers | 1 |
| Q-Former attention heads | 8 |
Table 7. Main model configuration of W2A.
For latent distillation, we extract the student action representation from the middle layer of the action DiT and align it with a selected hidden state of the teacher video DiT. The main distillation configuration is shown in Table 8.
| Item | Configuration |
| Teacher latent | Output hidden state of the 14th video DiT block at |
| Student latent | Output hidden state of the 5th action DiT block |
| PCA dimension | 482 |
| Teacher latent shape | |
| Teacher latent shape after PCA | |
| Student latent shape |
Table 8. Latent distillation configuration.
Training Recipe
Training-data scope. Throughout this paper, “scratch” and “finetuned” are distinguished solely by whether external robot-action data are used for policy pretraining. “Scratch” methods use robot-action data only from the target evaluation benchmark, whereas “finetuned” methods are pretrained on external robot-action data and then adapted to the target benchmark. This convention allows “scratch” methods to use generic video pretraining. In W2A, we adapt a pretrained video model using target-benchmark videos and train the action DiT from scratch using target-benchmark action data, without external robot-action pretraining.
Table 9 summarizes the training-data sources for representative methods. “Ext. RA” denotes external robot-action data, and “Generic video” denotes general video data. “Target action” and “Target video” refer to data from the target evaluation benchmark; target videos are counted only when explicitly used to train or adapt a video model. The “Ext. RA” column in Table 1 follows the same convention.
| Category | Method | Ext. RA | Generic video | Target action | Target video |
| Scratch | Diffusion Policy | ✗ | ✗ | ✓ | ✗ |
| Scratch | mimic-video | ✗ | ✓ | ✓ | ✓ |
| Scratch | DreamZero-5B | ✗ | ✓ | ✓ | ✓ |
| Scratch | W2A | ✗ | ✓ | ✓ | ✓ |
| Finetuned | Octo | ✓ | ✗ | ✓ | ✗ |
| Finetuned | OpenVLA | ✓ | ✗ | ✓ | ✗ |
| Finetuned | ✓ | ✗ | ✓ | ✗ |
Table 9. Training-data sources for representative methods. ✓ indicates use of the corresponding data source, and ✗ indicates no use.
Table 10 summarizes the training recipe. We train the action DiT and the spatio-temporal alignment module jointly using the flow matching objective and the latent distillation objective.
| Item | Configuration |
| Batch size | 128 |
| Training epochs | 100 |
| Optimizer | AdamW |
| Learning rate | |
| Weight decay | |
| Gradient clipping | 1.0 |
| Learning-rate schedule | Cosine annealing |
| EMA decay | 0.999 |
Table 10. Training configuration.
Inference Details
At inference time, W2A removes all training-only components, including the teacher video DiT and the spatio-temporal alignment module. The deployed policy only consists of the observation encoder, language embedding, action DiT, and action normalizer. By default, we use precomputed T5 embeddings as the text input. For measuring inference latency and GPU memory usage, we instead use the variant trained with a CLIP text encoder. The two variants achieve comparable policy performance.
After each prediction, only the first 8 actions are executed, and the unexecuted suffix is cached to warmstart the next prediction. This enables temporally consistent replanning while keeping inference video-free. Table 11 and Table 12 provide the configurations for flow matching and inference, respectively.
| Item | Configuration |
| Number of flow steps | 100 |
| Flow-time sampling | Logit-normal |
| Inference denoising steps | 20 |
| Distillation weight | 1.0 |
| Gaussian weighting center | |
| Gaussian weighting std. |
Table 11. Flow matching and distillation settings.
| Item | Configuration |
| Observation history | 2 steps |
| Predicted action chunk | 16 actions |
| Executed actions per prediction | 8 actions |
| Inference denoising steps | 20 |
| Warmstart time |
Table 12. Inference configuration.
Spatio-temporal Alignment Mechanism
We provide additional details of the spatio-temporal alignment mechanism used in Stage II. The goal is to organize the Q-Former output into the same spatio-temporal layout as the compressed teacher video latent , enabling position-wise distillation between corresponding spatio-temporal locations. In our current implementation, the Q-Former mainly serves as a structured information readout module.
In our implementation, and . Following the setting used in our LoRA adaptation of Cosmos-Predict2, the video VAE has a temporal compression ratio of , meaning that one video temporal token summarizes four future frames. Accordingly, we partition into temporal groups:
With and , we obtain temporal groups, each containing four action tokens.
On the teacher side, we exclude the temporal token corresponding to the current observation and use the following selected video temporal tokens in as distillation targets. For each temporal group , we introduce a set of learnable spatial query tokens arranged according to the spatial resolution of . These queries aggregate information from the action tokens within the same temporal group through cross-attention:
where denotes the spatial queries for the -th temporal group. This produces one aligned action representation for each spatial position of the compressed teacher video latent.
Applying this operation across all temporal groups yields
which matches the layout of the selected compressed teacher latent:
Both tensors are then flattened into for computing the cosine distillation loss. This flattening is only used for loss computation and preserves the correspondence between the same temporal group and the same spatial position.
Visualization and Discussion of Raw and PCA-Projected Video Latents
Figure 11 compares raw video latents and PCA-projected video latents. For visualization, we take the first three channels as RGB-like values, apply a symmetric normalization, and map them to . For the PCA-projected latent, these three channels correspond to the first three principal directions, while for the raw latent they correspond to the first three original feature channels.
Figure 11. Comparison of raw and PCA-projected video latents visualized as RGB feature maps.
The PCA-projected latent remains visually more structured and interpretable than the raw latent. In particular, it preserves clear scene layout and salient patterns in dynamic and task-relevant regions, such as the gripper and target objects. By contrast, the raw latent visualization is noisier and less structured. This suggests that PCA preserves the dominant information in the teacher latent while suppressing part of the redundant variations, which is consistent with our use of PCA to construct a compact teacher representation for distillation.
This also provides additional qualitative evidence that the compressed teacher video latent retains scene dynamics and task-relevant interaction cues that are transferred from the video DiT to the action DiT.
Quantitative Evidence of Distilled Task-Relevant World Priors
Task-Relevant Region Alignment
As a quantitative extension to the analysis of distilled world priors in § 5.3, we further measure whether the action representations shaped by W2A are more strongly aligned with task-relevant regions of the teacher video latent. Since the spatial components of world priors that are most relevant to manipulation should primarily involve the gripper and manipulated objects, we compute region-level statistics over the action-video latent similarity maps. Specifically, we use Segment Anything 2 (SAM2) [49] to automatically segment object masks from the corresponding video frames, and select the masks associated with the task-relevant objects. These masks are then downsampled to the teacher latent grid using area interpolation, resulting in a soft mask . The action-video alignment is measured using the same aligned-slot cosine similarity map as in § 5.3. Figure 12 shows examples of the SAM2-derived masks and the corresponding action-video alignment maps used in this analysis.

Figure 12. Examples of task-relevant masks and action-video alignment maps used for world-prior analysis. SAM2 provides frame-level masks, and warmer colors denote higher latent similarity.
We report two region-level metrics. The first is mask enrichment, which measures how much similarity mass falls inside the task-relevant mask after normalizing by the mask area. Given a similarity map , we convert it into a spatial probability distribution using softmax normalization, i.e., , and compute
where indexes spatial locations, , and is the softmax temperature. A uniform similarity distribution has an expected enrichment of 1.0×, while values larger than 1.0× indicate that the action-video latent similarity is concentrated in task-relevant regions beyond what would be expected from their area alone. The second metric is mask-background contrast, which measures whether the mask region has higher normalized similarity than the background. We first standardize the similarity map as , and then compute
A larger contrast indicates that the task-relevant region is more strongly aligned with the teacher video latent than the surrounding background.
As shown in Table 13, the non-distilled Action DiT baseline exhibits only marginal alignment with the task-relevant mask, with a mask enrichment close to the uniform-distribution baseline (0.98×) and an almost zero mask-background contrast (0.01). In contrast, W2A achieves a substantially higher mask enrichment of 3.87× and a much stronger mask-background contrast of 1.34. This indicates that the action-video latent similarity in W2A is not uniformly distributed over the scene, but is strongly concentrated in task-relevant object regions. These results provide quantitative evidence that W2A encourages the action model to internalize task-relevant world priors from the video DiT, especially those associated with manipulated objects and their interactions, rather than merely imposing a generic feature-level regularization.
| Method | Mask enrichment ↑ | Mask-bg contrast ↑ |
| Action DiT w/o distill. | 0.98× | 0.01 |
| W2A | 3.87× | 1.34 |
Table 13. Quantitative evidence of distilled task-relevant world priors. We measure whether action-video latent similarity is concentrated in task-relevant object regions. Mask enrichment is area-normalized, where a uniform similarity map has an expected value of 1.0×. Mask-bg contrast is computed on standardized similarity maps.
Linear Probe Test
To further test whether future dynamics are encoded in the action representation, we freeze the action DiT and train a linear probe to predict the future gripper and object image-plane coordinates from its action latents. We evaluate predictions at the current step and at future offsets and . We report the positional endpoint error (Pos EPE) in pixels, where lower values indicate more accurate prediction. The “w/o Distill” baseline uses the same action model without the latent distillation objective.
As shown in Table 14, W2A achieves lower Pos EPE at all prediction horizons than the non-distilled action model. The improvement becomes larger for farther future steps, suggesting that the distilled action representation retains information about how the gripper and manipulated objects evolve over time. This provides complementary evidence that W2A learns future-aware world priors rather than only improving instantaneous action prediction.
| Method | Pos endpoint error (EPE) (px) | |||
| Action DiT w/o distill. | 6.4 | 13.7 | 23.8 | |
| W2A | 5.9 | 8.2 | 13.1 |
Table 14. Results of the linear probe test for predicting future gripper and object image-plane coordinates. Pos EPE is measured in pixels ().
Discussion of the Q-Former and Latent Space Distillation
W2A is better viewed as a form of intermediate representation distillation rather than classical response-based distillation. While early knowledge distillation focused on soft outputs [50], later work showed that useful teacher knowledge can also be transferred through hidden features, attention, and relational structure [51, 52, 53, 54]. From this perspective, our method distills compact video-side latent representations into an action model, and therefore naturally belongs to the broader family of representation distillation.
The role of Q-Former in W2A should also be interpreted in light of our implementation. In BLIP-2, Q-Former is introduced as a lightweight querying transformer that uses learnable queries to extract useful information from frozen visual tokens [16]. More broadly, multimodal systems often employ lightweight bridge modules to connect pretrained backbones and downstream objectives [55, 56]. In our case, the Q-Former mainly serves as a structured information readout module that organizes action latents into teacher-compatible slots, which is consistent with our implementation details in Appendix § 10.§ 4. This design is also compatible with recent evidence that generative and diffusion models provide transferable intermediate representations that can be distilled into downstream models [57, 58].
Hardware
Figure 13 shows the hardware used in our real-world experiments. We use a Franka Research 3 robot as the manipulation platform. The robot is equipped with a front-view camera and a wrist-view camera, which provide the visual observations used by the policy. The policy runs at 15 Hz on the real robot and predicts action chunks in a receding-horizon manner.

Figure 13. Real-world hardware setup. We use a Franka Research 3 robot for policy execution and an Omega 7 haptic device for teleoperated demonstration collection.
For data collection, we use an Omega 7 haptic device to teleoperate the robot and collect demonstrations. The operator controls the robot end effector through the Omega 7 device, while the robot observations and actions are recorded to construct the real-world training dataset. This setup is used for both pick-and-place and cube-stacking tasks.
Detailed Simulation Results Across Random Seeds
As shown in Table 15 and Figure 14, W2A exhibits stable performance across different random seeds on both LIBERO and CALVIN. The Object suite shows the smallest variance across seeds, indicating highly stable performance on object-centric manipulation tasks. The Goal and Long suites exhibit relatively larger variation, which is expected since they involve more diverse goal configurations and longer-horizon execution. Nevertheless, W2A maintains strong performance under all three seeds, suggesting that the reported gains are not caused by a favorable random initialization.
| Seed | LIBERO | CALVIN | |||
| Spatial | Object | Goal | Long | Avg. Len | |
| 0 | 95.4 | 99.0 | 91.6 | 91.0 | 4.34 |
| 5 | 98.6 | 98.2 | 97.4 | 94.2 | 4.26 |
| 42 | 95.0 | 99.2 | 96.0 | 95.6 | 4.30 |
| Avg. | 96.3±1.97 | 98.8±0.53 | 95.0±3.03 | 93.6±2.36 | 4.30±0.04 |
Table 15. Results across different random seeds on LIBERO and CALVIN.
Figure 14. Seed-wise simulation results. Error bars denote standard deviation over three seeds.
On CALVIN ABC→D, W2A also shows stable long-horizon instruction-following performance, with an average sequence length of across three seeds. The small variance indicates that
W2A generalizes consistently to the unseen environment split. Overall, these results demonstrate that W2A achieves not only high average performance but also robust behavior across random seeds.
Detailed Real-world Results and More Demonstrations
Figure 15 shows additional real-world task cases used in our evaluation. The examples cover both pick-and-place and cube-stacking tasks. For pick-and-place, the robot needs to identify the target object from language instructions and place it into the specified receptacle or onto the target object. For cube-stacking, the robot must manipulate small objects and complete sequential placement under contact-rich interactions. Detailed per-subtask real-world results are reported in Table 16, where each entry shows the number of successful rollouts out of 30.
| Task Type | Subtask | Diffusion Policy | OpenVLA | W2A |
| PnP | Pick up the carrot and place it in the basket | 12/30 | 15/30 | 25/30 |
| PnP | Pick up the banana and place it in the basket | 6/30 | 15/30 | 28/30 |
| PnP | Pick up the white bowl and place it on the plate | 9/30 | 22/30 | 19/30 |
| PnP | Pick up the blue cup and place it on the plate | 8/30 | 9/30 | 22/30 |
| Cube | Stack the green cube on top of the red cube | 0/30 | 10/30 | 21/30 |
| Cube | Stack the purple cube on top of the red cube | 4/30 | 10/30 | 15/30 |
| Cube | Stack the green cube on top of the purple cube | 8/30 | 6/30 | 27/30 |
| Cube | Stack the red cube on top of the blue cube | 0/30 | 16/30 | 15/30 |
| Long | Open the top drawer and put the carrot in it | 1/30 | 6/30 | 21/30 |
| Long | Open the top drawer and put the banana in it | 1/30 | 3/30 | 14/30 |
| Long | Open the top drawer and put the white bowl in it | 1/30 | 0/30 | 10/30 |
| Long | Pick up the carrot and banana and place them in the basket | 0/30 | 13/30 | 20/30 |
| Practical Skills | Towel folding | 7/30 | 15/30 | 23/30 |
| Practical Skills | Table clearing | 2/30 | 8/30 | 19/30 |
| Practical Skills | USB insertion | 0/30 | 6/30 | 15/30 |
| Overall | 13.1% | 34.2% | 65.3% |
Table 16. Per-subtask results on real-world tasks. Each entry is reported as the number of successful rollouts out of 30. The overall row reports the average success rate over all subtasks.
Figure 15. Additional real-world task cases.
Generalization Behavior and Failure Mode Analysis
We further examine the compositional generalization behavior of W2A in real-world evaluation. As shown in Figure 16, W2A can complete new combinations of previously observed objects, receptacles, and manipulation skills, such as placing the white bowl into the basket and placing the banana onto the plate. These rollouts are not direct repetitions of training demonstrations: while the involved objects and primitive skills have been seen during training, the specific object-task pairings and resulting motion trajectories are unseen. This suggests that W2A can recombine learned interaction patterns and task-conditioned motion priors, rather than simply memorizing demonstrated trajectories.
Figure 16. Examples of compositional generalization in real-world evaluation.
However, this compositional generalization is not unlimited. When the task semantics or scene configuration move beyond the support of the training distribution, W2A can still fail, as shown in Figure 17. These failures mainly arise in two situations. First, W2A may fail on tasks or objects that are entirely unseen during training, such as novel object categories or unseen task specifications. In these cases, the policy cannot directly rely on previously learned object interaction patterns or task-conditioned motion priors. Second, W2A can also fail when the initial scene configuration differs too much from the training distribution, even if some of the involved objects or skills are familiar. Large deviations in object placement, relative spatial layout, or scene geometry can make the generated action sequence less reliable and may cause errors to accumulate over replanning steps. These observations suggest that the compositional generalization of W2A is still bounded by the support of the training distribution. Since W2A distills world priors into the action model and removes the video model at test time, it does not retain explicit online future imagination or strong recovery mechanisms for out-of-distribution situations. As a result, its robustness remains limited when both semantic novelty and geometric distribution shift become significant.
Figure 17. Representative failure cases in real-world evaluation. W2A may fail when facing unseen tasks or objects, or when the initial scene configuration deviates substantially from the training distribution.
Discussion of Related Efforts on Improving WAM Efficiency
Several recent video-action and world-action models have explored how to reduce the inference cost introduced by test-time world modeling. For example, UVA [8] improves action inference efficiency by learning a joint video-action latent representation while decoupling video and action decoding, allowing action prediction to bypass explicit future video generation at inference time. Other efficient WAM designs reduce deployment cost by reducing or skipping explicit future prediction [10], or by adopting an action-centered formulation in which future video generation becomes optional rather than a prerequisite for action decoding [40]. These works suggest that the benefits of video-based world modeling may come not only from test-time imagination, but also from the world-aware representations learned during training.
W2A is complementary to these efforts, but differs in its deployment-time computation graph. Rather than designing a more efficient world-action inference pipeline, W2A uses the video DiT only as a training-time teacher and distills its intermediate denoising representations into an action DiT. After training, the deployed policy does not run video generation, predict future visual latents, or execute a separate world-modeling branch. This makes W2A closer to a standard action model at deployment, while still benefiting from video-based world priors during training.
In summary, existing efficient WAMs mainly ask how to make test-time world modeling cheaper, whereas W2A asks whether useful world priors can be internalized into the action model so that test-time world modeling is no longer required.
References
- [1]M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. OpenVLA: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.arxiv.org/abs/2406.09246
- [2]K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024.arxiv.org/abs/2410.24164
- [3]K. Black, N. Brown, I. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, et al. π0.5: a vision-language-action model with open-world generalization. In 9th Annual Conference on Robot Learning (CoRL), 2025.arxiv.org/abs/2504.16054
- [4]J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. GR00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.arxiv.org/abs/2503.14734
- [5]W. Song, Z. Zhou, H. Zhao, J. Chen, P. Ding, H. Yan, Y. Huang, F. Tang, D. Wang, and H. Li. ReconVLA: Reconstructive vision-language-action model as effective robot perceiver. arXiv preprint arXiv:2508.10333, 2025.arxiv.org/abs/2508.10333
- [6]Y. Hu, Y. Guo, P. Wang, X. Chen, Y.-J. Wang, J. Zhang, K. Sreenath, C. Lu, and J. Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.14803, 2024.arxiv.org/abs/2412.14803
- [7]J. Pai, L. Achenbach, V. Montesinos, B. Forrai, O. Mees, and E. Nava. mimic-video: Video-action models for generalizable robot control beyond VLAs. arXiv preprint arXiv:2512.15692, 2025.arxiv.org/abs/2512.15692
- [8]S. Li, Y. Gao, D. Sadigh, and S. Song. Unified video action model. arXiv preprint arXiv:2503.00200, 2025.arxiv.org/abs/2503.00200
- [9]T. Ma, J. Zheng, Z. Wang, C. Jiang, A. Cui, J. Liang, and S. Yang. DiT4DiT: Jointly modeling video dynamics and actions for generalizable robot control. arXiv preprint arXiv:2603.10448, 2026.arxiv.org/abs/2603.10448
- [10]T. Yuan, Z. Dong, Y. Liu, and H. Zhao. Fast-WAM: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026.arxiv.org/abs/2603.16666
- [11]H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, H. Zhao, H. Liu, Z. Su, L. Ma, H. Su, and J. Zhu. Motus: A unified latent action world model. arXiv preprint arXiv:2512.13030, 2025.arxiv.org/abs/2512.13030
- [12]M. J. Kim, Y. Gao, T.-Y. Lin, Y.-C. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M.-Y. Liu, C. Finn, and J. Gu. Cosmos Policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163, 2026.arxiv.org/abs/2601.16163
- [13]W. Peebles and S. Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.arxiv.org/abs/2212.09748
- [14]H. Hotelling. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6):417, 1933.
- [15]Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.arxiv.org/abs/2210.02747
- [16]J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730–19742. PMLR, 2023.arxiv.org/abs/2301.12597
- [17]Q. Zhao, Y. Lu, M. J. Kim, Z. Fu, Z. Zhang, Y. Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1702–1713, 2025.arxiv.org/abs/2503.22020
- [18]M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zoutine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti, et al. Smolvla: A vision-language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844, 2025.arxiv.org/abs/2506.01844
- [19]Gemini Robotics Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025.arxiv.org/abs/2503.20020
- [20]S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu. RDT-1B: A diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024.arxiv.org/abs/2410.07864
- [21]T. Jiang, T. Yuan, Y. Liu, C. Lu, J. Cui, X. Liu, S. Cheng, J. Gao, H. Xu, and H. Zhao. Galaxea open-world dataset and g0 dual-system vla model. arXiv preprint arXiv:2509.00576, 2025.arxiv.org/abs/2509.00576
- [22]H. Zhao, J. Zhang, W. Song, P. Ding, and D. Wang. VLA²: Empowering vision-language-action models with an agentic framework for unseen concept manipulation, 2025.arxiv.org/abs/2510.14902
- [23]L. Zhong, Y. Liu, Y. Wei, Z. Xiong, M. Yao, S. Liu, and G. Ren. ACoT-VLA: Action chain-of-thought for vision-language-action models. arXiv preprint arXiv:2601.11404, 2026.arxiv.org/abs/2601.11404
- [24]W. Wu, F. Lu, Y. Wang, S. Yang, S. Liu, F. Wang, S. Ma, H. Sun, Y. Wang, Z. Qiu, H. Xiong, Z. Wang, S. Zhou, Y. Ren, K. Zhang, H. Yu, J. Zhao, Q. Zhu, R. Cheng, Y.-L. Li, Y. Huang, X. Zhu, Y. Shen, and K. Zheng. A pragmatic vla foundation model. arXiv preprint arXiv:2601.18692v1, 2026.arxiv.org/abs/2601.18692
- [25]Physical Intelligence, B. Ai, A. Amin, R. Aniceto, A. Balakrishna, G. Balke, K. Black, G. Bokinsky, S. Cao, T. Charbonnier, et al. π₀.₇: a steerable generalist robotic foundation model with emergent capabilities. arXiv preprint arXiv:2604.15483, 2026.arxiv.org/abs/2604.15483
- [26]A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.arxiv.org/abs/2212.06817
- [27]B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023.arxiv.org/abs/2307.15818
- [28]Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, H. Tan, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.arxiv.org/abs/2405.12213
- [29]Physical Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, et al. π*₀.₆: a vla that learns from experience. arXiv preprint arXiv:2511.14759, 2025.arxiv.org/abs/2511.14759
- [30]Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems, volume 36, 2023.arxiv.org/abs/2302.00111
- [31]H. Wu, Y. Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations, 2024.arxiv.org/abs/2312.13139
- [32]Y. Feng, H. Tan, X. Mao, C. Xiang, G. Liu, S. Huang, H. Su, and J. Zhu. Vidar: Embodied video diffusion model for generalist manipulation. arXiv preprint arXiv:2507.12898, 2025.arxiv.org/abs/2507.12898
- [33]L. Zheng, Z. Ouyang, C. Wang, J. Pan, and W. Zhang. From video to control: A survey of learning manipulation interfaces from temporal visual data. arXiv preprint arXiv:2604.04974, 2026.arxiv.org/abs/2604.04974
- [34]C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In Proceedings of Robotics: Science and Systems (RSS), 2025.arxiv.org/abs/2504.02792
- [35]MotuBrain Team, C. Xiang, F. Bao, H. Liu, H. Tan, H. Bi, J. Li, J. Liu, J. Pang, K. Jing, et al. MotuBrain: An advanced world action model for robot control. arXiv preprint arXiv:2604.27792, 2026.arxiv.org/abs/2604.27792
- [36]J. Guo, Q. Li, P. Li, Z. Chen, N. Sun, Y. Su, H. Wang, Y. Zhang, X. Li, and H. Liu. Unified 4D world action modeling from video priors with asynchronous denoising. arXiv preprint arXiv:2604.26694, 2026.arxiv.org/abs/2604.26694
- [37]Z. Zhang, Z. Li, B. Rahmati, R. H. Yang, Y. Ma, A. Rasouli, S. Pakdamansavoj, Y. Wu, L. Zhang, T. Cao, et al. Do world action models generalize better than vlas? a robustness study. arXiv preprint arXiv:2603.22078, 2026.arxiv.org/abs/2603.22078
- [38]L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu. Causal world modeling for robot control. arXiv preprint arXiv:2601.21998, 2026.arxiv.org/abs/2601.21998
- [39]S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, A. Malik, K. Lee, W. Liang, N. Ranawaka, J. Gu, Y. Xu, G. Wang, F. Hu, A. Narayan, J. Bjorck, J. Wang, G. Kim, D. Niu, R. Zheng, Y. Xie, J. Wu, Q. Wang, R. Julian, D. Xu, Y. Du, Y. Chebotar, S. Reed, J. Kautz, Y. Zhu, L. J. Fan, and J. Jang. World action models are zero-shot policies, 2026. URL https://arxiv.org/abs/2602.15922.
- [40]A. Ye, B. Wang, C. Ni, G. Huang, G. Zhao, H. Li, H. Li, J. Li, J. Lv, J. Liu, M. Cao, P. Li, Q. Deng, W. Mei, X. Wang, X. Chen, X. Zhou, Y. Wang, Y. Chang, Y. Li, Y. Zhou, Y. Ye, Z. Liu, and Z. Zhu. GigaWorld-Policy: An efficient action-centered world-action model. arXiv preprint arXiv:2603.17240, 2026.arxiv.org/abs/2603.17240
- [41]NVIDIA. Cosmos-Predict2: World simulation model for physical ai, 2025. URL https://github.com/nvidia-cosmos/cosmos-predict2.
- [42]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
- [43]K. Black, M. Galliker, and S. Levine. Real-time execution of action chunking flow policies. Advances in Neural Information Processing Systems, 38:33383–33407, 2026.arxiv.org/abs/2506.07339
- [44]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791, 2023.arxiv.org/abs/2306.03310
- [45]O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3):7327–7334, 2022.arxiv.org/abs/2112.03227
- [46]C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion Policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
- [47]K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.arxiv.org/abs/1512.03385
- [48]C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL http://jmlr.org/papers/v21/20-074.html.arxiv.org/abs/1910.10683
- [49]N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. URL https://arxiv.org/abs/2408.00714.
- [50]G. E. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015.arxiv.org/abs/1503.02531
- [51]A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. FitNets: Hints for thin deep nets. CoRR, abs/1412.6550, 2014. URL https://api.semanticscholar.org/CorpusID:2723173.DOI
- [52]S. Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017. URL https://arxiv.org/abs/1612.03928.
- [53]W. Park, D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019.arxiv.org/abs/1904.05068
- [54]X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. TinyBERT: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2019.arxiv.org/abs/1909.10351
- [55]J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Saman- gooee, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14198, 2022. URL https://api.semanticscholar.org/CorpusID:248476411.
- [56]X. Wang, J. Liang, C.-K. Wang, K. Deng, Y. Lou, M. Lin, and S. Yang. ViLA: Efficient video-language alignment for video question answering, 2024. URL https://arxiv.org/abs/2312.08367.
- [57]X. Yang and X. Wang. Diffusion model as representation learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18892–18903, 2023.arxiv.org/abs/2308.10916
- [58]Y. Deng, Y. Jin, X. Jia, J. Xue, G. Neumann, and G. Chัลvatzaki. Robot-DIFT: Distilling diffusion features for geometrically consistent visuomotor control. ArXiv, abs/2602.11934, 2026.













