Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction

Introduction

World Action Models (WAMs) have emerged as a promising approach to robotic foundation models, demonstrating that dense dynamics knowledge acquired through video pretraining can be adapted for robot control. Built on pretrained video diffusion backbones, WAMs jointly predict future visual observations and robot actions, extending video models’ capacity for dense world modeling to action learning [11, 12, ?, 10].

The visual prediction objective plays a central role in this adaptation by supervising world representation learning in the target domain. Removing it has been shown to substantially degrade control performance [5], while jointly optimizing visual and action prediction further shapes these representations for control [6]. Yet, the objective typically retains the formulation inherited from video pretraining, which was designed for visual generation [13, 14] rather than robot control.

Recent approaches seek to bridge this gap by introducing additional semantic [24, 26] or geometric supervision [21, 22, 23]. However, these approaches primarily rely on external pretrained models or incorporate additional modalities such as depth or 3D information. We instead ask how visual supervision within the pretrained video backbone’s native latent space can be tailored to better support action learning.

Our motivating study shows that shifting visual supervision toward dynamic regions can consistently improve policy performance across WAMs (§ 2.2). We interpret these gains as evidence that emphasizing dynamic regions can better align visual supervision with the demands of robot control. Dynamic regions tend to capture robot motion and object interactions, whereas static regions may contain visually complex but less action-relevant details, such as background textures. These findings motivate assigning distinct roles to dynamic and static regions in visual prediction for action learning.

Building on this insight, we propose Motion-Guided Asymmetric Prediction (MoGAP), a new visual prediction task for WAMs that treats dynamic regions as prediction targets and uses static regions as scene context. MoGAP establishes this distinction through asymmetric noise allocation [17, 1], applying higher noise levels to dynamic regions while preserving more information in static regions. The noise allocation is guided by latent motion scores directly computed from temporal differences in training videos, requiring neither external teacher models nor additional supervision modalities. MoGAP is jointly optimized with the standard WAM objective through a prediction path within the shared video diffusion backbone. This path is used only during training, leaving inference unchanged.

Through extensive experiments, we demonstrate that MoGAP consistently improves policy performance across WAM architectures in both simulated and real-world manipulation benchmarks. On GR-1 Tabletop, MoGAP increases the success rate of Fast-WAM (Wan2.2-5B) from 38.8% to 47.5% and that of Cosmos3-Nano from 52.3% to 62.3%, outperforming competitive VLA and WAM baselines as well as representation alignment methods using external teachers. On a real-world OpenArm humanoid, we find that MoGAP helps the policy better respond to task-relevant perturbations, such as target object relocation during execution, even without demonstrations of such perturbations.

In summary, our contributions are as follows:

• Through an intuitive motivating study, we show that the spatial allocation of visual supervision has a substantial impact on downstream control performance, establishing it as an important design consideration for WAMs.

• We propose Motion-Guided Asymmetric Prediction (MoGAP), a visual prediction task that trains WAMs to predict more heavily corrupted dynamic regions from cleaner static context, without requiring external supervision models or additional modalities.

• Through extensive experiments, we demonstrate that MoGAP delivers consistent and substantial gains over strong WAM baselines across diverse tasks, embodiments, and environments.

Method

We introduce Motion-Guided Asymmetric Prediction (MoGAP), a visual prediction task that promotes control-relevant representation learning in World Action Models (WAMs). We first describe the standard WAM framework in § 2.1. In § 2.2, we present a motivating study investigating how the visual prediction objective affects policy learning. Finally in § 2.3, we present our MoGAP, with an illustrated overview in Figure 2.

Preliminaries: World Action Models

At timestep $t$, World Action Models (WAMs) predict future visual observations ${\boldsymbol{o}}_{t+1:t+H+1}$ and an action chunk ${\boldsymbol{a}}_{t:t+H}$ conditioned on the current observation ${\boldsymbol{o}}_t$, task instruction ${\boldsymbol{c}}$, and proprioceptive state ${\boldsymbol{q}}_t$. Here, slice notation denotes half-open intervals, so both prediction sequences have horizon $H$. The model $f_\theta$ learns the conditional joint distribution

Building on pretrained video diffusion backbones, WAMs perform visual prediction in the latent space of a pretrained video autoencoder. We denote the encoded future observations by ${\boldsymbol{z}}_t = \mathcal{E}({\boldsymbol{o}}_{t+1:t+H+1})$, where $\mathcal{E}$ is the video encoder; the corresponding decoder maps predicted latents back to the observation space.

Recent approaches [?, 10, 5, 7] train visual and action prediction jointly through flow matching [13, 14]. At flow-matching timesteps $\tau_v,\tau_a \in [0,1]$, noisy visual latents and actions are constructed as

where $\boldsymbol{\epsilon}_v,\boldsymbol{\epsilon}_a \sim \mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ are sampled independently. The model predicts visual and action velocity fields, denoted by $(\hat{\boldsymbol{u}}_v,\hat{\boldsymbol{u}}_a)=f_\theta({\boldsymbol{z}}_t^{\tau_v},{\boldsymbol{a}}_t^{\tau_a};{\boldsymbol{o}}_t,{\boldsymbol{c}},{\boldsymbol{q}}_t)$, and is trained with the flow-matching objectives:

The overall training objective is

where $\lambda_{\mathrm{action}} > 0$ is a weighting coefficient that balances the two objectives.

Motivating Study

To investigate how visual supervision affects action learning, we conduct a small experiment fixing the supervision budget, and redistributing it spatially across tokens. For this, we compute a motion proxy signal directly from temporal differences in the clean video latents ${\boldsymbol{z}}_t$. For temporal slice $n$ and spatial latent location $p$, we compute a motion score $m_{n,p}=\|{\boldsymbol{z}}_{t,n,p}-{\boldsymbol{z}}_{t,n-1,p}\|_2$. We normalize the scores by their spatial mean and define spatial reallocation weights as follows:

where $\widetilde m_{n,p}$ is the normalized score, and $P$ is the number of spatial locations. Larger motion scores receive higher weights, directing more visual supervision toward regions with greater temporal change. Here, $\beta>0$ controls the emphasis on the high-change regions, while $\varepsilon>0$ maintains supervision in low-change regions.

We use these weights to define the dynamic-region prioritized visual objective:

where $\mathcal{L}_{\mathrm{video}}^{n,p}$ denotes the flow-matching error at temporal slice $n$ and spatial location $p$, and $N$ is the number of future latent slices.

For this study, we replace $\mathcal{L}_{\mathrm{video}}$ in (2.5) with $\mathcal{L}_{\mathrm{video}}'$ to redistribute visual supervision toward dynamic regions, leaving the action prediction objective unchanged. As shown in Figure 3, this reallocation improves the action policy performance on GR-1 tabletop, a simulated humanoid manipulation benchmark. We interpret this finding as evidence that visual regions need not contribute equally to action learning: dynamic regions tend to capture robot motion and object interactions, whereas static regions may contain visually complex but less control-relevant details, such as tabletop textures. This suggests that visual prediction can better support control by assigning different learning roles to dynamic and static regions.

Motion-Guided Asymmetric Prediction

Building on this insight, we introduce Motion-Guided Asymmetric Prediction (MoGAP), a new visual prediction task for WAMs that uses less corrupted static regions as context for predicting more heavily corrupted dynamic regions. While our motivating experiment prioritizes dynamic regions by reweighting prediction errors, MoGAP extends this principle to the prediction task itself by controlling how much visual information is available in dynamic and static regions.

Designing the asymmetric prediction task.  To construct this asymmetric prediction task, we exploit the interpretation of diffusion noising as partial masking [17], where higher noise levels reveal less information about the underlying clean context. Given the base visual timestep $\tau_v$, we independently draw $\xi\sim\mathcal{T}_v$, where $\mathcal{T}_v$ is the video timestep sampling distribution, and rescale it to the interval $[\tau_v,1]$:

This preserves the base noise level while increasing corruption in dynamic regions, with the noise gap scaled by the remaining interval $1-\tau_v$. We construct a binary motion mask $M_{n,p}$ by thresholding latent motion scores. The mask is mapped to the model’s spatial token grid to construct a dual-timestep assignment $\boldsymbol{\tau}_M$:

where $\bar m_n$ is the spatial mean. Using Gaussian noise $\boldsymbol{\epsilon}_v'\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ sampled independently of $\boldsymbol{\epsilon}_v$, we construct the asymmetrically noised visual latents as

The corresponding flow-matching target is $\boldsymbol{\epsilon}_v'-{\boldsymbol{z}}_t$. Dynamic regions are thus predicted under greater corruption, with less corrupted static regions providing scene context.

For these noisy inputs, we apply the video flow-matching loss in (2.3) only to moving tokens with $M_{n,p}=1$. Within each latent slice, we normalize by the number of selected locations so that the moving-region fraction does not change the total loss weight. Let $\mathcal{S}_n=\{p\mid M_{n,p}=1\}$, with all spatial locations selected when the mask is empty. The resulting objective is:

where $\hat{\boldsymbol{u}}_{v,n,p}^{\boldsymbol{\tau}_M}$ denotes the visual velocity prediction under the asymmetric timestep assignment.

Incorporating MoGAP into WAMs.  As MoGAP provides complementary motion-guided supervision to standard video-action prediction, we incorporate it as an auxiliary training path that augments the original WAM objective. We process the dual-timestep-noised visual latents ${\boldsymbol{z}}_t^{\boldsymbol{\tau}_M}$ alongside the original WAM inputs in the shared video diffusion backbone, without introducing additional parameters. To keep the original WAM prediction path independent of the auxiliary tokens, we apply an asymmetric attention mask between the two paths. As shown in  Figure 4, the original WAM tokens, including noisy future visual and action tokens, cannot attend to the MoGAP tokens. MoGAP tokens can attend to one another, the current observation, and noisy action representations, but not the original path’s noisy future visual tokens. The task instruction ${\boldsymbol{c}}$ conditions both paths through the existing conditioning mechanism.

We jointly optimize the standard WAM objectives with the objective:

where $\lambda_{\mathrm{MoGAP}}>0$ controls the contribution of motion-guided supervision. At inference time, the MoGAP tokens are omitted, leaving the original WAM inference procedure unchanged.

Experiments

In this section, we evaluate MoGAP across diverse manipulation settings, robot embodiments, and WAM architectures. We first describe the benchmarks and implementation details in § 3.1. In § 3.2, we study its impact on frontier WAM baselines, across multitask simulated manipulation benchmarks. We then assess MoGAP’s performance in real-world physical deployment, on an OpenArm Humanoid equipped with Inspire hands, in § 3.3. Finally, in § 3.4, we present informative analyses to better understand the effects of MoGAP.

Setup

Benchmarks.  We evaluate MoGAP on two multitask simulation benchmarks and a real-world manipulation setup, covering different embodiments, observation configurations, and environmental conditions.

• RoboCasa GR-1 Tabletop [8] consists of 24 tabletop hand manipulation tasks using the GR-1 humanoid. We use this benchmark for our main study, including comparisons with existing methods and ablation studies.

• RoboCasa-Kitchen [19] comprises 24 manipulation tasks using a single-arm robot with a gripper in kitchen environments. We use this benchmark to evaluate MoGAP in multi-view observation settings.

• Real-world evaluation uses an OpenArm humanoid with two Inspire RH56F1 hands. We evaluate both in-domain tasks and diverse out-of-domain conditions to assess the abilities of MoGAP in physical deployment.

Implementation details.  We adopt Cosmos3-Nano [7] as our primary WAM, generating 16 actions and 8 future video frames with a temporal stride of two in a single forward process. We train for 16K gradient steps for the multi-task simulation benchmarks and 8K steps for real-world experiments, respectively, with a global batch size of 256. We follow the original work for the training details, and set $\lambda_{\mathrm{MoGAP}} = 1.0$. For the simulated benchmarks, we evaluate each method over 50 trials per task, for a total of 1,200 rollouts. Further details are described in Appendix appendix:implementationdetails.

Method	PnP	Articulated
Average
VLA-based methods
GR00T-N1.6 [8]	55.4	24.2	47.6
StarVLA-$\alpha$ [3]	58.4	54.0	57.3
RLDX-1 [4]	59.0	57.7	58.7
WAM-based methods
Fast-WAM [5]	33.8	53.7	38.8
DiT4DiT [6]	50.9	50.3	50.8
Cosmos3 [7]	48.3	64.3	52.3
MoGAP (Ours)	60.8	66.7	62.3

Method	Teacher
Average
Cosmos3	–	52.3
w/ Geometry Forcing [15]	VGGT	53.9
w/ AGRA [24]	DINOv2	55.4
w/ TSA [9]	VideoMAEv2	56.8
w/ VideoREPA [2]	VideoMAEv2	57.8
w/ VideoREPA [2]	V-JEPA2.1-L	59.2
w/ MoGAP (Ours)	–	62.3

Simulated benchmark results

Comparison with state-of-the-art methods.  Table 1 summarizes the performance on RoboCasa GR-1 Tabletop. Cosmos3 already provides a strong baseline, outperforming the other WAM-based methods and remaining competitive with leading VLA-based methods. Despite this strong starting point, MoGAP improves the average success rate from 52.3% to 62.3%, a gain of 10.0 percentage points, surpassing all VLA and WAM-based methods in the comparison. The gains are particularly pronounced on pick-and-place tasks, which require precise manipulation of a wider variety of objects within each task, with success rates increasing from 48.3% to 60.8% (+12.5 percentage points). We attribute this improvement to MoGAP’s emphasis on fine-grained motion that produces only subtle visual changes but is important for manipulation, especially for precise positioning during grasping and placing. We further visualize this effect in our qualitative analysis (see fig:analysis).

Comparison to teacher-based supervision.  We compare MoGAP with teacher-based representation learning methods that leverage visual foundation models [27, 20, 16] or 3D foundation models [28] as teachers, using the same Cosmos3-Nano baseline model. As shown in Table 2, all the other methods improve over the baseline, yet MoGAP outperforms them without an external teacher. While external representations provide useful supervision, MoGAP shows that motion-guided regional noise allocation offers an effective alternative for adapting the pretrained video backbone’s native representations to robot control.

Method	24 Tasks
Average
gray!10 MoGAP	62.3
w/o auxiliary path	54.4
w/o noise asymmetry	55.3
reversed direction	55.2

Component analysis. tab:component_analysis studies individual components of MoGAP. As shown in the first two rows, replacing the main visual supervision with our method improves the success rate from 52.3% to 54.4%, while introducing it as an auxiliary branch yields a substantially larger gain. This suggests that the standard full-noise range supervision and our asymmetric supervision provide complementary learning signals. The third and fourth rows further show that removing the noise allocation asymmetry or reversing the asymmetry direction significantly reduces the success rate to 55.3% and 55.2%, respectively. These results demonstrate that asymmetric noise allocation toward predicting dynamic regions from static context is a crucial design choice for downstream performance.

Generalization across WAM architectures.  We further assess MoGAP on Fast-WAM [5], which combines a Wan2.2-5B [18] video diffusion backbone with a separately parameterized action expert and omits future visual prediction at inference. MoGAP improves the success rate from 38.8% to 47.5% (tab:fastwam). Together with the improvements on Cosmos3, these results demonstrate MoGAP’s effectiveness across different WAM architectures and video backbones.

Adaptation to multi-view configuration.  We also evaluate MoGAP on RoboCasa-Kitchen, a single-arm manipulation benchmark with three camera views. We apply motion-guided noise assignment to the two external camera views while assigning all wrist-camera tokens to the high-noise group, as camera motion can dominate the latent motion scores in the wrist view. With this adaptation, MoGAP consistently outperforms the Cosmos3 baseline while retaining the benefits of longer execution horizons (Figure 5). These results demonstrate that MoGAP extends to multi-view settings that include a moving wrist camera.

Real-world experiment results

Real-world experimental setup.  We conduct experiments on an OpenArm humanoid with two 7-DoF arms and two five-finger, 6-DoF hands and an egocentric stereo camera. We evaluate three task groups targeting different aspects of policy performance: In-Domain tasks for general manipulation, OOD Objects & Environments tasks for robustness to unseen objects and environments, and OOD Object Relocation tasks for evaluating the policy’s ability to focus on task-relevant regions (fig:realworld_task). For low-latency deployment, we use a single denoising step for both visual and action prediction without classifier-free guidance [29]. We precompute and cache text embeddings, removing the text encoder from the inference path, and omit the VAE decoder. Together, these choices enable Cosmos3-Nano, a 16B-parameter model, to run with an inference latency of 48ms on a single RTX 5090 GPU with 32GB of VRAM. For each method, we train a single policy on the same multitask dataset collected in our real-world setup. Further details are provided in Appendix appendix:realdetails.

Real-world evaluation results.  fig:realworld_result presents the results on the real-robot manipulation tasks. MoGAP consistently improves over the baseline policy across all three groups, demonstrating gains in both general manipulation and robustness to distribution shifts. Notably, we find that MoGAP yields particularly large gains on the object relocation tasks. In these trials, the baseline policy often continues its original trajectory and attempts to grasp at the object’s previous location, whereas the MoGAP-trained policy more successfully adjusts its trajectory toward the relocated object. This adaptation emerges despite the absence of object relocation demonstrations during training. We attribute this behavior in part to MoGAP’s emphasis on dynamic regions, which encourages the policy to respond more effectively to task-relevant visual changes in the environment.

Further Analysis

Motion score analysis.  We analyze the motion score derived from temporal differences between video latents. In fig:analysis, the scores highlight subtle arm and hand movements (rows 1 and 2) and remain focused on the robot and objects even under camera shifts caused by waist motion (row 3). We quantify this localization using region-wise AUROC, which evaluates how well the motion scores rank locations within each ground-truth region relative to the rest of the scene (tab:motionscore). The results show that the signal clearly assigns higher scores to the moving arm (including the hand) and the moving target object, while assigning relatively lower scores to background regions (i.e., the tabletop, surrounding environment, and distractor objects).

[l]Evaluation
Region & AUROC
Moving arm & hand	0.887
Manipulated object	0.834
Background	0.430

Trained policy analysis.  Training with the latent motion score supervision encourages WAMs to attend more closely to dynamic regions during action generation and improves their prediction of these regions. In fig:analysis, we visualize the action tokens’ attention, and observe that it is concentrated more strongly on the moving hand and arm. In the generated examples, the baseline generation often appears blurred around the moving arm, while ours preserves its structure relatively more clearly (rows 2 and 3). Quantitative results in tab:policyanalysis support this attention shift, showing greater attention to the moving arm and hand and the manipulated object, and reduced attention to the background. Further details about the analysis metrics are provided in Appendix appendix:analysisdetails.

Limitations.  MoGAP introduces additional tokens during training, increasing wall-clock training time by a factor of 1.26 in exchange for improved policy performance. To assess this trade-off, we provide computational details and a compute-matched comparison in Appendix appendix:computationdetails.

Related Work

World Action Models (WAMs).  WAMs adapt rich spatiotemporal priors from large-scale video data to robot control by learning to predict future visual observations and control actions together [11, 12, ?, 31, 10]. However, visual reconstruction alone may not fully align with the needs of robot control [25, 36]. Recent studies incorporate supervision from semantic encoders [24, 26], or from depth and 3D geometry [21, 22, 23, 35]. Such approaches augment visual supervision through pretrained representation models [27, 28], or additional prediction targets derived from geometry and motion estimators [32, 33, 34]. Our work explores a complementary direction by introducing an asymmetric prediction task for action learning within the pretrained video diffusion model’s native latent space, without external teachers or additional modalities.

Representation learning for video generative models.  Recent work explores representation learning within video generative models to improve prediction quality. One line of work aligns diffusion and flow model representations with external pretrained encoders [30], encouraging physically plausible predictions [2, 38] or incorporating 3D geometric structure [15]. Another line improves representations through self-supervision without external teachers [1, 37]. Our approach builds on these advances, drawing in particular on token-wise noise assignment [17, 1] to introduce motion-guided asymmetry tailored to action modeling for robot control.

Conclusion

In this work, we present Motion-Guided Asymmetric Prediction (MoGAP), a visual prediction task that adapts pretrained video diffusion models for robot control through motion-guided regional noise allocation. Our experiments demonstrate consistent improvements over strong WAM baselines in simulated and real-world manipulation settings, including improved adaptation to target object relocation tasks during execution. These findings highlight the importance of tailoring visual supervision for robot policy learning, particularly by emphasizing regions that capture task-relevant dynamics. We hope this work encourages further exploration of control-oriented visual prediction to translate pretrained video models’ world knowledge into more capable robotic policies.

Acknowledgments

This work was partly supported by Institute for Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019-II190075, Artificial Intelligence Graduate School Program (KAIST)), and RLWRLD Inc. We thank the members of Algorithmic General Intelligence Laboratory (AGI-LAB) of KAIST, for helpful support in conducting real-world experiments.