betaFeedback

WorksComputer Science

PrismWAM: Decoupling State Representations in World Action Models

Abstract

World Action Models (WAMs) typically represent the world through latent embeddings from pretrained video models, which entangle heterogeneous aspects of the world, from how it looks (e.g., texture and color) to how it is physically configured (e.g., occupancy and pose). This raises an underexplored question: what should constitute the state of a WAM?\textbf{what should constitute the state of a WAM?} We argue that the world state should separate how the world appears from how the world evolves under action: the former captures semantic context, while the latter is governed by physical state. To this end, we propose PrismWAM\textbf{PrismWAM}, which decouples physics from semantics while jointly modeling state and action. PrismWAM factorizes an embodied task into a Semantic Planning→Physical Rollout→Executable Action\mathrm{Semantic\ Planning}\rightarrow\mathrm{Physical\ Rollout}\rightarrow\mathrm{Executable\ Action} pipeline: given only the initial RGB observation, semantic planning learns to predict the final goal image, while physical rollout predicts the fine-grained evolution toward this goal in depth space; executable actions are then generated from the depth rollout. Without any embodied action pretraining or additional data augmentation, PrismWAM achieves strong manipulation performance in both simulation and the real world. Notably, PrismWAM even surpasses a broad range of models with large-scale embodied action pretraining (e.g., OpenWAM-αα). More broadly, we reframe World Action Models from modeling how the world looks to how it responds to action, paving the way for action-centric world models grounded in occupancy, flow, tactile sensing, proprioception, and beyond.

Equal contribution †Corresponding author

Project Page: https://prism-wam.github.io/

Introduction

World models have emerged as a promising paradigm for enabling autonomous agents to predict the consequences of their actions ([17, 22]). By learning how the world evolves over time, these models aim to provide a foundation for planning and action ([18]). Recent World Action Models (WAMs) extend this paradigm by jointly modeling video and action, often relying on pretrained video backbones to capture the rich visual dynamics of the physical world ([26] (Ye et al. 2026b; Ye et al. 2026a)). Yet, a key question remains underexplored: are video latents truly the most appropriate representations for world states?

Our key observation is that video latents capture a mixture of factors that are not equally relevant to action generation ([35]). While appearance and texture provide rich semantic context for understanding the scene, they are often invariant to the actions being modeled. For example, when an agent moves or grasps an object, its geometry and pose may change, while its color, texture, and intrinsic appearance typically remain unchanged. Therefore, video latents which entangles the physical state with the semantic context can make the representation substantially richer than what is necessary for modeling how the world evolves under action.

This motivates a simple principle: a good state representation in WAMs should distinguish between physical state and semantic context. The physical state refers to action-relevant properties such as geometry, pose, occupancy, contact, and potentially dynamics, whereas semantic context provides the appearance information necessary to interpret that state, such as object identity, category, and texture; see Figure 1 (a). The distinction is not between information that is useful and useless; rather, it decouples what actions can change from what helps us understand what is being acted upon.

Figure 1. Overview of PrismWAM. (a) Our key insight is to decouple physical state from semantic context, yielding state representations that are better aligned with action-induced dynamics. (b) We follow a causal workflow of what to do →\to how the world responds →\to how to act, corresponding to Semantic Planning →\to Physical Rollout →\to Executable Action. (c) Comparison with starVLA and FastWAM across five benchmarks.

Therefore, we propose PrismWAM to separate semantic context from physical dynamics through a Semantic Planning →\to Physical Rollout →\to Executable Action workflow (see Figure 1 (b)), rather than treating a single video latent as a monolithic state. Semantic planning infers what should happen in the end from RGB observations, physical rollout models how the world physically evolves toward the goal, and executable action determines how to make it happen. We instantiate this principle with a Mixture-of-Transformers (MoT) architecture built upon a pretrained video backbone (Team Wan et al. [48]), as shown in Figure 2. During training, the semantic planner predicts the target from the initial RGB observation and transfers its task-level knowledge to the physical rollout via attention layers, while the action branch learns to map the predicted physical evolution to continuous actions. We use depth as the physical representation for its accessibility, physical grounding, and spatial alignment with RGB. Notably, the semantic planner serves only as a training-time teacher, with action inference relying solely on the initial RGB observation and learned depth rollout.

Figure 2. PrismWAM decouples semantic planning from physical rollout. Left: the RGB stream learns task semantics through sparse supervision from a single intended-future frame, while the depth stream predicts a dense geometric trajectory jointly with actions. Right: chain attention routes observed-RGB features to the depth stream and physical rollout features to the action tower.

Extensive experiments across diverse simulation and real-world benchmarks validate the effectiveness and efficiency of PrismWAM. (1) Without action pretraining, PrismWAM substantially outperforms representative VLAs and WAMs under the same setting, improving the success rate over FastWAM by 43.82 percentage points (83.67 vs. 39.85) on RoboTwin-C2R and the score over starVLA by 29.83 points (36.23 vs. 6.40) on RoboDojo; see Figure 1(c). (2) PrismWAM outperforms several WAMs, including models with action pretraining, with particularly strong gains in precision manipulation and generalization: on RoboDojo Precision, it surpasses OpenWAM-α\alpha by 21.37 points (39.82 vs. 18.45), while improving the success rate on RoboTwin-C2R by 14.62 percentage points (83.67 vs. 69.05), showing a higher ceiling for complex tasks and novel scenes. (3) PrismWAM even matches or surpasses large-scale pretrained models on several benchmarks despite being trained with only ∼\sim2,300 hours of depth data. On RoboDojo, it improves the score by 16.00 points over G0.5 (36.23 vs. 20.23) and 16.16 points over Xiaomi-Robotics-1 (36.23 vs. 20.07), showing that scaling action pretraining is not the only path to stronger embodied foundation models; principled architectural design offers a more data-efficient alternative. Our key contributions are listed as follows:

  • A decoupling principle for World Action Models. We identify the entanglement of semantics and physics in video latents as a fundamental limitation of existing WAMs, forcing action generation to disentangle high-level task planning from fine-grained physical state transitions.

  • An effective and efficient WAM architecture. We instantiate the principle with a Mixture-of-Transformers (MoT), where an RGB semantic planner guides depth-based physical rollout, and develop a learning pipeline combining dense depth supervision with sparse RGB goal supervision, enabling fast inference without explicit RGB goal prediction.

  • Strong performance and data efficiency. PrismWAM outperforms existing VLAs and WAMs, with particularly strong gains in precision manipulation and generalization. It also surpasses a number of models with large-scale action pretraining, demonstrating that principled architectural design offers a more data-efficient path toward stronger embodied foundation models.

Related Works

Vision-Language-Action Policies. The general and embodied knowledge encoded in pretrained vision-language models (VLMs) [9, 1, 25, 49] (Xiaomi Embodied Intelligence Team 2025) provides a foundation for vision-language-action (VLA) models that map visual observations and language instructions to robot actions [4, 20, 3, 27, 37, 36, 46]. Internet-scale image–text pretraining provides semantic priors for complex instruction understanding and long-horizon manipulation [4, 3, 37, 36, 44]. However, such pretraining emphasizes recognition and semantic alignment, with limited explicit supervision for continuous state transitions and how actions change the environment.

World Action Models. Advances in video generative models [48, 35, 34] have paved the way for world action models (WAMs), a promising paradigm that jointly models future states and robot actions for generalist robotic manipulation. Most jointly model future RGB observations and action sequences [21] (Zhu et al. 2025). Some methods guide actions based on predicted future RGB observations [42, 2, 26, 49] (Ye et al. 2026b; Xu et al. 2026; Zhang et al. 2026b). Others use future prediction as training supervision and generate actions directly from observed context [11, 15] (Yuan et al. 2026; Ye et al. 2026a; Zhang et al. 2026a). Both approaches primarily learn state evolution in RGB appearance space, which entangles action-relevant geometry and pose with texture, illumination, and background. This requires modeling appearance details that may be irrelevant to control and inferring physical changes from visual variations that may not reflect motion [38] (Zhang et al. 2026a).

Physical Representations for Manipulation. Physical or spatial representations support precise manipulation and spatial generalization [45, 14, 19, 39] (Ze et al. 2024). Such information can directly guide action generation through depth or point-cloud features extracted from current observations [23, 19] (Yuan et al. 2025). Complementary approaches introduce physical or spatial structure through supervision, including representation alignment [24, 43] (Wu et al. 2026; Yang et al. 2026) and auxiliary prediction [16, 28] (Zhang et al. 2026a; Zhang et al. 2025).

Recent world action models take RGB and geometry as joint inputs and predict their future states together [32] (Zhao et al. 2026). This couples semantic and physical information, retaining the RGB-based modeling issues discussed above. PrismWAM separates their roles along a Semantic Planning →\to Physical Rollout →\to Executable Action chain, distinguishing semantic task understanding from the physical state evolution that guides action execution.

Method

PrismWAM follows a Semantic Planning →\to Physical Rollout →\to Executable Action chain for prediction and control. The RGB stream learns semantic planning representations from a sparse future-frame target, while the depth stream predicts a dense physical rollout that guides action generation. We first present the problem formulation § 3.1, then describe PrismWAM’s architecture, training, and inference § 3.2.

Problem Formulation

We formulate a world–action model (WAM) that jointly predicts world-state evolution and the corresponding actions. Let x0x_0 denote the current world state, x1:Fx_{1:F} its future world state trajectory, and a1:Ha_{1:H} the corresponding action sequence. Given task context c=(ℓ,s)c = (\ell, s), comprising the language instruction ℓ\ell and proprioceptive state ss, the policy can be expressed as

πθ(a1:H∣x0,c)=∫pθ(a1:H∣x1:F,x0,c)⏟action generationpθ(x1:F∣x0,c)⏟world-state evolution dx1:F.(1)\pi_\theta(a_{1:H} \mid x_0,c) = \int\underbrace{p_\theta(a_{1:H} \mid x_{1:F}, x_0, c)}_{\text{action generation}} \underbrace{p_\theta(x_{1:F} \mid x_0,c)}_{\text{world-state evolution}} \,\mathrm{d}x_{1:F}. \tag*{(1)}

Here, pθ(x1:F∣x0,c)p_\theta(x_{1:F} \mid x_0, c) models how the current state may evolve over the prediction horizon, while pθ(a1:H∣x1:F,x0,c)p_\theta(a_{1:H} \mid x_{1:F}, x_0, c) predicts actions conditioned on the current state and a future state trajectory. Marginalizing over possible future world state trajectories yields the policy’s action distribution. The future world state trajectory x1:Fx_{1:F} and action sequence a1:Ha_{1:H} cover the same time interval.

Semantic Planning and Physical Rollout. We use historical and current RGB observations r≤0r_{\leq0} for semantic planning and the current depth observation d0d_0 as the initial physical state. Semantic planning captures task intent and the intended scene configuration, while physical rollout models the intermediate geometric changes needed for action execution. Guided by RGB representations, the depth stream predicts the sequence d1:Fd_{1:F}, and the action tower jointly generates executable actions from the evolving rollout features.

Let zRz_R denote semantic planning representations extracted from r≤0r_{\leq0} and zDz_D the physical rollout features produced during depth prediction. The resulting policy is

πθ(a1:H∣r≤0,d0,c)=∬pθ(a1:H∣zD,c)⏟Executable Actionpθ(zD∣d0,zR,c)⏟Physical Rolloutpθ(zR∣r≤0,c)⏟Semantic Planning dzR dzD.(2)\pi_\theta(a_{1:H} \mid r_{\leq0}, d_0, c) = \iint\underbrace{p_\theta(a_{1:H} \mid z_D,c)}_{\text{Executable Action}} \underbrace{p_\theta(z_D \mid d_0,z_R,c)}_{\text{Physical Rollout}} \underbrace{p_\theta(z_R \mid r_{\leq0},c)}_{\text{Semantic Planning}} \,\mathrm{d}z_R\,\mathrm{d}z_D. \tag*{(2)}

PrismWAM

Architecture with Sparse–Dense Temporal Supervision. Building on the pretrained video backbone WAN (Team Wan et al. [48]), PrismWAM uses RGB and depth streams with asymmetric temporal supervision, coupled to an action tower through a mixture-of-transformers (MoT) architecture (Figure 2, left). The RGB stream predicts a single intended-future frame rFr_F, a sparse target for learning task semantics. The depth stream predicts the denser sequence d1:Fd_{1:F}, providing intermediate spatial states that guide action evolution. This assigns semantic planning to sparse RGB prediction and physical rollout to dense geometric prediction, whose representation is less entangled with texture and illumination.

The model receives RGB history r≤0r_{\leq0}, current depth d0d_0, instruction ℓ\ell, and proprioception ss. Camera views are resized into matching RGB and depth mosaics, and RGB values and clipped metric depth are normalized. The frozen RGB and depth VAEs encode observations and future training targets into latent representations, which modality-specific patch projections convert into visual tokens. Frozen T5 embeddings and projected, normalized proprioception form the context CC, which all branches access through cross-attention. Appendix § 8.§ 1 specifies input layouts and temporal sampling.

During joint training, the RGB token sequence combines clean history latents with noisy latents of the single future target, while the depth sequence combines clean current-depth latents with noisy future-depth latents. The action branch linearly projects each action vector in the noisy sequence into an action token. Gaussian corruption is applied only to prediction targets; observed RGB and depth remain fixed. Each branch receives its own noise-level embedding.

Chain Attention. A structured attention mask controls information flow between these token groups (Figure 2, right). Within each visual stream, future tokens attend to observed tokens and one another, while observed tokens cannot read future tokens. Depth tokens additionally read only the observed-RGB tokens. Action tokens attend to one another and to both observed and future depth tokens, with no direct access to RGB. These cross-modal connections are directed: RGB does not read depth or actions, and depth does not read actions. Thus, action prediction uses evolving depth features without accessing the future-RGB target.

Specifically, let hm(j)h_m^{(j)}, m∈{R,D,A}m \in\{R,D,A\}, denote the RGB, depth, and action features after jj layers, with hm(0)h_m^{(0)} their input embeddings and hR,≤0(j)h_{R,\leq0}^{(j)} the observed-RGB subset. Both visual streams and the action tower receive the encoded task context CC through cross-attention. With modality-specific noise levels σR,σD,σA\sigma_R,\sigma_D,\sigma_A, the three feature sequences update synchronously as

hR(j+1)=BV(j)(hR(j)∣C,σR),hD(j+1)=BV(j)(hD(j)∣hR,≤0(j);C,σD),hA(j+1)=BA(j)(hA(j)∣hD(j);C,σD,σA).(3)\begin{aligned} h_R^{(j+1)} &= \mathcal{B}_V^{(j)}\left(h_R^{(j)} \mid C,\sigma_R\right),\\ h_D^{(j+1)} &= \mathcal{B}_V^{(j)}\left(h_D^{(j)} \mid h_{R,\leq0}^{(j)};C,\sigma_D\right),\\ h_A^{(j+1)} &= \mathcal{B}_A^{(j)}\left(h_A^{(j)} \mid h_D^{(j)};C,\sigma_D,\sigma_A\right). \tag*{(3)} \end{aligned}

Here BV(j)\mathcal{B}_V^{(j)} is the visual block shared by the RGB and depth streams, and BA(j)\mathcal{B}_A^{(j)} is the independently parameterized action block. The visual streams retain modality-specific input, conditioning, and output projections. After the final layer, visual heads project tokens back to latent grids to predict RGB and depth flow velocities, while the action head predicts a velocity vector for each step in normalized action space. These outputs are supervised by the joint flow-matching objective below.

Training Details. We initialize the depth VAE from the pretrained Wan VAE and pretrain it on 2,300 hours of depth data (see Appendix § 8.§ 2 for the data mixture, losses, and optimization settings); both VAEs and T5 remain frozen during policy training. Policy training follows two stages. First, we adapt the pretrained Wan backbone to depth space by predicting depth sequences conditioned on current depth and language, using depth-prediction loss alone. We then jointly optimize the RGB and depth streams, action tower, and conditioning projections with sparse RGB goal supervision, dense intermediate depth supervision, and action supervision, connecting semantic planning, physical rollout, and executable action generation.

For joint training, let R≤0,RFR_{\leq0},R_F and D0,D1:FD_0,D_{1:F} denote the encoded RGB and depth observations and targets, and A1:HA_{1:H} the normalized action chunk; visual subscripts identify the source frames rather than the number of temporally compressed latent frames. For targets (XR,XD,XA)=(RF,D1:F,A1:H)(X_R,X_D,X_A)=(R_F,D_{1:F},A_{1:H}), we independently sample Gaussian noise ϵm\epsilon_m and noise levels σm∈[0,1]\sigma_m \in[0,1] across modalities:

X~m=(1−σm)Xm+σmϵm,vm=ϵm−Xm,ϵm∼N(0,I).(4)\widetilde{X}_m=(1-\sigma_m)X_m+\sigma_m\epsilon_m,\qquad v_m=\epsilon_m-X_m,\qquad\epsilon_m\sim\mathcal{N}(0,I). \tag*{(4)}

Conditioned on R≤0,D0,CR_{\leq0},D_0,C, the model predicts velocities v^m\widehat{v}_m and minimizes the joint flow-matching objective ([29]):

L=LR+LD+LA,Lm=E[w(σm)MSE⁡(v^m,vm)].(5)\mathcal{L}=\mathcal{L}_R+\mathcal{L}_D+\mathcal{L}_A,\qquad\mathcal{L}_m=\mathbb{E}\left[w(\sigma_m)\operatorname{MSE}(\widehat{v}_m,v_m)\right]. \tag*{(5)}

Here ww weights noise levels, and the expectation is over training examples, noise, and noise levels. MSE is averaged over valid prediction targets, excluding observation and padding. The noise schedule and weighting function are specified in Appendix § 8.§ 1; benchmark-specific budgets, data mixtures, and optimizer settings are collected in Table 5.

SettingRoboTwin C2RLIBERORoboDojo
Training dataCleanFour suitesARX X5 sim.
Tasks / total demos50 / 2,50040 / 2,00035 / 3,500
Demos per task50 clean50100
Optimizer updates: depth / joint15k / 15k15k / 15k15k / 40k
Global batch size2562561024
Views; mosaic (H×WH \times W)3; 384 ×\times 3202; 224 ×\times 4483; 384 ×\times 320
Depth frames959
RGB history frames559
Future-RGB targett+32t + 32t+16t + 16t+32t + 32
Action representation14-D absolute joints7-D EE deltas14-D abs. joints
Proprioceptive dimension14814
Target action indices[t+1,t+32][t + 1, t + 32][t,t+15][t, t + 15][t+1,t+32][t + 1, t + 32]
Action / state statisticsShared mean/stdSeparate min–maxShared mean/std
Horizon / replan (H/KH/K)32 / 2416 / 1632 / 24

Table 5. PrismWAM training and deployment settings. C2R: clean-to-random transfer; EE: end-effector; H/KH/K: action horizon/executed prefix.

Inference. At inference (Figure 2, lower right), the RGB stream encodes r≤0r_{\leq0} once and caches its features for semantic guidance, without generating future RGB frames. Future depth latents and the normalized action chunk are initialized from Gaussian noise and jointly denoised, with the observed RGB, current depth, and task context held fixed. The resulting depth latents can be decoded into a depth sequence for visualization, but action generation uses intermediate rollout features and does not require decoding depth maps. We denormalize the predicted action chunk, execute a prefix, and replan from the next observation.

Experiment

Without embodied action pretraining, we evaluate PrismWAM across five simulation benchmarks spanning both in-distribution manipulation and out-of-distribution transfer (§ 4.2). We further examine how our role-decoupled representations support control through robustness analyses, controlled ablations, and discussion (§ 4.3), and validate the approach in real-world manipulation (§ 4.4). Training settings and detailed benchmark results are provided in Appendices A.1 and B, respectively.

Experiment Setup

Benchmarks. We evaluate in-distribution manipulation and out-of-distribution transfer across five simulation benchmarks: RoboTwin clean-to-random (Chen et al., [8]), LIBERO (Liu et al., [30]), LIBERO-Plus (Fei et al., [12]), LIBERO-Pro (Zhou et al., 2025), and RoboDojo (Chen et al., [7]). LIBERO assesses in-distribution manipulation under its standard protocol. RoboTwin clean-to-random, LIBERO-Plus, and LIBERO-Pro probe transfer across diverse visual, spatial, semantic, task, and environmental shifts, while RoboDojo evaluates manipulation capabilities using both task scores and success rates.

Baselines. We compare PrismWAM with representative cross-benchmark VLAs and WAMs, including methods trained without embodied action pretraining (starVLA (StarVLA Community, [47]), FastWAM (Yuan et al., 2026)), methods that use embodied action pretraining (π0.5\pi_{0.5} (Physical Intelligence et al., [37]), OpenWAM-α\alpha (Wang et al., [49])), and methods that incorporate spatial information during pretraining or fine-tuning (X-WAM (Guo et al., [16]), SpatialForcing (Li et al., ...)) 2025b), 4D-WAM (Yang et al., 2026). Hatched bars in Figure 3 denote pretrained models and methods fine-tuned from them. We additionally include top-performing benchmark-specific methods (Giga-Brain Team et al., 2026; Chen et al., 2026a; Zhong et al., 2026; Kim et al., 2024; Black et al., 2024; Liu et al., 2026; Xiaomi Robotics Team et al., 2026; Dexmal Team, 2026). For RoboTwin-C2R, Figure 3 also includes ME-Dex-1.0 (Zhang et al., 2026c), a leading entry on the official leaderboard as of October 19, 2026 [41]. ME-Dex-1.0 uses tactile supervision during training and zero tactile input at evaluation. We also include InternW0-Δ (Miao et al., [33]), with publicly available fine-tuning and inference code and benchmark checkpoints. We use its reported clean/random average for RoboTwin-C2R and the October 17, 2026 leaderboard score for RoboDojo [40].

Figure 3. Cross-benchmark performance of PrismWAM and PrismWAM*. PrismWAM* denotes our exploratory scaled-up variant. Across five manipulation benchmarks, our methods outperform the cross-benchmark baselines starVLA and FastWAM and remain competitive with leading benchmark-specific methods and models using large-scale embodied action pretraining.

Main Results

Across five benchmarks, PrismWAM _substantially outperforms the cross-benchmark baselines starVLA and FastWAM_ without embodied action pretraining. On RoboTwin-C2R, PrismWAM achieves 53.16% success in randomized scenes, compared with starVLA at 3.16% (+50.00 pp) and FastWAM at 1.90% (+51.26 pp). These results demonstrate the strong zero-shot _generalization_ capability of PrismWAM without embodied action pretraining. PrismWAM further outperforms the available geometry-aware baselines on LIBERO, reaching 99.35% versus 98.60% for 4D-WAM (Table 2). These results suggest that _the key to effectively leveraging physical information is not only to incorporate it, but also to decouple it from semantic representations._ Moreover, PrismWAM _surpasses both pretrained and non-pretrained WAM baselines_ in most settings, achieving 70.57% clean/random average success on RoboTwin-C2R compared with 69.05% for pretrained OpenWAM-α. On RoboDojo, PrismWAM* achieves an overall score of 36.23 (Figure 3). Beyond WAM comparisons, PrismWAM remains competitive with large-scale pretrained policies (Figure 5). Results on RoboDojo also suggest our _data-efficiency_, achieving strong control without additional embodied backbone pretraining.

Table 2: LIBERO performance and transfer. Success rates (%).

MethodLIBERO SpatialObjectGoalLongOverallLIBERO-Plus CameraRobotLang.LightBg.NoiseLayoutOverallLIBERO-Pro ObjectPos.Sem.TaskEnv.Overall
w/ Embodied Action Pretraining
π₀.₅98.898.298.092.496.8571.2376.0685.8297.7296.1987.3286.2384.7685.4028.1595.5025.3051.4557.16
OpenWAM-α99.699.699.898.299.333.2175.8787.5797.0285.5939.8577.3868.7984.4526.1095.9525.2546.8055.71
w/o Embodied Action Pretraining
starVLA97.898.696.293.896.664.0457.1685.7593.7896.0080.9577.2577.8376.652.8094.8517.8045.0547.43
FastWAM98.2100.097.095.297.616.0146.2672.9381.9655.4839.4166.4952.5673.000.0582.9014.9031.7040.51
PrismWAM (ours)99.4100.098.299.899.3560.6067.6192.1399.6599.5467.4681.7079.4488.656.5598.5523.4555.2054.48
w/ Spatial Information
X-WAM96.898.695.495.696.6072.9246.3288.4893.6188.2962.9080.7974.8083.9521.2097.2517.7045.5553.13
Spatial-Forcing99.499.698.896.098.556.8537.9484.3995.3691.8265.9079.0271.1073.952.0594.459.8031.8542.42
4D-WAM98.899.598.697.498.645.1564.2690.6394.2957.7169.0879.2171.0180.7548.34

Table 2.

Figure 5. RoboDojo score and success rate.

Analysis and Discussion

RQ1: Does physical-state prediction offer more than appearance robustness? Depth encodes distance rather than texture or illumination, making it naturally insensitive to changes in color and lighting. This raises a key question: does physical-state prediction offer more than appearance robustness? Results across multiple benchmarks point to broader benefits for generalization and precise control. PrismWAM achieves 81.70% under layout perturbations on LIBERO-Plus, compared with 77.38% for OpenWAM-α\alpha and 66.49% for FastWAM; on LIBERO-Pro, it achieves 88.65% when object attributes such as size, shape, and color change, exceeding OpenWAM-α\alpha at 84.45% and FastWAM at 73.00% (Table 2). On RoboDojo’s challenging Precision tasks, PrismWAM substantially outperforms pretrained OpenWAM-α\alpha, achieving 19.50% success versus 9.25% (Table 1). These results suggest that physical-state prediction goes beyond appearance robustness: it supports broader generalization, as illustrated by robustness to changes in scene layout and object shape, and exposes physical structure for precise control.

Table 1: RoboTwin-C2R and RoboDojo performance. RoboTwin-C2R: success (%); RoboDojo: Score (/100) / SR (%). PrismWAM* denotes our exploratory scaled-up variant. RoboDojo results for both PrismWAM variants are means over seeds 0–2. Bold: best; _underline_: second best.

MethodRoboTwin-C2R CleanRand.Avg.RoboDojo Gen.PrecisionLong-H.MemoryOpenOverall
w/ Embodied Action Pretraining
π₀.₅70.7046.0058.3513.38/8.1712.40/5.5023.54/14.675.89/4.671.98/1.6711.44/6.93
OpenWAM-α89.4048.7069.0520.71/14.8318.45/9.2534.93/25.3310.41/9.111.41/1.0817.18/11.92
w/o Embodied Action Pretraining
starVLA46.523.1624.843.94/2.349.90/4.3314.15/6.503.34/2.440.67/0.586.40/3.24
FastWAM77.801.9039.852.34/1.111.96/0.009.14/5.173.55/3.440.42/0.423.48/2.03
PrismWAM (ours)87.9853.1670.5727.01/19.5628.70/19.5037.39/22.0830.46/30.111.92/1.5025.10/18.55
PrismWAM* (ours)91.8475.5083.6736.71/29.3339.82/31.0848.67/31.0050.96/50.674.98/4.4236.23/29.30
w/ Spatial Information
X-WAM70.0025.8047.907.39/3.336.72/1.8317.47/9.086.32/4.670.57/0.257.69/3.83
Spatial-Forcing77.2026.7451.9714.12/9.3417.32/10.5823.26/14.585.43/4.111.78/1.5812.38/8.04
4D-WAM81.5041.8061.6513.35/8.8318.08/10.0023.30/14.007.38/6.001.44/1.0012.71/7.97

Table 1.

RQ2: Can PrismWAM remain reliable with real-world-like corruption? Real-world depth sensors may produce measurement noise and missing values, motivating evaluation with real-world-like depth corruption. The corruption profiles, which are informed by the manufacturers’ specifications for Orbbec Gemini 335 and RealSense D435, are controlled stress tests rather than calibrated models of either sensor; full settings are provided in section A.4. As shown in Figure 4, we train and evaluate PrismWAM on RoboTwin-C2R across three corruption levels constructed from Metric Jitter, Edge Missing, and Patch Hole. On randomized scenes, all three corruption levels reduce success relative to uncorrupted depth. In the clean setting, light corruption yields a small improvement before performance declines at higher severity. Nevertheless, even at the strongest corruption level, the average performance drops by only 2.45 pp. These results demonstrate that PrismWAM remains robust to diverse forms of depth corruption.

Figure 4. Robustness to progressive depth corruption. Top: Depth observations across corruption levels. Bottom left: Corruption types and their level-wise composition. Bottom right: RoboTwin performance under progressively increasing corruption of the original depth observations.

RQ3: Do PrismWAM’s design choices work together effectively? We examine the contributions of representation composition, attention routing, and supervision and training through ablations on RoboTwin-C2R (Table 3). Full settings and results for all six ablations are reported in Appendix § 8.§ 3. Removing either the RGB or depth stream reduces both clean-scene success and transfer, supporting the complementary roles of semantic context and physical state. Direct RGB-to-action access improves clean-scene success but weakens transfer, while exposing depth to future RGB also reduces transfer, favoring the proposed chain from observed semantic context through physical rollout to action. Together, these results support representation decoupling through complementary streams, with the full design achieving the best clean–random average.

Table 3: Component ablations on 50 RoboTwin tasks, with 20 episodes per task and condition.

VariantCleanRandomAvg.
PrismWAM87.4054.1070.75
Representation composition
w/o RGB stream79.1040.1059.60
w/o depth stream83.002.4042.70
Attention routing
Observed RGB → Action89.1040.3064.70
Future RGB → Depth87.0050.0068.50
Supervision & training
w/o future-RGB supervision81.6051.4066.50
w/o depth-only stage89.3024.7057.00

Table 3. Component ablations on 50 RoboTwin tasks, with 20 episodes per task and condition.

Real-world Validation

Tasks and evaluation conditions. We compare PrismWAM with OpenWAM (Wang et al. [49]) on three real-robot tasks: Pick Up Plate, Ranking Block Size, and Open Microwave. These tasks cover bimanual plate handling, single-arm size-dependent ordering, and multistep bimanual retrieval, respectively. In-distribution (ID) trials use the original setups and task-specific initial arrangements. Out-of-distribution (OOD) trials vary illumination, distractors, and object appearance or placement while preserving each task’s goal. Figure 6 and Table 4 summarize the settings and results. Task instructions, training recipes, are detailed in Appendix § 8.§ 5.

Figure 6. Real-world evaluation. ID trajectories (left), representative OOD scenes (right), and OpenWAM versus PrismWAM success rates (bottom) across three tasks.

TaskOpenWAM IDOpenWAM OODPrismWAM IDPrismWAM OOD
Pick Up Plate4/5 (80.0)10/15 (66.7)5/5 (100.0)13/15 (86.7)
Ranking Block Size7/12 (58.3)3/10 (30.0)11/12 (91.7)6/10 (60.0)
Open Microwave9/10 (90.0)4/10 (40.0)8/10 (80.0)5/10 (50.0)
Pooled (3 tasks)20/27 (74.1)17/35 (48.6)24/27 (88.9)24/35 (68.6)

Table 4. Real-world success under ID and OOD conditions. Entries are successes/trials (%). Both models are evaluated on the same three tasks and trial counts. Pooled rates divide total successes by total trials across tasks. Bold marks the higher success rate within each task and condition.

PrismWAM achieves pooled success of 88.9% in ID and 68.6% in OOD, compared with 74.1% and 48.6% for OpenWAM, improving success by 14.8 and 20.0 percentage points, respectively. OOD gains hold across all three tasks: 86.7% versus 66.7% on Pick Up Plate, 60.0% versus 30.0% on Ranking Block Size, and 50.0% versus 40.0% on Open Microwave. In ID, PrismWAM improves plate handling and size ordering, while OpenWAM performs better on microwave retrieval (90.0% versus 80.0%). These results extend the scene-change robustness observed in simulation to physical manipulation, supporting the practical value of combining RGB semantic guidance with depth-based physical rollouts.

Conclusion

We introduce PrismWAM, a World Action Model built on the principle of decoupling semantic context from physical state. Rather than treating video latents as a monolithic world representation, PrismWAM decomposes world-action modeling into Semantic Planning → Physical Rollout → Executable Action: RGB future-frame prediction captures task-level intent, depth models physical state transitions, and the resulting rollout is mapped directly to continuous actions without RGB goal prediction at inference. Across diverse simulation and real-world manipulation benchmarks, PrismWAM achieves strong performance and generalization with only 2,300 hours of depth data and without embodied action pretraining, while matching or surpassing large-scale action-pretrained models on challenging settings. These results suggest that scaling action data is not the only path to stronger embodied foundation models; principled state representations and architectural design offer a more data-efficient alternative. More broadly, PrismWAM paves the way for a new generation of WAMs that shifts the focus from modeling how the world looks to how the world responds to action. While depth is one practical realization of this principle, it can extend to occupancy, flow, tactile sensing, proprioception, and other representations capturing action-induced state transitions.

Limitations and future work. At present, large-scale real-world manipulation data with synchronized depth observations remains relatively limited, which constrains the scale of depth pretraining compared with existing action-pretrained models. We view this primarily as a reflection of the current data landscape: as depth sensing becomes more accessible and embodied datasets incorporate richer physical modalities, such data is likely to become increasingly available. Physical-state pretraining may therefore emerge as a natural scaling direction for World Action Models, with PrismWAM taking an early step toward establishing this new paradigm.

AI USE STATEMENT

Generative AI tools were used to assist with manuscript drafting, language polishing, and structural editing, as well as with literature search and code development. AI-generated suggestions were used only as assistance and were reviewed, revised, and verified by the authors. All technical claims, experimental results, interpretations, and final written content were validated by the authors, who take full responsibility for the work.

ACKNOWLEDGMENTS

We thank Chong Kar Hing, Junwei Li, and Zexuan Wen for their assistance with the experiments, and Shihao Wang and Yu-Cheng Chou for their insightful ideas.

Training and Additional Experiments

PrismWAM Training Recipe

The main PrismWAM model has approximately 6B parameters and uses Wan2.2-5B [48]; PrismWAM* denotes the exploratory 15B variant, which is initialized from WAN-2.1-14B with additional video-only pretraining. Training comprises depth-only adaptation conditioned on current depth and language, followed by joint RGB/depth/action training; both VAEs and T5 remain frozen.

Optimization and evaluation. AdamW uses β=(0.9,0.95)\beta= (0.9, 0.95), ϵ=10−8\epsilon= 10^{-8}, weight decay 0.01, gradient clipping 1.0, and bf16 without gradient accumulation. Each stage resets the optimizer; all trainable modules share 5% linear warmup and cosine decay from 10−410^{-4} to 10−610^{-6}. We use seed 42, uniform task/window sampling, and checkpoints every 1000 updates. Main results use the final checkpoint and no sensor-depth corruption. LIBERO trains on Spatial/Object/Goal/Long; Plus/Pro reuse its checkpoint. Inference uses ten shift-5 Euler steps with cached observed RGB.

Inputs. Three-view mosaics place a 256 ×\times 320 head image above two 128 ×\times 160 wrist images; LIBERO places 224 ×\times 224 external/wrist images left/right. RGB/depth layouts are shared across training and evaluation. RoboTwin/LIBERO encode RGB history as one causal clip (two latent frames) and the future target separately (one latent frame); RoboDojo uses history-noise probability 0.25. Depth clips include the current frame; actions use consecutive steps. Out-of-range frames repeat boundary observations, with padded targets masked.

Flow matching. Independently per predicted modality, sample u∼U(0,1)u \sim\mathcal{U}(0, 1), set σ=5u/(1+4u)\sigma= 5u/(1 + 4u), and use timestep 1000σ1000\sigma. For the targets in eqs. (4) and (5),

w(σ)=y(σ)−ymin⁡yˉ−ymin⁡+10−10,y(σ)=exp⁡[−2(σ−1/2)2],(6)w(\sigma) = \frac{y(\sigma) - y_{\min}}{\bar{y} - y_{\min} + 10^{-10}}, \qquad y(\sigma) = \exp[-2(\sigma- 1/2)^2], \tag*{(6)}

where ymin⁡y_{\min} and yˉ\bar{y} are the minimum and mean on the 1000-point shifted grid u∈{1,0.999,…,0.001}u \in\{1, 0.999, \ldots, 0.001\}. Modality losses have unit coefficients and exclude conditioning frames and padded targets.

Depth-VAE Training Recipe

We train Wan2.2-compatible and Wan2.1-compatible single-channel depth VAEs with 48 and 16 latent channels, respectively. Each is initialized from its corresponding pretrained RGB VAE [48], averaging the RGB input and output boundary weights across channels while retaining matching internal weights. Their spatial compression factors are 16 and 8, respectively. We additionally train a head-view-only depth VAE for real-world setups without wrist-view depth. The depth VAEs remain frozen throughout downstream policy training.

Training data and mixture. We use processed AgiBotWorld-Beta depth [5] and simulated RoboTwin depth from clean and randomized demonstrations [8]. After additional filtering of AgiBotWorld-Beta, the combined corpus contains approximately 2,300 hours of depth sequences. We sample AgiBotWorld-Beta and RoboTwin with probabilities 0.75 and 0.25, respectively, then sample a valid temporal clip with replacement. Training augmentation comprises random temporal cropping, in-plane rotation within ±5∘\pm5^{\circ}, and horizontal flipping with probability 0.5, applied consistently across frames and views.

Table 6 lists the shared training settings. The training objective is

LVAE=∥d^−d∥1+∥d^−d∥22+10−6DKL(q(z∣d) ∥ N(0,I))(7)\mathcal{L}_{\mathrm{VAE}} = \lVert\hat{d} - d \rVert_{1} + \lVert\hat{d} - d \rVert_{2}^{2} + 10^{-6} D_{\mathrm{KL}}\left(q(z \mid d) \,\|\, \mathcal{N}(0,I)\right) \tag*{(7)}

with mean-reduced reconstruction terms, stochastic posterior sampling, and no additional valid-depth mask. Per-channel latent means and standard deviations are computed from encoder posterior means and stored with each checkpoint for downstream normalization.

SettingValue
Clip length / frame stride9 frames / 4
Temporal compression factor4
Depth clipping / normalization0–1500 mm / [−1, 1]
Global batch / gradient accumulation256 / 1
OptimizerAdamW, β=(0.9,0.95)\beta= (0.9, 0.95)
Weight decay / gradient clipping0.01 / 1.0
Precisionbf16
Warmup updates5000
Learning-rate scheduleLinear warmup, then cosine decay
Selected checkpoint50k updates

Table 6. Shared depth-VAE training settings.

Component Ablations

We ablate PrismWAM initialized from Wan2.2-5B, using clean-only RoboTwin demonstrations, global batch 256, and 30k updates (15k depth adaptation plus 15k joint training; 30k joint updates without the depth-only stage). Inference uses ten denoising steps and replans every 24 actions; the future-RGB-to-depth variant jointly denoises RGB, depth, and actions. We evaluate final checkpoints on 50 tasks with 20 episodes per task and Clean/Random condition (1000 each), compared with 100 episodes per task and condition in the main PrismWAM results. Evaluation uses unseen instructions, policy seed 0, instruction seed 42, and environment seeds starting at 100000.

VariantCleanRandomMeanΔ\Delta
PrismWAM (full)87.4054.1070.75—
Representation composition
w/o RGB stream79.1040.1059.60−11.15
w/o depth stream83.002.4042.70−28.05
Attention routing
Observed RGB →\rightarrow Action89.1040.3064.70−6.05
Future RGB →\rightarrow Depth87.0050.0068.50−2.25
Supervision & training
w/o future-RGB supervision81.6051.4066.50−4.25
w/o depth-only stage89.3024.7057.00−13.75

Table 7. RoboTwin success (%); Δ\Delta: Mean change from the full model (pp).

Depth Corruption

Perturbation Setting. We corrupt metric depth at three severity levels (Figure 4 and Table 8). Metric Jitter adds Gaussian noise whose standard deviation increases quadratically with depth, ranging from σ0=12\sigma_0 = 12–3636 mm at 0 m to σ1=20\sigma_1 = 20–6060 mm at 1 m, with uniform clip-/episode-wise offsets in [−25-25, 2525] and [−50-50, 5050] mm at Levels 2 and 3, respectively. Edge Missing removes pixels around depth discontinuities exceeding 30 mm, using dilation radii of 1, 5, or 7 pixels and nominal removal fractions of 24–25%, modulated by depth at Levels 2–3. Patch Hole is enabled at Levels 2–3: each camera receives at most one connected hole targeting 1.5% of valid pixels per event, with a duration of 4–8 frames and onset offsets of 16–32 frames within each 80-frame cycle.

Levelσ0/σ1\sigma_0/\sigma_1 (mm)Shift (mm)Edge radiusEdge fractionPatch areaCleanRandomMean
Clean depth0/00–0087.4054.1070.75
112/200125%089.8052.7071.25
224/40[−25-25, 2525]525%1.5%87.6053.2070.40
336/60[−50-50, 5050]724%1.5%86.5050.1068.30

Table 8. Depth corruption parameters and success rates (%). Edge radius is the native-resolution four-neighbor dilation count; edge fraction is measured within the edge band. Patch area is relative to valid pixels during active events.

Training and evaluation. Each severity level uses 15k depth-only updates followed by 15k joint updates. Evaluation uses the matching corruption on 50 RoboTwin tasks, with 20 rollouts per task in each Clean/Random condition (1000 per condition; 2000 per level).

Real-World Evaluation

Training. Both policies train on the same demonstrations for 30k total updates with a global batch size of 256 and seed 42. PrismWAM is initialized from Wan2.2-5B and uses 15k depth-only adaptation updates followed by 15k joint RGB/depth/action updates; OpenWAM-α\alpha uses 30k RGB/action updates. For setups without wrist-view depth, PrismWAM uses the separately pretrained head-view-only depth VAE with 256×320256 \times320 inputs (Appendix § 8.§ 2). The VAEs and T5 remain frozen during policy training. PrismWAM predicts absolute joint commands, and OpenWAM-α\alpha predicts absolute end-effector targets.

Embodiment and data. We use single-arm and dual-arm Piper-X configurations, each equipped with one head-view RGB-D camera (Orbbec Gemini 335). Ranking Block Size uses a single arm, while Pick Up Plate and Open Microwave use both arms. For these three tasks, we collect 100 demonstration episodes for Open Microwave, 200 for Pick Up Plate, and 180 for Ranking Block Size, with 30 episodes for each of the six size permutations. Each task has 50 goal-preserving training paraphrases.

Tasks and evaluation. Success requires (i) Pick Up Plate: pick up the plate with both arms and place it on the rack; (ii) Ranking Block Size: place the largest cube farthest from the robot, the medium cube centrally, and the smallest nearest; (iii) Open Microwave: open the door, move the inside basket to the tabletop on the microwave’s right, and leave the door open. ID uses the original setup, including all six size permutations for Ranking Block Size. As illustrated in Figure 6, OOD settings for all three tasks include flickering lights with ambient lights switched off and added distractor objects. Additional settings change plate colors for Pick Up Plate and tablecloths for Ranking Block Size and Open Microwave. Table 9 reports successes/trials; overall rates pool trials.

TaskOpenWAM-α\alpha IDOpenWAM-α\alpha OODPrismWAM IDPrismWAM OOD
Pick Up Plate4/510/155/513/15
Ranking Block Size7/123/1011/126/10
Open Microwave9/104/108/105/10

Table 9. Real-world successes/trials.

Detailed Results across Five Benchmarks

LIBERO, LIBERO-Plus, and LIBERO-Pro

We evaluate PrismWAM on LIBERO [30], LIBERO-Plus [12], and LIBERO-Pro (Zhou et al., 2025) with an action horizon of 16 and replanning every 16 actions. LIBERO-Plus and LIBERO-Pro reuse the checkpoint trained on standard LIBERO without additional fine-tuning. For LIBERO-Plus, we apply the RGB sensor-noise settings to depth observations using the same corruption parameters. LIBERO-Pro evaluates five perturbation types with 2000 episodes per type.

SuiteLIBERO StandardLIBERO-Plus
Success / TotalSR (%)Success / TotalSR (%)
Spatial497/50099.402075/240286.39
Object500/500100.002127/251884.47
Goal491/50098.201830/259170.63
Long499/50099.801936/251976.86
Overall1987/200099.357968/1003079.44

Table 10. PrismWAM on standard LIBERO and LIBERO-Plus by base suite.

PerturbationSuccess / TotalSR (%)
Camera viewpoints969/159960.60
Robot initial states1048/155067.61
Language instructions1416/153792.13
Light conditions1138/114299.65
Background textures1071/107699.54
Sensor noise (matched RGB/depth)1080/160167.46
Object layout1246/152581.70
Overall7968/1003079.44

Table 11. PrismWAM on LIBERO-Plus by perturbation. Overall pools episodes rather than averaging rounded category percentages.

RoboTwin-C2R: Per-task Results

RoboTwin-C2R trains the policy on clean demonstrations only and evaluates 50 tasks in Clean and Random conditions. PrismWAM and PrismWAM∗^{*} results in table 12 use 100 episodes per task and condition.

TaskPrismWAMPrismWAM∗^{*}
CleanRand.CleanRand.
Adjust Bottle985410097
Beat Block Hammer81489858
Blocks Ranking RGB87329372
Blocks Ranking Size56277055
Click Alarmclock1008610098
Click Bell1007810093
Dump Bin Bigbin84809282
Grab Roller1008910099
Handover Block99379866
Handover Mic952410080
Hanging Mug38175941
Lift Pot996710086
Move Can Pot991910047
Move Pillbottle Pad1001910078
Move Playingcard Away996510092
Move Stapler Pad3386433
Open Laptop91509483
Open Microwave5719443
Pick Diverse Bottles93789284
Pick Dual Bottles1009610096
Place A2B Left83399274
Place A2B Right80418870
Place Bread Basket94779578
Place Bread Skillet90669174
Place Burger Fries989096100
Place Can Basket81317961
Place Cans Plasticbox1009510099
Place Container Plate1006810093
Place Dual Shoes73439470
Place Empty Cup997010084
Place Fan88429768
Place Mouse Pad89299143
Place Object Basket86448878
Place Object Scale88379781
Place Object Stand97769990
Place Phone Stand88479475
Place Shoe92559995
Press Stapler92508580
Put Bottles Dustbin90829293
Put Object Cabinet89289052
Rotate QRcode83268949
Scan Object94729781
Shake Bottle1007910098
Shake Bottle Horizontally1008010097
Stack Blocks Three90339587
Stack Blocks Two986010098
Stack Bowls Three84598279
Stack Bowls Two94749687
Stamp Seal80239533
Turn Switch70496765
Mean over 50 tasks87.9853.1691.8475.50

Table 12. RoboTwin-C2R per-task SR (%) after clean-only training.

RoboDojo: Category, Seed, and Per-task Results

The simulation suite ([7]) has 42 canonical tasks in five categories, represented by 54 standard/randomized entries per seed. Generalization uses 25 episodes/entry/seed; other categories use 50 (2100 episodes/seed, 6300 over seeds 0–2). Both policies use simulator depth, ten denoising steps, and replanning every 24 actions. We evaluate both policies with three seeds (0–2). Table 13 reports category and overall Score/SR as mean ±\pm population standard deviation across seeds, together with each seed’s overall result; overall metrics average the five categories equally. Table 14 reports per-entry Score/SR averaged over the same three seeds.

Category / seedPrismWAM ScorePrismWAM SR (%)PrismWAM* ScorePrismWAM* SR (%)
Generalization27.01±1.1227.01 \pm1.1219.56±0.7519.56 \pm0.7536.71±0.9336.71 \pm0.9329.33±0.9829.33 \pm0.98
Precision28.70±0.9828.70 \pm0.9819.50±1.7719.50 \pm1.7739.82±2.2139.82 \pm2.2131.08±2.6631.08 \pm2.66
Long-horizon37.39±1.1537.39 \pm1.1522.08±0.7722.08 \pm0.7748.67±0.2648.67 \pm0.2631.00±0.7431.00 \pm0.74
Memory30.46±1.8030.46 \pm1.8030.11±1.8130.11 \pm1.8150.96±0.2050.96 \pm0.2050.67±0.2750.67 \pm0.27
Open1.92±0.461.92 \pm0.461.50±0.541.50 \pm0.544.98±0.524.98 \pm0.524.42±0.664.42 \pm0.66
Overall25.10±0.6825.10 \pm0.6818.55±0.7418.55 \pm0.7436.23±0.5536.23 \pm0.5529.30±0.6129.30 \pm0.61
Seed 025.2125.2118.5818.5835.5835.5828.7028.70
Seed 125.8625.8619.4319.4336.9236.9230.1330.13
Seed 224.2124.2117.6317.6336.1736.1729.0729.07

Table 13. RoboDojo category and seed results. Score (/100) and SR (%) are mean ±\pm population standard deviation over seeds 0–2; seed rows give each complete evaluation’s category-balanced metrics.

TaskPrismWAM ScorePrismWAM SRPrismWAM* ScorePrismWAM* SR
Generalization – standard
stack bowls80.5377.3388.6786.67
pour liquid into cup54.6754.6760.0060.00
push T0.000.001.331.33
make toast40.6722.6768.6760.00
pack objects into box25.734.0031.335.33
arrange largest number7.802.6712.475.33
fold clothes62.1358.6778.6777.33
sort nesting dolls by size14.6714.6732.0032.00
hang mugs42.5320.0073.6061.33
store laptop and headphones38.6722.6758.4046.67
sweep blocks2.672.678.008.00
stack blocks74.6072.0093.2092.00
Generalization – randomized
stack bowls (Rand.)31.8722.6721.208.00
pour liquid into cup (Rand.)45.3345.3336.0036.00
push T (Rand.)0.000.000.000.00
make toast (Rand.)27.0010.6722.6710.67
pack objects into box (Rand.)23.734.0024.872.67
arrange largest number (Rand.)2.730.008.931.33
fold clothes (Rand.)13.078.0086.4084.00
sort nesting dolls by size (Rand.)1.331.331.331.33
hang mugs (Rand.)11.270.0020.330.00
store laptop and headphones (Rand.)40.2722.6732.5317.33
sweep blocks (Rand.)0.000.005.335.33
stack blocks (Rand.)7.072.6715.131.33
TaskPrismWAM ScorePrismWAM SRPrismWAM* ScorePrismWAM* SR
Precision
fasten screws32.938.0058.6030.00
play Xylophone4.004.000.000.00
insert tubes74.6760.6782.5374.00
deposit coin15.339.3340.0034.67
plug in charger1.331.3310.6710.67
insert key14.400.0014.400.00
pour balls into vase38.6738.6758.0058.00
build tower48.2734.0054.3341.33
Long-horizon
put bottles into dustbin95.6092.6799.5099.33
fill egg holder7.170.009.000.00
play tic tac toe40.4310.6731.304.67
organize table59.837.3355.004.00
classify objects5.701.3353.2042.67
play stacking toy5.000.0042.9319.33
fill pen holder62.0341.3365.7345.33
make kong23.3323.3332.6732.67
Memory
cover blocks100.00100.00100.00100.00
press by number20.0020.0081.3381.33
match and pick from conveyor51.3351.3389.3389.33
imitate sorting sequence2.770.672.400.67
swap T8.678.6732.6732.67
swap blocks0.000.000.000.00
Open
align blocks0.000.000.000.00
classify objects by language1.070.006.002.67
solve equation0.000.000.000.00
pick from conveyor by image0.000.000.000.00
stack blocks by language7.605.3311.3310.67
store tools in toolbox0.000.000.500.00
general pickup6.676.6722.0022.00
pour by language0.000.000.000.00
Overall (five categories)25.1018.5536.2329.30

Table 14. All 54 RoboDojo evaluation entries: three-seed mean Score (/100) and SR (%). Rand. marks randomized Generalization entries.

References

  1. [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-VL technical report, 2025. URL https://arxiv.org/abs/2511.21631.
  2. [2]Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, and Jun Zhu. Motus: A unified latent action world model, 2025. URL https://arxiv.org/abs/2512.13030.
  3. [3]Kevin Black et al. π₀: A vision-language-action flow model for general robot control, 2024. URL https://arxiv.org/abs/2410.24164.
  4. [4]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Danny Driess, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023. URL https://arxiv.org/abs/2307.15818.
  5. [5]Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025. URL https://arxiv.org/abs/2503.06669.
  6. [7]Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan, Honghao Su, et al. RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies, 2026b. URL https://arxiv.org/abs/2607.04434.
  7. [8]Tianxing Chen et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation, 2025. URL https://arxiv.org/abs/2506.18088.
  8. [9]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/2507.06261.
  9. [11]Pengcheng Fang, Hongli Chen, and Xiahao Cai. Privileged foresight distillation: Zero-cost future correction for world action models, 2026. URL https://arxiv.org/abs/2604.25859.
  10. [12]Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. LIBERO-Plus: In-depth robust-ness analysis of vision-language-action models, 2025. URL https://arxiv.org/abs/2510.13626.
  11. [14]Ankit Goyal et al. RVT: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, 2023. URL https://arxiv.org/abs/2306.14896.
  12. [15]Qiwen Gu, Jifan Li, Bingjie Gao, Rui Chen, Jing Tang, Xiangxiang Chu, and Junqiao Zhao. The right future for action: Learning action-relevant predictive states in world action models, 2026. URL https://arxiv.org/abs/2609.23369.
  13. [16]Jun Guo, Qiwei Li, Peiyan Li, Zilong Chen, Nan Sun, Yifei Su, Heyun Wang, Yuan Zhang, Xinghang Li, and Huaping Liu. Unified 4D world action modeling from video priors with asynchronous denoising, 2026. URL https://arxiv.org/abs/2604.26694.
  14. [17]David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. URL https://arxiv.org/abs/1803.10122. Preprint.
  15. [18]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.arxiv.org/abs/1912.01603
  16. [19]Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3D Diffuser Actor: Policy diffusion with 3D scene representations, 2024. URL https://arxiv.org/abs/2402.10885.
  17. [20]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfield, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model, 2024. URL https://arxiv.org/abs/2406.09246.
  18. [21]Moo Jin Kim, Yihui Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, and Jinwei Gu. Cosmos Policy: Fine-tuning video models for visuomotor control and planning, 2026. URL https://arxiv.org/abs/2601.16163.
  19. [22]Yann LeCun. A path towards autonomous machine intelligence, 2022. URL https://openreview.net/pdf?id=BZ5a1r-kVsf. Position paper, version 0.9.2, June 27, 2022.
  20. [23]Chengmeng Li, Junjie Wen, Yan Peng, Yaxin Peng, Feifei Feng, and Yichen Zhu. PointVLA: Injecting the 3D world into vision-language-action models, 2025a. URL https://arxiv.org/abs/2503.07511.
  21. [24]Fuhao Li, Wenxuan Song, Han Zhao, Jingbo Wang, Pengxiang Ding, Donglin Wang, Long Zeng, and Haoang Li. Spatial forcing: Implicit spatial representation alignment for Vision-Language-Action model, 2025b. URL https://arxiv.org/abs/2510.12276.
  22. [25]Kehan Li, Bohan Hou, Minghao Zhu, et al. RynnBrain 1.1: Towards more capable and generalizable embodied foundation model, 2026a. URL https://arxiv.org/abs/2607.17977.
  23. [26]Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, Yujun Shen, and Yinghao Xu. Causal world modeling for robot control, 2026b. URL https://arxiv.org/abs/2601.21998.
  24. [27]Qixiu Li et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation, 2024. URL https://arxiv.org/abs/2411.19650.
  25. [28]Ying Li, Xiaobao Wei, Jiajun Cao, Hao Wang, Xiaowei Chi, Chengyu Bai, Qianpu Sun, Jiajun Li, Xiaojie Zhang, Peidong Jia, Jian Tang, Sirui Han, and Shanghang Zhang. WAM4D: Fast 4D world action model via spatial register tokens, 2026c. URL https://arxiv.org/abs/2606.14048.
  26. [29]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URL https://arxiv.org/abs/2210.02747.
  27. [30]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning, 2023. URL https://arxiv.org/abs/2306.03310.
  28. [32]Javier Alejandro Lopetegui Gonzalez, Paul Pacaud, and Cordelia Schmid. Spatially aware world action model via geometric latent diffusion, 2026. URL https://arxiv.org/abs/2609.02531.
  29. [33]Xingyu Miao, Zizun Li, Baole Fang, Kaiwen Song, Tenghui Wang, Hanxue Zhang, Yating Wang, Xudong Li, Yuping He, Xueyuan Wei, Chao Gao, Xijie Yang, Yingxiang Xu, Kerui Ren, Wengi Guo, Jianjun Zhou, Xinzhe Wang, Weiguang Zhao, Ni Yang, Zetao Cai, Yufei Xue, Hengjie Li, Zeyu He, Yuanzhen Zhou, Rong Fu, Jianyang Zhang, Siwei Cui, Fuxian Huang, Yunsong Zhou, Xing Gao, Yifei Yao, Qiaojun Yu, Kailin Li, Ming Zhou, Mu Huang, Xinyue Li, Wenze Cui, Bingqi Jiang, Xueyue Zhu, Junting Dong, Haoyu Guo, Tao Lu, Mulin Yu, Bowen Zhou, Bin Zhao, Tianfan Xue, Weinan Zhang, and Chunhua Shen. InternW0-Δ: A world action model bridging predictive dynamics and actions with 20k+ hours of open data, 2026. URL https://arxiv.org/abs/2609.31394.
  30. [34]NVIDIA. Cosmos 3: Omnimodal world models for physical AI. arXiv preprint arXiv:2606.02800, 2026. URL https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf.
  31. [35]NVIDIA et al. World simulation with video foundation models for physical AI, 2025a. URL https://arxiv.org/abs/2511.00062.
  32. [36]NVIDIA et al. GROOT N1: An open foundation model for generalist humanoid robots, 2025b. URL https://arxiv.org/abs/2503.14734.
  33. [37]Physical Intelligence, Kevin Black, Noah Brown, et al. π0.5: A vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054.
  34. [38]Lu Qiu, Yizhuo Li, Yi Chen, Yuying Ge, Yixiao Ge, and Xihui Liu. Making foresight actionable: Repurposing representation alignment in world action models, 2026. URL https://arxiv.org/abs/2606.12217.
  35. [39]Delin Qu et al. SpatialVLA: Exploring spatial representations for visual-language-action model, 2025. URL https://arxiv.org/abs/2501.15830.
  36. [40]RoboDojo Team. RoboDojo Official Leaderboard, 2026. URL https://robodojo-benchmark.com/LeaderBoard. Updated October 17, 2026; accessed October 19, 2026.
  37. [41]RoboTwin Team. RoboTwin 2.0 Official Leaderboard, 2026. URL https://robotwin-platform.github.io/leaderboard. Accessed October 19, 2026; ranked by mean clean and randomized success.
  38. [42]Yichao Shen, Fangyun Wei, Zhiying Du, Yaobo Liang, Yan Lu, Jiaolong Yang, Nanning Zheng, and Baijing Guo. VideoVLA: Video generators can be generalizable robot manipulators. In Advances in Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=UPHlqbZFZB.arxiv.org/abs/2512.06963
  39. [43]Jiaxin Shi, Xidong Zhang, Fucai Zhu, Zhe Li, Siyu Zhu, and Weihao Yuan. 3DThinkVLA: Endowing vision-language-action models with latent 3D priors via 3D-thinking-guided co-training, 2026. URL https://arxiv.org/abs/2606.04436.
  40. [44]Lucy Shi et al. Hi robot: Open-ended instruction following with hierarchical vision-language-action models, 2025. URL https://arxiv.org/abs/2502.19417.
  41. [45]Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, 2022. URL https://arxiv.org/abs/2209.05451.
  42. [46]Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zoutine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, Simon Alibert, Matthieu Cord, Thomas Wolf, and Remi Cadene. SmolVLA: A vision-language-action model for affordable and efficient robotics, 2025. URL https://arxiv.org/abs/2506.01844.
  43. [47]StarVLA Community. StarVLA: A lego-like codebase for vision-language-action model developing, 2026. URL https://arxiv.org/abs/2604.05014.
  44. [48]Team Wan et al. Wan: Open and advanced large-scale video generative models, 2025. URL https://arxiv.org/abs/2503.20314.
  45. [49]Yuran Wang, Siqiao Huang, Mingleyang Li, Chenhao Zhang, Jiaqi Liang, Weiyang Jin, Yue Chen, Xuemin Chi, Donghao Zhou, Qize Yu, Yu-Kai Wang, Yuhan Rui, Shenzhe Yao, Zhen Yuan, Zhenhao Shen, Kefei Zhu, Zijie Zhu, Ning Gao, Xiaowei Chi, Guanqi He, Shanghang Zhang, Hao Dong, Lin Shao, and Hang Zhao. OpenVLA: An open, modular exploration towards systematic world-action model pretraining, 2026a. URL https://arxiv.org/abs/2609.07398.

Paper details

vlnc:W_jed8bf5b
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents