WorksComputer Science
OpenNWM: A Generalist Navigation World Model with Latent Action Pretraining
Abstract
Navigation world models predict future observations conditioned on navigation actions to support planning, but their generalization across real-world environments remains constrained by the limited coverage of action-labeled data. Action-free navigation videos provide diverse visual experience, yet lack the recorded actions required for action-conditioned training. We introduce OpenNWM, a generalist navigation world model with latent action pretraining. It learns universal latent actions from large-scale action-free videos through visual prediction pretraining, and further aligns them with physical motions via action-conditioned post-training. To support video pretraining, we curate NavAnywhere, a large-scale action-free navigation video dataset comprising 17.5 million visual observations from 15 heterogeneous sources. It captures navigation by ground robots, humans, and drones, with diverse viewpoints and motion patterns across five broad scene types: residential areas, public and commercial spaces, urban settings, parks, and natural landscapes. Experiments demonstrate improved visual prediction on in-domain and out-of-domain benchmarks, together with improved navigation performance. These results show that latent-action pretraining enables diverse, action-free videos to improve dynamics learning and generalization for open-world navigation.
Introduction
Visual navigation is a fundamental capability that enables embodied agents to autonomously navigate and accomplish tasks in real world. Recent navigation policies generate actions from visual observations conditioned on goal images (Shah et al., 2023b;c; Sridhar et al., 2024) or language instructions (Song et al., 2026), demonstrating promising generalization across diverse environments and embodiments. Beyond direct policy learning, navigation world models explicitly model environmental dynamics by predicting future visual observations conditioned on candidate actions (Zhu et al., 2026; Yu et al., 2025), enabling agents to imagine and evaluate potential trajectories for flexible planning [2] (Zhang et al. 2026b). WMNav (Nie et al., 2025) integrates visionlanguage models into a world-model framework, using semantic predictions and memory feedback to guide object-goal navigation. However, existing NWMs rely heavily on action-labeled navigation trajectories, limiting their ability to exploit the breadth of visual experience required for open-world navigation.
Action-labeled trajectories provide a direct connection between navigation actions and their visual consequences, but collecting them across diverse environments and embodiments is costly [16]. In contrast, large-scale action-free videos cover diverse egocentric, driving, and scene exploration scenarios without requiring recorded actions [12, 1] (Liu et al. 2025; Ling et al. 2024). These videos contain rich visual and temporal information, yet do not directly supervise the relationship between executable actions and observed transitions (Wan et al., 2026). NWM partially addresses this gap through time-conditioned prediction on unlabeled videos [2], but a temporal offset specifies when to predict rather than explicitly describing the motion underlying a transition. This reveals a central tension: action-labeled trajectories provide control supervision but limited coverage, while action-free videos provide broad visual experience without explicit control signals.
Latent action models (LAMs) provide a promising direction by inferring proxy actions from visual transitions, enabling learning from videos without explicit action annotations [9, 11] (Schmidt & Jiang 2024; Ye et al. 2025). However, reconstruction-based latent actions may capture appearance changes and external dynamics rather than agent-controllable motion (Nikulin et al., 2025). Recent approaches such as villa-X and LatBot incorporate physical supervision to better connect visual transitions with executable motion [5, 22]. For navigation world models, this raises two connected questions: what supervision makes latent actions useful for learning navigation dynamics, and how can the resulting predictive dynamics be adapted to real navigation actions? Addressing both questions is important for turning diverse video experience into effective action-conditioned prediction and planning.
To this end, we introduce OpenNWM (see Figure 1), a generalist navigation world model built upon a universal latent-action space for open-world navigation. OpenNWM learns Universal Navigation Latent Actions (UNLAs) that work as an intermediate training signal for learning predictive dynamics from videos. By optimizing a unified visual-physical objective, OpenNWM learns latent representations that support both future visual prediction and physical motion prediction.
Figure 1. Overview of OpenNWM. OpenNWM learns navigation dynamics from heterogeneous action-free videos via latent-action pretraining, then adapts to real waypoint actions for action-conditioned prediction and planning.
We construct NavAnywhere, an action-free navigation video dataset comprising 17.5 million visual observations from 15 sources. It aggregates public datasets and online videos spanning ground robots, humans, and drones. The dataset covers residential environments, public and commercial spaces, urban and transportation settings, parks and gardens, and natural and off-road environments. We study the effects of latent-action pretraining and LAM supervision on navigation world modeling. As prediction quality alone does not ensure effective decision-making (Zhang et al., 2026a), we evaluate OpenNWM on in-domain and out-of-domain prediction, autoregressive rollouts, and planning under real actions. Our main contributions are as follows:
We introduce OpenNWM, a generalist navigation world model that learns a universal latent-action space from large-scale heterogeneous visual experiences, enabling scalable navigation learning across diverse environments, embodiments, and motion patterns.
We curate NavAnywhere, a large-scale heterogeneous navigation dataset comprising 17.5 million visual observations from 15 sources across diverse environments and embodiments. We further propose Universal Navigation Latent Actions (UNLAs), which disentangle agent motion from environmental dynamics and unify heterogeneous visual experiences with navigation signals within a shared latent space.
OpenNWM achieves state-of-the-art perceptual prediction and downstream navigation performance on the evaluated benchmarks. Systematic ablations further characterize how latent-action pretraining and different LAM supervision strategies contribute to dynamics modeling and cross-domain generalization.
Related Work
From Navigation Policies to Navigation World Models
Learning-based visual navigation has evolved from task-specific controllers to generalist policies that transfer across environments and embodiments. Early visual-goal navigation systems combined learned reachability models with topological memory, enabling long-horizon navigation from offline visual experience (Savinov et al., 2018; Shah et al., 2021b; 2022; Shah & Levine, 2022). This line of work subsequently converged on shared navigation interfaces: GNM provided a unified interface across heterogeneous robot trajectories through embodiment-agnostic waypoint prediction (Shah et al., 2023b); ViNT extended this interface with large-scale Transformer pretraining (Shah et al., 2023c); and NoMaD employed diffusion-based action generation to unify goal-conditioned navigation and goal-agnostic exploration (Sridhar et al., 2024). Together, these advances highlighted cross-dataset training for transferable navigation policies.
In parallel, pretrained vision–language models extend navigation beyond visual goals to semantic instructions and general tasks, integrating language, vision, and navigation modules for long-horizon reasoning and execution (Shah et al., 2023a; Zhou et al., 2024). Recent video-based navigation models integrate temporal visual context, language understanding, and action prediction, improving generalization across tasks, datasets, and sim-to-real transfer (Zhang et al., 2024; 2025). NaVILA further couples high-level VLA reasoning with embodiment-specific locomotion skills, enabling language-guided navigation for legged robots [7]. Meanwhile, automatic extraction of motion supervision from web-scale walking and driving videos expands the coverage of scenes and behaviors available for policy learning (Liu et al., 2025).
Unlike policy-centric methods that implicitly encode dynamics, Navigation World Models explicitly predict action-conditioned futures for planning [2]. OpenNWM adopts latent actions as a shared control interface, enabling the learning of controllable dynamics jointly from action-labeled trajectories and action-free videos.
Unified Embodied Learning from Heterogeneous Data
The development of generalist embodied models increasingly relies on jointly learning from diverse experiences across tasks, environments, and robot embodiments. Studies on large-scale robot datasets demonstrate that cross-dataset training facilitates knowledge transfer across robotic platforms, while flexible architectures accommodate heterogeneous observation modalities and action spaces (O’Neill et al., 2024; Octo Model Team et al., 2024). However, such unified learning paradigms still assume that every trajectory is paired with executable robot actions. Their scalability is therefore constrained by the high costs of robot deployment, teleoperated data collection, and alignment across heterogeneous action spaces.
Human egocentric recordings and Internet videos provide large-scale embodied experiences, spanning substantially more diverse scenes and behaviors. Recent studies show that carefully curated human videos can yield more transferable pretrained representations than an equivalent amount of robot data (Ma et al., 2026). Large-scale human-video pretraining also enables world models to acquire broad interaction and environmental priors before adaptation to specific robotic platforms [10]. However, human and Internet videos lack robot-compatible actions, while embodiment-specific pose estimation and motion retargeting hinder a unified control interface.
Latent actions unify heterogeneous embodied data beyond predefined action spaces, supporting action-free pretraining, task transfer, and visual-action alignment [4, 23, 5, 22] (Schmidt & Jiang 2024; Ye et al. 2025). OpenNWM extends latent-action learning beyond manipulation and task-specific models to generalist navigation, unifying action-free videos and heterogeneous motion signals through a controllable world-model interface.
Method
Problem Setup
Dataset. Given heterogeneous visual observations, we construct , where denotes a visual sequence, contains optional camera poses, and encodes the data source, embodiment, and sampling rate. For action-free videos, .
Learning Target. Given visual observations up to time , we denote the encoded visual states of the most recent frames as , where is the visual context length. For a prediction horizon , the target time is defined as . A latent action is inferred from the visual transition as , where discovers a compact latent representation of visual changes without requiring explicit action annotations. Conditioned on the inferred latent action, OpenNWM learns future visual dynamics by optimizing:
Here, denotes a diffusion-parameterized conditional distribution over future visual states. The latent action serves as a shared interface for visual transitions, enabling OpenNWM to model multimodal future dynamics without explicit action labels.
Discussion. Conventional Navigation World Models rely on action-labeled trajectories, while OpenNWM discovers and grounds latent actions from heterogeneous visual experiences, enabling unified learning from action-free videos and robot data.
Universal Navigation Latent Actions
Visual transitions may contain a mixture of agent motion, environmental dynamics, and observation variations. Latent actions learned solely from visual prediction may capture arbitrary changes rather than meaningful agent behaviors. We address this ambiguity by learning a continuous latent action representation that captures visual transitions while preserving controllable motion semantics.
Variational Transition Modeling. Given an initial observation and a future observation , we infer a continuous latent action representation that captures the corresponding visual transition. The resulting variational objective is formulated as:
Here, denotes a diagonal Gaussian posterior, is the latent prior, and is an auxiliary transition predictor. The reconstruction term preserves transition information, while KL regularization constrains the latent space. The predictor only regularizes latent action learning with high-fidelity generation handled by the Navigation World Model.
Physical Motion Grounding. Visual transitions may capture environmental and observation changes beyond controllable agent motion. For trajectories with physical annotations, we align latent actions with motion transitions and robot actions through the following objective:
Here, contains available future motion states. The predictor maps latent actions to physical transitions. This objective is applied only to trajectories with available physical supervision.
Heterogeneous Supervision. We learn the latent-action representation from all visual observations, while applying physical-motion supervision only to trajectories with available annotations. The resulting joint objective is formulated as:
Here, denotes the subset with available physical motion annotations, and controls the contribution of physical supervision. The objective is applied to all observation sequences, while the physical objective is activated for motion-annotated sequences to ground latent actions in navigation dynamics. The learned latent actions serve as unified conditioning variables for OpenNWM.
Latent-Conditioned Navigation World Modeling
Given the learned universal latent-action space, the posterior distribution infers latent actions from both action-free videos and motion-annotated trajectories. Each observation is encoded into a visual latent state , and the historical context at time is denoted by . We model the conditional distribution of future visual states given historical context and latent actions (see Figure 2).

Figure 2. Framework of OpenNWM.
Latent Conditioning. For a transition from to , we combine the embeddings of the latent action, temporal horizon, and diffusion timestep to construct the conditioning representation:
Here, , , and denote the embedding functions for the latent action, temporal horizon, and diffusion timestep, respectively. The conditioning representation is injected through adaptive layer normalization, while historical visual states are incorporated via cross-attention.
Conditional Diffusion Dynamics. At diffusion timestep , the forward process gradually perturbs the target future state with Gaussian noise according to a predefined noise schedule:
Here, denotes the noisy future state at diffusion timestep , and is determined by the noise schedule. Conditioned on the historical context and the latent-action conditioning representation , the denoising model predicts the future state:
Iterating the reverse diffusion process produces a multimodal distribution of future visual states conditioned on the inferred latent action.
Unified World-Model Learning. OpenNWM is optimized using a hybrid objective combining future-state prediction and a variational diffusion objective:
Here, the first term supervises future-state prediction, while denotes the variational lower bound for learning the reverse diffusion model, and controls its contribution. Since latent actions are available for all visual transitions, the same diffusion objective applies to both action-free and motion-annotated data. Physical annotations only ground the latent-action space, while OpenNWM generates multimodal future states for navigation planning.
Navigation Planning
Given the visual context and goal observation , the post-trained OpenNWM evaluates candidate real waypoint-action sequences through action-conditioned future visual rollouts. We retain energy-based receding-horizon planning while optimizing directly over real waypoint actions. Latent actions are used for world-model pretraining rather than as online planning variables.
Action-Conditioned Trajectory Optimization. Let denote a candidate real waypoint-action sequence over a planning horizon . Conditioned on the visual context
Algorithm 1 Action-Conditioned Receding-Horizon Planning
Require: Post-trained world model , visual encoder , goal observation , context
Require: Horizon , context length , candidates , elites , rollouts , CEM iterations
1: 2: while the goal is not reached do 3: Observe and update the encoded visual context using 4: Initialize 5: for do 6: for do 7: Sample , 8: for do 9: 10: 11: end for 12: end for 13: 14: Refit using 15: end for 16: 17: 18: 19: Execute through the robot controller and advance to the next observation 20: end while
and each candidate action sequence, OpenNWM rolls out the corresponding future visual states . We evaluate each candidate rollout using the following planning energy:
Here, measures the similarity between the predicted terminal state and the goal observation, while aggregates navigation constraints, including predicted-state safety and physical-action feasibility under the embodiment context . The optimal action sequence is selected by minimizing the expected trajectory energy:
Here, denotes the rollout distribution induced by the post-trained, real-action-conditioned world model, with the fixed temporal offset per planning step omitted for brevity. We solve this optimization using the cross-entropy method (CEM) (Rubinstein, 1997), iteratively updating a proposal distribution toward elite action sequences evaluated through world-model rollouts.
Action Execution. The first optimized action is executed by the robot. The auxiliary physical predictor is used only for motion grounding during latent-action learning and is not required for online planning. Upon receiving a new observation, OpenNWM updates its visual context and repeats action-space optimization. This receding-horizon procedure enables predictive planning directly in the real waypoint-action space.
Action-Conditioned Planning. OpenNWM solves goal-conditioned navigation via receding-horizon optimization over real waypoint-action sequences. Candidates are sampled from a Gaussian proposal distribution and evaluated through real-action-conditioned world-model rollouts. Their energies jointly account for goal similarity, predicted-state safety, and physical-action feasibility. The lowest-energy candidates are used to update the proposal distribution over CEM iterations. Only the first action of the selected sequence is passed to the robot controller before replanning from updated observations. Pseudocode is provided in Algorithm 1. We use planning steps, corresponding to 2 seconds with 0.25-second intervals, and evaluate candidate sequences with stochastic rollouts per candidate in each CEM iteration.
NavAnyWhere
We construct NavAnywhere, a large-scale heterogeneous visual navigation corpus, by integrating 15 open-source datasets. It contains 17.5M frames of observations, spanning action-free videos across diverse environments. NavAnywhere enables OpenNWM to jointly learn scalable visual dynamics and physically grounded latent actions. NavAnywhere details are in Appendix A; preprocessing and dataset usage are described in Appendix B.
Experiments
Experiment Setups
Datasets. We pretrain OpenNWM on NavAnywhere, a large-scale action-free video corpus covering diverse indoor, outdoor, urban, and natural environments. For downstream training, we use SCAND (Karnan et al. [20]), TartanDrive (Triest et al. (2022)), RECON Shah et al. (2021a), and HuRoN (SACSoN) (Hirose et al. [14]), whose pose annotations are converted into relative translations and rotations. Translational displacements are normalized by the average step length of each dataset, and backward-motion transitions are removed. We evaluate in-domain performance on these four datasets and assess generalization to unseen environments on Go Stanford, the TUM RGB-D SLAM Dataset, Office-Go2, and Planetary Rover. Office-Go2 contains real-world office navigation sequences that we collected using a quadruped robot, while Planetary Rover comprises lunar and Martian rover navigation data sourced from the Internet. Details are provided in the Appendix § 9.§ 2.
Evaluation Metrics. ATE and RPE assess trajectory accuracy and local pose consistency (Sturm et al. (2012)). PSNR, LPIPS (Zhang et al. (2018)), and DreamSim (Fu et al. [8]) measure visual fidelity, while FID (Heusel et al. [13]) assesses image distribution quality. We also evaluate LAM reconstruction under ID and OOD settings. Details are provided in the Appendix § 10.§ 2
Baselines. We compare OpenNWM with NWM (Bar et al. [2]) and RAE-NWM (Zhang et al. (2026b)) for visual prediction and planning, and include NWM CDiT-XL + Ego4D (Grauman et al. [12]) for visual prediction. Navigation baselines additionally include GNM (Shah et al. (2023b)), NoMaD (Sridhar et al. (2024)), CompACT (Kim et al. [21]), and the latent-action controller of Garrido et al. [11].
Implementation Details. As shown in Figure 3 we train the LAM on NavAnywhere together with motion-annotated trajectories from RECON, HuRoN, and SCAND, and use its inferred latent actions to pretrain the world model. We then post-train on four navigation datasets with camera poses for physical grounding. OpenNWM uses a CDiT-B/2 backbone and a Stable Diffusion VAE (Blattmann et al. [3]), with four context frames and four sampled prediction horizons per trajectory. Both LAM and NWM are trained on 8 NVIDIA L20 GPUs.

Figure 3. Three-stage training pipeline of OpenNWM. (a) Latent-action pretraining on action-free NavAnywhere videos. (b) Action-encoder warmup with the CDiT backbone frozen. (c) Joint post-training of the entire model.
Comparison Results
Visual Prediction. Table 1 reports ID averages over RECON, SCAND, HuRoN, and TartanDrive, and OOD PSNR on Go Stanford (Stanford), TUM RGB-D (TUM), Planetary Rover (Rover), and
| Method | Params. (M) | ID LPIPS↓ | ID DS↓ | ID PSNR↑ | OOD Stanford | OOD TUM | OOD Rover | OOD Go2 |
| NWM | 279 | 0.405 | 0.230 | 13.811 | 11.323 | 8.957 | 11.739 | 10.750 |
| NWM† | 1,099 | 0.419 | 0.227 | 13.523 | 11.444 | 8.884 | 11.906 | 11.083 |
| RAE-NWM | 867 | 0.442 | 0.254 | 13.957 | 11.528 | 9.007 | 9.449 | 10.500 |
| OpenNWM | 280 | 0.368 | 0.196 | 14.299 | 11.591 | 9.135 | 12.574 | <u>11.047</u> |
Table 1. Direct visual prediction at 4 s. DS: DreamSim; NWM†: CDiT-XL + Ego4D. ID/OOD denote in-domain/out-of-domain evaluation. Best and second-best results are bold and <u>underlined</u>, respectively.
Office Go2 (Go2). OpenNWM achieves the best results among the compared methods on all three metrics. Relative to the strongest baseline for each metric, it reduces LPIPS and DreamSim by 9.1% and 13.7%, respectively, and improves PSNR by 0.342 dB. These gains are obtained with a 280M-parameter world model, comparable in size to NWM (279M) and substantially smaller than RAE-NWM (867M) and NWM CDiT-XL + Ego4D (1,099M). These results suggest that OpenNWM learns a more accurate model of navigation dynamics, enabling more faithful action-conditioned predictions of future observations.
Navigation Performance. Table 2 evaluates navigation performance on RECON, SCAND, and Office Go2 using ATE and RPE. OpenNWM achieves the lowest ATE and the best or tied-best RPE across all three datasets. On the in-domain benchmarks, it reduces ATE from 1.13 to 1.12 on RECON and from 1.01 to 0.98 on SCAND relative to NWM. It also ties with NWM for the best RPE of 0.35 on RECON and achieves the lowest RPE of 0.27 on SCAND, compared with 0.28 for both NWM and RAE-NWM. Together with the visual prediction results, these findings support the effectiveness of OpenNWM as a predictive model for navigation planning.
| Method | Params. (M) | RECON ATE↓ | RECON RPE↓ | SCAND ATE↓ | SCAND RPE↓ | Office Go2 ATE↓ | Office Go2 RPE↓ |
| GNM | 9 | 1.87 | 0.73 | 2.12 | 0.61 | – | – |
| NoMaD | 19 | 1.95 | 0.53 | 2.24 | 0.49 | – | – |
| NWM | 279 | <u>1.13</u> | 0.35 | <u>1.01</u> | <u>0.28</u> | 3.03 | 0.68 |
| RAE-NWM | 867 | 1.36 | <u>0.37</u> | 1.14 | <u>0.28</u> | 3.01 | <u>0.67</u> |
| Garrido et al. [11] | – | 1.40 | 0.40 | – | – | – | – |
| CompACT | – | 1.33 | 0.39 | 1.36 | 0.34 | – | – |
| OpenNWM | 280 | 1.12 | 0.35 | 0.98 | 0.27 | 2.89 | 0.65 |
Table 2. Navigation performance on RECON, SCAND, and Office Go2 (OOD). Lower ATE and RPE indicate better performance. Best and second-best results are bold and <u>underlined</u>, respectively.
Out-of-Distribution Generalization. We evaluate generalization to unseen domains through both visual prediction and navigation planning. For visual prediction, OpenNWM achieves the highest PSNR on three of the four OOD datasets in Table 1, outperforming the strongest baseline on Go Stanford, TUM RGB-D, and Planetary Rover. On Office Go2, it ranks second with a PSNR of 11.047 dB, only 0.036 dB below NWM CDiT-XL + Ego4D. Beyond visual prediction, Office Go2 also serves as an OOD navigation benchmark in Table 2. OpenNWM achieves the lowest ATE of 2.89 and RPE of 0.65, improving upon the strongest navigation baseline RAE-NWM on this dataset. Notably, OpenNWM achieves the best reported navigation accuracy on Office Go2 despite ranking second in PSNR, highlighting the importance of assessing generalization through downstream planning as well as visual prediction. Overall, these results support the transferability of OpenNWM to the evaluated unseen navigation domains.
LAM Reconstruction and Generalization. As shown in Table 3, our LAM achieves the best RGB reconstruction results in both in-domain (ID) and out-of-domain (OOD) settings, demonstrating consistent advantages in pixel fidelity and perceptual similarity. For ID evaluation, our method achieves a PSNR of 21.071 dB and an LPIPS of 0.505, improving upon the strongest baseline for each metric by 2.121 dB and 2.1%, respectively. For OOD evaluation, it achieves a PSNR of 21.912 dB and an LPIPS of 0.472, outperforming DreamBoJ by 1.199 dB and 2.7%, respectively. These results suggest that the learned latents retain key visual information and generalize to unseen domains, supporting downstream world-model prediction.
| Model | Source | ID PSNR ↑ | ID LPIPS ↓ | OOD PSNR ↑ | OOD LPIPS ↓ |
| villa-X | Chen et al. 2026 | 15.327 | 0.516 | 17.560 | 0.550 |
| LARA | Liu et al. 2026 | 15.752 | 0.538 | 17.742 | 0.527 |
| Olaf-World | Jiang et al. 2026b | 14.764 | 0.635 | 16.599 | 0.614 |
| DiLA | Zhang et al. 2026c | 13.180 | 0.597 | 14.078 | 0.621 |
| DreamDojo | Gao et al. 2026 | 18.950 | 0.529 | 20.713 | 0.485 |
| Ours | 21.071 | 0.505 | 21.912 | 0.472 |
Table 3. Image reconstruction performance of LAMs. ID (in-domain): average over RECON, SCAND, and HuRoN. OOD (out-of-domain): unseen Go Stanford. Best results are in bold and second best are _underlined_.
Ablation Studies
To assess the effectiveness of our components, we focus on two questions: (1) Do latent actions benefit navigation world models? (2) Which LAM supervision best supports navigation world models? We evaluate visual prediction using LPIPS for direct prediction on Office Go2 at 4 s and autoregressive prediction on Go Stanford at 4 FPS up to 4 s. Due to computational constraints, we use a smaller subset of NavAnywhere for pretraining in these ablations. Details are provided in the Appendix C.1.
Pretraining and Conditioning Ablation. Table 4 compares different conditioning signals during pretraining. Joint time and latent-action conditioning performs best across all six metrics, reducing SCAND ATE from 1.045 with time conditioning alone to 0.950. This suggests that motion information complements the temporal prediction horizon.
Table 4: Pretraining ablation. Effect of different DiT conditioning signals. Best and second-best results are bold and underlined, respectively.
| DiT Conditioning | Reconstruction | Navigation | |||||
| Time | Latent Action | Direct | Autoregressive | RECON | SCAND | ||
| LPIPS | LPIPS | ATE | RPE | ATE | RPE | ||
| ✗ | ✗ | 0.628 | 0.667 | 1.198 | 0.367 | 1.252 | 0.316 |
| ✓ | ✗ | 0.595 | 0.678 | 1.174 | 0.365 | 1.045 | 0.286 |
| ✗ | ✓ | 0.609 | 0.667 | 1.198 | 0.359 | 1.067 | 0.289 |
| ✓ | ✓ | 0.587 | 0.658 | 1.117 | 0.356 | 0.950 | 0.267 |
Table 4. Pretraining ablation. Effect of different DiT conditioning signals. Best and second-best results are bold and underlined, respectively.
LAM Supervision Ablation. Table 5 compares world model performance using different LAMs learned with pixel-only, action-only, and joint supervision. As shown in Figure 4, Pixel+Action supervision achieves the highest Macro-6 Recall on four of five datasets. It consistently outperforms action-only supervision, suggesting that visual reconstruction complements physical motion grounding in aligning latent actions with navigation semantics. Joint supervision leads on five of six metrics. Figure 5 visualizes latent-cluster assignments in the physical motion space (, , ) under four LAM supervision paradigms. Compared with pixel-only supervision, joint supervision exhibits more coherent cluster organization with respect to translation and rotation, suggesting stronger alignment between latent actions and physical motion semantics. Together, these quantitative results and qualitative patterns suggest that visual prediction and physical-motion grounding provide complementary supervision for learning effective latent actions. The Appendix F provides LAM paradigm definitions, qualitative analysis, and semantic-consistency evaluation.
Figure 4. Semantic consistency of latent actions. Detailed metric definitions are provided in the appendix.
Figure 5. Latent-action clustering under four LAM supervision paradigms.
Table 5: LAM supervision ablation. Effect of different supervision on latent-action learning. Best/second-best results are bold/underlined.
| LAM Supervision | Reconstruction | Navigation | |||||
| Pixel | Action | Direct | Autoregressive | RECON | SCAND | ||
| LPIPS | LPIPS | ATE | RPE | ATE | RPE | ||
| ✗ | ✓ | 0.598 | 0.670 | 1.138 | 0.365 | 0.981 | 0.274 |
| ✓ | ✗ | 0.605 | 0.674 | 1.176 | 0.352 | 0.982 | 0.274 |
| ✓ | ✓ | 0.587 | 0.658 | 1.117 | 0.356 | 0.950 | 0.267 |
Table 5. LAM supervision ablation. Effect of different supervision on latent-action learning. Best/second-best results are bold/underlined.
Conclusion
We introduced OpenNWM, a generalist world model for open-world visual navigation. At its core is a shared latent-action interface that unifies learning from heterogeneous visual trajectories with different action spaces and levels of physical supervision. By coupling large-scale action-free video pretraining with physical-motion grounding, OpenNWM learns transition representations that are both predictive of future observations and aligned with executable navigation motion. These representations condition a diffusion world model and support receding-horizon planning after post-training. Experiments across diverse environments demonstrate strong performance in visual future prediction, navigation planning, and generalization to unseen scenes. Taken together, our results establish latent actions as a scalable means of converting open-domain videos into control-relevant navigation supervision, providing a promising foundation for generalist Navigation World Models. Future work will extend this framework to longer-horizon prediction, higher-dimensional action spaces, and online adaptation in dynamic environments.
AI USE STATEMENT
In this work, we have not used generative AI tools for any tasks with required disclosure, and all such tasks are not applicable to this work. We used generative AI tools only to polish the writing of parts of the manuscript. The authors reviewed all AI-assisted edits to ensure that they preserve the intended technical meaning and contain no unsupported claims. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.
REPRODUCIBILITY STATEMENT
For reproducibility, the main paper and appendices detail our methodology and implementation. We will release the complete source code and model weights.
NavAnywhere Dataset Details
We introduce NavAnywhere, a large-scale, action-free visual navigation dataset assembled from 15 sources, including public datasets and our in-house recordings. The collection consists of RGB videos and sampled image sequences without action annotations, comprising 70,756 sequences, 17,496,570 sampled frames, and 1,188.65 hours of source video. NavAnywhere covers a broad spectrum of environments encountered in everyday life, ranging from residential and public indoor spaces to urban streets, parks, and natural outdoor environments. (see Figure 6)
Figure 6. Overview of NavAnywhere and its complementary out-of-domain evaluation sets. NavAnywhere covers diverse everyday navigation environments. The complementary evaluation sets are separate from NavAnywhere and are described in Section B.2.
Dataset Composition
NavAnywhere brings together 15 sources, including public datasets and in-house recordings, covering a broad range of navigation environments, camera viewpoints, and motion patterns. Below, we briefly describe each source and its main characteristics to provide an overview of the dataset’s composition:
BotanicGarden (Liu et al., 2024): Robot-mounted observations of botanical gardens, featuring vegetation, pathways, and other outdoor structures.
CityWalker Liu et al. (2025): Urban navigation sequences featuring sidewalks, intersections, pedestrians, and vehicles.
DL3DV-10K (Ling et al., 2024): Scene-exploration videos spanning diverse indoor and outdoor environments, viewpoints, and camera motions.
Ego4D Grauman et al. [12]: A navigation-related subset of egocentric videos depicting everyday activities and movement through indoor and outdoor environments.
EgoWalk [1]: First-person walking videos covering urban, residential, public, and recreational environments.
KrishnaCam (Singh et al., 2016): Long-term egocentric recordings of daily activities, including walking, commuting, and visits to indoor and outdoor spaces.
LAVN [19]: A collection combining real-world recordings with simulated navigation sequences.
ROVER (Schmidt et al., 2025): Robot navigation recordings collected in gardens, parks, and campus environments across different seasons and illumination conditions.
RealEstate10K (Zhou et al., 2018): Real-estate video clips of residential interiors, exteriors, and surroundings, including drone footage.
SANPO (Waghmare et al., 2025): Real and synthetic egocentric navigation sequences covering sidewalks, roads, parks, and other outdoor environments.
The Great Outdoors [17]: Robot-mounted observations of natural and off-road environments, including forests, unpaved trails, and uneven terrain.
Walking Tours (Venkataramanan et al., 2024): Long-form internet videos depicting walking tours and outdoor exploration.
i2Nav-Robot (Tang et al., 2025): Robot navigation sequences covering campus buildings, streets, parking areas, and transitions between indoor and outdoor spaces.
uB-VisioGeoloc (Scalvini et al., 2024): Pedestrian-view urban sequences covering streets, public spaces, and parks, together with synthetic sequences.
In-house collected data: Our in-house collection of nine RGB videos, covering offices, residential neighborhoods, and public landmarks across indoor and outdoor environments under daytime and nighttime conditions.
Table 6 and Figure 7 reports the complete NavAnywhere dataset statistics.

Figure 7. Source composition of NavAnywhere. Statistics cover the 15 action-free sources.
| Dataset | Sequences | Frames | Hours |
| Ego4D | 1,619 | 13,088,098 | 908.95 |
| DL3DV-10K | 20,352 | 1,180,382 | 81.36 |
| KrishnaCam | 436 | 996,666 | 69.21 |
| RealEstate10K | 45,010 | 712,145 | 49.44 |
| LAVN | 312 | 443,244 | 30.78 |
| Walking Tours | 10 | 198,097 | 14.40 |
| SANPO | 2,662 | 128,233 | 8.91 |
| CityWalker | 18 | 102,083 | 7.09 |
| ROVER | 39 | 76,442 | 5.31 |
| The Great Outdoors | 17 | 69,162 | 4.80 |
| i2Nav-Robot | 10 | 55,202 | 3.83 |
| EgoWalk | 239 | 423,389 | 2.94 |
| BotanicGarden | 7 | 12,272 | 0.85 |
| uB-VisioGeoloc | 16 | 6,382 | 0.44 |
| In-house Collected | 9 | 4,773 | 0.33 |
| NavAnywhere total | 70,756 | 17,496,570 | 1,188.65 |
Table 6. NavAnywhere dataset statistics. Sequences denote processed video clips or trajectories, and frames denote sampled RGB images. Hours measure source-video duration. All 15 sources are used as RGB-only, action-free data.
Scene and Environment Diversity
NavAnywhere covers a wide range of indoor and outdoor environments, reflecting the diversity of scenes encountered in everyday navigation. We organize these environments into five broad scene categories, each characterized by distinct spatial layouts and environmental features, as described below:
Residential environments: Homes, residential buildings, and surrounding neighborhoods.
Public and commercial environments: Offices, schools, shops, restaurants, and cultural facilities.
Urban and transportation environments: Streets, sidewalks, intersections, parking areas, and transit spaces.
Parks and managed outdoor environments: Gardens, campuses, recreational areas, and maintained green spaces.
Natural and off-road environments: Forests, trails, uneven terrain, and unstructured outdoor spaces.
Across these categories, the constituent sources exhibit substantial variation in viewpoints, illumination conditions, scene layouts, and motion patterns.
Dataset Details and Preprocessing
In addition to NavAnywhere, we use five action-annotated navigation datasets and three complementary out-of-domain evaluation sets. These datasets are not part of NavAnywhere and are excluded from its corpus statistics. This section describes these datasets, the preprocessing procedures applied to all data sources, and their respective roles in OpenNWM training and evaluation.
Action-Annotated Datasets
Following the dataset usage in NWM [2], we separately use RECON, HuRoN, SCAND, TartanDrive, and Go Stanford. Their processed trajectories provide RGB observations, planar positions, and yaw, from which relative waypoint actions (, , ) are derived in the reference frame of the current observation.
Go Stanford is reserved exclusively for unseen-environment evaluation. The other four datasets provide motion supervision according to the LAM and OpenNWM training protocols described in Section B.4. Table 7 summarizes the processed collections.
| Dataset | Sequences | Frames | Data used |
| RECON | 11,835 | 610,911 | RGB + position/yaw |
| HuRoN | 2,839 | 230,911 | RGB + position/yaw |
| Go Stanford | 3,696 | 198,126 | RGB + position/yaw |
| SCAND | 604 | 103,457 | RGB + position/yaw |
| TartanDrive | 1,251 | 62,884 | RGB + position/yaw |
Table 7. Statistics of the action-annotated datasets. Sequences denote processed trajectories, and frames denote sampled RGB images. Go Stanford is used only for evaluation.
Complementary Out-of-Domain Evaluation Sets
We evaluate out-of-domain transfer on three challenging real-world datasets excluded from training: TUM RGB-D, Planetary Rover, and Office-Go2.
TUM RGB-D. We select five sequences from the TUM RGB-D SLAM benchmark ((Sturm et al., 2012)), using RGB observations and the provided ground-truth camera poses. The processed subset contains 1,189 sampled frames.
Planetary Rover. We curate extraterrestrial navigation observations from publicly available online sources, using released imagery and official pose metadata from the Chang’e-4/Yutu-2 lunar mission and the Tianwen-1/Zhurong Martian mission. The collection contains 66 lunar and 173 Martian RGB images, organized into 28 sequences with 211 adjacent transitions whose relative motions are derived from pose metadata.
Office-Go2. We collect navigation recordings in-house using a Unitree Go2 quadruped robot in a real office environment. The collection includes RGB videos and synchronized camera-pose measurements. Its seven recordings are processed into 15 sequences containing 2,292 sampled frames.
Data Processing
Temporal sampling. We organize sampled frames in temporal order within each sequence.
Most NavAnywhere sources are sampled at 4 Hz, corresponding to a frame interval of 0.25 s. EgoWalk, in contrast, is originally recorded at 100 Hz to capture relatively fast ego-motion. So EgoWalk sequences have an effective sampling rate of approximately 40 Hz, corresponding to a 0.025 s interval.
Image-pair sampling for LAM training. For LAM training, image pairs are sampled within the same sequence using signed frame offsets . These offsets correspond to temporal displacements of up to 2 s for 4 Hz sources and approximately 0.2 s for EgoWalk.
Context and target sampling for pretraining. For pretraining, each example contains four consecutive context frames and four target frames sampled at signed offsets within relative to the latest context frame. At 4 Hz, these offsets span s; for EgoWalk, they span approximately s. Latent-action conditioning is available for local image pairs satisfying .
Processing of action-annotated and evaluation data. The five external action-annotated datasets are converted to 4 Hz trajectories with image-aligned planar positions and yaw. TUM RGB-D and Office-Go2 pair sampled RGB frames with camera poses, while Planetary Rover retains the released observation order and endpoint poses without temporal image interpolation.
Dataset Usage in OpenNWM
Action-free learning from NavAnywhere. NavAnywhere provides action-free visual observations for latent-action learning and world-model pretraining. Its training inputs consist only of RGB observations, even when the original sources provide additional sensing modalities or annotations. Latent actions are inferred from image pairs by the learned latent action model.
Physical-motion supervision and evaluation. Physical-motion grounding of the LAM uses the training splits of RECON, HuRoN, and SCAND. Action-conditioned post-training uses RECON, HuRoN, SCAND, and TartanDrive. Go Stanford and all three complementary out-of-domain evaluation sets are excluded from these training stages.
Architecture and Implementation Details
Training and Inference Details
Table 8 summarizes the training and inference configurations of OpenNWM used in the main experiments.
| Module | Configuration | Value |
| Latent Action Model | Architecture | Pixel+Action spatiotemporal Transformer |
| Latent Action Model | Model size | M |
| Latent Action Model | Encoder / decoder blocks | 24 / 24 |
| Latent Action Model | Hidden width / attention heads | 1,024 / 16 |
| Latent Action Model | Latent-action dimension | 32 |
| Latent Action Model | Input frames | 2 |
| Latent Action Model | Input resolution / patch size | 240 × 320 / 16 × 16 |
| Latent Action Model | Temporal offsets | Integers in |
| Latent Action Model | Action-labeled batch fraction | 25% |
| Latent Action Model | RGB / action / KL loss weights | 1 / 0.171244 / |
| Latent Action Model | Training steps | 60K |
| Latent Action Model | Global batch size | 160 |
| Latent Action Model | Optimizer | AdamW |
| Latent Action Model | Peak learning rate / weight decay | / 0.01 |
| Latent Action Model | LR warm-up / schedule length | 12.8K / 100K steps |
| Latent Action Model | LR decay / minimum LR ratio | Cosine / 0.1 |
| Latent Action Model | Precision / training hardware | BF16 / 8× NVIDIA L20 |
| Navigation World Model | Backbone / model size | CDiT-B/2 / M |
| Navigation World Model | Image tokenizer | Frozen SD-VAE |
| Navigation World Model | Input resolution | 224 × 224 |
| Navigation World Model | Context frames / targets per observation | 4 / 4 |
| Navigation World Model | Target frame offsets | Integers in |
| Navigation World Model | Latent-action conditioning range | |
| Navigation World Model | Pretraining steps | 60K |
| Navigation World Model | Pretraining learning rate | |
| Navigation World Model | Physical-action adapter warm-up | 3K steps |
| Navigation World Model | Joint post-training steps | 100K |
| Navigation World Model | Post-training LR: adapter / backbone | / |
| Navigation World Model | Global observation / prediction batch | 384 / 1,536 |
| Navigation World Model | Optimizer / weight decay | AdamW / 0.01 |
| Navigation World Model | Gradient clipping | 2.0 |
| Navigation World Model | Precision / training hardware | BF16 / 8× NVIDIA L20 |
| Inference and Planning | Diffusion sampler / sampling steps | DDPM / 250 |
| Inference and Planning | Planning action representation | Physical motion (, , ) |
| Inference and Planning | Planning horizon | 2 s |
| Inference and Planning | Action interval / sequence length | 0.25 s / 8 |
| Inference and Planning | CEM candidates | 80 |
| Inference and Planning | CEM elites | 5 |
| Inference and Planning | CEM updates | 1 |
| Inference and Planning | Stochastic evaluations per candidate | 3 |
| Inference and Planning | Planning cost | LPIPS-Alex |
Table 8. Training and inference configurations of OpenNWM. Batch sizes are global, and image resolutions are reported as height × width.
Latent-action learning. Our Pixel+Action navigation LAM follows the spatiotemporal Transformer architecture of Gao et al. [10]. It contains approximately 709M parameters, comprising a 24-block spatiotemporal encoder and a 24-block spatial decoder, both with a hidden width of 1,024 and 16 attention heads. The encoder maps a pair of RGB frames to a 32-dimensional variational latent. Frames are center-cropped to a 4:3 aspect ratio and resized to 240 × 320 (height × width), with 16 × 16 patches. Frame pairs use uniformly sampled integer temporal offsets in .
LAM training combines NavAnywhere with the training splits of RECON, HuRoN, and SCAND, reserving 25% of each batch for action-labeled samples. The objective combines RGB reconstruction MSE, action MSE on labeled samples, and KL regularization, with respective weights 1, 0.171244, and . We train from scratch using AdamW with a global batch size of 160, a peak learning rate of , and weight decay of 0.01, using BF16 on eight NVIDIA L20 GPUs. The learning-rate schedule is configured for 100K steps, with 12.8K steps of linear warm-up followed by cosine decay toward 10% of the peak rate. We use the 60K-step LAM checkpoint to extract deterministic posterior means as latent actions for pretraining.
World-model pretraining. OpenNWM uses a CDiT-B/2 diffusion transformer with approximately 197M parameters and a frozen VAE tokenizer. Images are resized to 224 × 224. Each observation provides four context frames and four sampled prediction targets. Pretraining uses the NavAnywhere subset, with signed target frame offsets in . All samples receive relative-time conditioning. Valid pairs within additionally receive their layer-normalized 32-dimensional latent actions; pairs outside this range contribute to the diffusion objective without latent-action conditioning.
We use the 60K-step pretraining checkpoint, trained with AdamW at a learning rate of , weight decay of 0.01, and gradient clipping at 2.0. Training uses BF16 on eight NVIDIA L20 GPUs, with 48 observations per GPU, giving a global batch of 384 observations and 1,536 context-target pairs per update.
World model post-training. For post-training, we replace the latent action adapter with a newly initialized adapter for three-dimensional physical motion (, , ). We first train this adapter for 3K steps with the backbone frozen, followed by 100K steps of joint post-training of the adapter and backbone. The adapter learning rate is throughout post-training, and the backbone learning rate is during the joint phase. Both phases use AdamW with weight decay of 0.01 and the same batch size, precision, and hardware as pretraining.
Post-training uses the training splits of RECON, HuRoN, SCAND, and TartanDrive. Go Stanford is excluded from training. Physical-motion targets are expressed relative to the latest context frame, with translations divided by dataset-specific waypoint spacing before fixed min–max normalization.
Inference and planning. Visual prediction uses 250-step DDPM [15] sampling with fixed evaluation windows and seeds. Planning evaluates sequences of eight physical-motion actions at 0.25 s intervals, corresponding to a 2 s horizon. The cross-entropy method (CEM) uses 80 candidates, five elites, and one proposal update. Candidate costs are computed using LPIPS-Alex and averaged over three stochastic evaluations. Planning conditions the post-trained world model directly on physical motion.
Ablation settings. The ablations retain the same CDiT-B/2 backbone, image resolution, context and target counts, and training data sources as the main experiments. The main difference are less NavAnywhere data to train the LAM, 200K pretraining steps at a learning rate of , and a global batch of 128 observations. Post-training uses a 10K-step adapter warm-up followed by 100K joint updates, with adapter and backbone learning rates of and , respectively. The ablations use the same 250-step DDPM and CEM80 planning protocol described above.
Evaluation Metrics
We evaluate visual prediction quality using three metrics: Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS), and Fréchet Inception Distance (FID), covering pixel-level fidelity, structural consistency, perceptual similarity, and distributional similarity.
Pixel-Level Fidelity. Peak Signal-to-Noise Ratio (PSNR) measures reconstruction fidelity on a logarithmic scale. Given the predicted image and the ground-truth image , it is defined as:
where denotes the total number of image elements, including all spatial locations and color channels, and denotes the maximum possible intensity value under the adopted image representation. PSNR is expressed in decibels (dB), with higher values indicating higher pixel-level fidelity.
Perceptual Similarity. Learned Perceptual Image Patch Similarity (LPIPS) measures perceptual discrepancy in the feature space of a pretrained vision network. Given channel-normalized feature representations at layer , it is defined as:
where denotes the learned channel-wise weights, denotes element-wise multiplication, and is the spatial resolution of the corresponding feature map. Lower LPIPS indicates higher perceptual similarity, complementing pixel-level and structural measures.
Distributional Similarity. Fréchet Inception Distance (FID) [13] measures the discrepancy between the feature distributions of predicted and ground-truth image sets. Features are extracted using a pretrained Inception-v3 network, and each feature distribution is approximated by a multivariate Gaussian. Let and denote the empirical means and covariance matrices of the predicted and ground-truth image features, respectively. FID is defined as:
where denotes the matrix trace and denotes the matrix square root. Lower FID indicates closer agreement between the feature distributions of predicted and ground-truth images. Unlike PSNR and LPIPS, FID is computed over image sets rather than individual image pairs and does not directly measure prediction–target correspondence.
Together, these four metrics provide complementary assessments of visual prediction quality, from paired reconstruction fidelity to distribution-level similarity.
Extended Quantitative Results
Consistency across Datasets. Table 9 shows that OpenNWM improves over NWM on every visual prediction metric across all four in-domain datasets. In comparison, RAE-NWM achieves stronger results on HuRoN but underperforms NWM on TartanDrive, revealing greater variation in its relative performance across datasets. The advantage of OpenNWM therefore extends beyond an improvement in the average score: its gains are distributed across the evaluated sources without degrading any reported dataset–metric combination relative to NWM. This consistency also extends to navigation (Table 10), where OpenNWM reduces ATE on every dataset while matching or improving RPE.
| Method | RECON | SCAND | HuRoN | TartanDrive | ||||||||
| LPIPS↓ | DS↓ | PSNR↑ | LPIPS↓ | DS↓ | PSNR↑ | LPIPS↓ | DS↓ | PSNR↑ | LPIPS↓ | DS↓ | PSNR↑ | |
| NWM | 0.323 | 0.120 | 15.487 | 0.406 | 0.300 | 12.778 | 0.463 | 0.275 | 13.377 | 0.427 | 0.223 | 13.602 |
| NWM XL (with Ego4D) | 0.359 | 0.140 | 14.850 | 0.394 | 0.260 | 13.068 | 0.501 | 0.295 | 12.604 | 0.424 | 0.213 | 13.569 |
| RAE-NWM | 0.353 | 0.134 | 15.491 | 0.413 | 0.190 | 14.419 | 0.339 | 0.171 | 15.079 | 0.662 | 0.519 | 10.840 |
| OpenNWM | 0.320 | 0.116 | 15.614 | 0.385 | 0.271 | 13.247 | 0.362 | 0.193 | 14.329 | 0.405 | 0.203 | 14.005 |
Table 9. In-distribution (ID) direct visual prediction at the nominal 4 s horizon. DS: DreamSim. Best and second-best results are bold and underlined, respectively.
| Method | RECON | SCAND | TartanDrive | |||
| ATE | RPE | ATE | RPE | ATE | RPE | |
| GNM | 1.87 | 0.73 | 2.12 | 0.61 | 6.65 | 1.62 |
| NoMaD | 1.95 | 0.53 | 2.24 | 0.49 | 6.32 | 1.31 |
| NWM | <u>1.13</u> | 0.35 | <u>1.01</u> | <u>0.28</u> | <u>5.72</u> | <u>1.19</u> |
| RAE-NWM | 1.36 | <u>0.37</u> | 1.14 | <u>0.28</u> | 5.99 | 1.25 |
| Garrido et al. | 1.40 | 0.40 | – | – | – | – |
| CompACT | 1.33 | 0.39 | 1.36 | 0.34 | – | – |
| OpenNWM (Ours) | 1.12 | 0.35 | 0.98 | 0.27 | 5.53 | 1.16 |
Table 10. Navigation performance on RECON, SCAND, and TartanDrive. Best and second-best available results are bold and underlined, respectively.
Visual Fidelity and Navigation Utility. SCAND reveals a mismatch between visual prediction and navigation rankings: RAE-NWM achieves better PSNR and DreamSim than OpenNWM, yet OpenNWM obtains lower ATE and RPE. Moreover, the visual metrics themselves disagree, with LPIPS favoring OpenNWM. These results indicate that individual image-similarity metrics are incomplete proxies for downstream navigation performance. Direct prediction measures agreement with an observed future, but does not directly assess whether predicted transitions support effective action selection. The navigation results therefore provide complementary evidence of the model’s utility for control.
Scaling and Ablation Studies
We examine how the amount of unlabeled video used for latent-action learning affects the navigation world model. For this ablation, we compare four latent-action pretraining settings that retain 25%, 50%, 75%, or 100% of the unlabeled data pool. We denote this percentage by and keep the action-label availability fixed. Each resulting model is then fine-tuned with real actions using the same downstream architecture, hyperparameters, and fine-tuning schedule. We evaluate all four models on the same 500 RECON windows for direct 4 s prediction. We report the mean DreamSim over these predictions; lower values indicate greater perceptual similarity to the ground-truth images.
Figure 8 shows a monotonic decrease in RECON DreamSim at , 50%, 75%, and 100%, respectively. Using the full unlabeled data pool reduces DreamSim by 6.08% relative to the 25% setting. The improvement continues from 75% to 100%. This supports the benefit of scaling unlabeled video for perceptual prediction quality.

Figure 8. Scaling unlabeled data ablation. The horizontal axis gives the percentage of the unlabeled data pool used for latent-action pretraining, with action-label availability fixed.
Latent-Action Visualization and Analysis
We further analyze whether learned Universal Navigation Latent Actions (UNLAs) capture physically grounded navigation semantics and generalize across heterogeneous environments. Specifically, we investigate two aspects: (1) physical alignment between latent actions and induced motions, and (2) cross-dataset consistency of latent action semantics.
Latent action learning paradigms.
As shown in Figure 9, we compare four paradigms for learning continuous latent actions from visual transitions . Each infers a variational latent and applies KL regularization toward a standard Gaussian prior. Action predicts the observed navigation action from using an action head. Pixel reconstructs conditioned on . DINO feature operates in a frozen DINOv3 ((Simeoni et al., 2025)) feature space: its encoder infers from , where , and its decoder predicts from . Pixel + Action jointly optimizes pixel reconstruction and action prediction from the same latent, applying action supervision only to labeled samples. All prediction losses use mean squared error; the joint objective weights the action term by . These paradigms examine how action supervision, visual reconstruction, and their combination shape the learned representation.
Physical Alignment of Latent Actions
To investigate whether latent actions encode meaningful navigation dynamics, we analyze the correspondence between latent actions and their associated physical motions. Given an anchor observation and a target observation, each model generates a 32-dimensional latent action representation. For each supervision paradigm, we independently perform K-means clustering in the original 32-dimensional latent space (rather than the projected visualization space) with , obtaining eight latent action clusters .
Importantly, clustering is performed solely in the original latent space, rather than the three-dimensional motion space used for visualization. After clustering, each sample is visualized in the physical motion space (, , ), where each point represents an anchor-target observation pair and the color indicates its latent cluster assignment.
Figure 9. Four latent action learning paradigms. An encoder maps a visual transition to a variational latent . The prediction targets are (a) the observed action , (b) the future image , (c) frozen DINOv3 features , and (d) both the image and action. Reconstruction decoders additionally receive the current image or its features. In (d), pixel reconstruction uses all samples, while the action loss uses only labeled samples. All variants include the same form of KL regularization.
Compact clusters in the physical space indicate that samples sharing the same latent action cluster correspond to similar navigation motions. Conversely, highly overlapping clusters suggest that the latent action representation is affected by non-motion factors such as scene appearance, texture, or observation variations.
Figure 5 shows the clustering results under different supervision paradigms. Action supervision produces the most compact and clearly separated motion clusters, with different clusters aligning well with translational and rotational motion patterns. Joint Action-Pixel supervision achieves similar motion-oriented structures, suggesting that visual supervision improves representation richness while physical supervision preserves motion semantics.
DINO-based self-supervision discovers partial motion structures, but different clusters exhibit substantial overlap in the physical space. This indicates that semantic visual features can capture high-level scene variations but are insufficient to fully distinguish controllable agent motion from other visual changes. In contrast, Pixel-level self-supervision produces the most entangled clusters, where different latent clusters often correspond to similar physical motions. This suggests that pixel-level objectives tend to encode appearance-related variations rather than navigation-specific dynamics.
Note that the presence of eight clusters alone does not indicate meaningful latent actions, since K-means always produces the predefined number of clusters. The key criterion is whether samples within the same cluster exhibit consistent physical effects. The results demonstrate that physical supervision plays a critical role in grounding latent actions in controllable navigation semantics.
Cross-Dataset Consistency of Latent Action Semantics
Beyond within-dataset motion alignment, we further evaluate whether latent action semantics remain consistent across heterogeneous environments and datasets. Unlike the previous analysis, which only examines motion structure within each dataset, this evaluation tests whether the learned latent action space provides a transferable navigation interface.
For each supervision paradigm, K-means clustering is first performed on latent representations from in-domain datasets using the original 32-dimensional latent space. The obtained cluster assignments are then fixed and directly transferred to unseen datasets without additional clustering or adaptation. Specifically, we evaluate transfer from the in-domain datasets RECON, HuRoN, and SCAND to the out-of-domain datasets TartanDrive and Go Stanford.
To quantify semantic consistency, we discretize each continuous physical motion transition (, , ) into one of six navigation categories, where and denote the two planar displacement components and denotes the change in yaw angle. The category set is defined as , corresponding to forward, backward, forward-left, forward-right, backward-left, and backward-right, respectively.
Each latent cluster is labeled by its majority ground-truth motion category among in-domain samples, which determines predictions for samples in that cluster. We evaluate agreement with ground truth using Macro-6 Recall, the unweighted mean recall across all six categories:
where denotes the recall for category , is the number of samples whose ground-truth and predicted categories are both , and is the number of samples whose ground-truth category is but whose predicted category differs from . This metric assigns equal weight to each motion category, regardless of its frequency in the evaluation set.
As shown in Figure 10, Action-only supervision achieves strong motion-semantic consistency on in-domain datasets, demonstrating that explicit motion supervision can align latent actions with navigation behaviors. However, its performance decreases on unseen datasets, suggesting that motion semantics learned from action-labeled data may still be affected by dataset-specific distributions.

Figure 10. Visualization of the semantic separability of latent actions learned by four LAM paradigms across different navigation action categories. TartanDrive and Go Stanford are out-of-domain datasets.
Pixel-level self-supervision achieves weaker transfer performance, as visual representations may capture scene appearance and observation variations rather than consistent motion semantics. DINO-based self-supervision improves semantic transferability but still exhibits ambiguity in fine-grained navigation motions.
Joint Action-Pixel supervision achieves the strongest cross-dataset consistency by combining scalable visual supervision with physically grounded motion annotations. This improves within-dataset motion alignment and preserves navigation semantics across heterogeneous environments, supporting UNLAs as a shared, transferable interface between visual experience and navigation dynamics.
Qualitative Results
We present qualitative examples of OpenNWM’s visual predictions, navigation trajectories, and LAM image reconstructions across diverse environments.

Figure 11. Video generation examples on RECON. NWM, RAE-NWM and OpenNWM are conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 0 to 16 seconds, every 2 second.

Figure 12. Video generation examples on HuRoN. NWM, RAE-NWM and OpenNWM are conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 0 to 16 seconds, every 2 second.

Figure 13. Video generation examples on TartanDrive. NWM, RAE-NWM and OpenNWM are conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 0 to 16 seconds, every 2 second.

Figure 14. Video generation examples on Go Stanford (OOD). NWM, RAE-NWM and OpenNWM are conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 4 seconds at 4 FPS. We plot the generated results from 0 to 4 seconds, every 1 second.
Figure 15. Qualitative results of our LAM on HuRoN. Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error () are reported for each example.
Figure 16. Qualitative results of our LAM on RECON. Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error () are reported for each example.
Figure 17. Qualitative results of our LAM on SCAND. Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error () are reported for each example.
Figure 18. Qualitative results of our LAM on Go Stanford (OOD). Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error (°) are reported for each example.
Figure 19. Qualitative results of our LAM on TartanDrive (OOD). Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error () are reported for each example.
Figure 20. Qualitative navigation results of OpenNWM on HuRoN. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 21. Qualitative navigation results of OpenNWM on HuRoN. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 22. Qualitative navigation results of OpenNWM on Office-go2 (OOD). The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 23. Qualitative navigation results of OpenNWM on Office-go2 (OOD). The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 24. Qualitative navigation results of OpenNWM on RECON. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 25. Qualitative navigation results of OpenNWM on RECON. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 26. Qualitative navigation results of OpenNWM on SCAND. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 27. Qualitative navigation results of OpenNWM on SCAND. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 28. Qualitative navigation results of OpenNWM on Go Stanford (OOD). The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 29. Qualitative navigation results of OpenNWM on Go Stanford (OOD). The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 30. Qualitative navigation results of OpenNWM on TartanDrive. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 31. Qualitative navigation results of OpenNWM on TartanDrive. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
References
- [1]Timur Akhtyamov, Mohamad Al Mdfaa, Javier Antonio Ramirez, Sergey Bakulin, German Devyich, Denis Fatykhov, Alexander Mazurov, Kristina Zipa, Malik Mohrat, Pavel Kolesnik, Ivan Sosin, and Gonzalo Ferrer. EgoWalk: A multimodal dataset for robot navigation in the wild. arXiv preprint arXiv:2505.21282, 2025. URL https://arxiv.org/abs/2505.21282.
- [3]Andreas Blattmann, Tim Dockhorn, Sumith Kual, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.arxiv.org/abs/2311.15127
- [4]Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. UniVLA: Learning to act anywhere with task-centric latent actions. In Proceedings of Robotics: Science and Systems (RSS), Los Angeles, CA, USA, June 2025. doi: 10.15607/RSS.2025.XXI.014.DOI
- [5]Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, Jianyu Chen, and Jiang Bian. villa-X: Enhancing latent action modeling in vision-language-action models. arXiv preprint arXiv:2507.23682, 2025. URL https://arxiv.org/abs/2507.23682.
- [8]Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. DreamSim: Learning new dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, volume 36, 2023.
- [9]Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. AdaWorld: Learning adaptable world models with latent actions. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 18744–18771. PMLR, 2025. URL https://arxiv.org/abs/2503.18938.
- [10]Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, et al. Dreamdojo: A generational robot world model from large-scale human videos. arXiv preprint arXiv:2602.06949, 2026.arxiv.org/abs/2602.06949
- [11]Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, and Michael Rabbat. Learning latent action world models in the wild. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2601.05230
- [12]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18995–19012, 2022.arxiv.org/abs/2110.07058
- [13]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, pp. 6626–6637, 2017.
- [14]Noriaki Hirose, Dhruv Shah, Ajay Sridhar, and Sergey Levine. Sacson: Scalable autonomous control for social navigation. IEEE Robotics and Automation Letters, 9(1):49–56, 2023.arxiv.org/abs/2306.01874
- [15]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.arxiv.org/abs/2006.11239
- [16]Siqiao Huang, Jialong Wu, Qixing Zhou, Shangchen Miao, and Mingsheng Long. Vid2world: Crafting video diffusion models to interactive world models. arXiv preprint arXiv:2505.14357, 2025.arxiv.org/abs/2505.14357
- [17]Peng Jiang, Kasi Viswanath, Akhil Nagariya, George Chustz, Maggie Wigness, Philip Osteen, Timothy Overbye, Christian Ellis, Long Quang, Jia Huang, et al. Go: The great outdoors multimodal dataset. In 2026 IEEE Intelligent Vehicles Symposium (IV), pp. 1496–1502. IEEE, 2026a.arxiv.org/abs/2501.19274
- [19]Faith Johnson, Kristin Dana, Bryan Bo Cao, Shubham Jain, and Ashwin Ashok. A landmark-aware visual navigation dataset for map representation learning. In 2025 20th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 1026–1031. IEEE, 2025.DOI
- [21]Dongwon Kim, Gawon Seo, Jinsung Lee, Minsu Cho, and Suha Kwak. Planning in 8 tokens: A compact discrete tokenizer for latent world model. arXiv preprint arXiv:2603.05438, 2026.arxiv.org/abs/2603.05438
- [22]Zoulei Li, Xingyu Gao, Xiaofan Wang, and Jianlong Fu. LatBot: Distilling universal latent actions for vision-language-action models. arXiv preprint arXiv:2511.23034, 2025. URL https://arxiv.org/abs/2511.23034.
- [23]Yihan Lin, Haoyang Li, Yang Li, Haitao Shen, Yihan Zhao, Chao Shao, and Jing Zhang. From pixels to tokens: A systematic study of latent action supervision for vision-language-action models. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026. URL https://arxiv.org/abs/2605.04678.





















