betaFeedback

WorksComputer Science

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

Abstract

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.

Introduction

The philosophy of Heraclitus is often distilled into the aphorism panta rhei—everything flows. In the physical world, appearances and spatial configurations continually change, while objects maintain continuity over time. An embodied world model should capture both aspects. Recent generative video models (Team Wan et al., 2025; Team Seedance et al., 2026) predict increasingly coherent visual futures, but express these predictions in pixels, leaving the underlying 3D dynamics implicit. World action models [8, 5, 2] (NVIDIA 2026) connect future prediction to robotic behavior by incorporating actions into world modeling. However, action supervision is costly to collect at scale, and action representations are often tied to particular robot embodiments, complicating cross-platform transfer. These limitations motivate extending world modeling into the space of explicit 3D motion flow: a geometric representation of physical change that can connect visual prediction to planning and control across embodiments.

A complementary line of research approaches this space through motion modeling, progressing from tracking motion in observed videos (Lu et al., 2026; Luo et al., 2026) to predicting future trajectories from current observations [7] (Lee et al. 2026b). For generative motion prediction, producing dense 3D scene flow is computationally expensive. Existing scene-level methods instead predict motion on coarse spatial grids (Lee et al., 2026a; Li et al., 2026a). However, sparse scene flow still allocates motion tokens to task-irrelevant regions, while its reduced spatial resolution limits the fidelity of fine motion boundaries. Query-conditioned prediction offers a more targeted alternative, allocating motion tokens to specified 3D locations in task-critical regions while retaining global scene context as conditioning information. Yet learning generalizable dynamics at scale faces two challenges: high-quality 3D trajectory annotations are costly to acquire, while flow-only supervision provides a limited learning signal for physical dynamics, potentially limiting generalization across scenes and interactions. Jointly modeling visual evolution and explicit motion offers a natural way to enrich this supervision, bringing motion prediction into a generative world model.

To this end, we introduce WorldFlow, a generative world model that jointly predicts future video and query-conditioned 3D object trajectories through a unified Mixture-of-Transformers (MoT) backbone. Given an observed image, a language instruction, and sparse 3D query points, WorldFlow forecasts motion at the queried locations while using the full scene as visual context. We represent trajectories as continuous tokens encoding normalized frame-to-frame 3D displacements, enriched with initial point geometry and spatiotemporally aligned with video tokens. A unified rectified-flow objective jointly denoises both modalities, coupling trajectory learning with dense visual supervision of the same future.

To support training at scale, we assemble a heterogeneous corpus spanning simulated and real-world domains, comprising approximately 1.44 million object-level samples—equivalent to 1,224 hours of standardized 49-frame sequences at 16 FPS. We combine existing motion annotations with automated trajectory extraction, unify their representations, and apply source-specific quality control and fixed domain-level normalization for mixed-data training. We further introduce WorldFlowBench, comprising three evaluation sets, including SpatialCog for spatial reasoning in future motion prediction. On WorldFlowBench, WorldFlow achieves the lowest ADE and FDE across all three evaluation sets among non-static methods. Compared with its flow-only variant, joint video-flow generation reduces ADE by 48.5% on Sim7 and 55.9% on SpatialCog, with a 68.5% reduction in FDE on SpatialCog. These results support the value of complementary visual supervision for accurate 3D motion forecasting.

Our contributions are:

  • We introduce WorldFlow, a generative world model that jointly predicts video and query-conditioned 3D object motion through a unified MoT backbone, combining explicit trajectory supervision with dense visual supervision.

  • We construct a heterogeneous corpus of approximately 1.44 million object-level samples from simulated and real-world data, using unified motion representations and source-specific quality control to support large-scale training.

  • We introduce WorldFlowBench, comprising three evaluation sets including SpatialCog for spatial reasoning, and demonstrate improvements over existing motion prediction methods and video-generation-then-tracking pipelines.

Related Work

Predictive World Models: From Pixels to Actions

A fundamental objective of visual world modeling is to predict future physical states from current observations. In the visual domain, large-scale video generators establish rich appearance and motion priors to enable coherent future synthesis ((Team Wan et al., 2025; Team Seedance et al., 2026)), driven by architectural and training efficiencies ((Huang et al., 2025; Chen et al., 2025b)), and extending toward physical-world prediction, interactive simulation, and semantic control ((NVIDIA et al., 2025; Robbyant Team et al., 2026; Chen et al., 2026; Bian et al., 2025)). To bridge visual futures with robotic behavior, World Action Models (WAMs) directly incorporate the robot’s action space into predictive modeling, jointly forecasting future observations and actions to support end-to-end policy learning and interactive closed-loop control ([2] (Li et al. 2026c)), while expanding toward in-context task generalization ([8]), fast inference ([5]), and cross-embodiment transfer ((Li et al., 2026a)). Paralleling these efforts, unified multimodal models integrate visual understanding with multi-modal generation ((Deng et al., 2025)), paving the way for comprehensive frameworks that jointly process language, pixels, and actions for physical-world modeling ((NVIDIA, 2026)).

3D Motion Modeling: From Tracking to Forecasting

Point tracking stands as a classic paradigm focused on resolving dense, pixel-level correspondences and spatial linkages over observable video sequences ([6] (Lu et al. 2026; Karhade et al. 2026; Karaev et al. 2024; Karaev et al. 2025; Xiao et al. 2025; Luo et al. 2026)). Although these trackers provide direct measurements of surface history, they operate strictly on immediate visual evidence and remain susceptible to severe error accumulation under complex occlusions or non-rigid deformations. Concurrently, predictive motion forecasting has recently emerged to offer fresh research motivations tailored for downstream embodied and robotic applications. As a generative geometric interface, 3D motion flow bypasses immediate frame matching to predict future physical dynamics directly across objects and embodiments. Early work, General Flow ([4]), learned language-conditioned 3D point trajectories from human RGB-D videos as transferable manipulation affordances. NovaFlow (Li et al., 2025) and Dream2Flow (Dharmarajan et al., 2025) leveraged video generation to obtain motion priors, recovering actionable 3D object flow through tracking and geometric reconstruction. Alongside these video-based pipelines, direct motion models predict future geometry without synthesizing videos. TraceGen (Lee et al., 2026a) models compact 3D traces, while PointWorld (Huang et al., 2026) predicts action-conditioned full-scene flow and ObjectForesight (Soraki et al., 2026) forecasts rigid-object pose trajectories. More recently, RoboFlow4D (Lin et al., 2026), μ0\mu_0 (Lee et al., 2026b), and MolmoMotion ([7]) advanced flow-guided planning, interaction-trace generation, and language-conditioned object-point forecasting, respectively. Flow representations have also become integral to policy learning and visual world modeling: LaMP (Wang et al., 2026) introduces a latent 3D motion prior for action generation, EgoWAM (Li et al., 2026a) studies 3D flow as a world-prediction target for human-to-robot transfer, and Hydra-0 (Li et al., 2026b) uses image-plane action flow to condition video prediction and support robot control.

WorldFlow

Problem Formulation

Given a language instruction cc, an observed RGB frame I0I_0, and 3D query points Q={p0,i}i=1NQ = \{\mathbf{p}_{0,i}\}_{i=1}^{N} with corresponding image coordinates S0={s0,i}i=1NS_0 = \{s_{0,i}\}_{i=1}^{N}, WorldFlow jointly predicts future RGB frames I1:TI_{1:T} and 3D flow f1:Tf_{1:T}. All 3D positions are expressed in the initial camera coordinate system. We define flow as the frame-to-frame displacement of each query point:

ft,i=pt,i−pt−1,i,t=1,…,T.(1)f_{t,i} = \mathbf{p}_{t,i} - \mathbf{p}_{t-1,i}, \qquad t = 1,\ldots,T. \tag*{(1)}

The corresponding trajectories are recovered by accumulation:

pt,i=p0,i+∑τ=1tfτ,i.(2)\mathbf{p}_{t,i} = \mathbf{p}_{0,i} + \sum_{\tau=1}^{t} f_{\tau,i}. \tag*{(2)}

Our objective is to model the conditional joint distribution

pθ(I1:T,f1:T∣I0,Q,S0,c).(3)p_{\theta}\left(I_{1:T}, f_{1:T} \mid I_0, Q, S_0, c\right). \tag*{(3)}

Architecture

Our central design introduces 3D flow tokens as an additional generative modality for joint video-motion prediction. Built on the Cosmos-Nano framework (NVIDIA, 2026), WorldFlow retains the pretrained reasoner and extends the generator to jointly synthesize future video and 3D flow (Figure 2). The reasoner processes the language instruction, while the video VAE encodes the observed frame into clean latents. The flow tokenizer embeds noisy displacements together with initial query-point geometry and aligns their spatiotemporal positional coordinates with the video token grid (Section 3.3).

Overview of WorldFlow showing text prompt, observed frame, query points, pretrained reasoner, joint generator, shared multimodal attention, future video, and future flow

Figure 2. Overview of WorldFlow. Conditioned on text tokens tjt_j through the pretrained reasoner, clean observed-frame latents v0v_0, and query-point geometry, the generator jointly denoises video and flow from noise ϵj\epsilon_j and δk\delta_k through shared multimodal attention. Generated video latents vjv_j are decoded by the VAE, while generated 3D flow ftf_t is rescaled and accumulated from the initial query positions to recover metric trajectories. The 2D flow overlay shows generated 3D trajectories projected onto generated video frames. The 3D flow overlay shows generated 3D trajectories overlaid on a scene point cloud reconstructed from the initial frame’s ground-truth depth.

Within the Mixture-of-Transformers (MoT) architecture, video and flow tokens interact through full attention and attend to the reasoner’s causal context. At inference, future video latents and flow are initialized from noise and jointly denoised, conditioned on the clean observed-frame latents, language context, and query geometry. The VAE decodes the generated video latents into RGB frames, while the predicted flow is rescaled and accumulated to recover 3D trajectories. This joint generation process produces explicit motion alongside video without a separate tracking stage.

3D Flow Tokenization

We represent each query point’s displacement at each future frame as a continuous-valued token, augmented with initial 3D geometry and spatiotemporal positional encoding.

Normalization and sequence construction. To balance motion scales across domains, we normalize displacements by a domain-specific scalar estimated from training data and fixed throughout training:

fˉt,i=ft,iad,ad>0.(4)\bar{f}_{t,i} = \frac{f_{t,i}}{a_d}, \qquad a_d > 0. \tag*{(4)}

The normalized displacements are arranged in frame-major order, yielding TNTN three-dimensional vectors for token construction.

Query point geometry encoder. We encode each query point’s absolute initial position and its position relative to the query-set centroid using two parallel encoders. Each applies coordinate normalization, fixed Fourier features, and an MLP. Their outputs are concatenated and fused:

ei3D=MLP⁡([Eabs(p0,i);Erel(p0,i−μ)]),(5)\mathbf{e}^{3D}_i = \operatorname{MLP}\left([E_{\mathrm{abs}}(\mathbf{p}_{0,i}); E_{\mathrm{rel}}(\mathbf{p}_{0,i} - \boldsymbol{\mu})]\right), \tag*{(5)}

where EabsE_{\mathrm{abs}} and ErelE_{\mathrm{rel}} denote the two encoders, and μ\boldsymbol{\mu} is the query-set centroid. The geometry embedding is shared across all future tokens of the same query point.

3D Flow Tokenization

Flow token construction. A linear projection maps noisy normalized displacements into the generator’s hidden space, where they are combined with geometry and conditioning embeddings:

ht,i(σ)=Linear⁡(fˉt,i(σ))+ei3D+econd(σ).(6)\mathbf{h}_{t,i}^{(\sigma)} = \operatorname{Linear}\left(\bar{\mathbf{f}}_{t,i}^{(\sigma)}\right) + \mathbf{e}_i^{3\mathrm{D}} + \mathbf{e}_{\mathrm{cond}}(\sigma). \tag*{(6)}

Here, σ\sigma denotes the denoising noise level, distinct from the physical frame index tt. The conditioning embedding combines a learned flow-modality embedding with the noise-level embedding.

Spatiotemporal alignment. We assign flow tokens rotary positional coordinates using the same temporal and spatial conventions as video tokens. Each flow token is anchored to its future frame and the query point’s initial image patch:

rt,i=(tqt,⌊v0,iqsps⌋,⌊u0,iqsps⌋),(7)\mathbf{r}_{t,i} = \left(\frac{t}{q_t}, \left\lfloor\frac{v_{0,i}}{q_s p_s}\right\rfloor, \left\lfloor\frac{u_{0,i}}{q_s p_s}\right\rfloor\right), \tag*{(7)}

where (u0,i,v0,i)=s0,i(u_{0,i}, v_{0,i}) = \mathbf{s}_{0,i} denotes the initial image coordinates, qtq_t and qsq_s are the temporal and spatial VAE compression factors, and psp_s is the latent patch size. These coordinates determine the rotary transformations of attention queries and keys:

q~t,i=R(rt,i)qt,i,k~t,i=R(rt,i)kt,i.(8)\widetilde{\mathbf{q}}_{t,i} = R(\mathbf{r}_{t,i})\mathbf{q}_{t,i}, \qquad\widetilde{\mathbf{k}}_{t,i} = R(\mathbf{r}_{t,i})\mathbf{k}_{t,i}. \tag*{(8)}

The shared positional convention encodes relative spatiotemporal relationships between flow and video tokens within multimodal attention.

Training Objectives

Joint flow matching

We jointly train video and 3D flow generation using a weighted rectified-flow objective:

LFM=λvideoLvideo+λflowLflow,(9)\mathcal{L}_{\mathrm{FM}} = \lambda_{\mathrm{video}}\mathcal{L}_{\mathrm{video}} + \lambda_{\mathrm{flow}}\mathcal{L}_{\mathrm{flow}}, \tag*{(9)}

where Lvideo\mathcal{L}_{\mathrm{video}} and Lflow\mathcal{L}_{\mathrm{flow}} supervise vector-field prediction for video latents and normalized 3D flow, respectively. Both modalities share a per-sample noise level and are jointly denoised. Losses are computed separately for each modality, excluding conditioning positions from supervision.

Auxiliary motion objectives

To complement vector-field supervision with explicit constraints on predicted motion, we apply auxiliary objectives to clean normalized flow reconstructed from the predicted vector field. These objectives penalize frame-to-frame displacement error, cumulative displacement direction error, and relative endpoint error, encouraging both local accuracy and trajectory-level consistency.

Direction errors are weighted by ground-truth displacement magnitude, while points with small net displacement are excluded from endpoint supervision. All auxiliary terms operate in normalized coordinates and use noise-dependent weighting to emphasize lower-noise samples. The complete objective is

L=LFM+λdirLdir+λvmagLvmag+λendLend,(10)\mathcal{L} = \mathcal{L}_{\mathrm{FM}} + \lambda_{\mathrm{dir}}\mathcal{L}_{\mathrm{dir}} + \lambda_{\mathrm{vmag}}\mathcal{L}_{\mathrm{vmag}} + \lambda_{\mathrm{end}}\mathcal{L}_{\mathrm{end}}, \tag*{(10)}

where Ldir\mathcal{L}_{\mathrm{dir}}, Lvmag\mathcal{L}_{\mathrm{vmag}}, and Lend\mathcal{L}_{\mathrm{end}} denote the direction, displacement, and endpoint losses, respectively. Detailed formulations and derivations are provided in the supplementary material, and their contribution is evaluated through ablation experiments.

Experiments

Datasets

We assemble a heterogeneous corpus of paired videos and 3D point trajectories spanning simulated and real-world interactions. The training corpus contains approximately 1.44 million object-level samples, equivalent to 1,224 hours of standardized 49-frame sequences at 16 FPS. The data are organized into four collections:

  • Sim7 combines seven simulation datasets: ManiSkill ((Tao et al., 2024)), MIKASA ((Cherepanov et al., 2026)), MetaWorld ([3]), MetaFold ((Chen et al., 2025a)), RLBench ((James et al., 2020)), RoboCasa ((Nasiriany et al., 2024)), and RoboTwin ((Mu et al., 2025)), covering diverse robotic interactions and manipulation tasks.

  • Real3 comprises AgiBot World ((Bu et al., 2025)), EPIC-KITCHENS ((Damen et al., 2018)), and Something-Something V2 ((Goyal et al., 2017)), covering real-world robot and human interactions.

  • SpatialCog is our self-collected dataset of tabletop and household interactions, with an emphasis on spatial relations in instruction-conditioned motion.

  • MolmoMotion-1M ([7]) contributes curated 3D motion annotations from six sources: EgoDex ((Hoque et al., 2025)), Xperience ((Ropedia, 2026)), YT-VIS ((Yang et al., 2019)), Stereo4D ((Jin et al., 2025)), DROiD ((Khazatsky et al., 2024)), and MolmoSpaces ((Kim et al., 2026)).

Annotation and standardization. For Sim7 and Real3, we generate captions with Gemini-2.5 Pro (Comanici et al., 2025), obtain object masks with SAM 2 (Ravi et al., 2024), and extract 3D point trajectories using Track4World (Lu et al., 2026). SpatialCog pairs manually annotated captions with Track4World trajectories. For MolmoMotion-1M, we reuse the released motion annotations, apply source-specific geometry, scale, and trajectory-validity checks, and standardize the retained sequences into 49-frame clips. All collections use a shared representation consisting of video, a language instruction, initial 3D query points, and frame-to-frame 3D displacements. Additional processing details and per-source statistics are provided in the supplementary material.

Evaluation split. We introduce WorldFlowBench, a benchmark comprising 1,678 held-out evaluation samples: 396 from Sim7, 882 from Real3, and 400 from SpatialCog. The three subsets cover simulated interactions, real-world motion, and spatial-relation reasoning, respectively. All evaluation samples are excluded from training.

Training Configuration and Strategy

Implementation details. We train on 49-frame clips at 288×480288 \times480 resolution with N=64N = 64 query points. The video VAE uses temporal and spatial compression factors of qt=4q_t = 4 and qs=16q_s = 16, followed by 2×22 \times2 latent patchification (ps=2p_s = 2). Each clip therefore contains a 13×9×1513 \times9 \times15 video-token grid and 48×6448 \times64 flow tokens. Training uses 64 NVIDIA H800 GPUs across eight nodes, with a per-GPU batch size of two and no gradient accumulation, yielding a global batch size of 128.

Training strategy. We adopt a two-stage training schedule. The first stage initializes the model from pretrained Cosmos-Nano checkpoints and trains on the SpatialCog training split for 32,000 steps. The learning rate follows linear warmup and cosine decay, with a peak of 10−410^{-4}. The resulting exponential moving average (EMA) weights initialize the second stage.

The second stage trains on the heterogeneous data mixture for 51,100 steps across three curriculum phases: simulation-focused training, mixed real-world adaptation, and refinement on the SpatialCog training split. During real-world adaptation, sampling gradually shifts from MolmoMotion to Real3. Each phase prioritizes its target domain while retaining cross-domain replay, with smooth transitions between sampling distributions. After warmup, the learning rate reaches 10−410^{-4}, transitions to an 8×10−58 \times10^{-5} plateau during real-world adaptation, and decays through late adaptation and final refinement.

We expand task prompts into structured instructions specifying data origin, camera motion, actor type, and environment. For classifier-free guidance, text conditioning is dropped with probability 0.1, while visual and geometric conditioning is retained. At inference, we use 40 rectified-flow sampling steps with a guidance scale of 2.

Flow Benchmark

Evaluation Metrics

We evaluate 3D point trajectories in metric coordinates. Average displacement error (ADE) measures the mean Euclidean position error across query points and future frames, while final displacement error (FDE) measures the mean position error at the final frame. Both are reported in meters. PWT measures the fraction of point–frame pairs whose position error falls within a given distance threshold. We report its average over thresholds of 1, 2, and 5 cm to assess trajectory accuracy at fine spatial scales.

Quantitative Evaluation on WorldFlowBench

We evaluate all learning-based methods under a single-frame protocol, where one RGB observation and a language instruction are provided to forecast a common future horizon without motion history. All predictions are converted into metric 3D point trajectories in the initial camera coordinate system. We additionally report a Static reference that predicts zero flow and therefore keeps every query point at its initial position throughout the prediction horizon.

We compare against two families of learned baselines. The video-generation-then-tracking family includes NovaFlow (Li et al., 2025), as well as pipelines built on Cosmos3-Nano (NVIDIA, 2026) or LingBot-Video-MoE (Robbyant Team et al., 2026), paired with Track4World (T4W) (Lu et al., 2026) or 4RC (Luo et al., 2026) for 3D tracking. The direct flow-generation family includes MolmoMotion-AR-H1 [7], μ0\mu_0 (Lee et al., 2026b), and EgoWAM (Li et al., 2026a).

As shown in Table 1, WorldFlow achieves the lowest ADE and FDE among all evaluated learning-based methods on all three subsets. Relative to the strongest external learned baseline on each subset, it reduces ADE by approximately 46.9% on Sim7, 4.7% on Real3, and 54.2% on SpatialCog. It also achieves the highest PWT among learning-based methods on Sim7 and SpatialCog. On Real3, WorldFlow obtains a PWT of 0.311, compared with the best value of 0.332, while still delivering the lowest ADE and FDE. These results demonstrate consistent improvements in trajectory accuracy across simulated interactions, real-world motion, and spatial-reasoning tasks.

ParadigmModelSim7 ADE↓Sim7 FDE↓Sim7 PWT↑Real3 ADE↓Real3 FDE↓Real3 PWT↑SpatialCog ADE↓SpatialCog FDE↓SpatialCog PWT↑
Non-parametricStatic0.0960.2010.5730.0620.0860.4640.1430.2780.412
Video generation + 3D trackingNovaFlow0.1960.3180.2890.0950.1290.2930.1650.2950.320
Video generation + 3D trackingCosmos3-Nano + T4W0.1010.2020.4510.0880.1330.3100.1410.2570.398
Video generation + 3D trackingCosmos3-Nano + 4RC0.0960.1960.5310.0850.1220.3320.1410.2630.401
Video generation + 3D trackingLingBot-Video-MoE + T4W0.1050.2070.4250.0970.1450.2670.1340.2390.364
Video generation + 3D trackingLingBot-Video-MoE + 4RC0.1040.2070.4920.0920.1340.2950.1310.2320.376
Direct flow generationMolmoMotion-AR-H10.1760.3230.2800.1850.3350.2220.2820.5330.173
Direct flow generationμ0\mu_00.1860.3070.0690.1160.1560.1480.2130.2700.055
Direct flow generationEgoWAM0.1980.3360.0200.1300.1900.0560.2490.3830.080
AblationsWorldFlow (flow-only)0.0990.2060.4950.1010.1520.3200.1360.2480.372
AblationsWorldFlow + T4W0.0880.1740.4240.1000.1540.2630.0830.1380.403
AblationsWorldFlow + 4RC0.0880.1700.4830.0920.1320.2910.0980.1750.411
AblationsWorldFlow w/o auxiliary loss0.0630.1160.5270.0830.1150.2990.0620.0840.436
OursWorldFlow0.0510.0940.5640.0810.1130.3110.0600.0780.449

Table 1. 3D motion forecasting on WorldFlowBench. ADE and FDE measure average and final position errors in meters, respectively; lower is better. PWT is the mean point–frame hit rate over distance thresholds of 1, 2, and 5 cm; higher is better. Bold and underline mark the best and second-best results in each column among non-static methods, including ablations. Static is reported as a reference and excluded from ranking. T4W denotes Track4World. For WorldFlow + T4W/4RC, trajectories are extracted from the videos generated by WorldFlow using the corresponding tracker.

Ablation Studies

Flow-only versus joint generation. To evaluate the benefit of jointly modeling 3D motion and visual evolution, we compare WorldFlow with a flow-only variant using the same backbone, training data, and optimization recipe. The variant predicts only 3D flow, whereas the full model jointly generates video and flow. The full model reduces ADE by 48.5%, 19.8%, and 55.9% on Sim7, Real3, and SpatialCog, respectively, with lower FDE on all three subsets. On SpatialCog, FDE decreases from 0.248 to 0.078 m, a 68.5% reduction. PWT also improves on Sim7 and SpatialCog, with a small decrease on Real3. These results support the value of complementary visual supervision for learning accurate 3D motion.

Direct flow generation versus post-hoc tracking. To evaluate flow as an explicit generative modality, we compare directly generated trajectories with those recovered by Track4World and 4RC from videos produced by the same WorldFlow checkpoint. This comparison holds the video generator and conditioning inputs fixed, contrasting explicit motion generation with motion recovery from generated pixels. Direct flow generation achieves lower ADE and FDE and higher PWT than both tracking pipelines on all three subsets. On SpatialCog, it achieves an ADE of 0.060 m, compared with 0.083 m for Track4World and 0.098 m for 4RC. These results support incorporating explicit 3D flow into the world model’s generative output rather than recovering motion through a separate tracking stage.

Auxiliary motion supervision. We compare the full model with a variant trained without auxiliary motion objectives. These objectives supervise displacement, cumulative motion direction, and trajectory endpoints through the reconstructed clean flow. Adding them improves ADE, FDE, and PWT on all three subsets, with the largest ADE reduction on Sim7, from 0.063 to 0.051 m (19.0%). PWT increases from 0.527 to 0.564 on Sim7, from 0.299 to 0.311 on Real3, and from 0.436 to 0.449 on SpatialCog. These consistent improvements indicate that explicit supervision of clean motion complements the primary flow-matching objective.

Flow-Guided Robotic Manipulation

We evaluate the capability of 3D motion prediction models in robotic scenarios and their utility in guiding downstream execution through both simulation and real-robot experiments. Within modular robotic manipulation pipelines, we compare WorldFlow with μ0\mu_0 (Lee et al., 2026b), which is selected for its competitive offline trajectory prediction performance among reproducible direct motion predictors. Furthermore, RoboLab provides an out-of-distribution (OOD) testbed for WorldFlow to validate its generalization under novel conditions.

Simulation on RoboLab

We evaluate WorldFlow on RoboLab [1], a public manipulation benchmark providing OOD inference scenarios for the motion predictor. We use the pretrained checkpoint without RoboLabspecific post-training or fine-tuning. Specifically, we adopt the Simple task suite of RoboLab and evaluate all 64 tasks within it under Vague, Default, and Specific instructions. Each task-instruction configuration is tested across three trials, yielding 192 trials per instruction variant and 576 trials per motion predictor under each execution setting.

RoboLab uses a simulated Franka arm with a Robotiq 2F-85 two-finger gripper. Our pipeline comprises grasp acquisition, flow-guided transport, and placement. Grasping uses a rule-based policy with geometric checks, refined offline through GPT-6.1 Sol-assisted iteration and simulation feedback (OpenAI, 2026), then fixed for evaluation without online GPT calls. Qwen3.8-27B (Qwen Team, 2026) segments the instructed object; its mask and depth yield a point cloud for sampling 3D query points. Given these points, the image, and the instruction, the motion predictor generates object trajectories, which an optimizer converts into end-effector poses under grasp and kinematic constraints. The robot tracks these poses through inverse kinematics and releases the object at the predicted endpoint. Both predictors are evaluated under two settings using RoboLab’s automated task completion criterion. Within each setting, perception, grasping, trajectory optimization, and control modules are shared; only the motion predictor changes. Orch executes the full pipeline to measure end-to-end manipulation success. Attach replaces physical grasping with simulator-assisted attachment to prevent grasp failures and slippage during transport, allowing flow-guided task completion to be evaluated with reduced grasp-related interference.

Simulation results. Table 2 evaluates OOD manipulation from two perspectives. Within our pipelines, WorldFlow outperforms μ0\mu_0 across all three instruction variants and both settings. Under Orch, overall task success increases from 3.82% (22/576) to 38.54% (222/576), a gain of 34.72 percentage points. Under Attach, success increases from 9.20% (53/576) to 47.22% (272/576), a gain of 38.02 percentage points. These gains demonstrate more effective transfer of predicted object motion to OOD manipulation than μ0\mu_0. For system-level context, WorldFlow-Orch also exceeds the published success rates of the selected VLA and WAM policies across all three instruction variants. Its overall success rate is 3.23 percentage points higher than Cosmos3-Nano-Policy, the strongest of these references. Together, these results support the practical value of WorldFlow’s motion predictions for robotic manipulation in OOD environments.

MethodVagueDefaultSpecificOverall
Cosmos3-Nano-Policy (NVIDIA, 2026)23.28%40.63%42.03%35.31%
π0.5\pi_{0.5} (Black et al., 2025)16.25%29.69%29.84%25.26%
DreamZero [2]15.00%26.09%25.63%22.24%
GR00T N1.6 (NVIDIA et al., 2025)7.19%8.75%7.50%7.81%
μ0\mu_0-Orch3.13%3.65%4.69%3.82%
WorldFlow-Orch28.13%44.79%42.71%38.54%
μ0\mu_0-Attach8.33%8.33%10.94%9.20%
WorldFlow-Attach32.29%52.60%56.77%47.22%

Table 2. Task success on RoboLab Simple. Published policy results from the RoboLab leaderboard are reported under their original configurations for reference. Motion predictors are compared under two settings: Orch (full manipulation) and Attach (simulator-assisted object retention). Bold and underline mark the highest and second-highest values per column across the displayed settings.

Real-Robot Manipulation

Setup and execution. Our physical setup uses a 7-DoF xArm, an XHand dexterous hand, and a ZED Mini RGB-D camera. Given an RGB-D observation and a language instruction, Qwen3.8-27B (Qwen Team, 2026) identifies and segments the target object. We use the resulting mask and depth image to extract a 3D point cloud and sample query points for trajectory prediction. T(R,O)T(R,O) Grasp (Fei et al., 2025) generates the initial dexterous grasp pose. Following Dream2Flow (Dharmarajan et al., 2025), we fit a locally rigid patch near the grasp to the predicted object trajectories to obtain the corresponding arm trajectory. We evaluate 12 tasks covering pick and place, stacking, and relative placement, with three randomized trials per task and 36 executions per method. We evaluate flow prediction and robot execution separately. Flow prediction success is manually assessed based on whether the predicted object motion is plausible for the given instruction. Robot execution succeeds if the full pipeline completes the task on the physical robot. In each paired trial, we vary only the flow predictor and keep the initial layout, instruction, grasp result, and control settings identical.

Real-robot results. As shown in Table 3, WorldFlow achieves 97.2% flow prediction success (35/36) and 66.7% end-to-end execution success (24/36), compared with 25.0% (9/36) and 11.1% (4/36) for μ0\mu_0, respectively. Execution failures despite plausible motion predictions arise from grasping, trajectory optimization, and unstable placement during stacking. With the grasping and control modules held fixed, these results show that WorldFlow provides more reliable object-motion references, translating better flow predictions into higher real-world task completion.

GroupTask CaptionFlow PredictionRobot Execution
Pick and PlacePlace the watermelon doll into the small pink plate.3/30/33/30/3
Place the oolong tea into the large blue tray.3/31/32/30/3
Place the potato chips into the dusty pink basket.3/31/32/30/3
Place the yellow toy inside the red circle.3/31/32/31/3
StackingPlace the green biscuit box on the translucent box.3/31/33/30/3
Place the can of Red Bull on the translucent box.3/31/32/31/3
Place the beef curry on the paper towel.3/31/31/30/3
Place the yellow toy on the paper towel.3/32/30/30/3
Relative PlacementPlace the sunscreen in front of the tissue.2/30/31/30/3
Place the potato chips behind the Oreo box.3/30/32/31/3
Place the watermelon doll to the left of the cola.3/31/33/31/3
Place the oolong tea to the right of the small blue toy car.3/30/33/30/3
Overall success rate97.2%25.0%66.7%11.1%

Table 3. Real-world manipulation with different flow predictors. Each task is evaluated over three trials, yielding 36 trials per method. Flow prediction and robot execution are evaluated separately for each trial. Both methods use matched initial layouts, identical grasp results, and the same trajectory optimization and execution settings; only the flow predictor changes. Bold indicates the higher success rate.

Conclusion

We presented WorldFlow, a world model that jointly generates future video and metric 3D flow from a single observation, a language instruction, and 3D query points. Continuous flow tokens are grounded in video coordinates and initial point geometry, enabling both modalities to be jointly denoised by a shared rectified-flow generator. Experiments across diverse embodied domains demonstrate improved motion forecasting over direct flow predictors and video-generation-then-tracking pipelines.

The current formulation requires predefined query points and inherits noise from automatically generated trajectory supervision. Future work will explore automatic query selection, multimodal futures, and joint generation of 3D flow and actions to study the benefits of explicit motion representations for policy learning.

AI Use Statement

We used generative AI tools to assist with translation, provide feedback on experimental methodology and interpret experimental results. Gemini 2.5 Pro was used to generate captions for Sim7 and Real3, as described in the dataset section. We additionally used AI tools to assist with manuscript drafting, language editing, organization, and LaTeX formatting. The authors reviewed the AI-assisted content and checked technical descriptions, numerical comparisons, and conclusions against the methodology and experimental records. The authors take full responsibility for the final content of this work, including all AI-assisted text, claims, and artifacts.

Data Processing Details and Dataset Statistics

Data Processing Details

The main paper provides an overview of the data pipeline; here, we describe the source-specific processing details. For Sim7, Real3, and SpatialCog, we extract trajectories directly from the original videos using a standardized configuration of 49 RGB frames and 64 query points per object record. Tracked pixels are back-projected using depth and camera intrinsics, and camera poses transform the resulting 3D points into the first-frame camera coordinate system. The object mask in the first frame defines the visible partial surface from which the query points are selected. RGB frames and point trajectories are generated at the same temporal indices and therefore require no subsequent temporal realignment.

For MolmoMotion-1M, we reuse the released trajectories and associated geometry rather than extracting trajectories from the source videos. We transform the trajectories into the same first-frame camera coordinate system and standardize their point counts and temporal lengths. Records with fewer than eight valid object-point candidates are discarded. When more than 64 candidates are available, we use deterministic voxel-uniform sampling to maintain spatial coverage; for records with 8–63 candidates, we sample 64 points with replacement using a fixed random seed. RGB frames and point trajectories are aligned using identical temporal indices. Intervals longer than 49 frames are uniformly subsampled, whereas intervals containing 8–48 frames are extended to 49 frames by repeating the final RGB frame and point positions. The resulting padded transitions have zero ground-truth displacement. The repeated RGB frames and their zero-displacement targets are included in training supervision.

Quality control. Starting from 2.30 million raw candidate samples, we obtain approximately 1.44 million object-level training samples after format standardization, quality control, manifest validation, and exclusion of held-out evaluation samples. Our quality-control procedure assesses trajectories along four dimensions: geometric validity, temporal continuity, observation completeness, and motion plausibility. Geometric checks reject invalid camera parameters, non-finite 3D coordinates, and inconsistent changes between depth and 3D position. Temporal checks detect abrupt changes in point position, depth, or object extent. Observation checks limit the proportions of out-of-frame or unobserved trajectory points. Motion checks identify implausible displacements, velocities, and changes in spatial scale.

The criteria depend on the reliability of each source’s geometry and scale. For motion magnitude, SpatialCog and Real3 retain peak displacements in [0.05, 3] m, while metric MolmoMotion-1M subsets impose an upper limit of 3 m. To detect temporal discontinuities, Real3 limits the 99.5th percentile of frame-rate-normalized step displacement to 0.5 m; metric MolmoMotion-1M subsets additionally cap individual steps at 1 m and speed at 15 m/s. DROID also rejects concurrent discontinuities in depth and 3D position. For spatial consistency, Real3 and metric MolmoMotion-1M subsets limit the increase in RMS spatial radius to 0.25 m. For observation completeness, these sources cap the out-of-frame fraction at 5%; metric MolmoMotion-1M subsets also cap the unobserved fraction at 5%.

For YT-VIS, which lacks a reliable metric scale, we instead use relative step-to-range, adjacent-depth, and spatial-radius statistics without absolute displacement or speed thresholds. Simulation records are screened for complete geometry and valid task annotations. All motion and continuity checks are evaluated before temporal padding.

Figure I. Initial data volume and retention across sources before the training/evaluation split. (a) Number of candidate object records entering the reported processing stage, shown on a logarithmic scale. (b) Fraction retained after formatting, quality control, and manifest validation, before benchmark exclusion. For SpatialCog, 82,831 candidate records yield 82,588 valid records, corresponding to a retention rate of 99.7%. Initial counts describe source-specific candidate pools rather than the full original datasets. Colors indicate the four collections defined in the main paper. All counts refer to object records rather than unique videos.

Figure 1 summarizes the source-specific candidate counts and retention after formatting, quality control, and manifest validation, before the training/evaluation split. For SpatialCog, 82,831 post-prefilter candidate records yield 82,588 valid manifest records, corresponding to a retention rate of 99.7%. This pool includes all 400 SpatialCog evaluation samples in WorldFlowBench. These samples are excluded at training time, leaving 82,188 SpatialCog records for training.

3D Flow Predictions and motion modeling paradigms

Figure 1. 3D Flow Predictions and motion modeling paradigms. Left: 3D motion flow predicted by WorldFlow across simulated, robotic manipulation, human-object interaction, and in-the-wild environments. Right: (a) Tracking models recover dense scene flow from video. (b) Motion prediction models forecast future 3D flow from an image and text. Existing formulations can be broadly divided into sparse scene flow prediction and query-conditioned flow prediction. (c) WorldFlow is a world model conditioned on an image, text, and query points that jointly generates future video and query-conditioned 3D flow.

Dataset Statistics

Table 1 summarizes the processed data inventory. Real3 and SpatialCog exclude WorldFlowBench records; Sim7 uses its QC inventory before runtime exclusion. The reported 1,438,379 standardized object clips comprise 70,480,571 frame instances, or approximately 1,223.6 hours at 16 FPS. With 64 query points and 48 displacement steps per clip, they contain 92,056,256 point trajectories and 4,418,700,288 3D displacement vectors. These totals precede Sim7 runtime exclusion and may include padded frames, repeated query points, and multiple object records sharing the same source video.

The MolmoMotion-1M sources exhibit substantial variation in their native temporal coverage. Figure 2 reports their interval lengths before conversion to the common 49-frame representation and the fraction of records requiring tail padding.

To characterize the scale of the motion targets, we define the component RMS of record rr as

mr=(148⋅64⋅3∑t=148∑i=164∥fr,i,t∥22)1/2.(11)m_r = \left(\frac{1}{48 \cdot64 \cdot3}\sum_{t=1}^{48}\sum_{i=1}^{64}\lVert\mathbf{f}_{r,i,t} \rVert_2^2\right)^{1/2}. \tag*{(11)}

Figure 3 reports the distribution of mrm_r within each source together with the source-level aggregate RMS, Er[mr2]\sqrt{\mathbb{E}_r[m_r^2]}. These statistics are computed before flow normalization and include zero-motion transitions introduced by temporal padding. They characterize the numerical scale of the motion targets rather than physical speed or endpoint displacement.

Figure 3. Qualitative comparison on diverse real and simulated scenes. We visualize predicted 3D query-point trajectories from μ0\mu_0, MolmoMotion, and WorldFlow across SpatialCog, AgiBot, DROID, Stereo4D, Meta-World, RLBench, RoboTwin, and MetaFold. The trajectories are projected onto the corresponding input frames for visualization.

SourceCollectionDescriptionObject clipsStandardized duration (h)
Sim7Sim7Simulated robot manipulation drawn from seven simulation datasets.190,267161.9
AgiBot WorldReal3Real-world robot manipulation across diverse environments and tasks.631,873537.5
STH-V2Real3Human–object interactions covering a broad range of object motions.86,81173.8
EPIC-KITCHENSReal3Egocentric human activities and object interactions in kitchens.5,3374.5
SpatialCogSpatialCogTabletop and household interactions emphasizing spatial relations.82,18869.9
MolmoSpacesMolmoMotion-1MSimulated interactive scenes with released 3D motion annotations.164,970140.3
DROIDMolmoMotion-1MReal-world robot manipulation collected across multiple environments and tasks.19,73916.8
EgoDexMolmoMotion-1MEgocentric dexterous human–object interactions.126,850107.9
Stereo4DMolmoMotion-1MDynamic real-world scenes reconstructed from stereo observations.15,87913.5
XperienceMolmoMotion-1MEgocentric videos of human activities and object interactions.112,99296.1
YT-VISMolmoMotion-1MDiverse Internet videos with non-metric object trajectories.1,4731.3
Total——1,438,3791,223.6

Table I (PDF p. 19). Composition of the processed data inventory. Real3 and SpatialCog counts exclude WorldFlowBench evaluation records; Sim7 uses its QC inventory before runtime benchmark exclusion. Each object clip contains 49 frames standardized to 16 FPS. Standardized duration in hours is computed as Nclips×49/(16×3600)N_{\mathrm{clips}} \times49/(16 \times3600), where NclipsN_{\mathrm{clips}} denotes the reported number of object clips. This measures standardized data volume rather than deduplicated source-video duration, since different object records may share the same underlying video interval.

Training Details

Data Sampling and Curriculum

The second stage is initialized from the EMA weights of the SpatialCog warm-start checkpoint, with optimizer and scheduler states reset. We detail the source-sampling strategy, curriculum transitions, structured text conditioning, and learning-rate schedule used during its 51,100 optimization steps.

Hierarchical source sampling. We organize the training sources into five sampling domains. Simulation combines Sim7 and MolmoSpaces; Molmo-real comprises DROID, EgoDex, Stereo4D, and Xperience; Outdoor contains YT-VIS; and Real3 and SpatialCog retain their collection-level definitions. These groups determine curriculum sampling and differ from the four data collections described in the main paper.

At optimization step kk, the probability of selecting source ss in domain dd is

pk(s)=pk(d)Nsαd∑s′∈dNs′αd,αd={0.8,d=Real3,0.5,otherwise.(12)p_k(s) = p_k(d) \frac{N_s^{\alpha_d}}{\sum_{s' \in d} N_{s'}^{\alpha_d}}, \qquad\alpha_d = \begin{cases} 0.8, & d = \mathrm{Real3},\\ 0.5, & \text{otherwise}. \end{cases} \tag*{(12)}

Figure II. Native temporal coverage of the six retained MolmoMotion-1M subsets before conversion to 49 frames. (a) Native interval-length distributions; thin lines indicate P5–P95, thick bars indicate the interquartile range, and circles indicate the median. The dashed line marks the target length of 49 frames. (b) Percentage of records requiring tail padding. Intervals longer than 49 frames are uniformly subsampled.

Figure III. Pre-normalization motion-scale statistics. Left: median, 90th percentile, and 99th percentile of the per-record component RMS. Right: source-level aggregate component RMS. Both horizontal axes use logarithmic scales. SpatialCog statistics use 82,188 records after excluding all 400 evaluation samples; Real3 statistics also exclude benchmark records. Sim7, marked with an asterisk, uses its QC inventory before runtime benchmark exclusion and provides aggregate moments but no per-record percentile summary. Statistics include zero-motion transitions introduced by temporal padding. YT-VIS values follow the non-metric scale of its annotations.

Here, pk(d)p_k(d) is the curriculum-dependent domain probability, and NsN_s is the source count used to compute the sampler’s within-domain probabilities. The sublinear exponent reduces the dominance of larger sources within each domain. Sim7 is treated as a single source and retains the natural proportions of its constituent datasets.

Curriculum intervals. Table II specifies five sampling intervals that implement the three phases described in the main paper: simulation-focused training, mixed real-world adaptation, and SpatialCog refinement.

IntervalStepsSimulationMolmo-realOutdoorReal3SpatialCog
Simulation-focused[0,12,000)[0, 12{,}000)80%5%0%10%5%
Molmo-real adaptation[12,000,24,000)[12{,}000, 24{,}000)10%70%0%15%5%
Outdoor adaptation[24,000,24,100)[24{,}000, 24{,}100)5%20%50%25%0%
Real3 adaptation[24,100,48,100)[24{,}100, 48{,}100)10%10%0%75%5%
SpatialCog refinement[48,100,51,100)[48{,}100, 51{,}100)5%10%0%35%50%

Table II. Target domain-sampling probabilities for the second-stage curriculum. Adjacent distributions are linearly interpolated near interval boundaries using the windows specified in the text. Within each domain, sources are sampled according to Eq. equation 12.

Real-world adaptation first emphasizes Molmo-real, briefly introduces Outdoor data, and then prioritizes Real3. Each interval retains cross-domain replay.

We linearly interpolate adjacent domain-probability vectors over windows centered at steps 12,000, 24,000, 24,100, and 48,100, with widths of 1,000, 50, 50, and 1,000 steps, respectively. Outside these transition windows, domain probabilities remain fixed at the values in Table II. With a global batch size of 128, the nominal second-stage budget is 6,540,800 sample draws, including repeated visits to the same clips.

Structured text conditioning. Each prompt consists of the original task instruction followed by a context block with four categorical fields. Data origin distinguishes simulated from real-world observations. Actor type is specified as robot, human, mixed, or none. Camera context distinguishes egocentric, fixed external, and moving external views. Environment is specified as indoor, outdoor, or mixed. These attributes provide explicit context for interpreting the requested motion across heterogeneous sources.

We use sample-level context metadata when available and predefined source-level defaults otherwise. The context fields follow a fixed order: origin, actor, view, and scene. Training and inference use the same prompt format, which excludes dataset identifiers and flow-normalization values.

Learning-rate schedule. Figure IV shows the second-stage learning-rate schedule. The learning rate increases linearly to 10−410^{-4} over the first 500 steps and remains at this level during the main simulation-focused interval. Near the transition to real-world adaptation, it decreases to 8×10−58 \times10^{-5} and remains at this plateau until step 42,100, including the brief Outdoor interval. It then decays to 2×10−52 \times10^{-5} at step 48,100 and reaches 5×10−65 \times10^{-6} at the end of SpatialCog refinement.

Figure IV. Learning-rate schedule for second-stage training. Background bands indicate the curriculum intervals. The narrow gold band marks the 100-step Outdoor interval, during which the learning rate remains at 8×10−58 \times10^{-5}.

Auxiliary Motion Objectives

The auxiliary objectives supervise frame-to-frame displacement, cumulative displacement direction, and final displacement, all in normalized coordinates.

Clean-Flow Reconstruction

We reconstruct clean displacements from the predicted rectified-flow vector field for auxiliary supervision. Let x0\mathbf{x}_0 denote the noise-free displacement sequence and σ∈[0,1]\sigma\in[0,1] the noise level. Under the linear interpolation xσ=(1−σ)x0+σϵ\mathbf{x}_{\sigma} = (1-\sigma)\mathbf{x}_0 + \sigma\boldsymbol{\epsilon}, the target vector field is u=ϵ−x0\mathbf{u} = \boldsymbol{\epsilon} - \mathbf{x}_0. The clean-flow estimate is

x^0=xσ−σvθ.(13)\hat{\mathbf{x}}_0 = \mathbf{x}_{\sigma} - \sigma\mathbf{v}_{\theta}. \tag*{(13)}

The network retains its vector-field parameterization; auxiliary losses act on the reconstructed sequence. Here, x0\mathbf{x}_0 denotes the noise-free sequence, not the first video frame.

Substituting xσ=x0+σu\mathbf{x}_{\sigma} = \mathbf{x}_0 + \sigma\mathbf{u} into Eq. equation 13 gives x^0−x0=−σ(vθ−u)\hat{\mathbf{x}}_0 - \mathbf{x}_0 = -\sigma(\mathbf{v}_{\theta} - \mathbf{u}), and hence

∥x^0−x0∥F2=σ2∥vθ−u∥F2.(14)\lVert\hat{\mathbf{x}}_0 - \mathbf{x}_0\rVert_F^2 = \sigma^2\lVert\mathbf{v}_{\theta} - \mathbf{u}\rVert_F^2. \tag*{(14)}

Thus, squared displacement supervision is exactly a noise-dependent reweighting of the vector-field squared error.

Motion Supervision

Let fti\mathbf{f}_{ti} and f^ti\hat{\mathbf{f}}_{ti} denote the ground-truth and reconstructed normalized displacements of point ii at step tt. Their cumulative displacements from the initial position are dti=∑τ=1tfτi\mathbf{d}_{ti} = \sum_{\tau=1}^{t}\mathbf{f}_{\tau i} and d^ti=∑τ=1tf^τi\hat{\mathbf{d}}_{ti} = \sum_{\tau=1}^{t}\hat{\mathbf{f}}_{\tau i}. Each sequence contains T=48T = 48 displacement steps and N=64N = 64 points. We omit the batch index below for readability.

Adjacent-displacement loss. The displacement loss penalizes the squared difference between predicted and ground-truth motion at each step:

ℓvmag(t,i)=∥f^ti−fti∥22.(15)\ell_{\mathrm{vmag}}(t,i) = \lVert\hat{\mathbf{f}}_{ti} - \mathbf{f}_{ti}\rVert_2^2. \tag*{(15)}

This loss supervises both magnitude and direction of local 3D displacements, with its relation to vector-field supervision given by Eq. equation 14.

Cumulative-direction loss. Small local errors can accumulate into an incorrect overall direction. We therefore compare the directions of the cumulative displacements:

ℓdir(t,i)=1−cos⁡ϵ(d^ti,dti).(16)\ell_{\mathrm{dir}}(t,i) = 1 - \cos_{\epsilon}(\hat{\mathbf{d}}_{ti},\mathbf{d}_{ti}). \tag*{(16)}

Each error is weighted by the ground-truth magnitude ∥dti∥2\lVert\mathbf{d}_{ti}\rVert_2, reducing the contribution of points with little net displacement. For numerical stability, cos⁡ϵ\cos_{\epsilon} computes the dot product divided by the product of the two vector norms, with each norm clamped below at ϵ=10−6\epsilon= 10^{-6}.

Relative endpoint loss. To constrain the final predicted position, we measure endpoint error relative to the ground-truth net displacement:

ℓend(i)=∥d^Ti−dTi∥2max⁡(∥dTi∥2,δ),δ=0.1.(17)\ell_{\mathrm{end}}(i) = \frac{\lVert\hat{\mathbf{d}}_{Ti} - \mathbf{d}_{Ti}\rVert_2}{\max(\lVert\mathbf{d}_{Ti}\rVert_2,\delta)}, \qquad\delta= 0.1. \tag*{(17)}

Only points satisfying ∥dTi∥2≥δ\lVert\mathbf{d}_{Ti}\rVert_2 \ge\delta contribute. This excludes small-net-displacement trajectories, including stationary motion and motion that returns close to its starting position, for which relative endpoint error is unstable. The threshold is defined in normalized coordinates.

Weighting and reduction. For sample bb, all three objectives use the noise weight wb=1−σbw_b = 1 - \sigma_b. The displacement loss averages ℓvmag\ell_{\mathrm{vmag}} over samples, steps, and points with weights wbw_b. The direction loss averages ℓdir\ell_{\mathrm{dir}} over the same indices with weights wb∥dbti∥2w_b\lVert\mathbf{d}_{bt\mathit{i}}\rVert_2. The endpoint loss averages ℓend\ell_{\mathrm{end}} over eligible sample–point pairs with weights wbw_b. Each weighted sum is divided by its total weight plus 10−610^{-6}; if no endpoint is eligible, its loss is zero.

This weighting gives high-noise samples relatively less influence within each batch. For the squared displacement term, the coefficient on vector-field squared error before batch normalization is σ2(1−σ)\sigma^2(1-\sigma), by Eq. equation 14. The direction and endpoint terms impose additional trajectory-level constraints; they do not share the squared-error equivalence. Their effects are evaluated through the ablation experiments in the main paper.

Training configuration. For second-stage training, the reduced losses use λdir=0.1\lambda_{\mathrm{dir}} = 0.1, λvmag=0.1\lambda_{\mathrm{vmag}} = 0.1, and λend=0.05\lambda_{\mathrm{end}} = 0.05.

Video-Motion Consistency

We evaluate whether jointly generated videos and 3D trajectories describe consistent motion by comparing video-tracked query points with the corresponding projected 3D predictions. We report pixel-space discrepancies, sensitivity to tracking confidence, and qualitative correspondence across future frames.

Setup. We evaluate video-motion consistency on all 1,678 WorldFlowBench test clips (396 Sim7, 882 Real3, and 400 SpatialCog). For each input, WorldFlow jointly generates a video and 64 query-point trajectories in a single sampling call, covering 48 future frames at 480×288480 \times288 resolution. We use Track4World (Lu et al. (2026)) to track the initial query pixels in the generated video without access to the predicted flow, and Depth Anything 3 (Lin et al. (2025)) to estimate depth and camera poses. We express the poses relative to the first frame and calibrate their translation scale using the median ratio of input query depths to estimated first-frame depths. The predicted 3D trajectories are then projected using these poses and first-frame intrinsics. No ground-truth future geometry is used. All pixel distances are measured at 480×288480 \times288 resolution.

Video-motion consistency metric. For clip ss and frame tt, we retain finite, in-frame tracks whose visibility-confidence product exceeds τ\tau, forming the query set Vs,t\mathcal{V}_{s,t}. We measure the mean distance between tracked positions vs,i,t\mathbf{v}_{s,i,t} and corresponding projected predictions us,i,t\mathbf{u}_{s,i,t}:

ds(t)=1∣Vs,t∣∑i∈Vs,t∥us,i,t−vs,i,t∥2.(18)d_s(t) = \frac{1}{\lvert\mathcal{V}_{s,t} \rvert} \sum_{i \in\mathcal{V}_{s,t}} \lVert\mathbf{u}_{s,i,t} - \mathbf{v}_{s,i,t} \rVert_2. \tag*{(18)}

Filtering depends only on tracking: out-of-frame flow projections remain included, and nonprojectable predictions receive an image-diagonal penalty. We average clip-level distances equally over clips with eligible observations. At each frame, the retained proportion is computed against all query points across all clips.

Results and confidence sensitivity. WorldFlow exhibits smaller video-motion discrepancies on tracks assigned higher confidence by the tracker. As shown in Figure V, increasing the tracking threshold reduces the measured discrepancy in all three domains. Across WorldFlowBench, raising τ\tau from 0.5 to 0.9 decreases the mean pointwise distance over 48 future frames from 12.30 to 7.31 pixels. This mean averages eligible query-frame distances within each clip, then equally across supported clips. These results indicate closer correspondence between generated video and 3D motion on higher-confidence tracked observations. The figure also reports the proportion of observations retained at each threshold. Generated outputs and camera projections remain fixed; only the retained query subset changes. The measured discrepancies include residual tracking and camera-estimation errors.

Figure V. Video-motion consistency under tracking-confidence filtering. Top: mean pointwise pixel distance at 480 × 288 resolution. Bottom: the proportion of query points retained at each frame. Columns correspond to Sim7, Real3, and SpatialCog. Thresholds apply to the tracker visibility-confidence product. Generated videos, 3D predictions, and camera projections remain fixed across thresholds.

Discrepancies generally increase with the prediction horizon. At τ=0.9\tau= 0.9, frame-48 distances are 14.85, 27.39, and 14.27 pixels on Sim7, Real3, and SpatialCog, respectively, at 480 × 288 resolution. Figure VI complements these measurements by overlaying video-tracked points and corresponding projected 3D predictions at the same frame, using the same >0.9> 0.9 threshold and displaying no trajectory tails. The examples show RoboTwin manipulation (Sim7), a moving phone (Real3), and tabletop object transport (SpatialCog), illustrating the spatial correspondence and residual discrepancies between the two generated modalities over time.

Figure VI. Video-motion consistency across future frames. Rows show Sim7, Real3, and SpatialCog examples at frames 8, 16, 24, 32, 40, and 48. Green circles and magenta crosses mark tracked video points and corresponding projected 3D predictions, respectively. Up to eight fixed queries with tracking score > 0.9 are shown; dd and nn report mean pixel distance and count over all eligible queries. Sim7 is cropped for display; distances use 480×288480 \times288 coordinates.

References

  1. [1]Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, and Jonathan Tremblay. RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies. In Proceedings of Robotics: Science and Systems, Sydney, Australia, July 2026. URL https://arxiv.org/abs/2604.09860.
  2. [2]Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, et al. World Action Models are Zero-shot Policies. arXiv preprint arXiv:2602.15922, 2026. URL https://arxiv.org/abs/2602.15922.
  3. [3]Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094–1100. PMLR, 2020.arxiv.org/abs/1910.10897
  4. [4]Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. arXiv preprint arXiv:2401.11439, 2024.arxiv.org/abs/2401.11439
  5. [5]Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv preprint arXiv:2603.16666, 2026. URL https://arxiv.org/abs/2603.16666.
  6. [6]Bowei Zhang, Lei Ke, Adam W Harley, and Katerina Fragkiadaki. Tapip3d: Tracking any point in persistent 3d geometry. arXiv preprint arXiv:2504.14717, 2025.arxiv.org/abs/2504.14717
  7. [7]Jianing Zhang, Chenhao Zheng, Yajun Yang, Max Argus, Rustin Soraki, Winson Han, Taira Anderson, Chun-Liang Li, Shuo Liu, Jiafei Duan, et al. Molmotion: Forecasting point trajectories in 3d with language instruction. arXiv preprint arXiv:2606.18558, 2026.arxiv.org/abs/2606.18558
  8. [8]Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, and Yinghao Xu. Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization. arXiv preprint arXiv:2608.26103, 2026. URL https://arxiv.org/abs/2608.26103.

Paper details

vlnc:W_junufsng
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Gemini 2.5 Pro (bare)

View statement
Looking for related papers…
Contents