betaFeedback

WorksComputer Science

Joint World Action Models through Differentiable Trajectory Optimization

Abstract

Joint world action models (WAMs) generate future actions and states together, modeling world-action trajectories rather than actions alone. However, trajectory imitation does not ensure that these trajectories are task-effective or consistent with dynamics. We present world action trajectory optimization (WATO), which optimizes generated actions and future states jointly for control. WATO uses both as references in a dynamics-constrained trajectory optimization problem and differentiates the task loss through the optimized trajectory to train the generator. This makes generated trajectories task-aware and dynamics-consistent while using trajectory optimization only during training. At deployment, WATO directly generates candidate trajectories and ranks them with a world model, without iterative trajectory optimization. Across 77 visual control environments, WATO improves average success from 79.9%79.9\% for the imitation joint WAM to 91.7%91.7\%, outperforming strong model-based, goal-conditioned RL, and generative control baselines. Compared with cross-entropy method (CEM), WATO is on average 11.5×11.5\times faster and uses 600.5×600.5\times fewer world model evaluations at deployment.

Figure 1. WATO overview. Joint world-action trajectories are optimized during training and amortized into the generator, enabling deployment with generation and world model evaluation only.

Introduction

Generative policies learn distributions over actions or action trajectories from data, reducing online planning [15, 6, 29]. Recent world action models (WAMs) connect action generation with future prediction [46, 40]. We focus on joint WAMs that generate future actions and states or observations together. We treat them as a model class over world-action trajectories, independent of architecture or scale. Unlike generative policies, joint WAMs specify both how the agent acts and where the trajectory is intended to go. This provides richer information for control, but does not make the trajectory task-effective.

A joint WAM trained by trajectory imitation mainly reproduces the offline data distribution. Its generated actions need not reach the goal, and its generated states need not match the consequences of those actions under the control dynamics. A trajectory can therefore be data-consistent yet task-ineffective or dynamics-inconsistent. The problem is not only to improve the actions, but to optimize the world-action trajectory as a whole. This leads to our central question:

How should joint world-action trajectories be optimized for task performance while remaining consistent with action-conditioned dynamics? Several existing approaches provide natural alternatives, but each uses only part of the world-action trajectory. Model-based planners optimize actions through world model rollouts [7, 18], but omit generated future states and require online optimization. Task losses can be backpropagated through learned dynamics [11], but supervise only the actions. Consistency objectives can align generated states with world model predictions [13], but use them as prediction targets rather than control references. Optimized trajectories can also be distilled into a policy [31], but detached targets discard how the solution depends on the generated trajectory. None of these approaches uses both generated actions and future states to parameterize trajectory optimization and learn through its optimized solution.

We introduce world action trajectory optimization (WATO). WATO uses generated actions and future states as references in a dynamics-constrained trajectory optimization problem. The WAM proposes an intended trajectory, while a world model predicts the consequences of its actions [44, 34, 24, 17]. The task objective drives the trajectory toward the goal. The action reference specifies how to move, and the state reference specifies where to go. Trajectory optimization combines both references with the world model to produce a task-directed, dynamics-consistent trajectory.

The generated world-action trajectory does not only initialize the optimizer, but it also parameterizes the optimization objective. Changing either the actions or future states can therefore change the optimized trajectory. WATO differentiates the task loss through this solution using implicit differentiation [4, 36], providing task gradients to both generated actions and states. Flow matching provides offline trajectory imitation [23], while trajectory optimization makes the joint WAM task-aware.

Trajectory optimization is used only during training. At deployment, WATO directly generates candidate world-action trajectories and evaluates their action sequences with the world model, i.e. action-conditioned dynamics. Deployment performs a lightweight best-of-KK selection over these proposals, but no iterative trajectory optimization is required.

We evaluate WATO on 7 visual control environments spanning navigation, reaching, contact-rich manipulation, and multi-stage goal-conditioned control. WATO improves average success from 79.9% for the imitation-trained joint WAM to 91.7%, outperforming strong model-based, goal-conditioned RL, and generative control baselines. Compared with CEM, WATO is 11.5× faster and uses 600.5× fewer world model evaluations. We also show that differentiating through trajectory optimization outperforms direct rollout, consistency regularization, and distillation. Removing either the action or state reference reduces performance, showing that both are crucial for control.

Our contributions are threefold:

  • We study joint WAMs as generative models over world-action trajectories and formulate task-aware world-action generation as dynamics-constrained trajectory optimization.

  • We introduce WATO, which uses generated actions and future states as optimization references and differentiates the task loss through the optimized trajectory to train the generator.

  • We show across diverse control tasks that WATO improves joint world-action generation, outperforms strong model-based and learning-based baselines, and amortizes trajectory optimization into the generator for efficient deployment.

Preliminaries

Visual offline goal-conditioned control. We consider a reward-free offline dataset of NN visual trajectories, D={τ(n)}n=1N\mathcal{D} = \{\tau^{(n)}\}_{n=1}^{N}, where τ(n)=(o0(n),a0(n),…,oT(n))\tau^{(n)} = (o_0^{(n)}, a_0^{(n)}, \ldots, o_T^{(n)}), with visual observations ot∈Oo_t \in\mathcal{O} and primitive actions at∈Aa_t \in\mathcal{A}. We group BB consecutive actions into an action block Ah=(at+hB,…,at+(h+1)B−1)∈UA_h = (a_{t+hB}, \ldots, a_{t+(h+1)B-1}) \in\mathcal{U}, where U:=AB\mathcal{U} := \mathcal{A}^{B}. An action trajectory A=(A0,…,AH−1)∈UHA = (A_0, \ldots, A_{H-1}) \in\mathcal{U}^{H} spans L=BHL = BH environment steps.

We use pretrained latent world models following LeWM [24]. The encoder maps an observation to a latent state zt=Eθ(ot)z_t = E_{\theta}(o_t), and the action-conditioned predictor rolls out future latent states from an action trajectory AA given ztz_t, and AA, we write

Z^(A)=Rθ(zt,A)=(z^1,…,z^H).(1)\hat{Z}(A) = \mathcal{R}_{\theta}(z_t, A) = (\hat{z}_1, \ldots, \hat{z}_H). \tag*{(1)}

A goal observation ogo_g is encoded as g=Eθ(og)g = E_\theta(o_g). For a latent trajectory Z=(z1,…,zH)Z = (z_1, \ldots, z_H), we use the terminal goal cost J(Z,g)=∥zH−g∥22J(Z,g) = \lVert z_H - g \rVert_2^2. Control proceeds in receding horizon by repeatedly generating an action trajectory, executing a segment, and planning from the new observation.

Generative policies and joint WAMs. Let c=(zt,g)c = (z_t,g) denote the context that includes the current-state and goal condition. An action-only generative policy models A~∼qϕA(A∣c)\tilde{A} \sim q_\phi^A(A \mid c). We focus on joint WAMs, which instead model future actions and states together:

(A~,Z~)∼qϕ(A,Z∣c),Z~=(z~1,…,z~H).(2)(\tilde{A},\tilde{Z}) \sim q_\phi(A,Z \mid c), \qquad\tilde{Z} = (\tilde{z}_1,\ldots,\tilde{z}_H). \tag*{(2)}

A joint WAM therefore induces an action-only distribution through the marginal qϕ,A(A∣c)=∫qϕ(A,Z∣c) dZq_{\phi,A}(A \mid c) = \int q_\phi(A,Z \mid c)\,dZ. The distinction is the generated object: an action-only policy generates how to act, while a joint WAM additionally generates an intended future trajectory.

The generated future Z~\tilde{Z} is distinct from the world model prediction Z^(A~)=Rθ(zt,A~)\hat{Z}(\tilde{A}) = R_\theta(z_t,\tilde{A}). The joint WAM proposes both actions and future states, while the pretrained world model predicts the states induced by those actions. Throughout the paper, ⋅~\tilde{\cdot} denotes a generated proposal, ⋅^\hat{\cdot} a world model prediction, and ⋅∗\cdot^* an optimized trajectory.

We parameterize the joint WAM with conditional flow matching (Lipman et al. [23]). For data xx, noise ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), condition cc, and flow time s∼Unif⁡[0,1]s \sim\operatorname{Unif}[0,1], we use the linear path xs=(1−s)ϵ+sxx_s = (1-s)\epsilon+ sx and train vϕ(xs,s;c)v_\phi(x_s,s;c) toward x−ϵx-\epsilon. § 3 applies this objective to joint action-state trajectories and makes the resulting generator task-aware through trajectory optimization.

World Action Trajectory Optimization

Imitating World-Action Trajectories Is Not Enough

We first train a joint world action flow (WAF) to imitate state-action trajectories from the offline dataset. For a local trajectory, let X=(A,Z)X = (A,Z) and condition the generator on the current state and goal, c=(zt,g)c = (z_t,g). For each training window, we use its final observation as the goal. The WAF generates actions and future states jointly as (A~,Z~)∼qϕ(A,Z∣zt,g)(\tilde{A},\tilde{Z}) \sim q_\phi(A,Z \mid z_t,g).

We parameterize qϕq_\phi with conditional flow matching. Let ϵ=(ϵA,ϵZ)∼N(0,I)\epsilon= (\epsilon_A,\epsilon_Z) \sim\mathcal{N}(0,I), s∼Unif⁡[0,1]s \sim\operatorname{Unif}[0,1], and Xs=(1−s)ϵ+sXX_s = (1-s)\epsilon+ sX. The joint vector field vϕ=(vϕA,vϕZ)v_\phi= (v_\phi^A,v_\phi^Z) is trained with

LWAF=E[MSE⁡(vϕA(Xs,s;c),A−ϵA)+λWAFMSE⁡(vϕZ(Xs,s;c),Z−ϵZ)].(3)\mathcal{L}_{\mathrm{WAF}} = \mathbb{E}\left[\operatorname{MSE}\left(v_\phi^A(X_s,s;c), A-\epsilon_A\right) + \lambda_{\mathrm{WAF}}\operatorname{MSE}\left(v_\phi^Z(X_s,s;c), Z-\epsilon_Z\right)\right]. \tag*{(3)}

This objective imitates the joint state-action distribution in the offline data, but does not optimize trajectories for control. The generated actions need not reach the goal under the pretrained dynamics, so J(Rθ(zt,A~),g)J(R_\theta(z_t,\tilde{A}),g) can remain large. The generated states also need not match the consequences of those actions, so Z~\tilde{Z} need not equal Rθ(zt,A~)R_\theta(z_t,\tilde{A}). Imitation alone therefore provides neither task optimality nor per-sample dynamics-consistency.

Why Not Optimize the Generator Directly?

A simple way to provide task-awareness is to evaluate its generated actions through the world model. Direct rollout training minimizes Lrollout=J(Rθ(zt,A~),g)\mathcal{L}_{\mathrm{rollout}} = J(R_\theta(z_t,\tilde{A}),g). This provides a task gradient to A~\tilde{A}, but Z~\tilde{Z} does not affect action optimization. The joint WAF is thus used only as an action generator.

A stronger variant can also match the generated states to the world model rollout. For example, one can minimize Ldirect+cons=J(Rθ(zt,A~),g)+βdZ(Rθ(zt,A~),Z~)\mathcal{L}_{\mathrm{direct+cons}} = J(R_\theta(z_t,\tilde{A}),g) + \beta d_Z(R_\theta(z_t,\tilde{A}),\tilde{Z}). This encourages dynamics-consistency, but still trains Z~\tilde{Z} to follow the rollout rather than to guide trajectory optimization.

Another alternative is optimization distillation. Given a generated proposal τ~=(A~,Z~)\tilde{\tau} = (\tilde{A},\tilde{Z}), a trajectory optimizer produces an optimized trajectory τ∗\tau^*, which is then treated as a detached regression target. This transfers optimized trajectories to the generator, but discards how τ∗\tau^* changes with τ~\tilde{\tau}. The optimizer provides targets rather than gradients through the optimized solution.

These alternatives either leave the generated future outside trajectory optimization or use optimization only to produce detached targets. WATO instead uses the full generated state-action trajectory to define the optimization problem and learns through its optimized solution.

Dynamics-Constrained Trajectory Projection

WATO uses the generated trajectory as a reference for trajectory optimization rather than executing it directly. For the current state ztz_t, define the set of trajectories consistent with the pretrained world model as Fθ(zt)={(A,Z):A∈UH,Z=Rθ(zt,A)}\mathcal{F}_{\theta}(z_t) = \{(A,Z): A \in\mathcal{U}^{H}, Z = \mathcal{R}_{\theta}(z_t,A)\}. Given a generated reference τ~=(A~,Z~)\tilde{\tau} = (\tilde{A},\tilde{Z}), we define the dynamics-constrained trajectory projection

Πθ,g(τ~)=arg⁡min⁡(A,Z)∈Fθ(zt)  J(Z,g)+λAdA(A,A~)+λZdZ(Z,Z~)(4)\Pi_{\theta,g}(\tilde{\tau}) = \underset{(A,Z) \in\mathcal{F}_{\theta}(z_t)}{\arg\min}\; J(Z,g) + \lambda_A d_A(A,\tilde{A}) + \lambda_Z d_Z(Z,\tilde{Z}) \tag*{(4)}

The task cost drives the trajectory toward the goal. The action reference preserves how the WAM proposes to move, while the state reference preserves where it proposes to go. The world model constrains these references through action-conditioned dynamics.

We use dA(A,A~)=H−1∑h=0H−1∥Ah−A~h∥22d_A(A,\tilde{A}) = H^{-1}\sum_{h=0}^{H-1}\lVert A_h-\tilde{A}_h\rVert_2^2 and dZ(Z,Z~)=H−1∑h=1H∥NZ(zh)−NZ(z~h)∥22d_Z(Z,\tilde{Z}) = H^{-1}\sum_{h=1}^{H}\lVert\mathcal{N}_Z(z_h)-\mathcal{N}_Z(\tilde{z}_h)\rVert_2^2, where NZ\mathcal{N}_Z is a fixed latent standardization. We solve Equation (4) in reduced space, with the action trajectory as the only optimization variable and Z(A)=Rθ(zt,A)Z(A)=\mathcal{R}_{\theta}(z_t,A). The optimization becomes

A⋆=arg⁡min⁡A∈UH  J(Z(A),g)+λAdA(A,A~)+λZdZ(Z(A),Z~),Z⋆=Z(A⋆).(5)A^\star= \underset{A \in\mathcal{U}^{H}}{\arg\min}\; J(Z(A),g) + \lambda_A d_A(A,\tilde{A}) + \lambda_Z d_Z(Z(A),\tilde{Z}), \qquad Z^\star= Z(A^\star). \tag*{(5)}

The optimized trajectory is task-directed and dynamics-consistent while remaining close to the generated state-action trajectory. Importantly, Z~\tilde{Z} is a reference, not an independently optimized state trajectory. Dynamics-consistency follows from the world model rollout rather than an additional penalty. Because both A~\tilde{A} and Z~\tilde{Z} parameterize the optimization objective, changing either reference can change the optimized trajectory. We next characterize this dependence and show how generated future states shape action optimization through the learned dynamics.

Learning Through the Optimized Trajectory

The generated trajectory parameterizes the trajectory optimization problem, not only its initialization. Define the task objective F(A)=J(Rθ(zt,A),g)F(A) = J(\mathcal{R}_{\theta}(z_t,A),g) and the standardized rollout r(A)=vec⁡NZ(Rθ(zt,A))r(A) = \operatorname{vec}\mathcal{N}_Z(\mathcal{R}_{\theta}(z_t,A)). We similarly write r~=vec⁡NZ(Z~)\tilde{r} = \operatorname{vec}\mathcal{N}_Z(\tilde{Z}). The reduced objective in Equation (5) is Ψ(A;A~,r~)=F(A)+λAH−1∥A−A~∥22+λZH−1∥r(A)−r~∥22\Psi(A;\tilde{A},\tilde{r}) = F(A) + \lambda_A H^{-1}\lVert A-\tilde{A}\rVert_2^2 + \lambda_Z H^{-1}\lVert r(A)-\tilde{r}\rVert_2^2.

Let S⋆=∂r(A)/∂A∣A=A⋆S_\star= \left.\partial r(A)/\partial A\right|_{A=A^\star} denote the standardized world model rollout Jacobian. For an interior local solution, the first-order condition is

∇F(A⋆)+2λAH(A⋆−A~)+2λZHS⋆T(r(A⋆)−r~)=0.(6)\nabla F(A^\star) + \frac{2\lambda_A}{H}(A^\star-\tilde{A}) + \frac{2\lambda_Z}{H}S_\star^{\mathsf{T}}\left(r(A^\star)-\tilde{r}\right) = 0. \tag*{(6)}

The action reference acts directly in action space. The state reference acts through S⋆TS_\star^{\mathsf{T}}, which maps a future-state error into an action-space optimization direction. Generated future states therefore affect trajectory optimization through the action-conditioned world dynamics.

Proposition 1 (Dynamics-aware reference sensitivity). Suppose A⋆A^\star is an interior local solution, Ψ\Psi is twice continuously differentiable near A⋆A^\star, and H⋆:=∇AA2Ψ(A⋆;A~,r~)\mathcal{H}_\star:= \nabla^2_{AA}\Psi(A^\star;\tilde{A},\tilde{r}) is nonsingular. Then

∂A⋆∂A~=2λAHH⋆−1,∂A⋆∂r~=2λZHH⋆−1S⋆T(7)\frac{\partial A^\star}{\partial\tilde{A}} = \frac{2\lambda_A}{H}\mathcal{H}_\star^{-1}, \qquad\frac{\partial A^\star}{\partial\tilde{r}} = \frac{2\lambda_Z}{H}\mathcal{H}_\star^{-1}S_\star^{\mathsf{T}} \tag*{(7)}

Actions affect the optimized trajectory through the local optimization Hessian. Future states additionally pass through the rollout Jacobian, so their influence depends on how actions change the predicted future. Both references can thus receive task gradients through the optimized solution.

This structure also changes the local geometry of action optimization. The state-reference term contributes 2S⋆TS⋆2S_\star^{\mathsf{T}}S_\star plus a residual-weighted second-order term from the rollout. When the state-reference residual is small, the local Hessian is approximated by

H⋆≈∇2F(A⋆)+2λAHI+2λZHS⋆TS⋆.(8)\mathcal{H}_\star\approx\nabla^2 F(A^\star) + \frac{2\lambda_A}{H}I + \frac{2\lambda_Z}{H}S_\star^{\mathsf{T}}S_\star. \tag*{(8)}

Corollary 1 (Dynamics-induced local geometry). The action reference contributes isotropic curvature in action space, while the state reference contributes the positive-semidefinite term S⋆TS⋆S_\star^{\mathsf{T}} S_\star. Action perturbations are therefore weighted by how strongly they change the predicted future. Generated states thus shape the local geometry of trajectory optimization through the learned dynamics.

We compute these sensitivities by implicit differentiation of the trajectory optimality conditions. Further derivations and local analysis are given in Appendix § 11.

The WATO Objective

Let τ~ϕ=(A~,Z~)∼qϕ(A,Z∣zt,g)\tilde{\tau}_{\phi} = (\tilde{A}, \tilde{Z}) \sim q_{\phi}(A, Z \mid z_t, g) denote a generated trajectory and τ⋆=(A⋆,Z⋆)=Πθ,g(τ~ϕ)\tau^\star= (A^\star, Z^\star) = \Pi_{\theta,g}(\tilde{\tau}_{\phi}) its optimized trajectory. WATO trains the same joint WAF with both imitation and task supervision:

LWATO=LWAF+λtaskE[J(Z⋆,g)](9)\mathcal{L}_{\mathrm{WATO}} = \mathcal{L}_{\mathrm{WAF}} + \lambda_{\mathrm{task}}\mathbb{E}\left[J(Z^\star,g)\right] \tag*{(9)}

The flow matching term continues to imitate offline state-action trajectories, while the task term favors generated references that lead to better optimized trajectories.

The task gradient passes through the optimized trajectory as

dJ(Z⋆,g)dϕ=∂J∂τ⋆∂τ⋆∂τ~ϕ⏟solution sensitivity∂τ~ϕ∂ϕ(10)\frac{\mathrm{d}J(Z^\star,g)}{\mathrm{d}\phi} = \frac{\partial J}{\partial\tau^\star} \underbrace{\frac{\partial\tau^\star}{\partial\tilde{\tau}_{\phi}}}_{\text{solution sensitivity}} \frac{\partial\tilde{\tau}_{\phi}}{\partial\phi} \tag*{(10)}

By Proposition 1, action gradients pass through the local optimization Hessian, while state gradients additionally pass through the rollout Jacobian. The world model is differentiable when computing these gradients, but its parameters θ\theta remain fixed. WATO updates only the generator parameters ϕ\phi.

Training and Deployment

WATO uses two training stages. We first pretrain the joint WAF with LWAF\mathcal{L}_{\mathrm{WAF}} to imitate offline state-action trajectories. We then continue training the same generator with LWATO\mathcal{L}_{\mathrm{WATO}}, differentiating through trajectory optimization to provide task supervision.

Trajectory optimization is removed after training. At deployment, WATO generates KK independent proposals (A~(k),Z~(k))∼qϕ(A,Z∣zt,g)(\tilde{A}^{(k)},\tilde{Z}^{(k)}) \sim q_{\phi}(A,Z \mid z_t,g). Each action proposal is rolled out with the pretrained world model as Z^(k)=Rθ(zt,A~(k))\hat{Z}^{(k)} = \mathcal{R}_{\theta}(z_t,\tilde{A}^{(k)}). The controller selects k⋆=arg min⁡kJ(Z^(k),g)k^\star= \operatorname*{arg\,min}_{k} J(\hat{Z}^{(k)},g) and executes the selected action segment in receding horizon. The generated state trajectory is not directly scored at deployment. It contributes through joint generation and trajectory-optimization training.

Optimization is used to train the generator, not to operate the deployed controller. Deployment requires only trajectory generation and world model evaluation. Full algorithm is provided in Algorithm 1.

Remark: Optimization amortization. Under local regularity assumptions, small projection residuals bound the task-cost gap between generated and optimized trajectories and the generated trajectory’s inconsistency with the world model (see Appendix § 11.§ 4).

Related Work

Generative policies and WAMs. Generative policies learn distributions over actions or trajectories with multimodality ([15, 2, 6, 29]). WAMs extend action generation by jointly modeling future actions and states or observations ([46, 5, 40]). Recent work also studies how generated futures should be used at inference time ([42, 43]). WATO instead optimizes the generated world-action trajectory for control. Generated actions and future states jointly define a dynamics-constrained optimization problem, and the generator is trained through its solution.

World models and model-based planning. World models learn action-conditioned dynamics and use predicted futures to evaluate and/or optimize actions ([10, 12]

Algorithm 1 World Action Trajectory Optimization (WATO)

Require: Offline dataset D\mathcal{D}, pretrained world model Rθ\mathcal{R}_{\theta} (Equation (1))

▷ WAF pretraining

Pretrain joint WAF qϕ(A,Z∣zt,g)q_{\phi}(A, Z \mid z_t, g) with flow matching (Equation (3))

▷ WATO training loop

while not converged do

Sample (zt,g,A,Z)∼D(z_t, g, A, Z) \sim\mathcal{D}

Generate (A~,Z~)∼qϕ(A,Z∣zt,g)(\widetilde{A}, \widetilde{Z}) \sim q_{\phi}(A, Z \mid z_t, g) (Equation (2))

▷ Dynamics-constrained trajectory optimization

Compute (A⋆,Z⋆)=Πθ,g(A,Z~)(A^\star, Z^\star) = \Pi_{\theta,g}(A, \widetilde{Z}) (Equation (4))

▷ Learning through the optimized trajectory

Compute task loss J(Z⋆,g)J(Z^\star, g)

Differentiate through the optimized trajectory (Equation (10))

Update ϕ\phi with LWATO\mathcal{L}_{\mathrm{WATO}} (Equation (9))

▷ Deployment

Generate KK proposals (A~(k),Z~(k))∼qϕ(A,Z∣zt,g)(\widetilde{A}^{(k)}, \widetilde{Z}^{(k)}) \sim q_{\phi}(A, Z \mid z_t, g)

Roll out Z^(k)=Rθ(zt,A~(k))\widehat{Z}^{(k)} = \mathcal{R}_{\theta}(z_t, \widetilde{A}^{(k)}) for each proposal

Select k⋆=arg⁡min⁡kJ(Z^(k),g)k^\star= \arg\min_k J(\widehat{Z}^{(k)}, g)

Execute the selected action segment and replan

Sobal et al., [34]; Maes et al. [24]. Model-based planners commonly optimize action sequences online using sampling or gradient-based methods [32, 45]. WATO uses a pretrained world model to constrain trajectory optimization during training. Deployment only requires trajectory generation and world model evaluation, without online optimization.

Differentiable optimization and generative control. Differentiable optimization exposes how an optimized trajectory changes with model, cost, or policy parameters [4, 3]. Prior work uses this principle to train policies through optimization or predictive control [38, 39, 36]. Other methods combine generative models with trajectory optimization, model-based projections, or MPC [33, 25, 14, 22]. WATO uses both actions and future states to define the optimization objective, then differentiates task performance through its solution. Thus, differentiable optimization serves as a learning mechanism for WAMs rather than refinement only.

Offline goal-conditioned control. Offline goal-conditioned RL (GCRL) learns to reach goals from fixed trajectory data. Prior methods include goal-conditioned behavior cloning, value learning, quasimetric learning, and hierarchical RL [8, 20, 37, 26, 28]. WATO addresses the same downstream goal-reaching problem from a different direction. It starts from a joint WAF and makes its trajectories task-aware using a world model and differentiable trajectory optimization. We compare against these methods as goal-conditioned control baselines.

Experiments

We evaluate WATO along four questions. First, does world-action trajectory optimization improve goal-conditioned control over model-based, generative, and RL methods? Second, is differentiating through trajectory optimization more effective than direct task training or optimization distillation? Third, do generated future states improve control beyond action generation alone? Finally, is trajectory optimization successfully transferred into the generator during training?

Experimental Setup

Environments. We evaluate WATO on 7 environments covering distinct control challenges. TwoRooms tests visual navigation, Reacher tests continuous reaching, and PushT tests contact-rich pushing [24]. We additionally use four visual manipulation tasks from OGBench: cube-{single, double}, scene-play, and puzzle-3x3 [28]. Together, these environments cover navigation, reaching, manipulation, and multi-stage goal-conditioned control.

Evaluation. We evaluate goal reaching from offline data. Each initial observation is paired with a goal observation L=25L = 25 primitive steps later in a evaluation trajectory, and the controller receives a 50-step execution budget. Unless otherwise stated, WATO generates K=8K = 8 candidates using 8 flow integration steps (NFE =8= 8) at deployment. Success rate is the primary metric. We also report final goal error, latency, and world model evaluations in the appendix.

Baselines. We compare against three groups of methods. For goal-conditioned learning, we use GCBC (Ding et al., [8]), GCIVL (Kostrikov et al., [20]), GCIQL (Park et al., [28]), and QRL (Wang et al., [37]). For generative control, we compare with an flow policy, a joint WAF trained only by flow matching, and generative predictive control (GPC) (Kurtz & Burdick, [21]), adapted to our offline setting. For model-based control, we use CEM (Maes et al., [24]) and gradient-based MPC. GPC, CEM, gradient-based MPC, and WATO use the same pretrained LeWM. Model-based planners also use the same task cost. All methods use the same data, goal distribution, and evaluation protocol. Training details are given in Appendix § 14.§ 2.

How Good is WATO for Visual Offline Goal-Conditioned Control?

We first evaluate WATO across all 7 environments. Table 1 reports the success rates. WATO achieves the highest macro-average success of 91.7% and is best or tied for best on six of 7 environments, while GPC performs best on Reacher.

EnvironmentGCBCGCIVLGCIQLQRLFlow PolicyJoint WAFGPCCEMGrad. MPCWATO
TwoRooms81±281 \pm277±377 \pm380±580 \pm585±285 \pm279±279 \pm283±183 \pm187±287 \pm286±386 \pm392±392 \pm393±093 \pm0
Reacher74±374 \pm365±365 \pm371±471 \pm473±373 \pm364±564 \pm567±367 \pm391±291 \pm270±270 \pm281±481 \pm485±085 \pm0
PushT66±166 \pm145±245 \pm239±039 \pm058±158 \pm177±277 \pm284±484 \pm499±199 \pm182±282 \pm297±397 \pm3100±0100 \pm0
cube-single75±075 \pm045±345 \pm352±152 \pm172±372 \pm381±381 \pm388±188 \pm178±378 \pm368±868 \pm876±276 \pm299±199 \pm1
cube-double63±163 \pm121±421 \pm437±537 \pm560±160 \pm162±462 \pm489±289 \pm264±764 \pm758±358 \pm371±171 \pm190±390 \pm3
scene-play55±255 \pm232±232 \pm240±440 \pm458±058 \pm068±368 \pm378±378 \pm337±237 \pm242±1142 \pm1159±559 \pm594±194 \pm1
puzzle-3x351±251 \pm249±849 \pm837±637 \pm645±145 \pm154±354 \pm370±370 \pm339±539 \pm549±449 \pm461±261 \pm281±281 \pm2
Average66.447.750.964.469.379.970.765.076.791.7

Table 1. Overall control performance. Success rates across 7 visual control environments.

Compared with the joint WAF, WATO improves average success from 79.9% to 91.7%. This 11.8 percentage points (pp) gain shows that trajectory imitation alone does not fully exploit the generated state-action trajectory. Optimizing it during training makes the generator more effective for control.

WATO also outperforms online model-based planning. CEM and gradient-based MPC optimize actions at deployment using the same world model, achieving average success rates of 65.0% and 76.7%, respectively. WATO uses optimization only during training and deploys the generator without online optimization. Its gains therefore come from improving the generator rather than additional extensive test-time search. § 5.6 examines this transfer of optimization into the generator.

What Does WATO Generate?

Figure 2 separates single-step candidate quality from full closed-loop behavior. A good local proposal does not necessarily yield reliable control over repeated planning.

Figure 2. Visualizing control behavior in PushT. (a) Terminal object-position and orientation errors for candidates from a single planning step in one representative episode. Each candidate is replayed from the same state. (b) Goal-aligned object trajectories over 8 full episodes with heterogeneous initializations, with the goal at the origin.

At one planning step, WATO generates only K=8K = 8 concentrated, low-error candidates. CEM uses 300 rollouts but produces a broader candidate set, while the flow policy includes several large-error proposals. GCIQL and GCBC provide only a single proposal without candidate selection. WATO therefore achieves high-quality candidates with little test-time search.

Across eight heterogeneous episodes, WATO succeeds in 8/8, compared with 7/8 for the joint WAF, 6/8 for CEM, and 5/8 for the flow policy. Relative to the joint WAF, WATO yields a tighter candidate distribution and more reliable closed-loop trajectories. CEM also shows that more online search does not necessarily improve reliability. GCBC further illustrates the gap between local proposal quality and closed-loop behavior: its proposal in Figure 2(a) can lie near the goal, yet only 5/8 episodes succeed in Figure 2(b). Overall, WATO combines sample-efficient candidate generation with stable repeated planning.

How Beneficial is Differentiation Through Trajectory Optimization?

How Beneficial Is Differentiation through Trajectory Optimization?

We next ask whether WATO benefits specifically from differentiating through trajectory optimization. All variants use the same joint WAF, offline data, and world model. Only the training objective changes. Joint WAF uses flow matching alone. Direct rollout adds the goal-reaching loss through the world model rollout, supervising the generated actions. Direct rollout + consistency also matches states to the rollout. Distillation trains the generator toward detached trajectories produced by the same optimizer. WATO instead differentiates through the optimized solution, allowing both action and state references to receive gradients according to how they affect that solution.

Figure 3 shows a steady improvement across these objectives. Joint WAF achieves 79.9%, direct rollout 85.7%, direct rollout with consistency 87.4%, and distillation 88.0%. WATO reaches 91.7%, indicating that differentiating through optimization provides more effective supervision than direct task losses, consistency regularization, or detached optimized targets.

Figure 3. Training objective ablation. Average success across all environments.

Is Trajectory Optimization Well-Amortized through WATO?

WATO uses trajectory optimization during training but removes it at deployment. We test whether this optimization is transferred into the generator by applying the same trajectory optimization layer before and after WATO training.

Table 2 reports the additional success obtained from online trajectory optimization. Before WATO training, optimization improves success by 10.4 pp on average. After WATO training, the remaining gain drops to 1.3 pp. Thus, WATO learns to generate goal-reaching and dynamics-consistent trajectories that already capture most of the benefit of trajectory optimization.

EnvironmentBefore WATOAfter WATO
TwoRooms+8.0+1.0
Reacher+15.0+3.0
PushT+14.0+0.0
cube-single+9.0+1.0
cube-double+3.0+1.0
scene-play+14.0+1.0
puzzle-3×3+10.0+2.0
Average+10.4+1.3

Table 2. Remaining benefit of online trajectory optimization is small. Additional success from the same optimizer before and after WATO training, in percentage points.

Should We Generate Both States and Actions?

We next isolate the roles of generated actions and future states. The action-only variant removes dZd_Z, while the state-only variant removes dAd_A. They achieve average success rates of 85.7% and 85.0%, respectively, compared with 91.7% for full WATO. Thus, neither reference alone recovers the performance of their combination. Generated future states therefore contribute directly to trajectory optimization rather than serving only as auxiliary predictions. This result is consistent with Proposition 1: state references influence optimized actions through the world model rollout Jacobian. We further compare generated and rollout-derived state references to test whether generated futures provide benefits beyond dynamics-aware regularization (Appendix § 15.§ 1).

How Does Trajectory Optimization Shape the Generator?

We next examine how the generator changes during WATO training on three environments: PushT, cube-double, and scene-play. For each generated world-action trajectory, we measure dA(A⋆,A~)d_A(A^\star, \widetilde{A}), dZ(Z⋆,Z~)d_Z(Z^\star, \widetilde{Z}), and the task improvement ΔJ=J(Rθ(zt,A~),g)−J(Z⋆,g)\Delta J = J(R_\theta(z_t, \widetilde{A}), g) - J(Z^\star, g) (see § 3.3 for definitions).

Figure 4 shows that all three metrics decrease despite some fluctuations. From initialization to 10k WATO steps, the action-reference distance decreases by 39.3% on average across the three environments, the state-reference distance by 30.5%, and the remaining goal-reaching task improvement by 31.8%. The generator therefore moves closer to its optimized trajectories while requiring progressively less correction from trajectory optimization. This provides direct evidence that trajectory optimization is amortized into world-action generation.

Figure 4. Trajectory projection during WATO training. Training dynamics on three representative environments, evaluated every 1,000 WATO steps.

How Efficient is WATO at Deployment?

Table 3 compares WATO with CEM at deployment. WATO reduces latency in every environment, with speedups from 5.8× to 25.0× and a geometric mean of 11.5×. World model evaluations decrease more sharply, from 375× to 1125×, with a geometric mean of 600.5×.

EnvironmentLatencyWM evals
TwoRooms18.9×1125×
Reacher13.6×1125×
PushT25.0×1125×
cube-single5.8×375×
cube-double6.3×375×
scene-play12.2×375×
puzzle-3x39.2×375×
Geometric mean11.5×600.5×

Table 3. Efficiency relative to CEM at deployment. Ratios are TCEM/TWATOT_{\mathrm{CEM}}/T_{\mathrm{WATO}} and NCEM/NWATON_{\mathrm{CEM}}/N_{\mathrm{WATO}}. Higher is better.

The larger reduction in world model evaluations than in latency shows that model evaluations alone do not determine wall-clock time, because trajectory generation and execution remain. WATO and the joint WAF use the same deployment procedure, so WATO’s control gains require no additional test-time computation. Absolute latency, evaluation counts, and timing details are reported in Appendix § 17.

Discussion and Conclusion

WATO shows that trajectory optimization can train joint WAMs rather than only refine actions at deployment. Across 7 environments, it improves average success from 79.9% for the joint WAF to 91.7%, with ablations showing that both action and state references contribute. Decreasing projection residuals and optimization benefit further indicate that WATO transfers trajectory optimization into the generator. Generated future states also play a direct control role: as trajectory references, they shape optimized actions through the world model dynamics.

World models and joint WAMs therefore play complementary roles. The joint WAM proposes trajectories, while the pretrained world model provides action-conditioned dynamics during training and evaluates actions at deployment. WATO uses this model to improve generation without iterative online planning. Limitations include dependence on learned dynamics, the finite-horizon goal-conditioned setting, and evaluation only with flow matching. Extending WATO to robust longer-horizon control and other joint generative models remains open.

AI Use Statement

Generative AI was only used to assist with grammar checking and writing polish. All ideas, scientific content, experimental results, and conclusions are the authors’ own.

Reproducibility Statement

Implementation details, model hyperparameters, optimization procedures, and evaluation protocols are provided in the appendix. We also include anonymous code to reproduce the experiments.

World Action Models as a Generative Control Model Class

We make this perspective manifest by treating WAMs as a mathematical class of conditional trajectory models, independent of architecture or scale. Let c=(zt,g)c = (z_t, g) denote the current-state and goal condition. An action-only generative policy models qϕA(A∣c)q_{\phi}^{A}(A \mid c), a world model represents action-conditioned dynamics pθ(Z∣A,c)p_{\theta}(Z \mid A, c), and a joint WAM models qϕ(A,Z∣c)q_{\phi}(A, Z \mid c). What differs is the modeled object. A joint WAM generates both an action trajectory and an intended future trajectory, with action marginal qϕ,A(A∣c)=∫qϕ(A,Z∣c) dZq_{\phi,A}(A \mid c) = \int q_{\phi}(A, Z \mid c)\,\mathrm{d}Z.

Model class and scale are separate choices. A joint WAM does not require a particular architecture, generative parameterization, dataset scale, or pretraining regime. Autoregressive, diffusion, and flow-based models can all represent distributions over world-action trajectories. A joint WAM can likewise be task-specific or pretrained on heterogeneous data. Recent WAMs emphasize large-scale pretraining and generalization [46, 5, 40]. Here, we focus on the control problem that comes with generating a joint trajectory.

Joint generation exposes an additional control reference. An action-only generator gives downstream control only an action trajectory. A joint WAM also provides a future trajectory Z~\tilde{Z}. For this extra output to matter for control, it should affect the resulting actions rather than act only as an auxiliary prediction. WATO uses A~\tilde{A} and Z~\tilde{Z} as references in the trajectory projection Πθ,g\Pi_{\theta,g}, so both can shape the optimized action trajectory. Proposition 1 in § 3.4 makes this explicit: the state reference acts through the world model rollout Jacobian.

Scaling does not affect the control objective. Learning a broad trajectory distribution and choosing good behavior for a downstream task are different problems. A large joint WAM may represent diverse behaviors and futures, but imitation alone does not ensure that a generated trajectory minimizes the task objective or matches the dynamics used for control. Pretraining determines what trajectory distribution the model represents. WATO shapes how that distribution is used for control.

The formulation is independent of the generator parameterization. Our implementation uses flow matching, but WATO is defined at the level of generated world-action trajectories. The operator Πθ,g(A~,Z~)\Pi_{\theta,g}(\tilde{A}, \tilde{Z}) depends on action and state references and differentiable action-conditioned dynamics, not on the flow architecture itself. The same idea may therefore extend to other differentiable joint generative models, including larger pretrained WAMs. We leave this direction to future work.

Why WATO Should Be Your Favorite Controller

Our results suggest WATO as a default controller for visual offline goal-conditioned control with local goals or waypoints. This raises a broader question: why does WATO also outperform value-based GCRL methods designed for long-horizon goal reaching? We look at local versus long-horizon control, representation, candidate selection, and the way each method learns a controller.

Short horizons do not make policy learning trivial. OGBench uses goal relabeling during training and predefined start-goal tasks for evaluation [28]. Our evaluation instead samples goals 25 steps ahead in test trajectories and allows 50 execution steps. Each pair has an observed local connection, so little cross-trajectory stitching is needed. But the controller still needs accurate contact, pose control, and recovery from execution errors. Being able to propagate goal-reaching values over long horizons does not automatically give a strong local controller.

The gap is not explained by the encoder alone. The main comparisons use the same pretrained LeWM encoder to control for representation quality. End-to-end training with the official IMPALA-based setup raises GCIQL’s average success from 50.9% to 53.9%, with larger gains on scene-play and puzzle–3x3. So encoder training can help GCIQL, but it does not close the gap.

The advantage persists without candidate selection. WATO with K=1K = 1 achieves 88.7% average success and outperforms both GCIQL variants in all 7 environments. Increasing KK to 8 adds another 3.0 percentage points. With K=1K = 1, world model ranking cannot change the executed proposal, so best-of-KK selection alone cannot explain the advantage. Table 4 summarizes both controls.

EnvironmentGCIQL: Pretrained LeWMGCIQL: End-to-end trainingWATO: pretrained LeWM: K=1K = 1WATO: pretrained LeWM: K=8K = 8
TwoRooms80±580 \pm581±881 \pm890±390 \pm393±093 \pm0
Reacher71±471 \pm467±667 \pm682±582 \pm585±285 \pm2
PushT39±039 \pm038±538 \pm596±396 \pm3100±0100 \pm0
cube-single52±152 \pm156±756 \pm797±297 \pm299±199 \pm1
cube-double37±537 \pm540±1040 \pm1086±286 \pm290±390 \pm3
scene-play40±440 \pm451±551 \pm590±190 \pm194±194 \pm1
puzzle-3x337±637 \pm644±444 \pm480±380 \pm381±281 \pm2
Average50.953.988.791.7

Table 4. Encoder and candidate-count controls. Success rates (%, mean ± std over 4 seeds). End-to-end GCIQL uses the official IMPALA-based setup.

Trajectory supervision directly matches the local control problem. Each training window gives a current state, an attained future goal, and the actions and states in between. Goal-conditioned imitation uses the same kind of hindsight supervision [9], and GCRL also uses relabeled goals and dataset actions. So the key difference is not whether the methods see goal-reaching trajectories, but how they use them. GCIQL learns goal-conditioned values and extracts a policy using those values and a behavioral constraint [28]. WATO instead learns the joint trajectory and sends task gradients through its optimized solution. In this sense, it trades bootstrapped value learning for model-based trajectory supervision while keeping the imitation signal.

Simply matching the goal distribution throughout trajectories is not enough. GCBC and the flow policy achieve 66.4% and 69.3% average success, compared with 79.9% for Joint WAF and 91.7% for WATO. Neither hindsight imitation nor action-only generation reaches the performance of WATO.

One trajectory sample can still represent coordinated behavior. Reducing KK removes best-of-KK selection, but not the structure within a sampled trajectory. A joint WAF can generate an action sequence whose approach, contact, and subsequent motion follow one behavioral mode. This may help compared with extracting individual actions from a unimodal policy, because the continuous-action OGBench baselines use Gaussian policies evaluated at their means [28]. Still, our comparisons do not cleanly separate temporal coordination, policy expressiveness, and the training objective. We therefore view trajectory generation as a useful inductive bias, not as something a Markov policy could never represent.

Lower GCRL performance does not point to value error alone. The bottleneck could lie in value estimation, policy extraction, or generalization to states encountered during execution. Prior work shows that policy extraction and test-time generalization can limit offline RL even when value learning works well [27]. Our controls do not isolate these factors one by one, but they show that neither encoder choice nor best-of-KK selection explains the gap.

WATO takes a different route: it gets continuous task supervision through a learned rollout and trajectory optimization. This still depends on world model accuracy and on transferring optimized behavior into the unprojected generator, as analyzed in Appendix § 11.§ 4.

WATO as the default local controller. Taken together, our results point to a simple default: if the goal is to turn visual offline trajectories into a goal-conditioned controller, start with a joint WAM and optimize it for control. WATO outperforms the evaluated GCRL methods even when GCIQL trains its visual encoder end-to-end, and K=1K = 1 WATO remains far ahead without best-of-KK selection. The gap is therefore not explained by representation quality or test-time candidate selection alone. WATO also keeps the resulting controller simple at deployment, requiring only trajectory generation and a small number of world model evaluations.

This does not make WAMs and GCRL mutually exclusive. Generative policies can be combined with value learning [29], and values may be useful when the problem requires trajectory stitching or reasoning far beyond locally supported trajectories. But that is a different job from learning a strong local controller. For the controller itself, our results give a clear answer: directly learn the trajectory, optimize it for the task, and amortize that optimization into the generator.

For visual offline goal-conditioned control, WATO should be your favorite controller.

Theory and Derivations

This section develops the reference sensitivities and local geometry used in § 3.4. We also study the local stability of the trajectory projection and connect its residuals to optimization amortization. All results are local, since the world model rollout and trajectory optimization problem are generally nonlinear and nonconvex.

Reference Sensitivity

We start from the reduced WATO objective

Ψ(A;A~,r~)=F(A)+λAH∥A−A~∥22+λZH∥r(A)−r~∥22,(11)\Psi(A;\widetilde{A},\widetilde{r}) = F(A) + \frac{\lambda_A}{H}\lVert A-\widetilde{A}\rVert_2^2 + \frac{\lambda_Z}{H}\lVert r(A)-\widetilde{r}\rVert_2^2, \tag*{(11)}

where F(A)=J(Rθ(zt,A),g)F(A)=J(\mathcal{R}_{\theta}(z_t,A),g), r(A)=vec⁡NZ(Rθ(zt,A))r(A)=\operatorname{vec}\mathcal{N}_{Z}(\mathcal{R}_{\theta}(z_t,A)), and r~=vec⁡NZ(Z~)\widetilde{r}=\operatorname{vec}\mathcal{N}_{Z}(\widetilde{Z}). Let S(A)=∂r(A)/∂AS(A)=\partial r(A)/\partial A and write the first-order condition as

G(A;A~,r~):=∇AΨ(A;A~,r~)=0.(12)G(A;\widetilde{A},\widetilde{r}) := \nabla_A\Psi(A;\widetilde{A},\widetilde{r})=0. \tag*{(12)}

For an interior local solution A⋆A^\star, this becomes

∇F(A⋆)+2λAH(A⋆−A~)+2λZHS⋆T(r(A⋆)−r~)=0,(13)\nabla F(A^\star)+\frac{2\lambda_A}{H}(A^\star-\widetilde{A})+\frac{2\lambda_Z}{H}S_\star^{\mathsf{T}}(r(A^\star)-\widetilde{r})=0, \tag*{(13)}

where S⋆=S(A⋆)S_\star=S(A^\star). Let H⋆=∇AA2Ψ(A⋆;A~,r~)\mathcal{H}_\star=\nabla^2_{AA}\Psi(A^\star;\widetilde{A},\widetilde{r}). If H⋆\mathcal{H}_\star is nonsingular, the implicit function theorem gives

∂A⋆∂y=−H⋆−1∂G∂y,y∈{A,r~}.(14)\frac{\partial A^\star}{\partial y}=-\mathcal{H}_\star^{-1}\frac{\partial G}{\partial y},\qquad y\in\{A,\widetilde{r}\}. \tag*{(14)}

The action reference enters Equation (13) directly, with ∂G/∂A~=−(2λA/H)I\partial G/\partial\widetilde{A}=-(2\lambda_A/H)I. This gives

∂A⋆∂A~=2λAHH⋆−1.(15)\frac{\partial A^\star}{\partial\widetilde{A}}=\frac{2\lambda_A}{H}\mathcal{H}_\star^{-1}. \tag*{(15)}

The state reference enters through the world model rollout. Since ∂G/∂r~=−(2λZ/H)S⋆T\partial G/\partial\widetilde{r}=-(2\lambda_Z/H)S_\star^{\mathsf{T}}, we get

∂A⋆∂r~=2λZHH⋆−1S⋆T.(16)\frac{\partial A^\star}{\partial\widetilde{r}}=\frac{2\lambda_Z}{H}\mathcal{H}_\star^{-1}S_\star^{\mathsf{T}}. \tag*{(16)}

Together, Equation (15)-Equation (16) give Proposition 1. The action reference changes the optimized action trajectory through the local optimization Hessian, while the state reference additionally passes through S⋆TS_\star^{\mathsf{T}}, which maps changes in the predicted future back to action space. For box-constrained solutions, the same derivation applies locally on the free action coordinates when the active set is fixed.

Local Optimization Geometry

The Hessian of Equation (11) also shows how the two references shape the local optimization geometry. Define the state-reference residual e(A)=r(A)−r~e(A)=r(A)-\widetilde{r}. Holding r~\widetilde{r} fixed, its squared norm has gradient

∇A∥e(A)∥22=2S(A)Te(A).(17)\nabla_A\lVert e(A)\rVert_2^2=2S(A)^{\mathsf{T}}e(A). \tag*{(17)}

Writing ri(A)r_i(A) and ei(A)e_i(A) for the corresponding coordinates and differentiating again gives

∇A2∥e(A)∥22=2S(A)TS(A)+2∑iei(A)∇A2ri(A).(18)\nabla_A^2\lVert e(A)\rVert_2^2=2S(A)^{\mathsf{T}}S(A)+2\sum_i e_i(A)\nabla_A^2r_i(A). \tag*{(18)}

The exact Hessian at A⋆A^\star is

H⋆=∇2F(A⋆)+2λAHI+2λZH[S⋆TS⋆+∑ie⋆,i∇A2ri(A⋆)],(19)\mathcal{H}_\star=\nabla^2F(A^\star)+\frac{2\lambda_A}{H}I+\frac{2\lambda_Z}{H}\left[S_\star^{\mathsf{T}}S_\star+\sum_i e_{\star,i}\nabla_A^2r_i(A^\star)\right], \tag*{(19)}

where e⋆=r(A⋆)−r~e_{\star}=r(A^{\star})-\tilde{r}.

Approximating the state-reference Hessian by its Gauss-Newton term gives

H⋆≈∇2F(A⋆)+2λAHI+2λZHS⋆TS⋆.(20)\mathcal{H}_{\star}\approx\nabla^{2}F(A^{\star})+\frac{2\lambda_{A}}{H}I+\frac{2\lambda_{Z}}{H}S_{\star}^{\mathsf{T}}S_{\star}. \tag*{(20)}

The task Hessian ∇2F(A⋆)\nabla^{2}F(A^{\star}) is kept exact. If ∥∇A2ri(A)∥≤Mr\lVert\nabla^{2}_{A}r_{i}(A)\rVert\leq M_{r} for every coordinate in a neighborhood of A⋆A^{\star}, the omitted term satisfies

∥2λZH∑ie⋆,i∇A2ri(A⋆)∥≤2λZMrH∥e⋆∥1.(21)\left\lVert\frac{2\lambda_{Z}}{H}\sum_{i}e_{\star,i}\nabla^{2}_{A}r_{i}(A^{\star})\right\rVert\leq\frac{2\lambda_{Z}M_{r}}{H}\lVert e_{\star}\rVert_{1}. \tag*{(21)}

So the approximation error vanishes with the state-reference residual.

The action reference contributes isotropic curvature (2λA/H)I(2\lambda_{A}/H)I. The state-reference Gauss-Newton term is positive semidefinite since, for any action perturbation uu,

uTS⋆TS⋆u=∥S⋆u∥22≥0.(22)u^{\mathsf{T}}S_{\star}^{\mathsf{T}}S_{\star}u=\lVert S_{\star}u\rVert_{2}^{2}\geq0. \tag*{(22)}

It weights action directions by how strongly they change the standardized predicted trajectory. The task Hessian and residual-weighted term may still be indefinite, so this does not make the full objective convex.

Generated versus rollout-derived references. One interesting distinction is that this geometry does not require a WAM-generated future. For a fixed proposal A~\tilde{A}, replacing r~\tilde{r} with r(A~)r(\tilde{A}) gives the state-reference penalty λZH−1∥r(A)−r(A~)∥22\lambda_{Z}H^{-1}\lVert r(A)-r(\tilde{A})\rVert_{2}^{2}. At the same action trajectory AA, its Gauss-Newton contribution is still (2λZ/H)S(A)TS(A)(2\lambda_{Z}/H)S(A)^{\mathsf{T}}S(A). What changes is the reference itself, and hence the gradient, residual-weighted Hessian term, and optimized solution. Removing dZd_{Z} therefore tests the state-reference penalty as a whole, but does not separate generated-future information from dynamics-aware regularization. This motivates the generated versus rollout-derived reference comparison in Appendix § 15.§ 1.

Projection Stability

The same sensitivity expressions also tell us how stable the projection is locally. Assume that Ψ\Psi is locally μ\mu-strongly convex in AA around A⋆A^{\star}, so that

H⋆⪰μI,μ>0.(23)\mathcal{H}_{\star}\succeq\mu I,\qquad\mu>0. \tag*{(23)}

Then ∥H⋆−1∥≤1/μ\lVert\mathcal{H}_{\star}^{-1}\rVert\leq1/\mu. Using Equation (15),

∥∂A⋆∂A~∥≤2λAHμ.(24)\left\lVert\frac{\partial A^{\star}}{\partial\tilde{A}}\right\rVert\leq\frac{2\lambda_{A}}{H\mu}. \tag*{(24)}

Similarly,

∥∂A⋆∂r~∥≤2λZHμ∥S⋆∥.(25)\left\lVert\frac{\partial A^{\star}}{\partial\tilde{r}}\right\rVert\leq\frac{2\lambda_{Z}}{H\mu}\lVert S_{\star}\rVert. \tag*{(25)}

For sufficiently small perturbations (ΔA~,Δr~)(\Delta\tilde{A},\Delta\tilde{r}), these bounds give

∥ΔA⋆∥≤2Hμ(λA∥ΔA~∥+λZ∥S⋆∥∥Δr~∥)+o(∥(ΔA~,Δr~)∥).(26)\lVert\Delta A^{\star}\rVert\leq\frac{2}{H\mu}\left(\lambda_{A}\lVert\Delta\tilde{A}\rVert+\lambda_{Z}\lVert S_{\star}\rVert\lVert\Delta\tilde{r}\rVert\right)+o\left(\lVert(\Delta\tilde{A},\Delta\tilde{r})\rVert\right). \tag*{(26)}

The action-reference sensitivity is controlled by the local curvature of the optimization problem. For the state reference, the rollout Jacobian also matters: it has more influence in directions where small action changes produce larger changes in the future.

Optimization Amortization

There is one subtlety in WATO: training evaluates the projected task cost F(A⋆)F(A^{\star}), while deployment uses the unprojected proposal A~\tilde{A}. A good projected trajectory does not automatically imply a good raw proposal. For example, consider H=1H = 1, r(A)=Ar(A) = A, F(A)=A2F(A) = A^2, and λA=λZ=1\lambda_A = \lambda_Z = 1. The projection is

A⋆=arg⁡min⁡A[A2+(A−A~)2+(A−r~)2]=A~+r~3.(27)A^\star= \underset{A}{\arg\min}\left[A^2 + (A-\widetilde{A})^2 + (A-\widetilde{r})^2\right] = \frac{\widetilde{A}+\widetilde{r}}{3}. \tag*{(27)}

Setting A~=t\widetilde{A} = t and r~=−t\widetilde{r} = -t gives F(A⋆)=0F(A^\star) = 0 but F(A~)=t2F(\widetilde{A}) = t^2. This example only concerns the projected task term. WATO still keeps trajectory imitation through LWAF\mathcal{L}_{\mathrm{WAF}}. The useful question is therefore when a small projection residual is enough to transfer the optimized behavior back to the raw proposal.

Residual-based deployment bounds. For brevity, write dA=dA(A⋆,A~)d_A = d_A(A^\star,\widetilde{A}) and dZ=dZ(Z⋆,Z~)d_Z = d_Z(Z^\star,\widetilde{Z}), where Z⋆=Rθ(zt,A⋆)Z^\star= R_\theta(z_t,A^\star). By definition,

∥A⋆−A~∥2=HdA,∥r(A⋆)−r~∥2=HdZ.(28)\left\|A^\star-\widetilde{A}\right\|_2 = \sqrt{H d_A}, \qquad\left\|r(A^\star)-\widetilde{r}\right\|_2 = \sqrt{H d_Z}. \tag*{(28)}

Suppose A⋆A^\star is an exact interior local solution. Assume that ∇F\nabla F is LFL_F-Lipschitz and rr is LrL_r-Lipschitz on a neighborhood containing the segment between A⋆A^\star and A~\widetilde{A}. Below, matrix norms are spectral norms.

Residual-based deployment transfer.

The task-cost gap satisfies

∣F(A~)−F(A⋆)∣≤(2λA+HLF2)dA+2λZ∥S⋆∥dAdZ.(29)\left|F(\widetilde{A})-F(A^\star)\right| \leq\left(2\lambda_A+\frac{H L_F}{2}\right)d_A + 2\lambda_Z\left\|S_\star\right\|\sqrt{d_A d_Z}. \tag*{(29)}

The inconsistency between the generated future and the rollout of the generated actions satisfies

dZ(Rθ(zt,A~),Z~)≤LrdA+dZ.(30)\sqrt{d_Z\left(R_\theta(z_t,\widetilde{A}),\widetilde{Z}\right)} \leq L_r\sqrt{d_A}+\sqrt{d_Z}. \tag*{(30)}

Proof. The first-order condition in Equation (13) gives

∥∇F(A⋆)∥2≤2H(λAdA+λZ∥S⋆∥dZ).(31)\left\|\nabla F(A^\star)\right\|_2 \leq\frac{2}{\sqrt{H}}\left(\lambda_A\sqrt{d_A}+\lambda_Z\left\|S_\star\right\|\sqrt{d_Z}\right). \tag*{(31)}

Lipschitz continuity of ∇F\nabla F gives

∣F(A~)−F(A⋆)∣≤∥∇F(A⋆)∥2HdA+HLF2dA.(32)\left|F(\widetilde{A})-F(A^\star)\right| \leq\left\|\nabla F(A^\star)\right\|_2\sqrt{H d_A}+\frac{H L_F}{2}d_A. \tag*{(32)}

Substituting Equation (31) proves Equation (29). For the consistency bound, the triangle inequality gives

∥r(A~)−r~∥2≤Lr∥A~−A⋆∥2+∥r(A⋆)−r~∥2.(33)\left\|r(\widetilde{A})-\widetilde{r}\right\|_2 \leq L_r\left\|\widetilde{A}-A^\star\right\|_2+\left\|r(A^\star)-\widetilde{r}\right\|_2. \tag*{(33)}

Dividing by H\sqrt{H} gives Equation (30).

These bounds connect the reference distances in Figure 4 to two deployment quantities: the cost of removing optimization and the world model inconsistency of the generated trajectory. The second statement is with respect to the pretrained world model, not the true environment.

Stationarity and fixed points. Under the same assumptions,

∥∇F(A~)∥2≤2H[(λA+HLF2)dA+λZ∥S⋆∥dZ].(34)\left\|\nabla F(\widetilde{A})\right\|_2 \leq\frac{2}{\sqrt{H}}\left[\left(\lambda_A+\frac{H L_F}{2}\right)\sqrt{d_A}+\lambda_Z\left\|S_\star\right\|\sqrt{d_Z}\right]. \tag*{(34)}

This follows from ∥∇F(A~)∥2≤∥∇F(A⋆)∥2+LF∥A~−A⋆∥2\left\|\nabla F(\widetilde{A})\right\|_2 \leq\left\|\nabla F(A^\star)\right\|_2+L_F\left\|\widetilde{A}-A^\star\right\|_2 and Equation (31). So small residuals also bound the task gradient of the raw proposal.

If A⋆=A~A^\star= \widetilde{A} and r~=r(A~)\widetilde{r} = r(\widetilde{A}), both distances vanish and

∇AJ(Rθ(zt,A~),g)=0.(35)\nabla_A J\left(R_\theta(z_t,\widetilde{A}),g\right)=0. \tag*{(35)}

An interior dynamics-consistent fixed point is task-stationary and not necessarily globally optimal.

Active action bounds. When action bounds are active, the full task gradient need not vanish. Let NUH(A)N_{U_H}(A) denote the normal cone of the action box. Constrained optimality gives 0∈∇AΨ(A⋆;A~,r~)+NUH(A⋆)0 \in\nabla_A\Psi(A^\star;\tilde{A},\tilde{r}) + N_{U_H}(A^\star), which implies

dist⁡(0,∇F(A⋆)+NUH(A⋆))≤2H(λAdA+λZ∥S⋆∥dZ).(36)\operatorname{dist}\left(0,\nabla F(A^\star)+N_{U_H}(A^\star)\right) \le\frac{2}{\sqrt{H}}\left(\lambda_A\sqrt{d_A}+\lambda_Z\|S_\star\|\sqrt{d_Z}\right). \tag*{(36)}

At a dynamics-consistent fixed point, this becomes constrained task stationarity.

For feasible A~\tilde{A} and A⋆A^\star in a neighborhood where FF is CFC_F-Lipschitz, we can still bound the task-cost gap without assuming interior stationarity:

∣F(A~)−F(A⋆)∣≤CFHdA.(37)\left|F(\tilde{A})-F(A^\star)\right|\le C_F\sqrt{H d_A}. \tag*{(37)}

The dynamics-consistency bound in Equation (30) also continues to hold at active bounds, since its proof does not use stationarity.

Inexact optimization. The same picture extends to approximate solves. For an approximate interior solution, let ϵopt=∥∇AΨ(A⋆;A~,r~)∥2\epsilon_{\mathrm{opt}}=\|\nabla_A\Psi(A^\star;\tilde{A},\tilde{r})\|_2, with all distances and Jacobians evaluated at the returned trajectory. The right-hand sides of Equations (31) and (34) then gain ϵopt\epsilon_{\mathrm{opt}}, while Equation (29) gains ϵoptHdA\epsilon_{\mathrm{opt}}\sqrt{H d_A}. For a bound-active solution, Equation (36) similarly gains the corresponding KKT residual dist⁡(0,∇AΨ(A⋆;A~,r~)+NUH(A⋆))\operatorname{dist}(0,\nabla_A\Psi(A^\star;\tilde{A},\tilde{r})+N_{U_H}(A^\star)). The consistency and Lipschitz cost bounds do not require a converged projection.

Candidate selection under model error. Finally, consider the deployed best-of-KK rule. For KK generated candidates and their projected trajectories, let k⋆=arg⁡min⁡kF(A~(k))k^\star=\arg\min_k F(\tilde{A}^{(k)}). Let Fenv(A)F_{\mathrm{env}}(A) denote the terminal task cost obtained by executing AA in the environment, using the same encoder and goal. Suppose ∣Fenv(A)−F(A)∣≤ϵmodel|F_{\mathrm{env}}(A)-F(A)|\le\epsilon_{\mathrm{model}} for every generated and projected action trajectory in this candidate set. Let ηk≥0\eta_k\ge0 satisfy F(A~(k))≤F(A⋆(k))+ηkF(\tilde{A}^{(k)})\le F(A^{\star(k)})+\eta_k, as provided by the preceding task-cost bounds. Then

Fenv(A~(k⋆))≤min⁡k[Fenv(A⋆(k))+ηk]+2ϵmodel≤min⁡kFenv(A⋆(k))+max⁡kηk+2ϵmodel.(38)\begin{aligned} F_{\mathrm{env}}(\tilde{A}^{(k^\star)})\le\min_k\left[F_{\mathrm{env}}(A^{\star(k)})+\eta_k\right]+2\epsilon_{\mathrm{model}} \\ &\le\min_k F_{\mathrm{env}}(A^{\star(k)})+\max_k\eta_k+2\epsilon_{\mathrm{model}}. \tag*{(38)} \end{aligned}

This follows from F(A~(k⋆))≤F(A~(k⋆))≤F(A⋆(k⋆))+ηk⋆F(\tilde{A}^{(k^\star)})\le F(\tilde{A}^{(k^\star)})\le F(A^{\star(k^\star)})+\eta_{k^\star} and the model-error bound on the selected and projected trajectories. The bound separates three effects: projected candidate quality, the cost of removing optimization, and world model error.

These results describe what small projection residuals buy us: WATO does not explicitly minimize them. They are finite-horizon statements and do not imply convergence of WATO training, closed-loop stability, or environment-defined success.

Trajectory Optimization and Differentiation

WATO optimizes only the action trajectory and obtains future states from the world model rollout. The world model parameters remain fixed, while derivatives w.r.t. the action trajectory are retained.

Reduced-Space Optimization

We implement Equation (5) in standardized action coordinates. The action trajectory AA is the only optimization variable, with Z(A)=Rθ(zt,A)Z(A)=\mathcal{R}_{\theta}(z_t,A). We solve

A⋆=arg⁡min⁡A∈UstdHJ(Z(A),g)+λAdA(A,A~)+λZdZ(Z(A),Z~),Z⋆=Z(A⋆),(39)A^\star=\arg\min_{A\in\mathcal{U}_{\mathrm{std}}^H} J(Z(A),g)+\lambda_A d_A(A,\tilde{A})+\lambda_Z d_Z(Z(A),\tilde{Z}), \qquad Z^\star=Z(A^\star), \tag*{(39)}

where Ustd\mathcal{U}_{\mathrm{std}} denotes the action bounds in standardized coordinates. The optimized future Z⋆Z^\star is always produced by the world model.

We use

J(Z(A),g)=∥ZH(A)−g∥22,(40)J(Z(A),g)=\|Z_H(A)-g\|_2^2, \tag*{(40)}

and

dA(A,A~)=1H∑h=0H−1∥Ah−A~h∥2,dZ(Z,Z~)=1H∑h=1H∥NZ(zh)−NZ(z~h)∥2.(41)d_A(A,\widetilde{A}) = \frac{1}{H}\sum_{h=0}^{H-1}\lVert A_h-\widetilde{A}_h\rVert^2,\qquad d_Z(Z,\widetilde{Z}) = \frac{1}{H}\sum_{h=1}^{H}\lVert\mathcal{N}_Z(z_h)-\mathcal{N}_Z(\widetilde{z}_h)\rVert^2. \tag*{(41)}

Primitive actions are standardized using offline-data statistics before forming action blocks. Environment action bounds are transformed with the same statistics and imposed as box constraints. The action reference A~\widetilde{A} uses the same coordinates.

The state normalization is NZ(z)=(z−μZ)/σZ\mathcal{N}_Z(z)=(z-\mu_Z)/\sigma_Z, where μZ\mu_Z and σZ\sigma_Z are computed only from frozen-LeWM latents in the offline training split. This normalization is used only in dZd_Z; the task cost is evaluated in the native pretrained latent space. We use λA=10\lambda_A=10 and λZ=0.1\lambda_Z=0.1 in all experiments.

Numerical Solver

We initialize the primal solve from the clipped generated action reference A~\widetilde{A}. We run L-BFGS-B for at most 200 iterations under the standardized action bounds, followed by up to 25 projected-gradient polishing iterations. Each polishing iteration uses at most 12 backtracking trials.

Because the world model rollout is nonlinear, the optimization problem is generally nonconvex. We require local stationarity using

∥∇projΨ(A⋆;A~,r~)∥∞≤0.02,(42)\left\lVert\nabla_{\mathrm{proj}}\Psi(A^\star;\widetilde{A},\widetilde{r})\right\rVert_\infty\le0.02, \tag*{(42)}

where the projected gradient is taken with respect to the standardized action bounds.

During training, a task sample whose solution fails Equation (42) is rejected and retried with another reserved window and proposal. We allow at most four attempts per update. If none satisfies the stationarity criterion, the task update is aborted.

The trajectory optimizer is used only during WATO training. Deployment uses generated proposals and world model evaluation without L-BFGS-B refinement.

Implicit Differentiation

We differentiate the local solution through its optimality conditions rather than unrolling the primal solver. Let y=(A~,r~)y=(\widetilde{A},\widetilde{r}), where r~=vec⁡NZ(Z~)\widetilde{r}=\operatorname{vec}\mathcal{N}_Z(\widetilde{Z}), and let A⋆(y)A^\star(y) denote a local solution. For an interior regular solution, ∇AΨ(A⋆;y)=0\nabla_A\Psi(A^\star;y)=0. Define H=∇AA2Ψ(A⋆;y)H=\nabla^2_{AA}\Psi(A^\star;y) and B=∂y∇AΨ(A⋆;y)B=\partial_y\nabla_A\Psi(A^\star;y). Differentiating the stationarity condition gives

H∂A⋆∂y+B=0.(43)H\frac{\partial A^\star}{\partial y}+B=0. \tag*{(43)}

We only need the vector-Jacobian product with the task loss. Let

gtask=∇AJ(Rθ(zt,A),g)∣A=A⋆.(44)g_{\mathrm{task}}=\left.\nabla_A J\bigl(R_\theta(z_t,A),g\bigr)\right|_{A=A^\star}. \tag*{(44)}

Rather than forming H−1H^{-1}, we solve

HTv=gtask,∇yJ(Z⋆,g)=−BTv.(45)H^{\mathsf{T}}v=g_{\mathrm{task}},\qquad\nabla_y J(Z^\star,g)=-B^{\mathsf{T}}v. \tag*{(45)}

The resulting reference gradients are propagated through the joint generator to update ϕ\phi.

Let

JR=∂Rθ(zt,A)∂A∣A=A⋆,(46)J_R=\left.\frac{\partial R_\theta(z_t,A)}{\partial A}\right|_{A=A^\star}, \tag*{(46)}

and let DZD_Z denote the block-diagonal linear map induced by NZ\mathcal{N}_Z. The standardized rollout Jacobian is S⋆=DZJRS_\star=D_ZJ_R. The joint WAF directly outputs the normalized state reference r~\widetilde{r}, with task gradient

∇A~J(Z⋆,g)=2λAHv,∇r~J(Z⋆,g)=2λZHDZJRv.(47)\nabla_{\widetilde{A}}J(Z^\star,g)=\frac{2\lambda_A}{H}v,\qquad\nabla_{\widetilde{r}}J(Z^\star,g)=\frac{2\lambda_Z}{H}D_ZJ_Rv. \tag*{(47)}

In native latent coordinates,

∇ZJ(Z∗,g)=DZ⊤∇r~J(Z∗,g)=2λZHDZ⊤DZJRv.(48)\nabla_{Z}J(Z^*,g)=D_Z^\top\nabla_{\tilde{r}}J(Z^*,g)=\frac{2\lambda Z}{H}D_Z^\top D_ZJ_Rv. \tag*{(48)}

Thus, the generated future receives a task gradient through its effect on the optimized actions even though it is not an independent optimization variable.

For a box-constrained solution with a locally fixed active set, we differentiate only through the free action coordinates. Let EfreeE_{\mathrm{free}} embed the free coordinates into the full action vector. The theoretical adjoint system is

(Efree⊤HEfree)⊤vfree=Efree⊤gtask,v=Efreevfree.(49)\left(E_{\mathrm{free}}^\top H E_{\mathrm{free}}\right)^\top v_{\mathrm{free}}=E_{\mathrm{free}}^\top g_{\mathrm{task}},\qquad v=E_{\mathrm{free}}v_{\mathrm{free}}. \tag*{(49)}

This local derivative does not cover points where the active set changes.

In implementation, we solve the damped system

(Efree⊤HEfree+10−3I)⊤vfree=Efree⊤gtask.(50)\left(E_{\mathrm{free}}^\top H E_{\mathrm{free}}+10^{-3}I\right)^\top v_{\mathrm{free}}=E_{\mathrm{free}}^\top g_{\mathrm{task}}. \tag*{(50)}

We compute Hessian-vector products through the frozen world model and use conjugate gradients for at most 16 iterations with tolerance 10−410^{-4}. If conjugate gradients fails, we fall back to a dense solve and, when needed, a pseudoinverse. Damping and finite solver tolerance make the backward pass an approximation to the local sensitivity.

The world model parameters θ\theta are never updated by this gradient. Derivatives through RθR_\theta with respect to the action trajectory are retained.

World Model and WAF Implementation

We specify the pretrained world model, joint world-action generator, and WATO training procedure. The world model and joint WAF are trained before WATO. WATO updates only the generator parameters ϕ\phi and the world model parameters θ\theta remain fixed.

World Model

We use latent world models following LeWM [24]. Each model consists of a visual encoder EθE_\theta and an action-conditioned predictor FθF_\theta trained jointly end-to-end. The encoder maps an observation to a latent state, while the predictor takes an action block AhA_h and a finite latent context ChC_h and predicts the next latent state. Autoregressive prediction gives the rollout RθR_\theta defined in Equation (1).

The released LeWM architecture uses 224 ×\times 224 visual inputs, a ViT-Tiny encoder with patch size 14, and a six-layer action-conditioned Transformer predictor. For released checkpoints, five primitive actions are grouped into each action block. The predictor uses a finite latent context whose length depends on the environment.

LeWM is trained with next-latent prediction and SIGReg. The prediction loss is

Lpred=ED[MSE⁡(z^h+1,zh+1)],(51)\mathcal{L}_{\mathrm{pred}}=\mathbb{E}_{D}\left[\operatorname{MSE}\left(\hat{z}_{h+1},z_{h+1}\right)\right], \tag*{(51)}

and the full objective is LWM=Lpred+λsigLSIG\mathcal{L}_{\mathrm{WM}}=\mathcal{L}_{\mathrm{pred}}+\lambda_{\mathrm{sig}}\mathcal{L}_{\mathrm{SIG}}. Following LeWM, we use 1024 random projections for SIGReg and λsig=0.1\lambda_{\mathrm{sig}}=0.1.

We use released LeWM checkpoints when available. For environments without one, we train the same LeWM architecture and objective from scratch. We retain the default temporal abstraction of five primitive actions per latent transition and a maximum context of three latent states. For models trained in this work, the training entry reports the optimizer step of the selected validation checkpoint. Table 5 summarizes the configurations.

EnvironmentSourceBBlcl_cSelected checkpoint
TwoRoomsReleased LeWM51Released
ReacherReleased LeWM53Released
PushTReleased LeWM53Released
cube-singleReleased LeWM53Released
cube-doubleTrained in this work5392,052 updates
scene-playTrained in this work5376,252 updates
puzzle-3x3Trained in this work5368,724 updates

Table 5. World model configurations. BB is the number of primitive actions per latent transition and lcl_c is the maximum latent context length.

For world models trained in this work, we use AdamW with a learning rate of 5×10−55\times10^{-5}, weight decay of 10−310^{-3}, and batch size 128. We train for up to 12 epochs (approximately 92k optimizer updates) with a 1% linear warmup followed by cosine decay to zero, and clip the global gradient norm to 1.0. We select the checkpoint with the lowest teacher-forced validation loss, defined as prediction MSE plus 0.09 times the SIGReg loss.

All world models are trained before WATO. During WATO, gradients through EθE_\theta, FθF_\theta, and RθR_\theta with respect to their inputs are retained, but θ\theta is never updated.

The joint WAF models X=(A,Z)X=(A,Z), where A=(A0,…,AH−1)A=(A_0,\ldots,A_{H-1}) and Z=(z1,…,zH)Z=(z_1,\ldots,z_H), conditioned on the current and goal latents c=(zt,g)c=(z_t,g). A single conditional flow generates the action and future-state trajectories jointly rather than autoregressively over the horizon.

The vector field vϕv_\phi is parameterized by a Transformer with model width 256, 4 Transformer layers, 8 attention heads, feed-forward dimension 1024, no dropout, and a 64-dimensional flow-time embedding. Action and latent variables are projected to the common model width, processed jointly, and mapped back to their respective output spaces. The model has approximately 3.5M parameters.

Primitive actions are standardized before action blocks are formed. Future state targets are obtained by encoding dataset observations with the pretrained encoder. For each minibatch, we sample a horizon-HH trajectory window, form X=(A,Z)X=(A,Z), and train with the joint WAF objective in Equation (3). We use λWAF=1\lambda_{\mathrm{WAF}}=1, batch size 512, and 50,000 pretraining updates.

At generation time, Gaussian noise with the same structure as XX is integrated from flow time 0 to 1. We use Euler integration with NFE = 8. All action and state variables are updated jointly at each flow step. Unless otherwise stated, deployment draws K=8K=8 independent candidates.

WATO Training

WATO fine-tunes the pretrained joint WAF using the objective in Equation (9). The flow matching term preserves trajectory imitation, while the task term differentiates through the trajectory-optimization layer. Only the WAF parameters ϕ\phi are updated. The world model remains fixed.

We use AdamW with a learning rate of 5×10−55 \times10^{-5}, weight decay 10−410^{-4}, and a flow matching batch size of 64 for 10,000 WATO updates. The task term uses one projected trajectory per update with λtask=5×10−3\lambda_{\mathrm{task}}=5 \times10^{-3}. For each update, trajectories for the task term are generated with Euler integration using NFE = 2 and passed through the trajectory projection in Equation (4). We draw up to four proposals to obtain a stationarity-qualified trajectory. Only solutions satisfying the criterion in Equation (42) contribute to the task gradient.

We clip the global generator gradient norm to 1. The learning rate uses a linear warmup over the first 100 updates followed by cosine decay to zero. We select the final checkpoint using the minimum fixed-validation objective, LWAF+λtaskLtask\mathcal{L}_{\mathrm{WAF}}+\lambda_{\mathrm{task}}\mathcal{L}_{\mathrm{task}}.

Training Cost and Solver Statistics

WATO adds training-time computation because each task update solves a trajectory optimization problem and differentiates through its solution. Table 7 reports statistics from the full 10,000-update PushT run. All wall-clock times are measured on a single NVIDIA RTX Pro 6000 Blackwell GPU.

Training costSolver workloadSolver reliability
MetricValueMetricValueMetricValue
Joint WAF pretraining1.76 hMean L-BFGS-B iter.90Optimizer updates10,000
WATO training5.82 hMedian L-BFGS-B iter.84Task-solve attempts10,005
Total pipeline7.58 hInterquartile range53–116Rejected samples5
WATO / Joint WAF3.31×Maximum L-BFGS-B iter.200First-attempt success99.95%
Pipeline / Joint WAF-only4.31×Maximum polish iter.25Success within ≤4\leq4100%
FM-only continuation0.65 hTotal L-BFGS-B iter.8.8×1058.8 \times10^{5}Aborted updates0
WATO / FM-only9.0×Aborted-update rate0%
Direct rollout0.9–1.5 h
WATO / direct rollout3.9–6.5×

Table 7. Training cost and solver statistics on PushT. Statistics are from the full 10,000-update WATO run on a single GPU.

WATO fine-tuning takes 5.82 hours, compared with 1.76 hours for joint WAF pretraining. The primal solver uses 90 L-BFGS-B iterations on average, but the stationarity filter rarely rejects a solve: only 5 of 10,005 attempts are rejected, and no training update is aborted. The additional computation is confined to training; deployment uses generation, world model evaluation, and best-of-KK selection without iterative trajectory refinement.

SettingJoint WAF trainingWATO training
OptimizerAdamWAdamW
Weight decay10−410^{-4}10−410^{-4}
Learning rate3×10−43 \times10^{-4}5×10−55 \times10^{-5}
Batch size51264 (FM), 1 (task)
Updates50,00010,000
Warmup500 updates100 updates
Learning-rate scheduleCosine decayCosine decay
Task-loss flow integration–Euler, NFE = 2
Deployment flow integrationEuler, NFE = 8Euler, NFE = 8
λWAF\lambda_{\mathrm{WAF}}11
λtask\lambda_{\mathrm{task}}–5×10−35 \times10^{-3}
Gradient-norm clipping11
Checkpoint criterionMinimum validation FM lossMinimum validation weighted objective

Table 6. Joint WAF training and WATO training configuration. NFE = 2 is used to generate trajectories for the task loss during training, whereas all controllers use NFE = 8 at deployment.

Experimental Protocol

This section specifies goal construction, baseline training, environment-specific configurations, and compute measurement. Unless stated otherwise, all methods use the same offline data, evaluation goals, execution budget, and pretrained visual representation. Figure 5 shows representative observations from the seven evaluation environments.

Figure 5. Environments for WATO experiments

Goal Construction and Evaluation

We evaluate local visual goal reaching from evaluation trajectories. For an initial observation oto_t, the goal is the observation ot+Lo_{t+L} from the same trajectory, with L=25L = 25 environment steps. All evaluation episodes are nontrivial: the initial state never satisfies the goal criterion. Only pairs with at least LL remaining steps are used. This defines the same goal distribution across all methods.

Each episode has a 50-step execution budget. A controller repeatedly observes the current image, conditions on the fixed goal image, executes its selected action segment, and replans until success or the budget is exhausted. All methods use the same environment-specific success criterion.

Success rate is the primary metric. We also report the final latent goal error ∥Eθ(oT)−g∥2\lVert E_{\theta}(o_T) - g \rVert_2, where oTo_T is the final observation. All results use 4 seeds and identical start-goal pairs across methods.

Baseline Training

All baselines are trained on the same offline dataset and goal distribution as WATO. Evaluation uses identical start-goal pairs and the same execution budget. All methods use the same pretrained LeWM encoder to separate control performance from representation quality. Each baseline otherwise keeps its original objective, goal-relabeling procedure, and training scheme.

Goal-conditioned learning. GCBC ([8]) learns a goal-conditioned policy by behavioral cloning. GCIVL ([20]) and GCIQL ([28]) learn goal-conditioned value functions and policies from the same offline data. QRL [37] learns goal-conditioned control through its quasimetric objective. All four methods are trained from scratch for this goal distribution rather than evaluated with published final-goal checkpoints.

Generative control. The Flow policy generates only the action trajectory AA and serves as the action-only generative baseline. Joint WAF uses the joint generator qϕ(A,Z∣zt,g)q_{\phi}(A, Z \mid z_t, g) trained only with the flow matching objective in Equation (3). It isolates the effect of WATO training from joint trajectory imitation. For GPC [21], we adapt the method to the same offline setting and use the same pretrained LeWM dynamics model as WATO. All generative methods use the same goals and execution budget.

Model-based control. CEM [24, 18] optimizes action trajectories online using the same pretrained world model and terminal task cost JJ as WATO. It uses a population of 300 trajectories, retains 30 elites (an elite fraction of 0.1), and performs 30 optimization rounds for PushT and Reacher and 10 rounds for the OGBench manipulation environments. The initial standard deviation is 1, no warm start is used, and the final Gaussian mean is returned as the action plan. Gradient-based MPC uses 8 independently initialized action trajectories, 40 gradient steps, and a learning rate of 0.05. Neither method updates the world model during evaluation.

We use the released LeWM CEM hyperparameters without tuning. Hyperparameters for learned baselines are selected on validation tasks and then fixed for test evaluation.

Hyperparameters

We use the same WATO configuration across all environments. The planning horizon is H=5H = 5 latent transitions with B=5B = 5 primitive actions per block. At deployment, WATO generates K=8K = 8 candidates using Euler integration with NFE =8= 8. We set λA=10\lambda_A = 10, λZ=0.1\lambda_Z = 0.1, and λtask=5×10−3\lambda_{\text{task}} = 5 \times10^{-3}. The trajectory optimizer uses at most 200 L-BFGS-B iterations during WATO training and is not used at deployment. Ablations change only the component specified by each experiment.

Success Criteria

We evaluate success using environment-specific semantic state criteria rather than the latent-space terminal cost used for planning. The latent cost ranks candidate action trajectories, while success is computed from simulator state. Table 8 summarizes the criterion for each environment.

EnvironmentSuccess criterion
TwoRoomsThe Euclidean distance between the agent position and target position is less than 16 coordinate units.
ReacherThe maximum absolute joint-position error over the two target joints is less than 0.05 radians.
PushTThe positional error over the first four task-state coordinates is less than 20, and the wrapped block-orientation error is less than π/9\pi/9 radians.
cube-singleThe Euclidean distance between the cube position and target position is at most 0.04 meters.
cube-doubleBoth cubes are within 0.04 meters of their respective targets, i.e., max⁡i∈{1,2}∥pi−pig∥2≤0.04\max_{i\in\{1,2\}}\lVert p_i-p_i^g\rVert_2 \le0.04.
scene-playThe cube-position error is at most 0.04 meters, the absolute drawer and window position errors are each at most 0.04, and every discrete button state matches its target state. All conditions must hold simultaneously.
puzzle-3x3All nine binary button states match their target states, i.e., ∑i=191[si≠sig]=0\sum_{i=1}^{9}\mathbf{1}[s_i \ne s_i^g] = 0.

Table 8. Environment-specific semantic success criteria. These criteria use privileged simulator state and are distinct from the pretrained world model’s latent-space planning cost.

An episode is successful if its criterion is satisfied at least once during execution. WATO and the other WAF-based controllers terminate immediately upon success. The released Reacher CEM parity evaluator is an exception: it continues until the full episode budget is exhausted but keeps the first observed success for evaluation.

Full Results

This section reports the per-environment results underlying the aggregate comparisons in the main paper. We provide the full WATO ablations, the absolute success rates used to measure optimization amortization, and final goal errors.

Full WATO ablations

Table 9 reports the complete training-objective and reference-design ablations. All variants use the same joint WAF architecture, offline data, pretrained world model, and evaluation protocol.

EnvironmentJoint WAFDirectDirect + ConsistencyDistillationAction onlyState onlyRollout-derived referenceWATO
TwoRooms83±183 \pm185±185 \pm187±287 \pm290±090 \pm086±286 \pm288±388 \pm388±388 \pm393±0\mathbf{93 \pm0}
Reacher67±367 \pm377±577 \pm579±579 \pm581±481 \pm479±379 \pm376±476 \pm481±281 \pm285±2\mathbf{85 \pm2}
PushT84±484 \pm494±394 \pm395±395 \pm393±293 \pm295±395 \pm391±491 \pm496±496 \pm4100±0\mathbf{100 \pm0}
cube-single88±188 \pm196±296 \pm297±197 \pm196±196 \pm196±296 \pm294±294 \pm297±197 \pm199±1\mathbf{99 \pm1}
cube-double89±289 \pm287±387 \pm388±388 \pm386±286 \pm284±384 \pm385±385 \pm387±587 \pm590±3\mathbf{90 \pm3}
scene-play78±378 \pm390±490 \pm491±391 \pm391±391 \pm387±487 \pm489±489 \pm490±390 \pm394±1\mathbf{94 \pm1}
puzzle-3x370±370 \pm371±471 \pm475±275 \pm279±379 \pm373±473 \pm472±472 \pm476±276 \pm281±2\mathbf{81 \pm2}
Average79.985.787.488.085.785.087.991.7

Table 9. Full WATO ablation results. Results are success rates (%, mean ±\pm std over 4 seeds).

The training-objective ablation isolates how trajectory optimization enters learning. Joint WAF uses flow matching alone. Direct rollout adds task supervision through the world model, direct rollout with consistency additionally aligns generated states with the rollout, and distillation uses optimized trajectories as detached targets. WATO achieves the highest average success rate, improving from 79.9% for Joint WAF to 91.7%.

The reference-design ablation isolates the two components of the world-action trajectory. Using only the action reference achieves 85.7%, while using only the state reference achieves 85.0%. Their combination reaches 91.7%, showing that neither reference alone recovers full WATO performance.

Removing dZd_Z eliminates both its dynamics-aware regularization and the generated future reference. To separate these effects, the rollout-derived variant replaces Z~\widetilde{Z} with Rθ(zt,A~)R_{\theta}(z_t,\widetilde{A}) while retaining both reference penalties. The backward pass includes the dependence of this reference on A~\widetilde{A}. This retains dynamics-aware regularization without using the generated future as a control reference, as discussed in Appendix § 11.§ 2.

The average success rate of 87.9% place this variant 2.2 pp above the action-reference-only variant and 3.8 points below WATO. This indicate that dynamics-aware regularization explains part, but not all, of the state-reference benefit. The remaining gap support using generated futures to guide optimization beyond the consequences already predicted for the proposed actions. Conversely, performance comparable to WATO suggest that rollout-derived references capture most of this benefit.

Training Horizon and Test Goal Distance

The main experiments use a training horizon Ltrain=25L_{\mathrm{train}} = 25 and test goals D=25D = 25 primitive steps ahead. We additionally vary Ltrain∈{10,25,50,100}L_{\mathrm{train}} \in\{10,25,50,100\} and D∈{10,25,50,100}D \in\{10,25,50,100\} on PushT, cube-single, and scene-play. For each goal distance DD, the execution budget is 2D2D steps. All other settings are unchanged.

These horizons correspond to physical durations. In PushT, one environment step is 0.1 s, so Ltrain={10,25,50,100}L_{\mathrm{train}} = \{10, 25, 50, 100\} corresponds to {1,2.5,5,10}\{1, 2.5, 5, 10\} s. In cube-single and scene-play, one step is 0.05 s, giving {0.5,1.25,2.5,5}\{0.5, 1.25, 2.5, 5\} s. The execution budget is twice the test-goal duration.

Table 10 shows that WATO is not limited to short local control. The best training horizon generally grows with the test goal distance, although exact matching is not always optimal. For example, Ltrain=50L_{\mathrm{train}} = 50 achieves 82% success on PushT at D=50D = 50, corresponding to a 5 s trajectory. An average PushT episode spans roughly 125 steps, or 12.5 s, so a 50-step WATO horizon already covers about 40% of a full episode. A complete task therefore spans only about two to three such horizons, leaving less work for a separate high-level planner than a purely local-controller view would suggest.

EnvironmentLtrain\DL_{\mathrm{train}}\backslash D102550100
PushT10100805216
PushT25741007030
PushT5030668248
PushT1008245040
cube-single10100683210
cube-single2570944024
cube-single5024589244
cube-single1005184258
scene-play109764286
scene-play2568995014
scene-play5020557632
scene-play1004143552

Table 10. Training horizon and test goal distance. WATO success rates (%) for different training horizons LtrainL_{\mathrm{train}} and test goal distances DD, measured in primitive environment steps. Bold indicates the best training horizon for each environment and test goal distance.

Performance still decreases as the temporal scale grows. At D=100D = 100, the best success rates are 48%, 58%, and 52% on PushT, cube-single, and scene-play, respectively. Longer tasks with more objects, stages, or distant goals may therefore benefit from hierarchy or high-level planning. Such a planner, however, can operate over multi-second WATO controllers rather than primitive local actions. These results further support the perspective in Appendix § 10: WATO is a strong default controller over a broader temporal range than the main 25-step setting alone suggests.

Full Optimization-Amortization Results

The main paper reports the additional success obtained by applying trajectory optimization before and after WATO training. Table 11 reports the corresponding absolute success rates.

EnvironmentBefore WATO: Joint WAFBefore WATO: + Traj. Opt.After WATO: WATOAfter WATO: + Traj. Opt.
TwoRooms83919394
Reacher67828588
PushT8498100100
cube-single889799100
cube-double89929091
scene-play78929495
puzzle-3x370808183
Average79.990.391.793.0

Table 11. Absolute performance before and after trajectory optimization. The table expands the pp gains reported in Table 2. Results are success rates (%) from 100 tests.

Before WATO, trajectory optimization raises average success from 79.9% to 90.3%. After WATO, the same optimizer raises success only from 91.7% to 93.0%. Thus, most of the performance previously obtained through online optimization is already present in the WATO generator.

The absolute results also separate optimization quality from amortization. Online trajectory optimization is already strong when applied to the imitation-trained generator, reaching 90.3% average success. WATO reaches 91.7% without this online optimization, while an additional optimization step provides only a small remaining gain. This supports the interpretation that WATO transfers the useful effect of trajectory optimization into generation rather than relying on a weak optimizer.

Final Goal Error

Success is a thresholded measure and does not distinguish failures that terminate near the goal from those that remain far away. We therefore also report the final latent goal error ∥Eθ(oT)−g∥2\lVert E_\theta(o_T) - g \rVert_2, where oTo_T is the final observation after the execution budget. Lower values indicate trajectories that terminate closer to the target.

Table 12 reports the metric for the same methods and start-goal pairs as Table 1. We report results separately for each environment because latent scales need not be comparable across independently trained world models.

EnvironmentGCBCGCIVLGCIQLQRLFlow Policy
TwoRooms12.7±4.312.7 \pm4.313.6±4.713.6 \pm4.712.9±4.912.9 \pm4.911.9±4.211.9 \pm4.213.1±4.413.1 \pm4.4
Reacher124.3±24.8124.3 \pm24.8114.5±35.2114.5 \pm35.2125.0±15.1125.0 \pm15.1149.5±24.8149.5 \pm24.8162.8±35.7162.8 \pm35.7
PushT26.3±4.826.3 \pm4.833.0±6.233.0 \pm6.235.4±6.335.4 \pm6.328.6±5.228.6 \pm5.223.6±4.523.6 \pm4.5
cube-single14.0±4.214.0 \pm4.223.0±6.423.0 \pm6.420.5±5.620.5 \pm5.614.7±4.914.7 \pm4.912.7±4.512.7 \pm4.5
cube-double17.1±4.917.1 \pm4.934.1±8.734.1 \pm8.726.2±7.426.2 \pm7.418.0±5.118.0 \pm5.117.4±5.617.4 \pm5.6
scene-play3.5±1.63.5 \pm1.64.5±2.24.5 \pm2.25.0±1.15.0 \pm1.13.6±1.03.6 \pm1.03.7±1.13.7 \pm1.1
puzzle-3x320.8±5.820.8 \pm5.821.5±7.121.5 \pm7.126.2±7.626.2 \pm7.623.0±6.023.0 \pm6.019.8±5.819.8 \pm5.8
EnvironmentJoint WAFGPCCEMGrad. MPCWATO
TwoRooms12.3±4.012.3 \pm4.011.5±4.111.5 \pm4.165.5±14.365.5 \pm14.310.6±4.110.6 \pm4.110.4±3.510.4 \pm3.5
Reacher116.0±5.1116.0 \pm5.1110.8±4.0110.8 \pm4.0285.3±17.7285.3 \pm17.7112.7±4.7112.7 \pm4.792.9±4.292.9 \pm4.2
PushT22.1±4.622.1 \pm4.629.5±3.529.5 \pm3.570.0±14.970.0 \pm14.918.8±4.018.8 \pm4.09.3±3.39.3 \pm3.3
cube-single11.3±3.911.3 \pm3.913.4±4.613.4 \pm4.688.1±19.488.1 \pm19.413.8±4.513.8 \pm4.59.5±3.59.5 \pm3.5
cube-double11.1±4.011.1 \pm4.016.8±6.116.8 \pm6.193.9±21.493.9 \pm21.415.0±4.515.0 \pm4.511.0±4.211.0 \pm4.2
scene-play4.4±1.64.4 \pm1.64.2±2.84.2 \pm2.825.2±8.725.2 \pm8.75.3±2.95.3 \pm2.92.3±1.72.3 \pm1.7
puzzle-3x315.2±5.015.2 \pm5.025.4±7.225.4 \pm7.2120.5±24.6120.5 \pm24.617.7±5.217.7 \pm5.212.7±4.312.7 \pm4.3

Table 12. Final latent goal error. Lower is better. Results use the same start-goal pairs and execution budget as the success-rate evaluation.

Final goal error complements success rate but does not replace it. Success remains the primary metric because the evaluation criterion is defined in the environment, while latent distance measures how closely the final observation matches the goal under the shared representation.

Deployment Efficiency Details

This section expands the deployment-efficiency comparison in Sec. 5.8. We report per-environment latency, define the world model computation count, and study the effect of WATO sampling budget on success and deployment cost.

Per-Environment Latency

We measure the wall-clock cost of one planning step. WATO includes joint trajectory generation, world model evaluation of all generated action candidates, candidate selection, and action extraction. CEM includes all online optimization iterations and world model rollouts.

Table 13 reports absolute deployment latency in seconds. The corresponding relative speedups are reported in Table 3. Absolute latency varies across environments, so we report per-environment values rather than averaging raw times.

EnvironmentCEM (s)WATO (s)
TwoRooms1.9090.101
Reacher1.9990.147
PushT2.4000.096
cube-single1.2010.207
cube-double1.3990.222
scene-play1.2930.106
puzzle-3x31.9040.207

Table 13. Per-environment deployment latency. Wall-clock time in seconds per planning step.

World Model Evaluations

Wall-clock latency depends on hardware and batching, so we also report world model computation directly. We count one world model evaluation as one latent transition Fθ(Ch,Ah)F_{\theta}(C_h, A_h) for one candidate trajectory. A full horizon-HH rollout therefore uses HH evaluations.

WATO generates KK action candidates and rolls each candidate through the world model once. Its cost per planning step is therefore

NWMWATO=KH.(52)N_{\mathrm{WM}}^{\mathrm{WATO}} = KH. \tag*{(52)}

The flow matching-only joint WAF uses the same deployment procedure and therefore has the same world model count,

NWMJoint WAF=KH.(53)N_{\mathrm{WM}}^{\mathrm{Joint\ WAF}} = KH. \tag*{(53)}

Let NCEMN_{\mathrm{CEM}} denote the number of CEM iterations and MCEMM_{\mathrm{CEM}} the population size. Each iteration evaluates all candidates over the full horizon, giving

NWMCEM=NCEMMCEMH.(54)N_{\mathrm{WM}}^{\mathrm{CEM}} = N_{\mathrm{CEM}}M_{\mathrm{CEM}}H. \tag*{(54)}

For gradient-based MPC, let NgradN_{\mathrm{grad}} be the number of optimization steps and MinitM_{\mathrm{init}} the number of parallel initial action trajectories. Each optimization step requires a forward rollout through the horizon, so

NWMGrad. MPC=NgradMinitH.(55)N_{\mathrm{WM}}^{\mathrm{Grad.\ MPC}} = N_{\mathrm{grad}}M_{\mathrm{init}}H. \tag*{(55)}

Backward differentiation is included in wall-clock latency but not counted as an additional forward world model evaluation.

Because all compared methods use the same horizon HH, the relative forward-model cost is independent of HH. For example, the CEM-to-WATO ratio is

NWMCEMNWMWATO=NCEMMCEMK.(56)\frac{N_{\mathrm{WM}}^{\mathrm{CEM}}}{N_{\mathrm{WM}}^{\mathrm{WATO}}} = \frac{N_{\mathrm{CEM}}M_{\mathrm{CEM}}}{K}. \tag*{(56)}

This count is independent of batching: evaluating KK candidates in one batched forward pass still counts as KHKH latent transitions.

MethodCountRelative to WATO
Joint WAFKHKH1×1\times
CEMNCEMMCEMHN_{\mathrm{CEM}}M_{\mathrm{CEM}}HNCEMMCEMK\frac{N_{\mathrm{CEM}}M_{\mathrm{CEM}}}{K}
Grad. MPCNgradMinitHN_{\mathrm{grad}}M_{\mathrm{init}}HNgradMinitK\frac{N_{\mathrm{grad}}M_{\mathrm{init}}}{K}
WATOKHKH1×1\times

Table 14. World model computation per planning step. Counts are latent transition evaluations.

Success-Compute Tradeoff

WATO exposes two deployment-time compute parameters. The candidate count KK controls how many generated action trajectories are evaluated by the world model, while NFE controls the cost of flow integration. Increasing either can improve candidate quality or coverage at additional compute.

We evaluate K∈{1,4,8,16}K \in\{1, 4, 8, 16\} with NFE fixed at 8, and NFE ∈{4,8,16}\in\{4, 8, 16\} with KK fixed at 8. All other settings remain unchanged. Figure 6 reports average success across the 7 environments against deployment latency, with the number of world model evaluations shown for each setting.

Figure 6. WATO success-compute tradeoff. Left: varying the number of generated candidates KK with NFE =8= 8. Right: varying flow integration steps with K=8K = 8. The blue marker denotes the default setting used in the main experiments.

Increasing KK gives a clear gain from one to eight candidates, while the improvement from K=8K = 8 to K=16K = 16 is small despite the additional world model evaluations and latency. Increasing NFE shows a similar pattern: NFE =8= 8 captures most of the gain, with little improvement from doubling the integration steps to 16. We therefore use K=8K = 8 and NFE =8= 8 as the default, which lies near the knee of both tradeoffs.

Call for Research

WATO leaves further questions: what should we learn to do quickly, and what should we still plan online? A strong local controller lets us ask where reasoning, value learning, and additional computation are actually most useful.

A strong controller is only the beginning. Long-horizon goal reaching has benefited from hierarchical learning ([26, 28, 19]), but reliably composing local behaviors from offline visual data remains open. Reaching waypoints is not enough if the resulting state makes the next transition difficult. A high-level plan must choose reachable goals, leave the controller in useful states, and recover when execution deviates from the plan. This is more than simply increasing the horizon. We need work that local execution, subgoal selection, and their composition.

World action models beyond scale. WAM research need not be only about building larger foundation models. As discussed in Appendix § 9, joint WAMs also form a generative control model class whose objectives, references, and deployment rules deserve study. How should generated futures influence actions? When does joint generation help beyond an equally expressive action-only policy? Which control principles transfer across architectures and scales? Large-scale pretraining and control optimization address different questions. Better generation and better control need not come from the same design choices.

Put values and broad knowledge where they help most. Our results also raise a question about where value learning belongs. If local trajectories can already be generated reliably, values may be more useful for selecting subgoals and stitching behaviors than for extracting every motor action. Hierarchical goal-conditioned learning provides a natural starting point ([26]). High-level values should also reflect what the deployed low-level controller can actually execute, including its failures, rather than only what transitions appear in the dataset.

Large foundation models may similarly be most useful for semantic reasoning, task decomposition, and goal selection, while a dynamics-aware controller handles local execution. SayCan illustrates how broad knowledge can be grounded through executable skills and their values ([1]). At the same time, pretrained WAMs can directly generate low-level control ([40]). The question is therefore not only how much to scale, but where to put that scale.

Act fast, deliberate when needed. Thinking fast and slow [16] suggests allocating computation according to the decision. A familiar subgoal may need only one forward pass, while difficult contact may justify extra sampling or optimization. A useful extension of WATO would decide when direct generation is enough and when to refine a trajectory, increase sampling, or reconsider the goal. Progress, uncertainty, or disagreement between generated and predicted futures could provide such signals, but they should be validated against execution outcomes. The goal is to spend extra computation when it actually improves control.

Will more compute solve planning, or exploit model error? Faster hardware, GPU parallelism, and better solvers may make online trajectory optimization attractive. Systems such as cuRobo already show large gains from parallel optimization [35]. CEM is also not simply brute-force search: variants such as iCEM use temporal correlations and memory to improve sampling efficiency [30]. Future planners may further use learned proposals or references, erasing the distinction between online optimization and learned generation.

The harder question is whether computation or model error becomes the bottleneck. More search can reduce predicted cost without making the world model more accurate, and may even find trajectories that exploit prediction errors [41, 17]. WATO’s imitation loss and generated references guide optimization, but they do not guarantee model accuracy either. A generator can also learn model exploitation. Optimization error and model error therefore matter for both online planning and amortized control, as reflected in Appendix § 11.§ 4.

For local visual goals, WATO should be your favorite controller. The next challenges are:

where to go • when to think harder • how to leverage local abilities in a long horizon.

References

  1. [1]Michael Ahn, Anthony Brohan, Noah Brown, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.arxiv.org/abs/2204.01691
  2. [2]Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2211.15657
  3. [3]Brandon Amos and Denis Yarats. The differentiable cross-entropy method. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 291–302. PMLR, 2020.arxiv.org/abs/1909.12830
  4. [4]Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J. Zico Kolter. Differentiable MPC for end-to-end planning and control. In Advances in Neural Information Processing Systems, volume 31, pp. 8289–8300, 2018.arxiv.org/abs/1810.13400
  5. [5]Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, Deli Zhao, and Hao Chen. WorldVLA: Towards Autoregressive Action World Model, 2025.arxiv.org/abs/2506.21539
  6. [6]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin CM Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, 2023.
  7. [7]Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1805.12114
  8. [8]Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1906.05838
  9. [9]Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021.
  10. [10]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.arxiv.org/abs/1912.01603
  11. [11]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.arxiv.org/abs/1912.01603
  12. [12]Nick Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. In International Conference on Learning Representations, volume 2024, pp. 47376–47405, 2024.arxiv.org/abs/2310.16828
  13. [13]Hsiang-Wei Huang, Jianxu Shangguan, Junbin Lu, and Jenq-Neng Hwang. Leflow: Generative latent flow planning for world models. arXiv preprint arXiv:2608.24855, 2026.arxiv.org/abs/2608.24855
  14. [14]Simon Idoko, Prajyot Jadhav, and Arun Kumar Singh. Flow-Opt: Scalable centralized multi-robot trajectory optimization with flow matching and differentiable optimization. IEEE Transactions on Automation Science and Engineering, 2026.arxiv.org/abs/2510.09204
  15. [15]Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9902–9915. PMLR, 2022.arxiv.org/abs/2205.09991
  16. [16]Daniel Kahneman, Patrick Egan, et al. Thinking, fast and slow, volume 1. Farrar, Straus and Giroux New York, 2011.
  17. [17]Mintae Kim and Koushil Sreenath. In-distribution imagination for model-based offline reinforcement learning. arXiv preprint arXiv:2609.38673, 2026a.arxiv.org/abs/2609.38673
  18. [18]Mintae Kim and Koushil Sreenath. Wombet: World model-based experience transfer for robust and sample-efficient reinforcement learning. In Annual Learning for Dynamics and Control Conference, pp. 1121–1133. PMLR, 2026b.arxiv.org/abs/2604.08958
  19. [19]Mintae Kim, Jiaze Cai, and Koushil Sreenath. Roverfly: Robust and versatile implicit hybrid control of quadrotor-payload systems. arXiv preprint arXiv:2509.11149, 2025.arxiv.org/abs/2509.11149
  20. [20]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. 2022.arxiv.org/abs/2110.06169
  21. [21]Vince Kurtz and Joel W. Burdick. Generative predictive control: Flow matching policies for dynamic and difficult-to-demonstrate tasks. In 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.
  22. [22]Zeyang Li, Kaveh Alim, and Navid Azizan. Hardflow: Hard-constrained sampling for flow-matching models via trajectory optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.DOI
  23. [23]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
  24. [24]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. LeWorld-Model: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels, 2026.DOI
  25. [25]Kazuki Mizuta and Karen Leung. Unified generation-refinement planning: Bridging guided flow matching and sampling-based MPC for social navigation. In 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.arxiv.org/abs/2508.01192
  26. [26]Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36:34866–34891, 2023.arxiv.org/abs/2307.11949
  27. [27]Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline RL? In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2406.09329
  28. [28]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, volume 2025, pp. 94937–94982, 2025a.arxiv.org/abs/2410.20092
  29. [29]Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning (ICML), 2025b.arxiv.org/abs/2502.02538
  30. [30]Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. arXiv preprint arXiv:2008.06389, 2020.arxiv.org/abs/2008.06389
  31. [31]Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, and Georg Martius. Extracting strong policies for robotics tasks from zero-order trajectory optimizers. In International Conference on Learning Representations, 2021.
  32. [32]Thomas Power and Dmitry Berenson. Variational inference MPC using normalizing flows and out-of-distribution projection. In Proceedings of Robotics: Science and Systems, 2022.arxiv.org/abs/2205.04667
  33. [33]Ralf Römer, Alexander von Rohr, and Angela P Schoellig. Diffusion predictive control with constraints. arXiv preprint arXiv:2412.09342, 2024.arxiv.org/abs/2412.09342
  34. [34]Vlad Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestriero, Tim G. J. Rudner, and Yann LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2502.14819
  35. [35]Balakumar Sundaralingam, Siva Kumar Sastry Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, Nathan Ratliff, and Dieter Fox. cuRobo: Parallelized collision-free minimum-jerk robot motion generation. arXiv preprint arXiv:2310.17274, 2023.arxiv.org/abs/2310.17274
  36. [36]Weikang Wan, Ziyu Wang, Yufei Wang, Zackory Erickson, and David Held. DiffTORI: Differentiable trajectory optimization for deep reinforcement and imitation learning. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2402.05421
  37. [37]Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pp. 36411–36430. PMLR, 2023.arxiv.org/abs/2304.01203
  38. [38]Zhengtong Xu and Yu She. LeTO: Learning constrained visuomotor policy with differentiable trajectory optimization. IEEE Transactions on Automation Science and Engineering, 2024.arxiv.org/abs/2401.17500
  39. [39]Zhengtong Xu, Zichen Miao, Qiang Qiu, Zhe Zhang, and Yu She. DiffOG: Differentiable policy trajectory optimization with generalizability. IEEE Transactions on Robotics, 2025.arxiv.org/abs/2504.13807
  40. [40]Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, Ayaan Malik, Kyungmin Lee, William Liang, Nadun Ranawaka, Jiasheng Gu, Yinzheng Xu, Guanzhi Wang, Fengyuan Hu, Avnish Narayan, Johan Bjorck, Jing Wang, Gwanghyun Kim, Dantong Niu, Ruijie Zheng, Yuqi Xie, Jimmy Wu, Qi Wang, Ryan Julian, Danfei Xu, Yilun Du, Yevgen Chebotar, Scott Reed, Jan Kautz, Yuke Zhu, Linxi ”Jim” Fan, and Joel Yang. World Action Models are Zero-shot Policies, 2026.arxiv.org/abs/2602.15922
  41. [41]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2005.13239
  42. [42]Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026.arxiv.org/abs/2603.16666
  43. [43]Weiheng Zhao, Haoyi Jiang, Xin Shi, Liu Liu, Fan Huang, Zhizhong Su, Wei Sui, and Xinggang Wang. Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Models, 2026.arxiv.org/abs/2608.04404
  44. [44]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 79115–79135. PMLR, 2025a.arxiv.org/abs/2411.04983
  45. [45]Guangyao Zhou, Sivaramakrishnan Swaminathan, Rajkumar Vasudeva Raju, J. Swaroop Guntupalli, Wolfgang Lehrach, Joseph Ortiz, Antoine Dedieu, Miguel Lázaro-Gredilla, and Kevin Murphy. Diffusion model predictive control. Transactions on Machine Learning Research, 2025b.arxiv.org/abs/2410.05364
  46. [46]Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In Proceedings of Robotics: Science and Systems, 2025.arxiv.org/abs/2504.02792

Paper details

vlnc:W_kg5hy6uu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI

View statement
Looking for related papers…
Contents