betaFeedback

WorksComputer Science

AnyGoal: On World Model Generalization Beyond Expert Rollouts

Abstract

The goal of a world model is to capture the dynamics of an environment. This has the great appeal that it allows agents to plan and simulate a course of actions with the help of a world model. For these uses to be reliable, however, the learned dynamics must behave well beyond the specific state-action combinations observed in expert rollouts. Despite this, the relationship between data coverage, dynamics generalization, and downstream planning performance remains underexplored. We study this relationship through the lens of compositional generalization: can a world model recombine familiar state and action factors into valid dynamics? To answer this, we introduce AnyGoal, a benchmark of goal-shifted task variants built from widely used world-model environments, together with an open-source library. Each task is designed to be within reach of a world model that has learned the underlying dynamics: individual state factors and actions remain within their training support, but their combinations are underrepresented in expert trajectories. Across multiple environments and world model families, we find that current models consistently exhibit failures under these recombinations. By examining dynamics and representation failures across model families, interaction regimes, and evaluation metrics, we study where current world models generalize and where challenges remain for goal-shifted prediction and planning. Together, this framework, benchmark, and analysis provide a basis for evaluating progress toward world models that generalize to new planning tasks.

Introduction

A key goal of physical intelligence is the ability to plan and perform new tasks. Routine tasks, such as driving to work, can be automatized and executed without much cognitive effort. Yet novel tasks requires us to plan and simulate the consequences of our actions. World models [14, 31, 29, 59, 58] (Vlaskov et al. 2026) aim to learn action-conditioned dynamics of environment by predicting how the world evolves under actions rather than task-specific behavior [42]. Therefore, a good world model should not merely memorize behavioral biases present in the training distribution. However, in-distribution evaluation offers limited evidence of reusable dynamics. Figure 1 illustrates a simple but strong example of this caveat. In Push-T, the agent must push a block to a target pose, so expert rollouts strongly couple far-from-block states with actions that move the agent toward the block. A model trained on Push-T may therefore learn this rollout regularity rather than the action-conditioned effect of the supplied action, as reflected by the action-sensitivity plot in Figure 1. When evaluated on Avoid-T, which requires avoiding the block using familiar 2D navigation, the model fails.

Example of generalization failure under goal shift

Figure 1. Example of generalization failure under goal shift. A world model trained on Push-T performs poorly on Avoid-T, despite Avoid-T requiring only familiar 2D navigation and having individually familiar states and actions. In Push-T, action interventions reveal reduced model sensitivity in biased regions, suggesting that the model learns shortcuts rather than reusable dynamics.

If the training data fully covered the relevant joint state–action space, then a model could in principle predict correctly on that support. However, such coverage is unrealistic in many control problems, especially when the relevant space is high-dimensional. First, offline datasets for world-model training are often collected from goal-directed policies that reach task-relevant states [4], which can introduce strong state–action correlations. One possible remedy is to mix data from multiple policies [57]. However, even with such mixtures, it can be difficult to determine whether the resulting dataset sufficiently covers the relevant state–action space, especially when the true reachable set is unknown. Even when a coverage gap is identified, collecting targeted rollouts to fill it can be difficult in offline settings. This makes it necessary to study generalization under incomplete coverage, rather than assuming that data scale alone will resolve the issue.

Generalization of world models has been studied to some extent. Closest to our setting, Hansen & Wang [15] study hallucination detection and mitigation in world models, while we evaluate generalization in a statistically controlled space with explicit state–action support checks. Other OOD evaluations introduce states or factors outside the training support [30, 47, 27]. Such settings are valuable for testing robustness to novel factors or extrapolation, but they may go beyond the generalization expected from a reusable dynamics model.

What remains missing is a systematic evaluation of the generalization that a good world model should support. We formalize this criterion as compositional generalization [50] and link it to goal shifts and behavior bias in the training data. We further show that the bias is empirically realized through targeted action interventions on familiar but under-covered state–action combinations, eliciting degenerate model responses. We then propose AnyGoal (Fig. 2), a benchmark of goal-shifted task variants built from widely used world-model environments. To remain in the compositional-generalization regime, AnyGoal preserves the observed marginal support of states, actions, task-relevant factors, and physical rules, while changing the joint combinations induced by the training behavior distribution and targeting under-covered state–action regions. With AnyGoal, we test whether current world models can generalize to new tasks without requiring extrapolation to unseen states, actions, or visual factors. Across this controlled setting, we observe broad failures across modeling paradigms, including generative world models [59, 29] and JEPA-style architectures [31, 16, 58]. Our results then characterize this degradation across several axes, showing that failures can arise from both dynamics prediction and representation learning, and that their severity depends on task complexity, evaluation metric, and model family. Finally, we outline a theoretical direction toward compositional generalization for world models, framing what future models may need in order to move beyond behavior-biased rollouts.

Related Works

Compositional generalization diagnostics and benchmarks. Compositional generalization tests whether a model can recombine familiar components into unseen configurations, a key challenge when possible compositions exceed finite data coverage. Early sequence-to-sequence benchmarks such as SCAN [24] and CFQ [18] exposed this failure mode through controlled compositional splits. The same diagnostic was subsequently adopted in vision and multimodal settings, with benchmarks such as CLEVR [17], Winoground [45], and CVR [54] testing whether familiar objects, attributes, and relations can be composed correctly. These failures remain prominent in modern large-scale models: ConceptMix [51] shows that text-to-image models degrade as prompts require more familiar concepts to be composed jointly, while recent vision–language work continues to reveal failures in object–attribute and relational composition [1]. At the same time, compositional evaluation is becoming broader and more systematic, with scalable protocols for probing generalization across controlled compositions [5], and similar shifts being used as diagnostics in mechanistic interpretability [34]. AnyGoal brings this perspective to world models, where goal-directed trajectories naturally induce restricted joint coverage over otherwise familiar physical states and actions.

World-model generalization and support mismatch. Recent world-model work has studied generalization through object-centric compositional dynamics [56], composable causal components for unseen environments [46], multi-task world models [13], environment-level evaluation queries [48], action-conditioned planning [55], and coverage-related hallucination failures [15]. However, less is known about whether expert-trained world models generalize to new goal-induced recombinations of familiar states and actions. This question is closely connected to support mismatch in offline reinforcement learning, where fixed behavior datasets can lead methods to fail when evaluating poorly covered state–action regions [10, 21, 26]. Offline RL addresses support mismatch by constraining or regularizing policies toward covered regions (Gao et al., 2025b;a; Laroche et al., 2019; Chen et al., 2025; Kumar et al., 2020; Tang et al., 2020). Model-based offline RL extends this issue to learned dynamics, motivating uncertainty- or pessimism-based objectives for poorly covered regions [52, 19, 53, 38]. D4RL further shows that dataset composition strongly affects offline policy improvement [9]. AnyGoal is complementary to these approaches: it evaluates whether the world model learns reusable dynamics under goal shifts.

Compositional Generalization for World Models

When should we say that a world model has learned reusable dynamics, rather than only the training behavior? In this section, we formalize this criterion as compositional generalization.

Notation. We use uppercase letters for random variables and lowercase letters for their realizations. Let zt=(z1,t,…,zK,t)z_t = (z_{1,t}, \ldots, z_{K,t}) denote true underlying state of the environment, where each component corresponds to a semantic factor of variation [39], such as object position or end-effector pose. We model observations as st=f(zt)s_t = f(z_t), where ff maps ztz_t to the RGB observation sts_t presented to the world model. An action-paired trajectory τ=(s0,a0,s1,a1,…,sH)\tau= (s_0, a_0, s_1, a_1, \ldots, s_H) can therefore equivalently be represented through the corresponding sequence of physical factors and actions, (z0,a0,z1,a1,…,zH)(z_0, a_0, z_1, a_1, \ldots, z_H), where HH denotes horizon.1

Action-Conditioned Prediction Can Degenerate into Behavior Prediction

An action-conditioned world model should learn a transition predictor T^\hat{T} that receives sts_t2 and an action ata_t, and predicts how the observation changes under the supplied action, T^(st,at)≈st+1\hat{T}(s_t, a_t) \approx s_{t+1}. We consider the common offline setting in which data are collected by a behavior policy π\pi, so that the observed action satisfies at∼π(⋅∣st)a_t \sim\pi(\cdot\mid s_t). Thus, the observed transitions reflect the state–action pairings induced by π\pi, rather than arbitrary interventions at each state:

Tπ(st+1∣st)=∫T(st+1∣st,a) π(a∣st) da.(1)T^\pi(s_{t+1} \mid s_t) = \int T(s_{t+1} \mid s_t, a)\,\pi(a \mid s_t)\,da. \tag*{(1)}

A predictor trained only on such data may degenerate into an action-ignorant model for state regions with restricted conditional action support, fitting the observed transitions by approximating

T^(st+1∣st,at)≈Tπ(st+1∣st),(2)\hat{T}(s_{t+1} \mid s_t, a_t) \approx T^\pi(s_{t+1} \mid s_t), \tag*{(2)}

where the dependence on the supplied action ata_t is effectively lost.

Compositional Distribution Shifts in World Models

Now, let gtrg_{\mathrm{tr}} and gteg_{\mathrm{te}} denote the training and test goals. Each goal induces a behavior policy, πtr\pi_{\mathrm{tr}} or πte\pi_{\mathrm{te}}, which generates actions along the same environment dynamics TT. We denote the resulting trajectory distributions by ptrp_{\mathrm{tr}} and ptep_{\mathrm{te}}, respectively. We define the supports of physical factors and actions observed across trajectories, writing the training joint state–action support as

Ctr:=⋃t=0H−1supp⁡ptr(Z1,t,…,ZK,t,At).(3)\mathcal{C}^{\mathrm{tr}} := \bigcup_{t=0}^{H-1} \operatorname{supp}_{p_{\mathrm{tr}}}(Z_{1,t}, \ldots, Z_{K,t}, A_t). \tag*{(3)}

We also define the marginal support of each factor as Zktr:=⋃t=0H−1supp⁡ptr(Zk,t)\mathcal{Z}^{\mathrm{tr}}_k := \bigcup_{t=0}^{H-1} \operatorname{supp}_{p_{\mathrm{tr}}}(Z_{k,t}), and the action support as Atr:=⋃t=0H−1supp⁡ptr(At)\mathcal{A}^{\mathrm{tr}} := \bigcup_{t=0}^{H-1} \operatorname{supp}_{p_{\mathrm{tr}}}(A_t). These supports pool over temporal position: a physical factor is considered observed during training if it occurs at any point along a trajectory drawn from ptrp_{\mathrm{tr}}. The test supports Cte\mathcal{C}^{\mathrm{te}}, Zkte\mathcal{Z}^{\mathrm{te}}_k, and Ate\mathcal{A}^{\mathrm{te}} are defined analogously under ptep_{\mathrm{te}}. We define the combinatorial space as the Cartesian product of the marginal training supports:

C×:=(∏k=1KZktr)×Atr.(4)\mathcal{C}^{\times} := \left(\prod_{k=1}^{K} \mathcal{Z}^{\mathrm{tr}}_k\right) \times\mathcal{A}^{\mathrm{tr}}. \tag*{(4)}
Compositional support structure

Figure 3. Compositional support structure. Cte⊆C×\mathcal{C}^{\mathrm{te}} \subseteq\mathcal{C}^{\times} and Cte⊄Ctr\mathcal{C}^{\mathrm{te}} \not\subset\mathcal{C}^{\mathrm{tr}} is necessary.

We study compositional distribution shifts [28, 32] where the underlying transition dynamics are invariant across the train and test distribution, i.e., Ttr(st+1∣st,at)=Tte(st+1∣st,at)T_{\mathrm{tr}}(s_{t+1} \mid s_t, a_t) = T_{\mathrm{te}}(s_{t+1} \mid s_t, a_t), but the test support introduces state, action pairs not present in the training support (Cte⊄Ctr\mathcal{C}^{\mathrm{te}} \not\subset\mathcal{C}^{\mathrm{tr}}). Any-Goal targets the combinatorial gap, regions of the combinatorial space C×\mathcal{C}^{\times} that are absent from the training joint support Ctr\mathcal{C}^{\mathrm{tr}}. Since test trajectories need not cover the entire gap or lie wholly within it, we require Cte⊆C×\mathcal{C}^{\mathrm{te}} \subseteq\mathcal{C}^{\times} and choose goals with substantial mass in under-covered regions of C×\mathcal{C}^{\times}. This means that the test distribution does not introduce novel physical factors or actions, only novel combinations of observed physical factors and actions. Importantly, training-distribution accuracy does not identify the extension of the learned transition function to C×∖Ctr\mathcal{C}^{\times} \setminus\mathcal{C}^{\mathrm{tr}}: predictors that agree on the training support may disagree on novel compositions [50]. This motivates evaluating the combinatorial gap directly. We discuss more about why it is an essential test in Appendix § 9.

Factor-wise marginal and state-action joint distribution comparisons for AnyGoal target variants

Figure 4. AnyGoal targets the combinatorial gap. (a) Example factor-wise support check for Avoid-T relative to Push-T using KDE. Darker regions indicate higher density, and gray regions mark areas whose density is below 1% of the peak reference density. (b) Estimated overlap between each target variant and the base-task training distribution via KDE.

AnyGoal: Controlled Goal-Shift Evaluation

We introduce AnyGoal, a benchmark for testing whether world models generalize beyond the goal-directed trajectories on which they are trained. AnyGoal uses the controlled goal shifts discussed in § 3.2 and builds on four widely used world-model tasks: TwoRoom [41], Push-T [7], DMC Manipulator Insert-Peg [44], and OGBench Cube [36]. AnyGoal contains eight variants: one free-space variant per base task and one or two contact-rich variants except TwoRoom. We release 400 episodes per variant, with data-collection tools for generating additional datasets. Figure 2 summarizes the tasks, with details in Appendix § 10.

Figure 2. Overview of AnyGoal. Each task preserves the environment dynamics of the corresponding base tasks and keep states and actions individually within training coverage. Throughout the paper, AnyGoal is used solely as a test set for evaluating generalization.

AnyGoal Targets the Combinatorial Gap

Validating Cte⊆C×\mathcal{C}^{\mathrm{te}} \subseteq\mathcal{C}^{\times}. For each base–target task pair, we apply three factor-wise support checks to verify that the target distribution remains marginally in-distribution relative to its reference distribution. The range check requires all query samples to lie within the coordinate-wise reference range. The Mahalanobis check uses the reference mean and covariance, thresholds distances at the reference 0.99-quantile, and requires at least 95% of query samples to fall below it. The KDE check thresholds reference densities at the 0.01-quantile and requires at least 95% of query samples to lie above it. The final support verdict is the conjunction of all three checks. All AnyGoal variants are designed to pass these factor-wise checks, which indicates that all tasks lie within the combinatorial space of training data. Appendix § 11 provides the full details of pass-criterion definitions, the verified factor groups for each base task, and the complete score of range, Mahalanobis, and KDE results.

Measuring combinatorial-gap coverage of Cte\mathcal{C}^{\mathrm{te}}. We then estimate how much each target distribution overlaps with the base-task joint support. Unlike the factor-wise checks above, this joint test is meant to diagnose the combinatorial gap: individual factors may be marginally familiar while their full state–action combination is rare under the training distribution. For each base–target pair, we concatenate the action, agent state, and task-object or target state, fit a full-dimensional Gaussian KDE to the reference vectors, and set the density threshold to the empirical 0.01-quantile of log-densities. We then report the fraction of target vectors whose log-density exceeds this threshold. Lower joint scores indicate less overlap with the base-task joint distribution and greater coverage of the combinatorial gap. Full vector definitions and KDE implementation details are in Appendix § 12.

Relative Evaluation Against Base Tasks

We evaluate rollout prediction error and downstream MPC planning. Models are trained only on the base dataset and evaluated on both base and goal-shifted trajectories to measure relative changes. Rollout-error changes more directly diagnose whether action-conditioned dynamics generalize to unseen state–action combinations. However, planning success remains important. For JEPA-style models, it provides complementary evidence that the representation retains task-relevant information. For generative models, low rollout error can miss small but task-critical observation changes. We therefore report both metrics over seeds 0, 1, 2. Evaluation details are in Appendix § 16.

Prediction Error. We measure MSE in each model’s native prediction space, which may be a learned latent space, oracle state space, or pixel space depending on the model class. We evaluate autoregressive prediction over horizons of 5 and 25 environment steps, with frame skip of 5. For each seed, we sample 100 episodes and average over all valid starting points within each episode.

MPC Planning Success Rate. We evaluate downstream control using the same CEM-based MPC implementation from Stable-WorldModel [30] for all models. We use identical planning configurations, with a planning horizon of 25 environment steps. The full 25-step action sequence is executed before replanning.

AnyGoal Library and Evaluation Interface

We release AnyGoal as an open-source Python library for reproducible goal-shift evaluation. The library provides dataset access, task-specific data collection, and standardized rollout-prediction and MPC-planning evaluation across supported world models. Its unified CLI allows researchers to evaluate the same base-trained model on both the original task and unseen goals, with explicit model, dataset, and seed selection. The library also exposes Python APIs for dataset collection, model training, and evaluation. We provide examples and reproducibility instructions in Appendix § 19, with complete documentation available online.

# Original training goal
anygoal eval planning --model lewm --dataset pusht --seed 0

# Unseen goal
anygoal eval planning --model lewm --dataset avoidt --seed 0
Command-line evaluation interface for original and unseen goals

Figure 5. The same base-trained model is evaluated on Push-T and Avoid-T using the standardized planning interface.

Experiment and Results

We evaluate a diverse model suite spanning JEPA-style and generative objectives, learned and frozen visual encoders, and action-conditioning mechanisms (Table 1). To separate dynamics learning from visual representation failure, we also include oracle-state predictors which replace visual encoders with oracle states. For consistency, all models use frame skip 5 and one-step prediction. Model details and oracle-state variables for each environment are provided in Appendix F, and the full numerical results are summarized in Appendix § 17.

ModelFamilyRepresentationAction conditioning
End-to-End World Models
LeWorldModelJEPALearned latentAdaLN
Sensorimotor WMJEPALearned latentAdaLN
DINO-WMJEPAFrozen DINO featuresFeature concatenation
DriftWorldGenerativePixelsConditional generator
IRASimGenerativeFrozen SDXL VAEDiT conditioning
Predictor-Only Models
AdaLNTransformerOracle stateAdaLN
Action-ConcatenatedTransformerOracle stateFeature concatenation
Toy-DriftWorldGenerativeOracle stateConditional generator

Table 1. End-to-End World models and oracle state predictors evaluated in our experiments.

End-to-End World Models. LeWorldModel (LeWM) [31] is an end-to-end JEPA that jointly learns a visual encoder and an action-conditioned transformer predictor from pixels. It

uses SIGReg ([2]) for anti-collapse regularization and conditions the predictor through AdaLN ([37]). Sensorimotor World Model (SMWM) ([16]) uses the same encoder–predictor family as LeWM with an inverse-dynamics objective that encourages representations to retain controllable changes. DINO-WM ([58]) decouples perception from dynamics by using a frozen DINOv2 ([33]) patch encoder and predicting future patch representations, with actions concatenated to the patch tokens. DriftWorld ([29]) is a generative model that uses an action-conditioned spatial–temporal U-Net to predict future observations directly from observation history, actions, and noise. IRASim ([59]) is a latent generative model that encodes observations with a frozen pretrained VAE and uses a Diffusion Transformer ([37]) to predict future latents conditioned on action trajectories.

Oracle-State Predictor. AdaLN predictor follows the architecture used by LeWM and SMWM. Action-Concatenated predictor follows the DINO-WM predictor, where an embedded action is concatenated to the patch representation before being processed by a causal transformer. Toy-DriftWorld is adapted from DriftWorld’s action-conditioned generative backbone. We use a reduced DriftWorld-style model and replace pixel inputs and outputs with true state inputs and next-state targets.

Dimensions of Goal-Shift Performance

Goal-shift performance cannot be summarized by a single number. The same degradation can reflect different sources depending on whether it appears in prediction or planning, in latent or pixel space, in free-space or contact-rich tasks, and with or without oracle state access. We therefore organize the results around five recurring dimensions for interpreting the results:

  1. Failure source: Predictor failures can be isolated from encoder failures via oracle-state predictors.

  2. Evaluation metric: Rollout prediction and planning are showing different aspects.

  3. Task complexity: Free-space and contact-rich tasks impose different dynamics challenges.

  4. Representation quality: Learned representations sometimes obscure task-relevant dynamics, although they sometimes can improve planning geometry.

  5. Prediction space: JEPA-style and generative models show distinct trends, but comparisons should be interpreted cautiously because prediction errors are measured in different spaces.

Dynamics Generalization Beyond Representation Failure

To examine dynamics generalization, we use the oracle states provided by AnyGoal, which expose task-relevant factors for action-conditioned next-step prediction and remove the need to infer them from observations under goal shifts. Overall, oracle-state predictors show that the AnyGoal gap cannot be reduced to visual representation failure.

Oracle-state predictors reveal an interaction-dependent dynamics gap. Figure 6 shows that removing visual abstraction does not yield a uniform outcome across AnyGoal variants. In freespace variants such as TR-Route, Avoid-T, Avoid-Peg, and Cube-Hop, MPC performance is preserved or improves relative to the corresponding base task. In contrast, contact-rich variants consistently degrade, despite using the same privileged state inputs. This suggests that the difficulty is not merely access to the relevant state factors (Klindt et al., [20]; Shi et al., [40]), but whether the goal shift requires precise object interaction. This asymmetry is consistent with prior work showing that compositional generalization requires constraints on the learned function. Additive or interaction-free settings are especially favorable, since factor-specific mechanisms can be learned from observed compositions and reused under novel recombinations (Dong & Ma, [8]; Lachapelle et al., [23]; Wiedemer et al., [49]; Mahajan et al., [32]). This intuition aligns with the independent causal mechanisms principle, under which reusable mechanisms remain autonomous as their joint occurrence changes (Parascandolo et al., [35]). Contact-rich variants depart from this favorable regime: action effects may no longer decompose into reusable factor-specific mechanisms. Compositional generalization may still be possible, but only with additional structure on these interactions (Brady et al., [3]). The same asymmetry appears in rollout prediction. Contact-rich variants show much larger error increases, often by one to three orders of magnitude, even with oracle states. Free-space variants degrade far less, and in some cases match or improve on the base-task error. Thus, removing representation learning does not make goal-shifted dynamics prediction uniformly reliable.

Oracle-state predictor comparison across tasks

Figure 6. Oracle-state predictors reveal dynamics-level failures. Even with task-relevant state factors provided directly, goal shifts still degrade prediction and planning, showing that the AnyGoal gap cannot be reduced to visual representation failure. See Appendix § 17.§ 1 for absolute metrics.

Relative prediction error and relative planning performance across task variants

Figure 7. End-to-end degradation reflects representation effects. Compared with oracle-state predictors, end-to-end models degrade more under goal shifts, suggesting additional failures from observation-level shift and learned representation geometry. See Appendix I.1 for absolute metrics.

Metric alignment depends on task complexity. We additionally observe that rollout error and planning performance sometimes diverge in free-space variants where planning can tolerate imperfect prediction. The closer agreement in contact-rich settings may reflect the structure of the planning problem. Rollout error measures whether the model predicts the full trajectory accurately, whereas MPC only needs to find some action sequence that reaches the goal, regardless of the exact intermediate path. Free-space tasks often admit many such trajectories, while contact-rich tasks allow fewer viable sequences because success also depends on the correct object interaction.

Representation Effects in End-to-End World Models

End-to-end world models can degrade on both metric under the goal shift even when oracle-state predictors remain stable. Such cases show that the AnyGoal gap is not solely a failure of action-conditioned dynamics, but can also reflect encoder failure under unseen factor combinations. Due to space constraints, we discuss two main observations, and more are in Appendix I.3.

Factor recombination can induce observation-level shifts. AnyGoal controls generalization at the level of physical factors, but end-to-end models observe rendered inputs s=f(z)s=f(z). Recombining marginally familiar factors can therefore create observations outside the joint visual support of expert rollouts, adding an encoder-inference challenge beyond dynamics prediction. (See Appendix I.2 for details.) Crucially, correct factor inference on training observations need not extend to unseen compositions. Even when factors are identifiable in distribution, an encoder may fail to invert novel recombined observations without compositional-consistency constraints [49]. Thus, end-to-end world models face a representation-level generalization problem before dynamics prediction is considered. End-to-end models must therefore infer task-relevant factors from observations generated by unseen factor combinations, in addition to learning dynamics.

Analysis of goal shifts without observation-level shift. TwoRoom provides a useful case where end-to-end degradation cannot be attributed to an observation-level shift. Because agent position is the only changing factor and TR-Route positions remain marginally within the TwoRoom training support, the corresponding observations stay visually in support, enabling a controlled comparison between end-to-end models and oracle-state predictors. Figure 8 (a) summarizes results from TwoRoom and TR-Route across different model families. We observed that rollout prediction improves across model families, while planning degrades for end-to-end models and remains stable for oracle-state predictors. This suggests that lower prediction error does not necessarily translate into better control: the learned latent space may still distort task-relevant state differences or action effects needed for planning. Thus, even visually in-support goal shifts can expose a mismatch between predictive accuracy and planning utility.

Figure 8. TwoRoom as a Visually In-Support Goal-Shift Case Study. (a) Rollout prediction improves across model families, while planning degrades for end-to-end models and remains stable for oracle-state predictors. (b) The 2D t-SNE projections illustrate that learned representations can organize the same underlying agent positions with substantially different geometry, which can affect prediction and planning.

Diagnosing Behavior-Policy Shortcuts Through Action Sensitivity

As discussed in § 3.1, a world model trained on expert rollouts may exploit correlations between states and the actions typically taken by the expert, rather than learning the effects of the supplied action. We investigate whether this failure mode is reflected in the learned models by measuring their sensitivity to interventional action inputs.

Interventional action sensitivity. To test whether the model responds to the supplied action rather than relying on the behavior policy’s preferred transition, we intervene on the action input while holding the state fixed. For each training state ss with observed action aa, we replace aa with actions from a fixed interventional set Acf\mathcal{A}_{\mathrm{cf}} and measure the resulting change in the model’s predicted representation:

Sθ(s)=1∣Acf∣∑a′∈Acf∥ϕθ(s,a)−ϕθ(s,a′)∥2,(5)S_{\theta}(s) = \frac{1}{\lvert\mathcal{A}_{\mathrm{cf}} \rvert} \sum_{a' \in\mathcal{A}_{\mathrm{cf}}} \left\lVert\phi_{\theta}(s,a) - \phi_{\theta}(s,a') \right\rVert_2 , \tag*{(5)}

where ϕθ(s,a)\phi_{\theta}(s,a) denotes the model representation used for the sensitivity measurement. A smaller score indicates that the model is less sensitive to changes in the supplied action, although this does not necessarily imply incorrect predictions. We evaluate this score across training states and relate it to an environment-specific coordinate along which expert behavior is strongly biased. More details are in Appendix E.

Goal bias and reduced action sensitivity. Figure 9 shows that action sensitivity varies systematically with the state regions associated with expert behavior. In TwoRoom, expert trajectories favor motion toward the narrow passage. In Push-T, the agent typically approaches the block before pushing it. In Insert-Peg, the agent carries the grasped peg toward the target and in Cube, vertical actions are prevalent at low end-effector heights. Across these environments, we observe reduced sensitivity to interventional actions in regions where the training behavior is strongly constrained by the goal and these observations are consistent with the degeneracy described in § 3.1.

Action sensitivity against environment-specific bias coordinates for TwoRoom, Push-T, Insert-Peg, and Cube

Figure 9. Goal-biased training support is associated with reduced action sensitivity. Each panel plots model action sensitivity against an environment-specific bias coordinate induced by the expert policy. The analysis suggests that the model can partially rely on a goal-directed prior rather than the supplied action.

Towards world models that generalize compositionally

Our oracle-state experiments show that compositional failures cannot be attributed only to visual representation learning. Even when task-relevant physical state is provided directly, goal shifts remain difficult, particularly when prediction involves contact-rich interactions. This raises a broader question for future work: what assumptions on the dynamics, data, and learned representation are sufficient for a world model to generalize compositionally? We see two distinct challenges.

  1. Identifying mechanisms. Fitting training trajectories does not ensure that a predictor has learned reusable action-conditioned mechanisms. This becomes harder in interaction-rich dynamics where action effects can vary with state and regime. Characterizing when such mechanisms can be identified from offline trajectories remains an open problem.

  1. Extrapolation beyond the training support. Even if the correct mechanisms are identified, their behavior on novel state–action compositions is generally not determined without additional structural assumptions. Interaction-free dynamics provide a favorable case for compositional generalization because factor effects can often be reused independently, but world models must also handle interaction-rich dynamics where action effects vary with the joint physical configuration and regime. In this setting, extrapolation is fundamentally harder because the relevant interactions may change across compositions, and is a fruitful direction for future work.

A simple illustrative case. One favorable case for extrapolation in interaction-rich dynamics arises when, within each interaction regime, each transition mechanism depends only on a small subset of the state and action variables. Consider regime-dependent dynamics of the following form zt+1=Trt(zt,at)z_{t+1} = T_{r_t}(z_t, a_t), where rtr_t denotes an interaction regime, such as free motion, contact, or grasping. Both the autonomous state evolution and the effect of the action may therefore change across regimes. Suppose that, within a fixed regime rr, the transition of each next-state factor zk,t+1z_{k,t+1} depends only on a subset of the coordinates of (zt,at)(z_t, a_t). Writing xt=(zt,at)x_t = (z_t, a_t), this sparse dependence can be expressed as follows, ∥∇xTk,rt(x)∥0≪dim⁡(x)\lVert\nabla_x T_{k,r_t}(x) \rVert_0 \ll\dim(x).

Let uku_k denote the relevant state and action coordinates for kk-th transition mechanism Tk,rtT_{k,r_t}, so that we may write zk,t+1=Tk,rt(uk,t)z_{k,t+1} = T_{k,r_t}(u_{k,t}). This allows a simple case where compositional shift in the full state–action space need not require extrapolation for each transition mechanism.

Cte⊈Ctr,whileUkte⊆Uktr∀k\mathcal{C}^{\mathrm{te}} \not\subseteq\mathcal{C}^{\mathrm{tr}}, \qquad\text{while} \qquad U_k^{\mathrm{te}} \subseteq U_k^{\mathrm{tr}} \quad\forall k

where UktrU_k^{\mathrm{tr}} and UkteU_k^{\mathrm{te}} denote the train and test supports of uku_k for the fixed regime rr. Thus, the complete state–action composition may be unseen, while every input relevant to predicting each next-state factor remains within its training support. Hence, if the corresponding mechanisms have been correctly learned, we can generalize to novel compositions as we do not need to extrapolate any individual mechanism beyond its training support. Appendix § 18 (Proposition 1) gives a formal statement.

Conclusion

We introduced AnyGoal, a benchmark for evaluating whether world models generalize to new behavioral goals by recombining familiar states and actions. Across a diverse set of models, goal-shift generalization remains difficult even when the relevant state and action factors are marginally covered by the training distribution. Together, these analyses show that goal-shift failures are heterogeneous, reflecting interactions among representation, dynamics prediction, task structure, and evaluation metric. More broadly, AnyGoal connects world-model evaluation with compositional generalization. It asks for prediction on novel joint state–action combinations within the product of marginal training supports, a reasonable but unresolved requirement for reusable dynamics. Our theoretical perspective suggests that global coverage of all state–action combinations is neither realistic nor necessary. Instead, world models need coverage of local mechanism inputs, together with objectives or architectures that identify action-conditioned mechanisms rather than behavior-policy shortcuts. By isolating goal-induced recombination as a failure mode, AnyGoal provides a controlled basis for evaluating progress toward world models that learn reusable dynamics beyond expert rollouts.

Acknowledgments

David Klindt acknowledges the CSHL GPU cluster with assistance from the US National Institutes of Health Grant S10OD028632-01. Divyat Mahajan acknowledges support via FRQNT doctoral scholarship (https://doi.org/10.69777/354785) for his graduate studies.

Compositional Generalization: Essential World-Model Test

We formalize why our compositional evaluation is an essential test for world models. The key distinction is between extrapolating to unseen factors and recombining observed ones. Good action-conditioned world model should predict the effect of an observed action on familiar physical factors, even when that particular combination was not selected by the behavior policy.

For clarity, consider a deterministic oracle-state setting with true controlled dynamics zt+1=T⋆(zt,at)z_{t+1} = T^{\star}(z_t, a_t). Suppose the offline dataset is collected by a behavior policy a=πE(z)a = \pi_E(z). Then the observed state–action support lies on the policy graph,

ΓπE={(z,a):a=πE(z)}.(6)\Gamma_{\pi_E} = \{(z, a) : a = \pi_E(z)\}. \tag*{(6)}

Thus, expert rollouts cover only the state–action pairings induced by the behavior policy, not arbitrary interventions with observed actions.

Zero Training Error Does Not Imply Action-Conditioned Dynamics. A predictive world model trained on (z,a,z′)∼ptr(z, a, z') \sim p_{\mathrm{tr}} is constrained only on the state–action pairs generated by the behavior policy. Since observed actions satisfy a=πE(z)a = \pi_E(z), the training loss constrains T^(z,πE(z))\hat{T}(z, \pi_E(z)), but not T^(z,a)\hat{T}(z, a) for alternative observed actions at the same state. Consequently, the action-ignorant predictor

T^deg(z,a)=T⋆(z,πE(z))(7)\hat{T}_{\mathrm{deg}}(z, a) = T^{\star}(z, \pi_E(z)) \tag*{(7)}

achieves zero prediction error on the behavior dataset while ignoring the queried action aa. Thus, expert-rollout prediction cannot distinguish a controlled dynamics model from a predictor that only models behavior-policy transitions.

Compositional Evaluation Requires Action-Conditioned Dynamics Compositional evaluation rules out this degenerate solution. Assume actions have nonzero effect within the observed marginal supports: for a set of oracle states with positive measure, there exist observed actions a,a′∈Atra, a' \in\mathcal{A}^{\mathrm{tr}} such that

T⋆(z,a)≠T⋆(z,a′).T^{\star}(z, a) \ne T^{\star}(z, a').

Now consider any action-ignorant predictor T^(z,a)=h(z)\hat{T}(z, a) = h(z). If it were correct on the relevant compositional state–action space, then for every observed action a∈Atra \in\mathcal{A}^{\mathrm{tr}}, h(z)=T⋆(z,a)h(z) = T^{\star}(z, a). Therefore, for any two observed actions a,a′a, a',

T⋆(z,a)=h(z)=T⋆(z,a′).(8)T^{\star}(z, a) = h(z) = T^{\star}(z, a'). \tag*{(8)}

which contradicts the action-relevance assumption. Hence, no action-ignorant model can be correct on recombinations of observed state and action factors when actions causally affect the next state.

Interpretation. Expert rollouts test whether a model predicts what happens under the behavior policy. Compositional state–action evaluation instead tests whether the model predicts what happens when observed actions are applied to familiar physical factors, including combinations not generated by the behavior policy. This is an essential requirement for model-based control, since planning must evaluate candidate actions other than the behavior action.

AnyGoal Task Details

Two Room and Its Variants

TwoRoom is a simple navigation environment with two rooms connected by a narrow slot. The agent must move through the slot and reach a target location, with little to no meaningful interaction with the environment beyond collision with walls. Because the dynamics are essentially planar motion, this environment isolates whether models can preserve action-conditioned movement under goal-biased trajectory data.

TwoRoom Route is an interaction-light navigation variant. The agent must follow a piecewise-linear triangular route defined by a start point, an intermediate waypoint, and an invisible endpoint, rather than moving directly through the slot toward the target. Episodes mix clean, noisy, and variable-speed actions within the observed action support. This variant tests whether the model preserves planar action-conditioned dynamics when familiar positions and actions are recombined outside the dominant slot-crossing behavior of the base dataset.

PushT and Its Variants

PushT is a planar manipulation environment where the agent must push a T-shaped block into a fixed T-shaped goal pose. Unlike TwoRoom, the base task already requires meaningful contact, object translation, and object rotation. The expert trajectories are strongly goal-directed, coupling agent motion with the block pose and the target configuration. This environment tests whether world models learn action-conditioned contact dynamics rather than only the dominant expert pushing manifold.

AvoidT is a non-interactive variant. The agent must avoid touching the physical T-block and move to one of four corner regions. This variant keeps the agent and action factors familiar, but changes the required behavior from goal-directed pushing to free-space navigation around the object. It primarily tests whether the model preserves the action effect when the expert pushing objective is no longer appropriate.

StackT is an interaction-rich variant. The physical T-block must be placed so that its head overlaps with the fixed goal T, while its rotation is opposite to the original PushT goal orientation, creating a plus-shaped target configuration. This variant requires contact and rotation dynamics under a goal that differs from the original training objective.

PushT Anti-goal is also interaction-rich and directly reverses the dominant goal bias in the base dataset. The physical T-block is initialized near the original PushT goal pose, but the agent must push it away from the fixed goal region and move it toward one of four corner regions, with rotation opposite to the original goal orientation. This tests whether the model can predict familiar pushing dynamics when the required behavior contradicts the original expert trajectory distribution.

Insert-Peg and Its Variants

Insert-Peg is a planar manipulation environment where an arm must move a peg toward a sampled slot and insert it into the target opening. The task involves articulated robot motion, object contact, and alignment between the peg and the slot. Expert trajectories are strongly structured around successful insertion behavior. This environment tests whether models can learn action-conditioned manipulation dynamics under goal-biased contact-rich data.

Avoid-Peg is an interaction-light variant. The arm must move around the target box in a specified hugging direction while avoiding contact with the peg. This changes the goal-conditioned arm behavior without requiring peg manipulation, testing whether the model can predict familiar arm motion under a non-insertion objective.

Extract-Peg is an interaction-rich variant. The arm must extract the peg from the target box, move it through a waypoint, and place it at a destination, with episodes balanced between left- and right-side box placements. This reverses the dominant insertion behavior while keeping the peg, box, workspace, and action space familiar.

OGBench Cube and Its Variants

Cube is a robot manipulation environment where the end-effector must interact with a cube and move it toward a target configuration. The base behavior follows a structured pick-and-place pattern, including approach, grasp, lift, transport, and release. This creates a goal-biased trajectory distribution in which robot motion, gripper state, cube pose, and target location are tightly coupled. The environment tests whether models learn manipulation dynamics beyond this dominant expert sequence.

CubeHop is a non-interactive variant. The robot does not intentionally move the cube. Instead, the end-effector moves between waypoint and target positions while gradually changing its yaw. This variant changes the goal-conditioned end-effector behavior while avoiding object interaction. It tests whether the model can predict familiar robot motion and yaw changes outside the dominant cube-manipulation trajectory distribution.

CubeDiagonal is an interaction-rich variant. The robot manipulates the cube using a diagonal grasp configuration, where the gripper contacts opposite edges of the cube instead of the standard parallel faces. This introduces a novel contact regime with different grasp geometry while keeping the object, task objective, and action space unchanged. It tests whether the model can compose familiar manipulation factors under unseen interaction configurations.

Statistical Test 1: Marginal Distribution Support

Distribution Support Criteria

We describe the statistical criteria used to decide whether a query distribution remains inside the support of a reference distribution. Let P={pn}n=1NP = \{p_n\}_{n=1}^{N} denote the reference or training point set, and let Q={qm}m=1MQ = \{q_m\}_{m=1}^{M} denote the query point set.

For each factor, we compute the empirical mean μP=1N∑n=1Npn\mu_P = \frac{1}{N}\sum_{n=1}^{N}p_n and regularized covariance ΣP=Cov⁡(P)+λI\Sigma_P = \operatorname{Cov}(P) + \lambda I, where λ>0\lambda> 0 is a small diagonal regularizer used to ensure that the covariance matrix is invertible. This is particularly important for low-rank or manifold-valued factors, such as circular variables represented by sine and cosine coordinates.

Now we measure three metric respectively. First, the Mahalanobis criterion checks whether query points remain within the reference distribution’s main covariance structure.

Empirical Mahalanobis criterion. The empirical squared Mahalanobis distance of a point xx from the reference distribution is

dP2(x)=(x−μP)TΣP−1(x−μP).(9)d_P^2(x) = (x-\mu_P)^{\mathsf{T}}\Sigma_P^{-1}(x-\mu_P). \tag*{(9)}

We compute the reference threshold as the 99th percentile of the empirical Mahalanobis distances of the reference points, τ0.99=Quantile⁡0.99({dP2(pn)}n=1N)\tau_{0.99} = \operatorname{Quantile}_{0.99}(\{d_P^2(p_n)\}_{n=1}^{N}). The query set passes the Mahalanobis support criterion if at least 95% of its points lie below this reference threshold:

Mahalanobis⁡0.99(Q;P)=[1M∑m=1M1[dP2(qm)≤τ0.99]≥0.95].(10)\operatorname{Mahalanobis}_{0.99}(Q;P) = \left[\frac{1}{M}\sum_{m=1}^{M}\mathbf{1}\left[d_P^2(q_m) \le\tau_{0.99}\right] \ge0.95\right]. \tag*{(10)}

For a one-dimensional factor, this criterion reduces to the regularized squared zz-score dP2(x)=(x−μP)2/(σP2+λ)d_P^2(x) = (x-\mu_P)^2/(\sigma_P^2+\lambda).

Second, the KDE criterion checks whether query points lie in regions with sufficient empirical reference density, beyond an ellipsoidal approximation.

Histogram-binned KDE criterion. We estimate a reference density ρP\rho_P with a histogram-binned Gaussian KDE fit only on the reference set PP. Query points are then evaluated under this fixed density estimate. The histogram grid is constructed from the pooled reference–query range with a small padding, and Gaussian smoothing uses Scott’s rule with a lower bandwidth clip to avoid degenerate bins.

After fitting the KDE, we evaluate the density of all reference samples and set the threshold to the empirical first percentile,

δ0.01=Quantile⁡0.01({ρP(pn)}n=1N).\delta_{0.01} = \operatorname{Quantile}_{0.01}\left(\{\rho_P(p_n)\}_{n=1}^{N}\right).

The query set passes the KDE criterion if at least 95% of query points have density no smaller than this reference-sample threshold:

KDE⁡0.01(Q;P)=[1M∑m=1M1[ρP(qm)≥δ0.01]≥0.95].(11)\operatorname{KDE}_{0.01}(Q;P) = \left[\frac{1}{M}\sum_{m=1}^{M}\mathbf{1}\left[\rho_P(q_m) \ge\delta_{0.01}\right] \ge0.95\right]. \tag*{(11)}

For periodic yaw variables, KDE smoothing is applied with wrapped boundary conditions.

Lastly, the range criterion checks whether query points avoid direct coordinate-wise extrapolation beyond the observed marginal range.

Observed-range criterion. As a complementary support check, we compute an axis-wise observed-range test. For each query point qmq_m, we check whether all of its coordinates lie inside the coordinate-wise minimum and maximum of the reference set, i.e., qm∈[min⁡(P),max⁡(P)]q_m \in[\min(P),\max(P)] iff min⁡npn,j≤qm,j≤max⁡npn,j\min_n p_{n,j} \le q_{m,j} \le\max_n p_{n,j} for all jj. The observed-range criterion is

range⁡(Q;P)=[1M∑m=1M1[qm∈[min⁡(P),max⁡(P)]]≥0.99].(12)\operatorname{range}(Q;P)=\left[\frac{1}{M}\sum_{m=1}^{M}\mathbf{1}[q_m\in[\min(P),\max(P)]]\ge0.99\right]. \tag*{(12)}

For circular variables, such as yaw, the observed-range criterion is ignored because the interval boundary is representation-dependent. In that case, the Mahalanobis and KDE criteria are used without the range check.

The final verdict is the conjunction of all non-ignored criteria:

pass⁡(Q;P)=range⁡(Q;P)∧Mahalanobis⁡0.99(Q;P)∧KDE⁡0.01(Q;P).(13)\operatorname{pass}(Q;P)=\operatorname{range}(Q;P)\land\operatorname{Mahalanobis}_{0.99}(Q;P)\land\operatorname{KDE}_{0.01}(Q;P). \tag*{(13)}

If the observed-range criterion is ignored for a circular factor, the verdict is computed over the remaining criteria:

pass⁡(Q;P)=Mahalanobis⁡0.99(Q;P)∧KDE⁡0.01(Q;P).(14)\operatorname{pass}(Q;P)=\operatorname{Mahalanobis}_{0.99}(Q;P)\land\operatorname{KDE}_{0.01}(Q;P). \tag*{(14)}

This design separates marginal factor support from full trajectory-distribution typicality. The support checks are applied at the level of semantic factor groups to verify that each physical factor remains within the reference support. Combinatorial novelty can still arise when individually supported factors appear in new joint configurations that are rare or absent under the expert trajectory distribution.

Verified Factor Groups

We apply the three support criteria above to task-relevant factor groups rather than to the full joint state–action vector. This verifies marginal coverage of the physical factors and actions used in each goal-shifted variant, while allowing the full combinations to remain novel. Table 2 lists the factor groups checked for each reference–query pair.

Reference dataset PPQuery dataset QQFactor groups checked
TwoRoomTR-RouteAgent position (2D); action position delta (2D)
PushTAvoid-T; Stack-T; Push-T AntigoalAgent position (2D); block position (2D); action (2D); block angle.
Insert-PegAvoid-Peg; Extract-PegArm root angle; Arm shoulder angle; Arm elbow angle; Arm wrist angle; Finger angle; Fingertip angle; Thumb angle; Thumbtip angle; hand, peg, and target position (2D); hand, peg, and target orientation (2D); 5 motor commands (1D) independently.
CubeCube-Hop; Cube-DiagonalEnd-effector position (3D); block position (3D); block quaternion (4D); action position delta (3D); gripper opening; gripper contact; end-effector yaw; action yaw delta; action gripper delta; 6 joint pos independently

Table 2. Every independent factor is in-distribution with respect to the reference dataset. Factor groups used for support verification. For each reference dataset PP, we compare the corresponding goal-shifted variant datasets QQ against the listed task-relevant factors..

Support-Verification Results

Table 3 reports the full support-verification results for the factor groups in Table 2. For each reference–query pair, we report the pass rate under the Mahalanobis, KDE, and observed-range criteria, together with the final conjunction verdict. All checked factors pass, confirming that the AnyGoal variants remain marginally in distribution with respect to the corresponding base datasets.

Table 3: Distribution-support verification between each reference dataset (PP) and query dataset (QQ). A dash indicates that the range criterion is not applied to circular variables. (All Passed)

PPQQFactor of variantsMahalanobis 0.99KDERangeFinal verdict
TwoRoomTR-Routeagent pos (2D)0.9701.0001.000pass
TwoRoomTR-Routeaction (2D)1.0000.9571.000pass
PushTAvoid-Tblock angle0.9700.963–pass
PushTAvoid-Tagent pos (2D)0.9680.9841.000pass
PushTAvoid-Tblock pos (2D)1.0000.9901.000pass
PushTAvoid-Taction (2D)1.0000.9951.000pass
PushTStack-Tblock angle0.9980.989–pass
PushTStack-Tagent pos (2D)0.9920.9891.000pass
PushTStack-Tblock pos (2D)1.0000.9991.000pass
PushTStack-Taction (2D)1.0000.9541.000pass
PushTPush-T Antigoalblock angle1.0000.988–pass
PushTPush-T Antigoalagent pos (2D)0.9880.9861.000pass
PushTPush-T Antigoalblock pos (2D)1.0001.0001.000pass
PushTPush-T Antigoalaction (2D)1.0000.9891.000pass
Insert-PegAvoid-Pegarm root angle0.9800.9631.000pass
Insert-PegAvoid-Pegarm shoulder angle1.0000.9991.000pass
Insert-PegAvoid-Pegarm elbow angle0.9990.9981.000pass
Insert-PegAvoid-Pegarm wrist angle1.0001.0001.000pass
Insert-PegAvoid-Pegfinger angle1.0001.0001.000pass
Insert-PegAvoid-Pegfingertip angle0.9960.9981.000pass
Insert-PegAvoid-Pegthumb angle1.0001.0001.000pass
Insert-PegAvoid-Pegthumbtip angle0.9940.9971.000pass
Insert-PegAvoid-Peghand position (x,z)0.9781.0001.000pass
Insert-PegAvoid-Peghand orientation (qw,qy)1.0001.0001.000pass
Insert-PegAvoid-Pegpeg position (x,z)1.0001.0001.000pass
Insert-PegAvoid-Pegpeg orientation (qw,qy)0.9990.9991.000pass
Insert-PegAvoid-Pegtarget position (x,z)1.0001.0001.000pass
Insert-PegAvoid-Pegtarget orientation (qw,qy)1.0001.0001.000pass
Insert-PegAvoid-Pegmotor command 01.0000.9821.000pass
Insert-PegAvoid-Pegmotor command 11.0001.0001.000pass
Insert-PegAvoid-Pegmotor command 21.0000.9911.000pass
Insert-PegAvoid-Pegmotor command 31.0001.0001.000pass
Insert-PegAvoid-Pegmotor command 41.0001.0001.000pass
Insert-PegExtract-Pegarm root angle0.9960.9751.000pass
Insert-PegExtract-Pegarm shoulder angle0.9890.9831.000pass
Insert-PegExtract-Pegarm elbow angle0.9880.9881.000pass
Insert-PegExtract-Pegarm wrist angle1.0000.9981.000pass
Insert-PegExtract-Pegfinger angle1.0001.0001.000pass
Insert-PegExtract-Pegfingertip angle0.9890.9911.000pass
Insert-PegExtract-Pegthumb angle1.0000.9971.000pass
Insert-PegExtract-Pegthumbtip angle0.9660.9601.000pass
Insert-PegExtract-Peghand position (x,z)1.0000.9981.000pass
Insert-PegExtract-Peghand orientation (qw,qy)1.0001.0001.000pass
Insert-PegExtract-Pegpeg position (x,z)0.9980.9971.000pass
Insert-PegExtract-Pegpeg orientation (qw,qy)0.9970.9971.000pass
Insert-PegExtract-Pegtarget position (x,z)0.9830.9781.000pass
Insert-PegExtract-Pegtarget orientation (qw,qy)1.0001.0001.000pass
Insert-PegExtract-Pegmotor command 01.0001.0001.000pass
Insert-PegExtract-Pegmotor command 11.0001.0001.000pass
Insert-PegExtract-Pegmotor command 21.0000.9871.000pass
Insert-PegExtract-Pegmotor command 31.0000.9981.000pass
Insert-PegExtract-Pegmotor command 41.0001.0001.000pass
CubeCube-Hopjoint pos 00.9950.9951.000pass
CubeCube-Hopjoint pos 10.9970.9981.000pass
CubeCube-Hopjoint pos 20.9740.9741.000pass
CubeCube-Hopjoint pos 30.9900.9911.000pass
CubeCube-Hopjoint pos 41.0001.0001.000pass
CubeCube-Hopjoint pos 50.9570.9601.000pass
CubeCube-Hopgripper opening0.9801.0001.000pass
CubeCube-Hopgripper contact1.0001.0001.000pass
CubeCube-HopEE yaw0.9580.961–pass
CubeCube-Hopdelta yaw1.0001.0001.000pass
PPQQFactor of variantsMahalanobis 0.99KDERangeFinal verdict
delta gripper1.0001.0001.000pass
EE pos (3D)0.9690.9701.000pass
block pos (3D)1.0001.0001.000pass
block quat (4D)1.0001.0001.000pass
delta pos (3D)1.0001.0001.000pass
CubeCube-Diagonaljoint pos 00.9890.9901.000pass
joint pos 10.9920.9911.000pass
joint pos 20.9900.9901.000pass
joint pos 30.9910.9921.000pass
joint pos 40.9780.9791.000pass
joint pos 51.0001.0001.000pass
gripper opening0.9890.9841.000pass
gripper contact1.0000.9851.000pass
EE yaw1.0001.000–pass
delta yaw1.0000.9921.000pass
delta gripper0.9950.9951.000pass
EE pos (3D)0.9890.9901.000pass
block pos (3D)0.9910.9901.000pass
block quat (4D)0.9970.9871.000pass
delta pos (3D)0.9890.9901.000pass

Table 3. Table 3 continued.

Statistical Test 2: Full-Dimensional Joint Distribution Support

We complement the factor-wise support tests in Table 3 with a full-dimensional joint test. The factor-wise tests verify whether each task-relevant factor or factor group is marginally supported by the reference data. In contrast, the joint test asks whether the complete state–action combination lies in a high-density region of the reference distribution. This test uses the concatenated vectors directly, without coordinate marginalization or worst-pair reduction.

For each transition, let aa denote the action, sagents_{\mathrm{agent}} the agent-observed state, and sothers_{\mathrm{other}} the task-object and target state. We evaluate three joint vectors:

zagent=sagent,zall=[sagent,sother],za,all=[a,sagent,sother].z_{\mathrm{agent}} = s_{\mathrm{agent}}, \qquad z_{\mathrm{all}} = [s_{\mathrm{agent}, s_{\mathrm{other}}], \qquad z_{a,\mathrm{all}} = [a, s_{\mathrm{agent}}, s_{\mathrm{other}}].}

Only rows that are finite in every coordinate of the corresponding vector are retained. Thus, terminal rows with undefined actions contribute to the state-only vectors but not to the action–state vector.

Familysagents_{\mathrm{agent}}salls_{\mathrm{all}}[a,sall][a,s_{\mathrm{all}}]
TwoRoomposition (2D)position (2D)4D
PushTagent position (2D)+ block pose (5D total)7D
CubeEE/gripper state (6D)+ block pose (13D total)18D
Manipulatorjoints + hand pose (20D)+ peg/target poses (28D total)33D

Table 4. Components and dimensionalities of the full-dimensional joint KDE.

For each reference vector set, we quantize each coordinate using reference-based limits with small padding, and assign zero density to target values outside these limits. We then whiten the quantized vectors using the full reference covariance with a small diagonal regularizer. This preserves correlations between action and state coordinates when computing kernel distances.

We fit a full-dimensional Gaussian KDE to the whitened reference vectors using Scott’s bandwidth rule. The density is evaluated without reference subsampling, dimension dropping, nearest-neighbor truncation, or pairwise marginalization. The density threshold is calibrated only from the reference samples:

τ=Q0.01({log⁡p^(xi)}i=1N).(15)\tau= Q_{0.01}\left(\left\{\log\hat{p}(x_i)\right\}_{i=1}^{N}\right). \tag*{(15)}

For target vectors Y={yj}j=1MY = \{y_j\}_{j=1}^{M}, we report the fraction whose log-density exceeds this reference-sample threshold:

ID⁡(Y;X)=1M∑j=1M[log⁡p^(yj)≥τ].(16)\operatorname{ID}(Y;X) = \frac{1}{M}\sum_{j=1}^{M} \left[ \log\hat{p}(y_j) \ge\tau\right]. \tag*{(16)}

The joint score is therefore a full-vector KDE support percentage. Individual factors can pass their marginal support tests while the concatenated vector has a low joint score, indicating that the corresponding combination of familiar values is rare under the reference distribution.

Interventional Action Set

For each environment, we choose a fixed interventional action set that remains within the empirical action support while covering the task-relevant intervention directions as uniformly as possible. For environments with direct 2D controls, this is a regular grid in action space. For higher-dimensional manipulation actions, we vary only the dimensions needed for the intended spatial intervention and keep the remaining dimensions fixed.

TwoRoom. For TwoRoom, actions are two-dimensional position deltas, so interventional interventions are defined directly in the action space. We use a fixed 3×33 \times3 grid that is uniformly spread over the two action dimensions and includes the zero action as the center intervention.

Selected TwoRoom interventional actions.

AcfTwoRoom={(−1−1−10−110−100011−11011)}={−1,0,1}×{−1,0,1}.\mathcal{A}_{\mathrm{cf}}^{\mathrm{TwoRoom}} = \left\{ \begin{pmatrix} -1 & -1\\ -1 & 0\\ -1 & 1\\ 0 & -1\\ 0 & 0\\ 0 & 1\\ 1 & -1\\ 1 & 0\\ 1 & 1 \end{pmatrix} \right\} = \{-1,0,1\} \times\{-1,0,1\}.

Each row is a 2D action.

Push-T. For Push-T, actions are also two-dimensional planar controls. We therefore use a fixed 3×33 \times3 grid in the planar action space. The grid is chosen to provide evenly distributed in-distribution interventions at the empirical action scale of the dataset.

Selected Push-T interventional actions.

AcfPushT={(000.250−0.2500.25−0.250.250.25−0.25−0.25−0.250.250−0.2500.25)}.\mathcal{A}_{\mathrm{cf}}^{\mathrm{PushT}} = \left\{ \begin{pmatrix} 0 & 0\\ 0.25 & 0\\ -0.25 & 0\\ 0.25 & -0.25\\ 0.25 & 0.25\\ -0.25 & -0.25\\ -0.25 & 0.25\\ 0 & -0.25\\ 0 & 0.25 \end{pmatrix} \right\}.

Each row is a 2D planar action.

Insert-Peg. Unlike the 2D navigation and planar pushing tasks, Insert-Peg actions are 5-dimensional motor commands rather than direct task-space displacements. Therefore, interventional actions cannot be chosen by simply placing actions on a Euclidean grid in task space. Instead, we select actions by their realized effect on the hand endpoint.

We first choose a representative state near the center of the expert-state distribution. Each candidate action is sampled from the dataset and repeated for five environment control steps from this same state. We then measure the resulting hand endpoint in the xx-zz plane. To ensure that interventions remain in distribution, we retain only actions whose 20-nearest-neighbor distance in the raw 5D action space is below the 95th-percentile threshold estimated from held-out dataset actions.

Among the retained actions, we select a fixed set of nine interventional actions whose realized endpoints form the widest approximately uniform 3×33 \times3 grid. The zero action is assigned to the center, and the remaining actions are chosen to maximize grid spacing while keeping each endpoint within 10% of its corresponding ideal grid point. The resulting grid spacing is 0.0358, with a maximum endpoint mismatch of 9.95% of the grid spacing.

Selected Insert-Peg actions.

AcfInsert={(0.6728−0.9814−0.9932−1.00000.99000.1353−1.00000.6557−0.99721.0000−0.6023−0.12040.31420.9943−1.00001.0000−0.8002−1.0000−1.00000.97390.00000.00000.00000.00000.0000−0.77971.0000−0.66720.6553−1.00000.9130−0.0082−1.0000−0.98201.0000−0.05131.0000−1.00000.57040.1936−0.58330.89580.82340.05820.1618)}.\mathcal{A}_{\mathrm{cf}}^{\mathrm{Insert}} = \left\{ \begin{pmatrix} 0.6728 & -0.9814 & -0.9932 & -1.0000 & 0.9900 \\ 0.1353 & -1.0000 & 0.6557 & -0.9972 & 1.0000 \\ -0.6023 & -0.1204 & 0.3142 & 0.9943 & -1.0000 \\ 1.0000 & -0.8002 & -1.0000 & -1.0000 & 0.9739 \\ 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 \\ -0.7797 & 1.0000 & -0.6672 & 0.6553 & -1.0000 \\ 0.9130 & -0.0082 & -1.0000 & -0.9820 & 1.0000 \\ -0.0513 & 1.0000 & -1.0000 & 0.5704 & 0.1936 \\ -0.5833 & 0.8958 & 0.8234 & 0.0582 & 0.1618 \end{pmatrix} \right\}.

Each row is a 5D motor command. The zero action is the center intervention.

Cube. For Cube, the raw action is five-dimensional, but the interventional protocols are designed to probe spatial sensitivity of end-effector motion. We therefore vary only translational action dimensions and keep Δyaw\Delta yaw and Δgripper\Delta\mathrm{gripper} fixed at zero. The horizontal protocol uses an xx-yy grid.

Selected Cube horizontal interventional actions.

AcfCube-xy={(−0.5427−0.9504000−0.54270000−0.54270.95040000−0.95040000000000.95040000.5427−0.95040000.542700000.54270.9504000)}.\mathcal{A}_{\mathrm{cf}}^{\mathrm{Cube\text{-}xy}} = \left\{ \begin{pmatrix} -0.5427 & -0.9504 & 0 & 0 & 0 \\ -0.5427 & 0 & 0 & 0 & 0 \\ -0.5427 & 0.9504 & 0 & 0 & 0 \\ 0 & -0.9504 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0.9504 & 0 & 0 & 0 \\ 0.5427 & -0.9504 & 0 & 0 & 0 \\ 0.5427 & 0 & 0 & 0 & 0 \\ 0.5427 & 0.9504 & 0 & 0 & 0 \end{pmatrix} \right\}.

Each row is a normalized 5D Cube action (ax,ay,az,Δyaw,Δgripper)(a_x, a_y, a_z, \Delta\mathrm{yaw}, \Delta\mathrm{gripper}).

Implementation Detail. Within original LeWM architecture, we measure action sensitivity using hidden representations from the predictor blocks. Let flf_l denote the last-token hidden representation after the ll-th ConditionalBlock,

fl=output[:, -1, :]f_l = \texttt{output[:, -1, :]}

obtained from a forward hook on model.predictor.transformer.layers[layer_idx]\texttt{model.predictor.transformer.layers[layer\_idx]}. Thus, flf_l is not the representation immediately after AdaLN modulation; it is the output of the full AdaLN-conditioned transformer block, including the attention and MLP residual updates. In all reported layer-wise analyses, we average the sensitivity scores across these block outputs. For the final prediction representation, we use pred[:, -1, :]\texttt{pred[:, -1, :]}, the output of model.predict(...)\texttt{model.predict(...)} after the final pred_proj\texttt{pred\_proj} layer.

Model Explanation

Our model selection is designed to span several substantially different approaches to visual world modeling rather than to concentrate on a single architectural family. In particular, we evaluate both joint-embedding predictive architectures (JEPAs) and generative visual world models. Within the JEPA family, we include methods that jointly learn perception and dynamics from the offline dataset as well as a method that keeps a large pretrained visual encoder frozen. Within the generative family, we include both a direct action-conditioned generator and a latent video diffusion model with an explicit visual encoder–decoder. This diversity is important for our central question: whether goal-induced combinatorial bias is specific to a particular representation or generative objective, or can arise more broadly when action-conditioned prediction is trained on a biased offline distribution. For consistency, all models use frame skip 5, history length 3, and one-step prediction, except Sensorimotor-WM, where we keep the original history length 1 to preserve its inverse-dynamics setup.

End-to-End World Models

LeWorldModel (LeWM). LeWorldModel [31] is an end-to-end joint-embedding predictive architecture trained directly from pixels and actions. An image encoder maps observations to compact latent representations, and an autoregressive transformer predictor models the next latent representation conditioned on the action. Actions are injected into each transformer layer through adaptive layer normalization (AdaLN-zero). Unlike approaches that rely on a frozen encoder, stop-gradient target network, or exponential-moving-average encoder, LeWM jointly optimizes its encoder and predictor.

A central design choice in LeWM is its anti-collapse regularization. In addition to the next-embedding mean-squared prediction loss, LeWM applies the Sketched-Isotropic-Gaussian Regularizer (SIGReg), which encourages the empirical latent distribution to match an isotropic Gaussian. The resulting model is particularly useful for our study because it represents a recent example of a fully end-to-end JEPA in which both the visual representation and dynamics predictor are shaped by the same offline trajectories.

Sensorimotor World Model (SMWM). The Sensorimotor World Model [16] is also an end-to-end JEPA, but uses a different principle to prevent representation collapse. Rather than prescribing the marginal geometry of the latent representation, SMWM introduces an inverse dynamics model that predicts the executed action from consecutive latent states. The encoder, forward dynamics model, and inverse model are optimized jointly using forward-prediction and inverse-dynamics losses.

The inverse objective requires consecutive representations to preserve information that is useful for recovering the action that produced the transition. Consequently, the learned representation is explicitly biased toward controllable, action-relevant structure. Architecturally, SMWM adopts the LeWM encoder and transformer forward predictor, including AdaLN-zero action conditioning, while replacing Gaussian latent regularization with inverse dynamics as its anti-collapse mechanism. This makes SMWM particularly relevant to our benchmark: it provides a strong test of whether explicitly encouraging action-aware representations is sufficient to prevent failures induced by goal-biased state–action coverage.

DINO World Model (DINO-WM). DINO-WM [58] separates representation learning from dynamics learning. Instead of jointly learning an image encoder from the offline control dataset, it uses pretrained DINOv2 patch features as the observation representation and keeps the DINOv2 encoder frozen during both world-model training and evaluation. A causal ViT-based transition model then predicts future patch representations from a history of latent observations and actions.

DINO-WM also differs in how actions enter the predictor. The action is first mapped through an MLP and the resulting action representation is concatenated to each spatial patch representation before the transformer processes the latent state. Thus, compared with LeWM and SMWM, DINO-WM provides a useful separation between perception and dynamics: the visual representation is inherited from large-scale pretraining, while only the action-conditioned transition model is learned from the potentially goal-biased offline trajectories. This lets us test whether combinatorial failures persist even when the visual encoder itself is not learned from the biased task dataset.

DriftWorld. DriftWorld [29] represents a substantially different, reconstruction-based family of world models. It is an action-conditioned generative model based on drifting rather than a JEPA or iterative diffusion model. Its core generator is a spatial–temporal U-Net that receives observation history, Gaussian noise, and a future action sequence and generates the corresponding future visual observations. Actions condition the U-Net frame-by-frame through FiLM, such that each future frame is explicitly conditioned on its corresponding action.

For relatively simple simulated domains such as PushT, DriftWorld operates directly in pixel space and does not require a learned visual state encoder between the observations and the action-conditioned generator. Its drifting objective moves generated samples toward ground-truth future observations and away from generated negatives, enabling future video generation in a single forward pass at inference time. DriftWorld therefore provides an important contrast to the latent JEPA models: prediction and visual generation are implemented by a single action-conditioned generative backbone rather than an explicit encoder followed by a latent dynamics predictor.

IRASim. IRASim [59] represents the latent generative end of our model suite. It formulates world modeling for robot manipulation as trajectory-conditioned video generation. Frames are first compressed using a pretrained VAE, after which a Diffusion Transformer (DiT) models future video latents conditioned on historical observations and a robot action trajectory. The generated latents are subsequently decoded back to RGB observations.

A defining component of IRASim is its fine-grained action conditioning. Rather than conditioning the complete video only on a global trajectory embedding, IRASim can associate individual actions with their corresponding generated frames through frame-level adaptive normalization. This architecture gives us a generative model that is structurally different from DriftWorld: visual prediction occurs in an encoded latent space, the dynamics generator is a diffusion transformer, and an explicit encoder–decoder surrounds the dynamics model.

Predictor-Only Models from True States

End-to-end evaluation mixes perception and dynamics-prediction errors. To isolate the latter, we use predictor-only models that replace the visual encoder with oracle states from simulator metadata and proprioceptive signals. These oracle states contain task-relevant physical variables, such as positions, orientations, agent or gripper configurations, and other environment-specific state factors needed for one-step prediction.

We refer to these inputs as oracle state representations rather than unique or minimal Markov states. Their role is operational: they provide the predictor with relevant physical information directly, removing visual representation learning as the explanation for prediction failure. We instantiate three predictor families derived from the end-to-end models above. The two transformer-based predictor-only models are trained with next-state mean-squared error,

Lpred=∥s^t+1−st+1∥22.(17)\mathcal{L}_{\mathrm{pred}} = \left\lVert\hat{s}_{t+1} - s_{t+1} \right\rVert_2^2. \tag*{(17)}

Toy-DriftWorld instead retains the drifting particle loss used by its generative backbone.

AdaLN Transformer Predictor. Our first predictor follows the forward-model architecture shared by LeWM and SMWM. Ground-truth state factors replace the learned visual embedding, while the action is injected into the transformer through AdaLN-zero. The transformer predicts the next true state directly. Because LeWM and SMWM use the same forward-predictor family, a single predictor-only architecture represents both models in this controlled setting. Their end-to-end distinction—SIGReg versus inverse-dynamics representation regularization—is intentionally removed once the encoder is replaced by the oracle state.

Action-Duplicated Transformer Predictor. Our second predictor follows DINO-WM’s transition-model conditioning strategy. The true state replaces DINO patch features, while an embedded action is attached to the state representation before processing by a causal transformer. This preserves the central architectural distinction of DINO-WM’s predictor: action information enters through the predictor input rather than modulating transformer layers through AdaLN. The model is again trained only to regress the next true state with MSE.

DriftWorld-Style Generative Predictor. Our third predictor is derived from the action-conditioned generative backbone of DriftWorld. Because DriftWorld does not require a separate visual encoder for the simple simulated setting, its U-Net constitutes the principal predictive computation. For the predictor-only experiment, we use a reduced-size version of this backbone and replace pixel observations and outputs with the corresponding true-state representation. The predictor retains the drifting particle loss while operating on oracle-state inputs and next-state targets.

Oracle States

Oracle-state predictors replace RGB observations with task-relevant physical state variables from each base environment.

Base environmentDim.Oracle state variables
TwoRoom2DAgent position (x,y)(x, y).
Push-T10DGoal position (x,y)(x, y), goal orientation (sin⁡θ,cos⁡θ)(\sin\theta, \cos\theta), block position (x,y)(x, y), block orientation (sin⁡θ,cos⁡θ)(\sin\theta, \cos\theta), and agent position (x,y)(x, y).
Cube15DEnd-effector position (x,y,z)(x, y, z), end-effector yaw (cos⁡θ,sin⁡θ)(\cos\theta, \sin\theta), gripper opening, gripper contact, gripper velocity, block position (x,y,z)(x, y, z), and block quaternion (w,x,y,z)(w, x, y, z).
Manipulator Insert-Peg33DArm joint positions, touch sensors, hand pose (x,z,qw,qy)(x, z, q_w, q_y), peg pose (x,z,qw,qy)(x, z, q_w, q_y), and target pose (x,z,qw,qy)(x, z, q_w, q_y).

Table 5. Oracle state variables used by predictor-only models for each base environment.

Model Architecture Details

For LeWM Maes et al. [31] and Sensorimotor World-Model Ivashkov et al. [16], we use the authors’ official implementations. For DINO-WM Zhou et al. [58], we use the Stable-WorldModel Maes et al. [30] implementation without proprioceptive inputs. For DriftWorld Lu et al. [29], we use the authors’ implementation with two minor modifications: we integrate frame skip into the action-conditioning interface and add a pixel-space cost function for MPC planning, without changing the main architecture. We use the Push-T model configuration for TwoRoom and Push-T, and the Robomimic model configuration for the Cube and Manipulator environments. For IRASim Zhu et al. [59], we use the authors’ implementation with the same frame-skip integration and pixel-space planning cost, and use IRASim-XL/2 for all environments.

For the oracle-state experiments, we use three toy predictor families derived from the corresponding end-to-end modeling motifs. Toy-AdaLN keeps the LeWM transformer size unchanged, replacing only the pixel-latent input and output with true-state projections. Toy Action-Duplicate preserves the DINO-WM-style action duplication, causal masking, and shifted prediction design, but adapts it to scalar true-state tokens with a smaller 192D predictor instead of the original high-dimensional patch-token predictor. Toy-DriftWorld preserves the stochastic particle-generation objective and drifting loss, but replaces the image U-Net with a compact vector-valued residual FiLM MLP operating on oracle states.

Training Details

AdaLN predictors and Action duplicated predictors use AdamW, batch size 256, weight decay 10−310^{-3}, linear warmup followed by cosine decay, gradient clipping at 1.0, and an MSE loss over all shifted state predictions. Toy-DriftWorld is trained with AdamW, batch size 128, learning rate 10−410^{-4}, weight decay 10−210^{-2}, a 500-step warmup followed by a constant schedule, gradient clipping at 2.0, and the drifting particle loss. For end-to-end world models, we follow the original settings.

Detailed Planning and Evaluation Configuration

All planning experiments use the same MPC evaluation pipeline based on the Stable World Model library Maes et al. [30], with a Cross-Entropy Method (CEM) solver.

DatasetModelLRConfigured durationSelected checkpoint
Push-TLeWM--official LeWM checkpoint
Push-TSensorimotor-WM1×10−41 \times10^{-4}10 epochsepoch 10
Push-TDINO-WM--official DINO-WM checkpoint
Push-TDriftWorld5×10−45 \times10^{-4}max 500k stepsstep 500k
Push-TIRASim5×10−55 \times10^{-5}max 300k stepsstep 300k
CubeLeWM--official LeWM checkpoint
CubeSensorimotor-WM1×10−41 \times10^{-4}10 epochsepoch 10
CubeDINO-WM--official DINO-WM checkpoint
CubeDriftWorld5×10−55 \times10^{-5}300-epoch capstep 100k
CubeIRASim5×10−55 \times10^{-5}max 300k stepsstep 300k
TwoRoomLeWM--official LeWM checkpoint
TwoRoomSensorimotor-WM1×10−41 \times10^{-4}10 epochsepoch 10
TwoRoomDINO-WM--official DINO-WM checkpoint
TwoRoomDriftWorld1×10−41 \times10^{-4}300-epoch capstep 500k
TwoRoomIRASim5×10−55 \times10^{-5}max 300k stepsstep 300k
Insert-PegLeWM5×10−55 \times10^{-5}10 epochsepoch 10
Insert-PegSensorimotor-WM1×10−41 \times10^{-4}10 epochsepoch 10
Insert-PegDINO-WM5×10−45 \times10^{-4}10 epochsepoch 10
Insert-PegDriftWorld5×10−55 \times10^{-5}max 100k stepsstep 100k
Insert-PegIRASim5×10−55 \times10^{-5}max 300k stepsstep 300k

Table 6. Training details and selected checkpoints for end-to-end world models.

DatasetModelselected LRConfigured durationSelected checkpoint
Push-TToy-AdaLN3×10−43 \times10^{-4}100 epochsepoch 100
Push-TToy-Action-Duplicate3×10−43 \times10^{-4}100 epochsepoch 100
Push-TToy-DriftWorld1×10−41 \times10^{-4}50 epochs, max 500k stepsstep 500k
CubeToy-AdaLN3×10−43 \times10^{-4}100 epochsepoch 100
CubeToy-Action-Duplicate3×10−43 \times10^{-4}100 epochsepoch 100
CubeToy-DriftWorld1×10−41 \times10^{-4}50 epochs, max 500k stepsstep 500k
TwoRoomToy-AdaLN3×10−43 \times10^{-4}100 epochsepoch 100
TwoRoomToy-Action-Duplicate3×10−43 \times10^{-4}100 epochsepoch 100
TwoRoomToy-DriftWorld1×10−41 \times10^{-4}50 epochs, max 500k stepsepoch 50
Insert-PegToy-AdaLN3×10−43 \times10^{-4}100 epochsepoch 100
Insert-PegToy-Action-Duplicate3×10−43 \times10^{-4}100 epochsepoch 100
Insert-PegToy-DriftWorld1×10−41 \times10^{-4}50 epochs, max 500k stepsepoch 50

Table 7. Training details for the selected oracle-state predictor checkpoints.

Planning Evaluation Criteria

Table 8 summarizes the success criteria used for each environment. Some details are below.

EnvironmentPlanning success criterion
TwoRoomThe 2D Euclidean distance between the agent position and the goal agent position is less than 16 pixels. Target and door positions are not used.
PushTThe 4D Euclidean error over agent and block positions is less than 20 pixels, and the block angle error is less than π/9\pi/9 radians. The angle error accounts for object symmetries.
CubeThe rollout must simultaneously satisfy: end-effector 3D position error ≤0.04\le0.04 m, end-effector yaw error ≤0.10\le0.10 rad, and cube 3D position error ≤0.04\le0.04 m. Cube orientation, gripper opening, contact, and velocity are not used.
Insert PegThe rollout must simultaneously satisfy: hand 2D position error ≤0.04\le0.04 m, hand planar angle error ≤0.10\le0.10 rad, peg 2D position error ≤0.04\le0.04 m, and peg planar angle error ≤0.10\le0.10 rad. Arm joints, touch, velocity, and the physical target or slot pose are not used.

Table 8. Planning success criteria. Goals are defined by the dataset future state 25 raw steps after the initial state. An episode is successful if the criterion is satisfied at least once within 50 environment steps.

TwoRoom. For TwoRoom, the goal is the dataset future agent position. A rollout is successful when ∥pt−pg∥2<16\lVert p_t - p_g \rVert_2 < 16, where ptp_t is the current agent position and pgp_g is the goal agent position. Distances are measured in pixels in the 224×224224 \times224 environment coordinate system. Target and door positions are not used.

PushT. For PushT, the position error is computed over the 4D vector [agentx,agenty,blockx,blocky][\mathrm{agent}_x,\mathrm{agent}_y,\mathrm{block}_x,\mathrm{block}_y]. A rollout is successful when both ∥st−sgpos∥2<20\lVert s_t - s_g^{\mathrm{pos}} \rVert_2 < 20 and dangle(θt,θg)<π/9d_{\mathrm{angle}}(\theta_t,\theta_g) < \pi/9 hold at the same timestep. The angle distance accounts for object symmetries: square and + shapes use 90° symmetry, Z and I shapes use 180° symmetry, and other shapes use 360°.

Cube. For Cube, the goal is defined by the dataset future end-effector pose and cube position. A rollout is successful when ∥ptee−pgee∥2≤0.04\lVert p^{ee}_{t} - p^{ee}_{g} \rVert_{2} \le0.04 m, ∣wrap⁡(ψgee−ψtee)∣≤0.10\lvert\operatorname{wrap}(\psi^{ee}_{g} - \psi^{ee}_{t}) \rvert\le0.10 rad, and ∥ptcube−pgcube∥2≤0.04\lVert p^{cube}_{t} - p^{cube}_{g} \rVert_{2} \le0.04 m all hold at the same timestep. Cube orientation, gripper opening, contact, and velocity are not used. Once satisfied, episode-level success remains true. Cube rollouts are not terminated early by this criterion.

Manipulator Insert Peg. For Manipulator Insert Peg, the goal is defined by the dataset future hand and peg poses. Positions are measured in the (x,z)(x,z) plane, and planar angles are computed from quaternion components (qw,qy)(q_w,q_y) as 2atan2⁡(qy,qw)2\operatorname{atan2}(q_y,q_w). A rollout is successful when the hand position, hand angle, peg position, and peg angle errors are all within their thresholds: 0.04 m for positions and 0.10 rad for angles. Arm joints, touch sensors, velocity, and the physical target or slot pose are not used. This dataset-goal success metric differs from the native task success based on the DeepMind Control shaped reward.

Configuration

We evaluated all true-state dynamics models using the same goal-conditioned receding-horizon controller. For each episode, the initial state and goal were sampled from the same dataset trajectory with a temporal offset of 25 environment steps. Planning was performed in the normalized action space using the cross-entropy method (CEM). Each candidate consisted of five action blocks, where each block contained five consecutive environment actions, yielding an effective planning horizon of 25 environment steps. At every planning round, CEM evaluated 300 candidate sequences for 30 iterations and refitted a factorized Gaussian distribution to the 30 lowest-cost elites. The objective was the squared Euclidean distance between the terminal predicted state and the goal in the model’s normalized state space. The final mean sequence was executed open-loop for all five action blocks before replanning. Episodes were evaluated for at most 50 environment steps, corresponding to at most two planning rounds.

Results

Full Results

This section reports the full numerical results used in the main analysis. We include absolute MPC planning success rates complementing the relative comparisons shown in the main text, as well as a table of relative prediction errors with absolute base-environment rollout errors. For the five-step prediction error, corresponding to 25 environment steps. Avoid-T is omitted because its episodes are too short to support this horizon.

Base-EnvHorizonLeWMSMWMDINO-WMDriftWorldIRASimAdaLNAction Dup.Toy-Drift
TwoRoom50.0651880.0000580.0338410.0001540.0038380.000004660.00001440.000051
TwoRoom250.5118430.0007540.232770.0021640.0236460.0001450.0006360.00111671
PushT50.007430.029330.051240.0013310.0058130.0000050.0000280.000066
PushT250.167390.230390.486490.0185990.0501630.0002290.0010680.00388751
Insert Peg50.0007170.0029130.1727370.0014480.009390.0035260.0093560.054486
Insert Peg250.0106780.0247291.4103280.0131130.05930.0535530.1492510.404549
Cube50.004310.002030.152490.000950.010270.0000560.0000450.0521936
Cube250.078750.01930.897570.0110230.0721070.0004860.0003640.458214

Table 9. Absolute base-task rollout MSE at 5- and 25-step prediction horizons.

EnvVariantLeWMSMWMDINO-WMDriftWorldIRASimAdaLNAction Dup.Toy-Drift
TwoRoomTR-Route1.080.270.970.010.900.051.030.01
TwoRoomAvoidT1.421.591.140.710.771.191.281.32
PushTStackT8.012.122.382.541.36475.61115.5770.45
PushTAnti-goal4.791.912.061.641.0951.9513.0720.02
PushTAvoid-Peg6.831.881.423.181.011.450.620.08
Insert PegExtract-Peg4.432.991.312.591.0953.5320.013.55
CubeCubeHop22.5383.971.461.381.1913.9630.120.06
CubeCube Diagonal1.705.831.121.090.97940.341026.480.58

Table 10. Base-relative one-step (5 environment step) prediction error. Values report the error ratio with respect to the corresponding base task.

EnvVariantLeWMSMWMDINO-WMDriftWorldIRASimAdaLNAction Dup.Toy-Drift
TwoRoomTR-Route0.940.190.760.010.820.060.820.01
PushTStackT5.562.332.222.341.44158.7947.5922.09
PushTAnti-goal4.272.091.911.721.2830.0312.698.96
PushTAvoid-Peg10.545.522.102.941.062.971.340.36
Insert PegExtract-Peg4.087.401.742.751.1934.3912.524.49
CubeCubeHop21.92103.721.751.451.3732.24181.400.10
CubeCube Diagonal1.5411.321.121.050.97841.691202.220.46

Table 11. Base-relative five-step (25 environment step) prediction error. Values report the error ratio with respect to the corresponding base task.

Factor Support versus Observation Support

Our compositional shifts are defined at the level of physical factors: test queries recombine factor values that are individually observed in the training data. However, this does not imply that the resulting observations are in-distribution for an end-to-end encoder.

Let ZtrZ^{\mathrm{tr}} denote the joint support of physical factors under the training trajectories, and let Z×=∏k=1KZktrZ^{\times} = \prod_{k=1}^{K} Z_{k}^{\mathrm{tr}} denote the Cartesian product of their marginal supports. A compositional query may satisfy Ztte⊆Z×Z_{t}^{\mathrm{te}} \subseteq Z^{\times} while still containing factor combinations absent from ZtrZ^{\mathrm{tr}}.

TaskLeWMSMWMDINO-WMDriftWorldIRASimAdaLNAction Dup.Drift Toy
TwoRoom89.30100.00100.0090.0076.00100.0099.30100.00
TR-Route80.67100.00100.0073.3362.00100.00100.00100.00
Push-T84.7084.7074.0043.305.3092.7092.0089.30
AvoidT27.3385.3311.330.000.00100.0099.3399.33
StackT46.0032.0040.6719.334.0070.6767.3354.00
Push-T Antigoal56.6750.6742.0014.005.3381.3380.6776.00
Insert-Peg26.6770.0035.3363.332.4374.0071.3371.33
Avoid-Peg28.0074.0037.3368.6735.3392.0089.3380.67
Extract-Peg6.0024.0011.3318.679.3316.6725.3319.33
Cube30.7072.7060.0030.7010.7030.0020.7025.30
Cube-Hop7.3375.3388.6748.676.0095.3358.0078.67
Cube-Diagonal27.3352.0055.3334.677.3318.6711.3318.67

Table 12. Absolute planning success rate (%) across different models and tasks.

End-to-end world models observe s=f(z)s = f(z), where ff is the rendering map. Thus the corresponding observation supports are Str=f(Ztr)S^{\mathrm{tr}} = f(Z^{\mathrm{tr}}) and S×=f(Z×)S^{\times} = f(Z^{\times}). When ff mixes factors, unseen factor combinations in Z×∖ZtrZ^{\times} \setminus Z^{\mathrm{tr}} can induce observations in S×∖StrS^{\times} \setminus S^{\mathrm{tr}}.

Therefore, a query can be factor-wise in support while still creating an observation-level shift for an end-to-end encoder. This is why we separately analyze oracle-state predictors: they remove the encoder-inference problem and isolate whether the learned dynamics model can generalize across recombined state-action factors.

Additional Observations

Prediction Space Affects Metric Interpretation We do not use the rollout results to directly rank JEPA-style and generative world models. These models predict in different spaces: JEPA-style models are evaluated in latent space, whereas generative models are evaluated in pixel or reconstruction space. As a result, absolute prediction errors are not directly comparable across model families, and we primarily interpret rollout degradation relative to each model’s own base-task performance.

Under this relative comparison, generative models often show smaller rollout degradation under some goal shifts, while several JEPA-style models show larger increases. However, this tendency does not translate consistently to planning: a model can show stable rollout error but poor MPC success, or larger rollout degradation while remaining competitive in control. We therefore treat prediction-space effects as an interpretation caveat rather than a model-family ranking. The results suggest that rollout robustness and downstream control utility are related but distinct properties.

Oracle state access does not guarantee better planning. Interestingly, access to the oracle state does not always lead to stronger planning performance. In Cube, several end-to-end models achieve higher planning success than their oracle-state counterparts, both on the base task and on CubeDiagonal. This should not be interpreted as evidence that learned latents are generally superior to physical state representations, since the end-to-end and oracle predictors differ in more than just their input space.

Rather, these results suggest that state observability and planning suitability are distinct properties. Oracle states preserve the underlying physical factors, but the resulting geometry may not be the most favorable one for the planner’s cost and optimization procedure. Conversely, a learned representation may, in some cases, emphasize task-relevant variation or compress nuisance factors in a way that makes goal-directed optimization easier. We therefore treat planning performance as depending not only on how much state information is available, but also on how that information is organized in the representation space.

Toward Compositional Generalization in World Models

Our experiments empirically show that removing visual abstraction is not sufficient for reliable goal-shift generalization: free-space dynamics often transfer across new state–action compositions, whereas contact-rich dynamics remain substantially harder even when the predictor receives taskrelevant physical state directly. This raises a dynamics-level question: what structure must a world model learn for its dynamics to remain reusable under compositional shifts?

Ideally, we want to solve AnyGoal by learning reusable dynamics that extrapolate to the combinatorial space. That is, we want a learned transition function T^\hat{T} that matches the observed training transitions on CtrC^{\mathrm{tr}}, while also predicting correctly on novel compositions in C×∖CtrC^{\times}\setminus C^{\mathrm{tr}}. We separate this question into two parts: identifying reusable mechanisms on the training support, and extrapolating those mechanisms to novel compositions.

Note. The discussion below should be read as a conditional statement, not a claim that reusable mechanisms are identifiable from expert rollouts alone. Rather, we give one possible sufficient condition: if sparse regime-conditioned local mechanisms are identified on training support and test goals query only locally covered active inputs, then the model can generalize to globally novel state–action compositions.

Identifying Reusable Mechanisms on the Training Support

To discuss identifiability, we first need to specify what structure the true transition may have. One observation from our results is that world models often face multiple regimes, such as free motion, contact, grasping, or collision.

In a single-regime setting with little interaction, such as free-space navigation of an agent, one may expect a transition of the form zt+1=f(zt)+g(at)z_{t+1}=f(z_t)+g(a_t). In this favorable case, identifiability and compositional generalization are easier to obtain, for example through an additive function class (Dong & Ma, 2022; Lachapelle et al., 2023; Wiedemer et al., 2024b;a; Mahajan et al., 2025).

However, world-model dynamics are often regime-dependent and more complex than a purely additive and separated form. The action effect is usually better written as gr(zt,at)g_r(z_t,a_t) rather than g(at)g(a_t), meaning that the transition cannot be decomposed into simple independent state-only and action-only terms. A simple additive form is therefore too limited for the setting we want to characterize.

We instead consider a factorized transition with regime-dependent local mechanisms. For each factor kk, let

zk,t+1=Tk,rt(uk,t),uk,t=(zpa(k),t,apa(k),t).z_{k,t+1}=T_{k,r_t}(u_{k,t}),\qquad u_{k,t}=(z_{\mathrm{pa}(k),t},a_{\mathrm{pa}(k),t}).

where pa(k)\mathrm{pa}(k) denotes the local parents of factor kk, and rt∈{1,…,R}r_t\in\{1,\ldots,R\} denotes a discrete local regime. The transition kernel is specified as follows

Trt(zt,at)=(T1,rt(u1,t),…,TK,rt(uK,t)).T_{r_t}(z_t,a_t)=\left(T_{1,r_t}(u_{1,t}),\ldots,T_{K,r_t}(u_{K,t})\right).

Since multiple regime–mechanism decompositions can explain the same training transitions, we next state the favorable assumptions under which the discussion applies: a Markov, goal-invariant oracle state, and training-support identification of the relevant local mechanisms up to observational equivalence.

Assumption 1 (Markov and invariant oracle state) The oracle state ztz_t is Markov, and the transition law is invariant across training and test goals:

p(zt+1∣z≤t,a≤t)=p(zt+1∣zt,at),Ttr(zt+1∣zt,at)=Tte(zt+1∣zt,at).p(z_{t+1}\mid z_{\leq t},a_{\leq t})=p(z_{t+1}\mid z_t,a_t),\qquad T_{\mathrm{tr}}(z_{t+1}\mid z_t,a_t)=T_{\mathrm{te}}(z_{t+1}\mid z_t,a_t).

Thus, goal shifts change the behavior distribution, but not the environment dynamics.

Assumption 2 (Identification of local mechanisms) For each state factor kk, suppose that the interaction regime rtr_t and the local input uk,t=uk(zt,at)u_{k,t}=u_k(z_t,a_t) are observed or identifiable from the training transitions, and that the corresponding regime can be consistently inferred at prediction time.

zk,t+1=Tk,rt(uk,t),z^k,t+1=T^k,rt(uk,t).z_{k,t+1}=T_{k,r_t}(u_{k,t}),\qquad\hat{z}_{k,t+1}=\hat{T}_{k,r_t}(u_{k,t}).

For each factor and regime, define the training support

Uk,rtr:=⋃t=0H−1supp⁡ptr(Uk,t∣Rt=r).\mathcal{U}_{k,r}^{\mathrm{tr}}:=\bigcup_{t=0}^{H-1}\operatorname{supp}_{p_{\mathrm{tr}}}(U_{k,t}\mid R_t=r).

Let Ltr(T~)\mathcal{L}_{\mathrm{tr}}(\widetilde{T}) denote the population objective used to train the world model. We assume that this objective identifies the regime-conditioned mechanisms on their training support: any population minimizer T~\widetilde{T} satisfies

T~k,r(u)=Tk,r(u),∀u∈Uk,rtr,\widetilde{T}_{k,r}(u) = T_{k,r}(u), \qquad\forall u \in\mathcal{U}^{\mathrm{tr}}_{k,r},

for every k,rk,r, up to observational equivalence and regime relabeling.

For a model T^\widehat{T} learned from finite data, we allow estimation and optimization error

ϵk,r:=sup⁡u∈Uk,rtr∥T^k,r(u)−Tk,r(u)∥2,\epsilon_{k,r} := \sup_{u \in\mathcal{U}^{\mathrm{tr}}_{k,r}} \lVert\widehat{T}_{k,r}(u) - T_{k,r}(u) \rVert_2,

and assume ϵk,r→0\epsilon_{k,r} \to0 as the amount of relevant training data grows and the training objective is optimized consistently.

Extrapolating Identified Mechanisms to Novel Compositions

Training-support identification does not by itself imply compositional generalization. Two mechanisms can agree on all observed training transitions and still disagree on a novel state–action composition induced by a goal shift. Recombination therefore requires additional structure that separates coordinates that affect a mechanism from coordinates that are only part of the global state.

Additive or separable functions provide one favorable case, which allow us to reuse the learned mechanisms for each factor zk,t+1z_{k,t+1}, but contact-rich world-model dynamics are often regime-dependent rather than purely additive. We propose a simpler scenario where each transition mechanisms Tk,r,tT_{k,r,t} depends sparsely on the inputs st,ats_t, a_t. For a fixed regime rr, let xt=(zt,at)x_t = (z_t, a_t), then

zk,t+1=Tk,r,t(xt).z_{k,t+1} = T_{k,r,t}(x_t).

Sparsity of ∇xTk,r,t(x)\nabla_x T_{k,r,t}(x) implies that only a subset of coordinates affects the kk-th factor transition, while the remaining coordinates may change the global composition without changing this local mechanism.

Assumption 3 (Sparse local mechanisms) For each factor kk and regime rr, there exists a fixed coordinate set Pk⊆{1,…,dim⁡(x)}P_k \subseteq\{1,\ldots,\dim(x)\}, with ∣Pk∣≪dim⁡(x)\lvert P_k\rvert\ll\dim(x), such that

Tk,r(x)=Tk,r(xPk).T_{k,r}(x) = T_{k,r}(x_{P_k}).

Equivalently,

∂Tk,r(x)∂xj=0∀j∉Pk.\frac{\partial T_{k,r}(x)}{\partial x_j} = 0 \qquad\forall j \notin P_k.

We write uk,t=xPk,tu_{k,t} = x_{P_k,t} for the inputs relevant to the kk-th transition mechanism.

Sparse local mechanism can allow us to create favorable cases when a goal shift creates a novel global composition without creating novel local mechanism inputs.

Assumption 4 (Local support coverage) Let

Uk,rte:=⋃t=0H−1supp⁡pte(Uk,t∣Rt=r)\mathcal{U}^{\mathrm{te}}_{k,r} := \bigcup_{t=0}^{H-1} \operatorname{supp}_{p_{\mathrm{te}}}(U_{k,t} \mid R_t = r)

denote the test support of the inputs of transition mechanism for factor kk. For every factor kk, assume

Uk,rte⊆Uk,rtr.\mathcal{U}^{\mathrm{te}}_{k,r} \subseteq\mathcal{U}^{\mathrm{tr}}_{k,r}.

Thus, the global state–action composition may be novel, but every local mechanism is queried only on local inputs covered during training.

Intuition. Within a fixed regime, sparse local support separates active & inactive coordinates. The active coordinates uku_k determine the transition mechanism for the factor zkz_k, while coordinates outside uku_k can be recombined to create globally novel state–action compositions without changing that local mechanism. Thus, the sparser the mechanism, the larger the set of globally novel compositions that can be covered by the same local transition rule.

Proposition 1 (Local active-coordinate coverage controls global error) Let Ltr(T^)\mathcal{L}_{\mathrm{tr}}(\widehat{T}) denote the learning objective used to train the world model. Suppose that we perfectly optimize the world model on training support, then under assumption 1 and 2, we identify the learned model such that the true transition and learned transition share the same regime-conditioned local decomposition,

T(zt,at)=(T1,rt(uk,t),…,TK,rt(uk,t)),T(z_t,a_t) = (T_{1,r_t}(u_{k,t}),\ldots,T_{K,r_t}(u_{k,t})),

and

T^(zt,at)=(T^1,rt(uk,t),…,T^K,rt(uk,t)).\widehat{T}(z_t,a_t) = (\widehat{T}_{1,r_t}(u_{k,t}),\ldots,\widehat{T}_{K,r_t}(u_{k,t})).

Assume that, for every factor kk and regime rr, we satisfy the assumption of sparse local mechanisms (assumption 3) and local support coverage (assumption 4).

Then, for any test composition (zt,at)∈Cte(z_t,a_t) \in\mathcal{C}^{\mathrm{te}},

∥T^(zt,at)−T(zt,at)∥2≤(∑k=1Kϵk,rt2)1/2.\left\lVert\widehat{T}(z_t,a_t)-T(z_t,a_t)\right\rVert_2 \le\left(\sum_{k=1}^{K}\epsilon_{k,r_t}^{2}\right)^{1/2}.

where

ϵk,r:=sup⁡u∈Uk,rtr∥T^k,r(u)−Tk,r(u)∥2.\epsilon_{k,r} := \sup_{u\in U_{k,r}^{\mathrm{tr}}}\left\lVert\widehat{T}_{k,r}(u)-T_{k,r}(u)\right\rVert_2.

Consequently, if ϵk,r→0\epsilon_{k,r} \to0 for all k,rk,r, then the world model generalizes to novel combinations present in the test support.

Proof. Take any test composition (zt,at)∈Cte(z_t,a_t) \in\mathcal{C}^{\mathrm{te}} with interaction regime rtr_t. By Assumption 4, for every factor kk,

uk,t∈Uk,rtte⊆Uk,rttr.u_{k,t} \in U_{k,r_t}^{\mathrm{te}} \subseteq U_{k,r_t}^{\mathrm{tr}}.

Thus, each local mechanism is evaluated within its regime-conditioned training support. By Assumption 2,

∥T^k,rt(uk,t)−Tk,rt(uk,t)∥2≤ϵk,rt.\left\lVert\widehat{T}_{k,r_t}(u_{k,t})-T_{k,r_t}(u_{k,t})\right\rVert_2 \le\epsilon_{k,r_t}.

Since the true and learned transitions share the same factorized decomposition,

∥T^(zt,at)−T(zt,at)∥22=∑k=1K∥T^k,rt(uk,t)−Tk,rt(uk,t)∥22\left\lVert\widehat{T}(z_t,a_t)-T(z_t,a_t)\right\rVert_2^2 = \sum_{k=1}^{K}\left\lVert\widehat{T}_{k,r_t}(u_{k,t})-T_{k,r_t}(u_{k,t})\right\rVert_2^2
≤∑k=1Kϵk,rt2.\le\sum_{k=1}^{K}\epsilon_{k,r_t}^2.

Taking the square root completes the proof.

Discussion. This construction should be read as an illustrative sufficient condition rather than a complete solution. Sparse local mechanisms reduce a high-dimensional global coverage problem to smaller local coverage problems, but they do not by themselves solve mechanism identification, regime discovery, or extrapolation to genuinely new local interactions. The broader challenge is to develop world-model architectures, objectives, and data-collection strategies that identify reusable action-conditioned mechanisms and impose enough structure on their interactions to support generalization beyond expert rollouts.

AnyGoal Library and Reproducibility

We release AnyGoal as an open-source Python library for collecting goal-shifted datasets, training world models, and evaluating rollout prediction and MPC planning. The library provides both Python APIs and a unified CLI, allowing researchers to reproduce the benchmark experiments and evaluate additional models under the same protocol. AnyGoal builds on Stable-WorldModel ([30]) and retains the original model implementations and numerical protocols. Complete installation instructions and API references are available in the online documentation.

Installation and Dataset Access

The library can be installed with evaluation dependencies using uv.

pip install uv
uv venv --python 3.10
source .venv/bin/activate
uv pip install "anygoal[eval]"

The default installation provides dataset access, collection, validation, and simulation environments, while the eval and all extras add evaluation and training dependencies, respectively.

The CLI exposes a registry of supported datasets and models.

# Inspect available datasets and models
anygoal list datasets
anygoal list models

# Inspect default data and checkpoint paths
anygoal config path data
anygoal config path checkpoints

Datasets and pretrained checkpoints can be downloaded from Hugging Face through the artifact interface.

# Download training and test datasets
anygoal artifacts download --dataset pusht
anygoal artifacts download --dataset avoidt

# Download a pretrained model checkpoint
anygoal artifacts download --model lewm

Model integrations use the corresponding upstream implementations. Their source code can be installed separately.

anygoal models list
anygoal models install lewm

Dataset Collection Beyond the released evaluation episodes, AnyGoal provides tools for generating additional goal-shifted trajectories. Dataset collection is available through both Python and CLI interfaces, which use the same in-package collectors.

The following example collects 400 Avoid-T episodes using eight parallel environments.

import anygoal

dataset = anygoal.collect(
    "avoidt",
    num_episodes=400,
    num_parallel=8,
    seed=0,
)

The equivalent collection procedure can be invoked from the terminal.

anygoal collect avoidt --num-episodes 400 --num-parallel 8

# Collect another goal-shifted variant
anygoal collect cubediagonal --output /path/to/cubediagonal.h5

World-Model Evaluation

AnyGoal provides standardized interfaces for evaluating a base-trained world model on both its original task and goal-shifted variants. The same model checkpoint can be used for rollout prediction and downstream MPC planning, enabling comparisons across evaluation metrics.

Planning evaluation. The following commands evaluate LeWM on Push-T and Avoid-T.

# Original training goal
anygoal eval planning --model lewm --dataset pusht --seed 0

# Unseen goal
anygoal eval planning --model lewm --dataset avoidt --seed 0

Rollout evaluation. The corresponding rollout-prediction evaluation uses the same model and dataset identifiers.

anygoal eval rollout --model lewm --dataset pusht --seed 0
anygoal eval rollout --model lewm --dataset avoidt --seed 0

By default, the evaluation commands resolve the checkpoint associated with the selected model and base-task family through the shared artifact catalog. Users can also specify a checkpoint explicitly with --checkpoint or download missing artifacts with --download.

The evaluation interfaces additionally support explicit random seeds, allowing comparisons across repeated runs.

Reproducibility and Extensibility

AnyGoal organizes datasets, checkpoints, and supported model implementations through shared registries. Dataset and model identifiers, artifact locations, and evaluation configurations can be specified explicitly, reducing ambiguity when reproducing experiments. The library exposes its core functionality through Python APIs, including anygoal.collect, anygoal.train, anygoal.evaluate_rollout, and anygoal.plan. These interfaces complement the CLI and support integration into existing research workflows.

The released benchmark contains 400 episodes per goal-shifted variant, while the collection tools allow researchers to construct additional evaluation datasets. Together with the standardized evaluation interfaces, this enables future world models to be tested on the same controlled goal shifts without reproducing the environment and evaluation infrastructure. The source code, configuration files, and implementation instructions are available at github.com/galilai-group/AnyGoal.

References

  1. [1]Rim Assouel, Pietro Astolfi, Florian Bordes, Michal Drozdzal, and Adriana Romero-Soriano. Object-centric binding in contrastive language-image pretraining. Advances in Neural Information Processing Systems, 38:131418–131458, 2026.arxiv.org/abs/2502.14113
  2. [2]Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics, 2025. URL https://arxiv.org/abs/2511.08544.
  3. [3]Jack Brady, Julius von Kügelgen, Sébastien Lachapelle, Simon Buchholz, Thomas Kipf, and Wieland Brendel. Interaction asymmetry: A general principle for learning composable abstractions. In International Conference on Learning Representations, volume 2025, pp. 46606–46660, 2025.arxiv.org/abs/2411.07784
  4. [4]Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1lJ1nJR5Ym.arxiv.org/abs/1810.12894
  5. [5]Giacomo Campsampiero, Pietro Barbiero, Michael Hersche, Roger Wattenhofer, and Abbas Rahimi. Scalable evaluation and neural models for compositional generalization. Advances in Neural Information Processing Systems, 38:7341–7401, 2026.
  6. [7]Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfield, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. Int. J. Rob. Res., 44(10–11):1684–1704, September 2025. ISSN 0278-3649. doi: 10.1177/02783649241273668. URL https://doi.org/10.1177/02783649241273668.
  7. [8]Kefan Dong and Tengyu Ma. First steps toward understanding the extrapolation of nonlinear models to unseen domains. arXiv preprint arXiv:2211.11719, 2022.arxiv.org/abs/2211.11719
  8. [9]Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021. URL https://arxiv.org/abs/2004.07219.
  9. [10]Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052–2062. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/fujimoto19a.html.arxiv.org/abs/1812.02900
  10. [13]Ignat Georgiev, Varun Giridhar, Nick Hansen, and Animesh Garg. Pwm: Policy learning with multi-task world models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, pp. 13737–13757, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/240ea1741b205ea295721d55184ac43b-Paper-Conference.pdf.arxiv.org/abs/2407.02466
  11. [14]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/2de5d16682c3c35007e4e92982f1a2ba-Paper.pdf.arxiv.org/abs/1809.01999
  12. [15]Nicklas Hansen and Xiaolong Wang. Hallucination in world models is predictable and preventable. arXiv preprint arXiv:2606.27326, 2026.arxiv.org/abs/2606.27326
  13. [16]Petr Ivashkov, Randall Balestriero, and Bernhard Schölkopf. Sensorimotor world models: Perception for action via inverse dynamics, 2026. URL https://arxiv.org/abs/2606.20104.
  14. [17]Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.arxiv.org/abs/1612.06890
  15. [18]Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In ICLR, 2020. URL https://arxiv.org/pdf/1912.09713.pdf.
  16. [19]Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21810–21823. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/e7efa4f864ae9b88d43527f4b14f750f-Paper.pdf.
  17. [20]David Klindt, Yann LeCun, and Randall Balestriero. When does lejépa learn a world model?, 2026. URL https://arxiv.org/abs/2605.26379.
  18. [21]Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/c2073ffa77b5357a498057413bb09d3a-Paper.pdf.arxiv.org/abs/1906.00949
  19. [23]Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian-product extrapolation. Advances in Neural Information Processing Systems, 36:25112–25150, 2023.arxiv.org/abs/2307.02598
  20. [24]Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2873–2882. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/lake18a.html.arxiv.org/abs/1711.00350
  21. [26]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020. URL https://arxiv.org/abs/2005.01643.
  22. [27]Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified Video Action Model. In Proceedings of Robotics: Science and Systems, LosAngeles, CA, USA, June 2025. doi: 10.15607/RSS.2025.XXI.074.
  23. [28]Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, and Francesco Locatello. Causal triplet: An open challenge for intervention-centric causal representation learning. In Conference on Causal Learning and Reasoning, pp. 553–573. PMLR, 2023.arxiv.org/abs/2301.05169
  24. [29]Susie Lu, Haonan Chen, Weirui Ye, and Yilun Du. Driftworld: Fast world modeling through drifting, 2026. URL https://arxiv.org/abs/2607.15065.
  25. [30]Lucas Maes, Quentin Le Lidec, Dan Haramati, Nassim Massaudi, Damien Scieur, Yann LeCun, and Randall Balestriero. stable-worldmodel-v1: Reproducible world modeling research and evaluation, 2026a. URL https://arxiv.org/abs/2602.08968.
  26. [31]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworld-model: Stable end-to-end joint-embedding predictive architecture from pixels, 2026b. URL https://arxiv.org/abs/2603.19312.
  27. [32]Divyat Mahajan, Mohammad Pezeshki, Charles Arnal, Ioannis Mitliagkas, Kartik Ahuja, and Pascal Vincent. Compositional risk minimization. In Proceedings of the 42nd International Conference on Machine Learning, pp. 42503–42553. PMLR, 2025. URL https://proceedings.mlr.press/v267/mahajan25a.html.arxiv.org/abs/2410.06303
  28. [33]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=a68SUt6zF't. Featured Certification.arxiv.org/abs/2304.07193
  29. [34]Viktória Barin Pacela, Shruti Joshi, Isabela Camacho, Simon Lacoste-Julien, and David Klindt. Stop probing, start coding: Why linear probes and sparse autoencoders fail at compositional generalisation. arXiv preprint arXiv:2603.28744, 2026.arxiv.org/abs/2603.28744
  30. [35]Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pp. 4036–4044. PMLR, 2018.arxiv.org/abs/1712.00961
  31. [36]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, pp. 94937–94982, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/ecd92623ac899357312aa891583699-Paper-Conference.pdf.arxiv.org/abs/2410.20092
  32. [37]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195–4205, October 2023.arxiv.org/abs/2212.09748
  33. [38]Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 16082–16097. Curran Associates, Inc., 2022. doi: 10.52202/068431-1170. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/6691c5e4a199b72dffd9c90acb63bcd6-Paper-Conference.pdf.DOI
  34. [39]B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612–634, 2021. doi: 10.1109/JPROC.2021.3058954. URL https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9363924. *equal contribution.DOI
  35. [40]Yuhong Shi, Zhenhao Chu, Jie Wei, Jun Hao, Jianyi Liu, and Lingwen Fu. Overcoming statistical bias in action-controllable world models, 2026. URL https://arxiv.org/abs/2608.04653.
  36. [41]Uladzislau Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestiero, Tim G. J. Rudner, and Yann LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.), Advances in Neural Information Processing Systems, volume 38, Main Conference, pp. 43905–43941. Curran Associates, Inc., 2025. doi: 10.52202/085713-1465. URL https://proceedings.neurips.cc/paper_files/paper/2025/file/3e7cf447f21cd11c846463affefce665-Paper-Conference.pdf.DOI
  37. [42]Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. The MIT Press, Cambridge, MA, 2nd edition, 2018.
  38. [44]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018. URL https://arxiv.org/abs/1801.00690.
  39. [45]Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5238–5248, June 2022.arxiv.org/abs/2204.03162
  40. [46]Xinyue Wang and Biwei Huang. Modeling unseen environments with language-guided composable causal components in reinforcement learning. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, pp. 48784–48812, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/79d86433c2acd12b6fa98553435d226e-Paper-Conference.pdf.arxiv.org/abs/2505.08361
  41. [47]Ying Wang, Oumayma Bounou, Yann LeCun, and Mengye Ren. Adajepa: An adaptive latent world model, 2026. URL https://arxiv.org/abs/2606.32026.
  42. [48]Archana Warrier, Dat Nguyen, Michelangelo Naim, Moksh Jain, Yichao Liang, Karen Schroeder, Cambridge Yang, Joshua B. Tenenbaum, Sebastian Josef Vollmer, Kevin Ellis, and Zenna Tavares. Benchmarking world-model learning with environment-level queries. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=Ny6UYZ8ysN.arxiv.org/abs/2510.19788
  43. [49]Thaddäus Wiedemer, Jack Brady, Alexander Panfilov, Attila Juhos, Matthias Bethge, and Wieland Brendel. Provable compositional generalization for object-centric learning. In International Conference on Learning Representations, volume 2024, pp. 20137–20164, 2024a.arxiv.org/abs/2310.05327
  44. [50]Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36, 2024b.arxiv.org/abs/2307.05596
  45. [51]Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty. Advances in Neural Information Processing Systems, 37:86004–86047, 2024.arxiv.org/abs/2408.14339
  46. [52]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14129–14142. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/a328252ce0df73e204b7e67cbef0d0a-Paper.pdf.arxiv.org/abs/2005.13239
  47. [53]Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 28954–28967. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/f29a179746902e331572c483c45e5086-Paper.pdf.arxiv.org/abs/2102.08363
  48. [54]Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. In S. Koy ejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 29776–29788. Curran Associates, Inc., 2022. doi: 10.52202/068431-2159. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/c08ee8fe3d19521f3bfa4102898329fd-Paper-Datasets_and_Benchmarks.pdf.DOI
  49. [55]Xiangcheng Zhang and Yilun Du. World action planner: Generalizable decision-making with action-conditioned world models, 2026. URL https://arxiv.org/abs/2607.27599.
  50. [56]Lin feng Zhao, Lingzhi Kong, Robin Walters, and Lawson L.S. Wong. Toward compositional generalization in object-oriented world modeling. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 26841–26864. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/zhao22b.html.arxiv.org/abs/2204.13661
  51. [57]Yi Zhao, Aidan Scannell, Wenshuai Zhao, Yuxin Hou, Tianyu Cui, Le Chen, Dieter Büchler, Arno Solin, Juho Kannala, and Joni Pajarinен. Efficient reinforcement learning by guiding world models with non-curated data. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), International Conference on Learning Representations, volume 2026, pp. 79246–79280, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/7ff013b7e372ba5b790352ccd6908f03-Paper-Conference.pdf.arxiv.org/abs/2502.19544
  52. [58]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=D5RNACOZEI.arxiv.org/abs/2411.04983
  53. [59]Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chi-Hou Cheang, and Tao Kong. Irasim: A fine-grained world model for robot manipulation. 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9834–9844, 2024. URL https://api.semanticscholar.org/CorpusID:270620594.arxiv.org/abs/2406.14540

Paper details

vlnc:W_yzs6v8un
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents