Training Controllable World Models via Action Influence Maximization

Website: princeton-prism.github.io/action-influence-maximization

Code: github.com/princeton-prism/action-influence-maximization

Introduction

Action-conditioned world models promise to predict what would happen if a robot took different actions at a given state, enabling policy improvement in novel problems. Yet, when queried with out-of-distribution actions, models trained on expert demonstrations can predict expert-like futures or hallucinate implausible alternatives. This failure of controllability (the ability to faithfully predict how changing actions affect outcomes) limits their usefulness for improving robot policies.

Scaling the quantity and diversity of training data can mitigate this [1, 2], but is limited by the cost of collecting robotics data. To make world models competitive with on-robot experience, the practical question is how to obtain the most useful world-model training data from each real robot interaction. Despite this, optimal data collection for robotic world models is relatively underexplored compared to policy improving data [3, 4, 5]. This paper investigates two key hypotheses: Does the data collection distribution strongly influence the downstream world model controllability and fidelity? And, supposing it consequential, can we optimize the data collection distribution to directly improve world model controllability and fidelity? This work suggests that the answer to both questions is yes.

This has a plausible and intuitive explanation: expert demonstrations in high-dimensional action spaces (e.g. human teleoperation) achieve their goals through a narrow sequence of actions. The more sensitive success is to action variability, the more deterministic expert demonstrations must be. Successful demonstrations can thus provide insufficient variation in actions to reveal evidence of how alternative actions would change outcomes. A world model fitting this data can rationally ignore action influence and usually predict expert-like outcomes regardless of action command [1]. In contrast, a non-expert learner may visit different states and try actions underrepresented in expert demonstration. Predicting consequences of varying actions in these states are precisely the situations where downstream policy improvement stands to benefit the most and where the world model has the least data support. We see evidence for this (Figure 6 and Figure 8): scaling passive data collection improves controllability for different eval distributions, but the most efficient collection strategy is eval-distribution dependent.

To address this, we propose Action Influence Maximization (AIM), an active data collection algorithm that uses a data-derived intrinsic reward. Put simply, AIM estimates which variations in action consequences would be poorly explained by the current dataset using a theoretically-motivated, learned intrinsic reward, and then focuses data collection on those state-action pairs as a form of embodied active learning. To do this, we connect conditional dataset mutual information (MI) to the controllability of any downstream world model, and derive a tractable learned pointwise reward estimator using data density estimation to optimize this property directly. AIM achieves up to an $85 \times$ improvement in data efficiency in a high-dimensional, robot manipulation task: adding just 500 actively collected episodes to 3k expert demonstrations matches the end-effector controllability of a world model trained on 300k expert episodes.

Related Work

Learning a model to predict the future consequences of actions is a well studied problem in decision-making [6, 7, 8, 9, 10, 11, 12, 13, 14]. Recent advances in generative modeling [15, 16, 17, 18, 19, 20, 21] and self-supervised representation learning [22, 23, 24, 25] have significantly improved the scalability of learned predictive models, enabling a new family of action-conditioned generative world models [26, 27, 28, 29, 30, 31]. This family of methods absorbs heterogeneous internet-scale video and robotics datasets, and aims to maximize distributional similarity between generated videos and the dataset [32, 33, 34, 35]. In robotics, this goal is misaligned: these models’ main utility is improving learner policies [2, 13, 14, 23, 27, 29, 36, 37, 38], where a rollout is only useful if it faithfully reproduces action-induced changes in task-relevant features that are generally unknown at training time. Our studied world model builds on DreamerV4 [26] and Diffusion Forcing [18], with architectural and training details described in Appendix § 11.

A surrogate for task-relevant fidelity is a world model’s controllability: faithful, action-responsive simulation over long horizons [1, 2]. Prior work has improved this fidelity by improving temporal consistency [18, 39], incorporating data from autonomous play [1] or failure [2], enforcing policy conservatism to avoid problematic extrapolation [26], performing self-consistency based post-training [40, 41], or using latent representations that are easier to predict [23, 42, 43]. Still, techniques from other domains have seen limited application to modern world models, such as active data collection [3, 44, 45, 46, 47] or sim-and-real co-training [48].

Exploration has a long history in reinforcement learning (RL) [49, 50, 51, 52, 53] and MI skill learning [44, 54, 55, 56]. In model-based RL, prediction errors [50, 51, 57] or ensemble disagreement [45, 53] provide an intrinsic reward for collecting data that a particular model struggles to predict. In contrast, in active learning [58], it is sometimes assumed that a small core set of data can efficiently summarize a data distribution [59]. Toneva et al. [60]’s “unforgettable examples” motivate this perspective: sample learning difficulty and utility is largely consistent across models. This suggests that data utility is intrinsic to the data, independent of the derived model, and can guide collection [61, 62]. This is similar to information-theoretic experimental design [63, 64] and persistent excitation in adaptive control [65, 66]. Our work is most closely related to PlayWorld [1], a method for refining generative world models via on-policy goal-directed “play data” rollouts, and contemporaneous work by Hansen and Wang [47] which uses heuristic measures of prediction instability as a curiosity signal to guide active collection. In contrast, we propose that the model and its data are largely separable, and optimize the data itself by maximizing intrinsic dataset action influence.

Data Collection with Action Influence Maximization

To turn this intuition into a principled collection algorithm, we first quantify world model controllability and connect it to dataset action information. Then, we derive an intrinsic reward to maximize this quantity, and provide a practical batch online RL implementation of the theoretical result.

How does data collection shape world model controllability?

A general proxy for task-specific simulation fidelity is to maximize the controllability of the world model: if the world model accurately represents how different actions influence the next state, and downstream rewards are primarily evaluated on features which are at least somewhat controllable, then maximizing the action controllability of the learned state transition should act as a valid surrogate for any task-specific fidelity we choose. Formally, we define a measure of world model action controllability as the difference in prediction error when the chosen action is known or not, dubbed the explained action information score:

Without knowledge of the action, the model averages its predictions over the evaluation policy: $\widehat{p}_{\theta,\pi_{\mathrm{eval}}}(s' \mid s) = \int_{\mathcal{A}} \pi_{\mathrm{eval}}(a \mid s) \widehat{p}_{\theta}(s' \mid s,a)\, da$, where $s, s'$ are MDP states, $a \in\mathcal{A}$ is an action, $p$ is the true transition kernel, and $\widehat{p}_{\theta}$ its learned approximation. The evaluation occupancy is $d_{\mathrm{eval}}(s,a) = \rho_{\mathrm{eval}}(s)\pi_{\mathrm{eval}}(a \mid s)$, where $\pi_{\mathrm{eval}}$ represents the mixture of learner policies encountered during policy improvement and $\rho_{\mathrm{eval}}$ their aggregate state occupancy. A larger score implies a world model more accurately predicts transitions when it sees the action than if it was held out, plausibly improving action responsive task-relevant transition fidelity indirectly. The observed correlation between explained action information and simulation fidelity supports this interpretation (Figure 3).

Our data scaling experiments in Figure 6 indicate that most data collection strategies will saturate in the data limit long before saturating the simulation fidelity of the downstream task evaluation. Near-deterministic expert policy data provides few examples of alternate actions in a state (Figure 4), making $\widehat{p}_{\theta,\pi_{\mathrm{eval}}}(s' \mid s) \approx\widehat{p}_{\theta}(s' \mid s,a)$ an equally plausible but degenerate explanation of the data.

If world model fidelity is mostly about how well its training distribution covers $d_{\mathrm{eval}}(s,a)$, then expert-oriented data on one task might be maximally bad for improving a weak learner. A weak learner definitionally concentrates very little of its state-action occupancy on highly rewarding states, but a world model trained on expert-like data will primarily predict plausible expert-like futures with action insensitivity or blindly extrapolate. Figure 8 illustrates this action insensitivity at low data and its improvement with more expert data. Adding failure data is likely an insufficient remedy: the world model must accurately model every learner evaluation distribution, from the weakest policy to the final strongest. Failure data adds a distribution in between, which may or may not bridge the gap between the learner’s distribution and the expert distribution.

This results in a sharp, testable hypothesis: passive data scaling is perhaps only as effective as the data’s natural coverage of evaluation states and the action consequences revealed at those states. Such incidental coverage can have very poor sample complexity in high-dimensional search spaces. In contrast, data-driven active data collection can focus scarce data sampling on high-utility transitions to maximize coverage of diverse state-action pairs, avoid duplication of data, and direct exploration toward “interesting” data. A reward derived from our controllability metric could guide this exploration in a principled manner.

An Objective for Collecting Informative Data

Data plays such a large role in world model capabilities (or lack thereof) that it seems possible that an upper bound on a world model’s capabilities could be quantified from the data alone. A key observation using Bayes’ rule and KL nonnegativity allows us to quantify the intrinsic dataset controllability as conditional mutual information, which upper-bounds the explained action information in Equation (1):

The AIM Reward

Proof and regularity conditions: Appendix § 7.§ 1.

This yields two key observations: world model capability can be analyzed from properties of the dataset before training a world model, and, when the bound is tight, world model controllability can be directly optimized by improving the return on the corresponding conditional mutual information maximizing contextual bandits problem

selecting $\pi$ to maximize $I(\rho_{\mathrm{eval}},\pi)$ on the evaluation distribution, with $\rho_{\mathrm{eval}}$ held fixed.

Since, in general, $\pi_{\mathrm{eval}}$ is not known at training time in the open-ended task policy improvement setting, this optimization problem can be seen as maximizing the diversity of distinguishable action consequences we can achieve from the evaluation state distribution $\rho_{\mathrm{eval}}$. This is equivalent to maximizing a single-step conditional mutual information between actions and next states given the current state, which is identical to the one-step empowerment objective [67, 68]. This provides an interesting insight: without additional inductive biases to enable extrapolation, reliable evaluation of learners requires the dataset to reveal the action consequences encountered under the learner’s evaluation distribution. Scaling passive data collection (e.g. passive video) is unlikely to recover missing action consequences without additional interactive data collection to quantify the effects of an agent’s actions on those outcomes. Equation (3) therefore provides a framing for improving world model data across downstream tasks.

The AIM Reward

Equation (3) motivates an intervention to test our hypothesis that active data collection can address the problem of passive incidental coverage by reframing the optimization problem of Equation (3) into a tractable active collection surrogate. Since we are trying to perform a targeted search for the data we are missing, we need to rely on data we have to identify those absences. This data generally has the form of a prior data distribution of observed transitions, often called a replay buffer, $d_k(s,a,s') = d_k(s)\beta_k(a \mid s)p(s' \mid s,a)$, which contains the previously visited data. Using this, we propose the following data density-based reward for AIM, combining a reference state coverage and mutual information term:

Here $q_k(a \mid s,s')$ is the inverse dynamics under the current replay policies $\beta_k$. Its utility is intuitive: the reference state coverage term will push the collected data to be closer to the eval state distribution and away from previously collected data, and the pointwise mutual information term rewards transitions whose outcomes make the chosen action more identifiable than under the existing policy. But it has a deeper connection to the mutual information contextual bandits problem itself:

Proof: Appendix A.3.

Thus optimizing the AIM reward in the contextual bandits setting iteratively increases dataset mutual information, and thus an upper bound on world model controllability.

Because the update reweights the prior replay data, expert demonstrations bias initial collection toward informative variations around expert behaviors in regions of interest. Optimizing this iteratively can be seen as a guided search for action consequences: it first explores variations around what humans have revealed is interesting, then spreads outward as those consequences become represented in the replay buffer and the re-estimated reward favors alternatives that remain underrepresented. This provides a way to progressively bridge the expert distribution to varying degrees of degraded policies, the exact qualities of data we desired for accurate policy evaluation.

Estimating and Optimizing the AIM Reward in Practice

Directly optimizing the AIM contextual bandits problem of Theorem 2 is not practical: it requires the ability to sample from the evaluation distribution $\rho_{\mathrm{eval}}(s)$ arbitrarily and sample a transition from the true dynamics $p(s' \mid s,a)$ for each state-action pair. Given the structure of equation (5), we can instead estimate the AIM reward function from the replay buffer and optimize a conservative sequential RL problem as a surrogate for the contextual bandits problem. Estimating the AIM reward is challenging for three reasons: it’s non-stationary as further data is collected, requires sufficiently accurate likelihood estimation of $q_{\beta_k}(a \mid s, s^\prime)$, $\beta(a \mid s)$, and $d_k(s)$, and the evaluation distribution $\rho_{\mathrm{eval}}(s)$ is not generally known at training.

To address the non-stationarity, we estimate the learned AIM reward function in a batch online RL loop which collects a batch of experimental data, updates learned density reward models to account for the new trajectories, and performs conservative RL to extract an AIM reward maximizing policy solution similar to equation (5). This is described in detail in Algorithm 1. This approach approximates the learned densities as piecewise constant in collection time to allow for amortizing the training of larger learned flow matching reward models [17, 70]. This can be thought of abstractly as linearizing around the current dataset (refitting the density reward models), taking a conservative step along an improvement direction under this linearization (data collection with the reward maximizing policy), and then relinearizing at the new replay buffer dataset (refitting again). Periodic re-estimation ensures the learned reward remains aligned with dataset utility as the data distribution evolves during collection. We approximate the policy improvement in Equation (5) by performing per-iteration value estimation via offline RL through an Implicit Q-Learning Critic [71] and use AWR [72] for policy extraction into a diffusion policy actor [70].

For the reward estimation itself, this type of variational mutual information maximization has deep roots in pre-existing theoretical literature [69], but practical estimation of density likelihoods over high dimensional data has been historically challenging. We make a number of principled decisions to inherit the strengths of modern generative modeling to our task. We estimate all likelihoods with a flow matching model [17] with the Continuous Normalizing Flows FFJORD-Hutchinson estimator [73, 74] and amortize the divergence estimation cost through training time self-distillation [19]. This has a nice property of averaging single point Hutchinson sample estimates in the weights throughout training, making the likelihood estimation more reliable. While this allows for exact likelihood and thus reward estimation, the practical differences in dimensionality between the state and action likelihoods creates numerical problems with balancing and training an RL critic on the AIM reward function. We address this by statistical standardization of the mutual information and reference state coverage terms, and clip AWR weights. This reduces the sharpness of rare tail support events and improves effective sample size in AWR, while still giving utility-driven rewards. Theoretically this can be interpreted as adjusting the sharpness of the weighting of the reference state coverage term, and as a trust region on the mutual information reward.

Finally, in absence of a specific desired evaluation distribution, we make $\rho_{\mathrm{eval}}(s)$ a uniform distribution over the target state space. We find this simplification works reasonably well, but future work could explore more explicitly designing this distribution to the target learner distribution to further improve sample efficiency.

Evaluation

§ 3 proposes a compelling theoretical framework for analyzing world model controllability and its relation to data: the dataset mutual information (or lack of it) determines the environment faithful controllability exposed to a world model and largely determines its useful fidelity for policy learners. To analyze this framework empirically, we study how the collection distribution affects world model fidelity, whether AIM improves data efficiency, and which parts of AIM contribute to that improvement. We hold architecture and training details fixed within each of the following environments:

Four Rooms: Four Rooms [75, 76] provides a fully observable, deterministic environment where we can exhaustively evaluate every state-action transition. We train identical flow-matching world models [17, 77] from scratch on several expert policy datasets: uniform random actions, sim-privileged Oracle PPO [78], partially observable PPO, and Early Stopped PPO. We evaluate simulator fidelity by querying the learned model at all state-action pairs for an environment and matching predicted observations to all sim states via maximum likelihood. This setting isolates the role of the transition distribution from partial observability, data contamination from large scale training, and bias towards any specific evaluation policy distribution.

Push-T: Push-T tests whether the same data-collection principles apply in high dimensional, pixel-based robot manipulation. We train a DreamerV4-Like latent Diffusion Forcing model [17, 18, 20, 26, 79] from scratch and compare datasets collected by expert, random, random-walk, synthetic RaC-like recovery [5], intrinsic motivation [44, 50, 54, 55], and AIM policies. Exhaustive transition evaluation is infeasible in high-dimensional manipulation; we evaluate evenly sampled autoregressive horizons (H=10) on two “learner” target policy distributions: rollouts from a weak on-policy Behavior-Cloned (BC) Diffusion policy [70] and an “Anti-Push-T” Push-Away policy which is trained to disperse the Tee to unique positions away from the normal goal. Our controllability metric is end-effector masked pixel recall to isolate action-dependent controllability. Additional metrics, details on the data generating policies, and world model architectures are reported in the Appendix.

Mutual Information and Controllability

First we briefly analyze the presumption that the dataset mutual information limits the model controllability. We train 68 world models, 8 data scales in Push-T and 9 in Four Rooms, with 4 passive data collection strategies for each. Figure 3 summarizes the results from Four Rooms: across data scaling, the primary predictor of simulator fidelity is the world models explained action influence ($r = 0.979$). Push-T shows similar scaling behavior (see Figure 6), though in the data limit, different data collection policies converge to different levels of controllability. This is compatible with our model: different fixed collection distributions will leave different portions of the state space systematically under sampled regardless of the data’s utility. Even random behavior becomes less effective as the dimensionality of the state space grows and the initial state distribution becomes less uniform (see Appendix § 14.§ 2). This motivates active data collection both empirically and theoretically.

Active Data Collection with AIM

Now that we have established that passive data collection has diminishing returns beyond a certain point in modeling high dimensional control, we can evaluate another prediction of our theory: maximizing action information through active collection should provide a substantial boost in data efficiency for world model improvement. We implement the approximate AIM sequential collection algorithm from § 3.5, and collect 100-episode batches over four rounds in Four Rooms and five in Push-T to augment a base expert collection distribution of ∼ 3K expert demonstrations. We compare this at equal rollout budget with both the active and passive data collection policies described in § 4, training a total of 73 world models, and report our results in Figure 5.

In Four Rooms, uniform random is predictably strongest, consistent with Figure 6 and our prior analysis. However, fixed-start Random performs much worse, showing that random’s advantage comes largely from its initial state coverage rather than action entropy alone. In Push-T, random, recovery, and intrinsic motivation baselines plateau or degrade on end-effector controllability, while AIM improves consistently between collections. With 3k expert trajectories plus 500 AIM collections, the model matches the 300k-expert baseline on BC rollouts controllability, and approaches the 19k-expert baseline on Anti-Push-T rollouts. The flat baselines do not indicate failed exploration. These policies collect meaningful data and perform competitively on other fidelity metrics (see Figure 12 and 13) and at larger data scales, but their additional rollouts appear to transfer poorly to the target controllability metrics in the low-data regime.

Our theory suggests why broader data coverage need not improve downstream world-model utility, especially in the low data regime: alternative collection policies may add trajectories far from the expert distribution, creating “islands” of action consequences that the model struggles to connect to existing data, leaving intermediate evaluation behaviors poorly supported. AIM can instead be thought of as “growing” a successively broader region of action explainability around existing data. For policies which vary over learning, or alternate tasks with behavior similar to the prior data, this data “bridge” may provide better support than broader but disconnected exploration. Two questions follow naturally: which parts of AIM contribute to this improvement, and how well does it generalize across conditions?

Ablations

To test the robustness of our conclusions, we evaluate higher-resolution Push-T world models over a longer horizon ($H = 33$), with end-effector, Tee, and joint $\min\{\mathrm{EE}, \mathrm{Tee}\}$ recall, and arena-style A/B human preference ratings [80] (1545 comparisons, 20 annotators). We also add a BC-Failure

test set that interleaves BC and Push Away policies as a proxy for suboptimal behavior during policy improvement. Table 1 summarizes the results. Adding 500 AIM episodes improves BC joint recall by 69% over PPO-3K and surpasses the 300K-expert baseline by 54%, and we find human preference is largely consistent with our recall metrics (BC Arena Elo increases from 732 to 1190).

Experiment

BC Recall (H=33, 95% CI)

Preference

BC Failure Recall (H=33, 95% CI)

Preference

Push Away Recall (H=33, 95% CI)

Preference

EE

TEE

Joint

ELO

EE

TEE

Joint

ELO

EE

TEE

Joint

ELO

PPO-300K

.275±.013

.400±.024

.224±.024

843±155

.554±.019

.491±.028

.432±.026

1040±185

.332±.023

.282±.026

.198±.018

686±128

PPO-3K

.267±.011

.322±.022

.203±.011

732±155

.517±.020

.424±.030

.378±.028

824±171

.297±.023

.190±.017

.147±.015

889±132

PPO-3K+AIM500

.485±.014

.403±.023

.344±.018

1190±167

.564±.018

.454±.031

.411±.029

1049±185

.375±.016

.201±.020

.165±.016

1029±134

PPO-3K+Recovery500

.272±.012

.378±.026

.215±.012

979±155

.545±.022

.455±.030

.406±.027

989±200

.314±.023

.216±.023

.169±.016

735±128

PPO-3K+ICM500

.222±.011

.393±.025

.186±.011

868±145

.506±.022

.441±.029

.383±.027

934±186

.267±.020

.242±.023

.158±.016

924±141

PPO-3K+DIAYN500

.233±.009

.364±.027

.186±.010

693±145

.522±.021

.438±.031

.388±.029

976±167

.310±.024

.186±.018

.143±.015

741±129

Recovery-3K

.315±.013

.422±.026

.266±.014

1059±158

.598±.018

.512±.029

.466±.026

1099±202

.396±.018

.276±.026

.226±.018

1250±147

Recovery-3K+AIM500

.511±.014

.417±.024

.361±.019

1412±183

.613±.016

.498±.028

.459±.026

1147±212

.421±.019

.264±.024

.223±.018

1317±178

AIM w/o re-estimation

.526±.013

.428±.023

.379±.019

1388±174

.552±.020

.464±.031

.416±.029

1047±169

.357±.017

.226±.022

.183±.018

1092±138

AIM w/o MI reward

.481±.013

.405±.024

.344±.019

1107±208

.566±.017

.458±.030

.416±.028

964±151

.389±.016

.231±.023

.192±.019

1076±207

AIM w/o Novelty reward

.483±.014

.401±.025

.340±.019

1376±295

.559±.018

.457±.030

.411±.028

1128±191

.377±.016

.224±.023

.182±.019

1048±184

To test if AIM is exploiting some characteristic of the low variability of PPO-3K for performance gains, we ablate swapping the PPO-3K base data with Recovery-3K, while keeping the same AIM data collected with the PPO-3K reward model, improving BC joint recall by 36% over Recovery-3K. This suggests AIM targets action information not easily acquired through common passive approaches via play-like structured behavior. We also ablate reward re-estimation by collecting 500 trajectories with the first learned reward model, improving BC joint recall by 10% over iterative AIM. This suggests that the initial reward model remains valid for longer than our analysis in § 3.5 predicted. When we ablate the AIM reward terms independently, we see the novelty reward outperforming on the very off-policy Push Away task (16% higher joint recall than AIM) where state coverage makes the most difference. Meanwhile, MI-only reward scores a bit higher in human evaluation on BC and BC-Failure evaluation sets (+16%/+8% over AIM), collecting more discriminative action consequences near the expert distribution. Together these results support the hypothesis that small data interventions can provide significant improvements in world model controllability and fidelity, while leaving ample room to further optimize this type of active collection to improve sample efficiency in robotics.

Conclusion and Limitations

In this work, we propose a new theoretical perspective relating the quality of an action conditioned world model to quality of its data: a world model is primarily limited by the action consequences explained by its data, not the scale of its data. We then show we can directly optimize the coverage of action consequences using Action Influence Maximization (AIM), a theoretically motivated batch online reinforcement learning algorithm that directs a data collection policy to probe the environment in ways which are expected to have predictable consequences but are unexplained by the existing dataset. AIM shows that data-focused active exploration in high-dimensional, pixel-based robotic environments is not only feasible, but results in as much as $85 \times$ data scaling efficiency compared to passive expert collection approaches. The key insight is simple: scaling helps mainly if it scales action information, not simply the number of transitions. This suggests a path toward data-efficient, intrinsic-motivation-driven continual improvement of generalist world models and embodied policies, where robots actively collect the data they need to improve their own capabilities.

Limitations. A key limitation of AIM is the implementation complexity and compute intensity of retraining multiple learned density reward models. The no-reestimation ablation suggests fewer updates could suffice, and the learned reward could also be used offline to filter datasets [59]. Simpler, more efficient implementation will be key in scaling AIM-like methods to pretraining scale. Additionally, our sequential RL approximation is not guaranteed to achieve improvement in the AIM contextual bandits problem, suggesting further theoretical work may be needed to bridge this limitation in more complex settings where optimizing RL is more challenging. Our analysis also leaves the effects of post-training on improving data efficiency unaddressed, which may allow for inference of action consequences absent from interaction data. Our work represents a proof of concept of the power of active data collection for robotics in simulation, and transferring this to continual improvement of generalist models remains to be shown by future work.

Acknowledgments

This research was supported in part by Microsoft Research and Toyota Research Institute (TRI); this article solely reflects the opinions and conclusions of its authors, and not TRI nor any other Toyota entity. We thank the Google TPU Research Cloud (TRC) Program, NVIDIA Academic Grant program, and Gemini Academic Program for their generous support of our research. Samuel M. Bateman is supported by the National Science Foundation CISE Graduate Fellowships under Grant No. 2313998. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF. The authors would like to thank Antonio Crivello, Duy Nguyen, Jeremy Bao, and Mingtong Zhang for providing invaluable feedback on the manuscript and presentation. Additionally, we would like to thank our amazing friends and colleagues for volunteering their time to help annotate world model rollout preferences.

Proofs

Proof of Theorem 1

Proof. Assume the KL prediction errors in Equation (1) have finite expectations, and the relevant log ratios are well defined with finite expectations. Expanding the KL divergences in Equation (1), we have

We can then write the mutual information of the ground-truth transitions over a given state marginal $\rho$ as

Using the variational posteriors,

we can write the difference between $I(\rho,\pi)$ and $\hat{I}(\rho,\pi)$ as:

Applying Bayes rule to both expressions, and plugging in the identity $\rho(s)\pi(a\mid s)p(s'\mid s,a)=\rho(s)p_{\pi}(s'\mid s)q_{\pi}(a\mid s,s')$, we have

where the last inequality holds because of the non-negative property of KL divergence [81]. This implies that the explained action information is upper bounded by the true data MI $\hat{I}(\rho,\pi)\leq I(\rho,\pi)$, and dataset action influence limits world model explainable controllability over the dataset. □

Improving Replay Buffer Baseline Lower Bound Improves Action Information

Proof. Recall the definition of the variational posterior in Eq. A.3. We can use it to rewrite the ground-truth data mutual information as

Using the variational lower bound proposed by Barber and Agakov [69], we have

where $q_k(a\mid s,s') \triangleq\frac{\beta_k(a\mid s)p(s'\mid s,a)}{p_{\beta_k}(s'\mid s)}$ is a variational posterior defined by the behavioral policy $\beta_k$.

We note that the lower bound $L_k^\rho(\pi) \le I(\rho,\pi)$ is tight if $\pi(a\mid s)=\beta_k(a\mid s)$ by plugging in the definition of $q_k(a\mid s,s')$: $L_k^\rho(\beta_k)=I(\rho,\beta_k)$. For any other policy $\pi^+$ for which $L_k^\rho(\beta_k)<L_k^\rho(\pi^+)$, we have

For any valid policy improvement $\pi^+$ on the lower bound, the dataset mutual information will improve as well. The same result holds with non-strict inequalities.

Proof of Theorem 2

Proof. Use the conditions in Appendix A.2, and assume $\rho$ and $d_k$ are mutually absolutely continuous. The replay-weighted contextual bandits problem is

We can define the log ratio as an intrinsic reward $r_{\mathrm{info}}^k(s,a,s')=\log q_k(a\mid s,s')-\log\beta_k(a\mid s)$. Rearranging terms in $L_k^\rho(\pi)$, we have

which is precisely the Lagrangian form of a KL-constrained optimization problem with the Lagrange multiplier $\eta= 1$ [82]. Its unique maximizer is a Boltzmann tilt of the following form [81]:

where $\mathbb{E}_p$ averages over $s' \sim p(\cdot\mid s,a)$.

Separately, we can reweight the observed samples with a Boltzmann tilt of the state marginal distribution: $\rho(s)/d_k(s) = \exp(\log\rho(s) - \log d_k(s))$, resulting in the following equality:

These identical weights induce the same conditional policy $\pi_k^\star(a \mid s) \propto\beta_k(a \mid s)\exp\left(\mathbb{E}_p[r_{\mathrm{AIM}}^k]\right)$, which is also the statewise optimizer of $J_k^d$. Thus both problems share $\pi_k^\star$ as an optimizer.

We apply Lemma 1 to the exponential optimizer, implying

Improvement is strict if $\pi_k^\star\ne\beta_k$ on states with positive evaluation probability. □

Scaling Passive Data Collection

In Figure 6, we analyze scaling behavior of passive data collection policies: in Four Rooms, uniform random data is both theoretically and practically strongest: uniform actions with uniform state resets provides near optimal evidence for maximizing explained action information score and thus controllability in fully observable, deterministic settings, approximating exhaustive coverage. Expert-like PPO policies require much more data because their policy optimality purposefully underweights sampling many action informative alternate transitions. Push-T shows the same qualitative lesson in a harder setting: broader non-expert data improves controllability, but the best data source depends on the specific eval rollout distribution. These results show that a better data generating policy can be worth 3× the scaling efficiency per sample in improved controllability. Random exploration works well when coverage is easy, but becomes less effective in high-dimensional manipulation with biased starts and partial observability. Additionally, in Four Rooms, explained action information correlates strongly with state-extracted simulation fidelity when evaluated exhaustively over the state space, supporting our hypothesis that controllability is a useful surrogate for task-relevant fidelity.

Policy Details

Algorithm 1 Batch AIM collection

Require: Replay $\mathcal{D}_0$, density checkpoint $M_{-1}$, rounds $K$, batch sizes $B_k$

1. for $k = 0, \ldots, K - 1$ do

2. Fit reward and annotate replay using Algorithm 2: $(M_k, \widehat{\mathcal{D}}_k) \leftarrow\operatorname{FitReward}(\mathcal{D}_k, M_{k-1})$

3. Initialize collector $\pi_k$ from $M_k$’s behavior chunk policy; reset IQL critic and value heads

4. Fit IQL, behavior cloning, and AWR flow matching on $\widehat{\mathcal{D}}_k$; enable AWR after critic warmup

5. Collect $B_k$ environment trajectories $\mathcal{T}_k$ using $\pi_k$

6. $\mathcal{D}_{k+1} \leftarrow\mathcal{D}_k \cup\mathcal{T}_k$

7. end for

8. return $\mathcal{D}_K$ for downstream world-model training

Algorithm 2 Reward fitting and annotation

Require: Replay $\mathcal{D}_k$, prior checkpoint $M_{k-1}$

FitReward$(\mathcal{D}_k, M_{k-1})$:

1. Warm-start from $M_{k-1}$; fit $\hat{q}_k$, $\hat{\beta}_k$, $\hat{d}_k$, and a behavior chunk policy on $\mathcal{D}_k$

2. Jointly train divergence heads for the three reward densities

3. Select checkpoint $M_k$ using held-out likelihood; freeze it

4. Integrate flows and learned divergences to score all replay transitions

5. Standardize each term using refreshed global replay statistics:

   $$\hat{r}_k = z_k(\log\hat{q}_k - \log\hat{\beta}_k) - z_k(\log\hat{d}_k)$$

6. Annotate $\mathcal{D}_k$ with $\hat{r}_k$ to obtain $\widehat{\mathcal{D}}_k$, fixed for policy fitting

7. return $(M_k, \widehat{\mathcal{D}}_k)$

We train a number of policies throughout this work across both discrete and continuous domains.

For the discrete domain Four Rooms, we implement our expert policies in the Stoix RL framework [84] via a Multi-Layer Perceptron (MLP) which consumes local field of view simulator state features and any extra provided state information (e.g. absolute position, goal position, and door positions for the Oracle variant). These policies predict a softmax action distribution which is then sampled from to provide stochasticity. These are then nominally trained for 300 million environmental transitions via PPO [78] to achieve reward saturation (See Table 2 for details). For our behavior cloning policies, we instead use a pixel-based softmax policy with a Stable Diffusion Variational Auto-encoder (VAE) [20, 79] and a ViT [92] on top to provide feature tokens which are cross attended to by a learned policy decoder token. Our IQL [71] implementation similarly takes the same ViT encoded tokens and cross attends with a readout decoder transformer which takes in a learned value token

Expert Policy Name	Steps Trained	Entropy	Eval Return
POMDP PPO	300M	0.054	0.657
Early Stop PPO	50M	0.252	0.587
State Oracle PPO	300M	0.022	0.839

and two learned Q tokens for pessimism. The output decoder tokens are fed into individual MLPs to readout the estimated policy or value estimate for usage. The VAE encoder remains fixed throughout training, everything else is trained end-to-end.

For the continuous domain Push-T, we construct the expert policy based on the Maniskill [83] simulator state-based continuous PPO [78] implementation and likewise use their dense Push-T reward to learn a policy. For the "Push Away" Push-T task we invert the goal footprint reward to encourage placement of the tee on the table in non-overlapping positions with the goal, providing a near symmetric inverse of the original Push-T task in terms of value. We also add an intrinsic novelty reward $r^{\mathrm{nov}}_{t,e}$ to encourage diverse positions. It is implemented as the following

which is computed by the minimum distance to a random subset of $n = 32$ states sampled uniformly from all parallel trajectory rollouts, thereby acting like a pairwise contrastive term [93, 94] to encourage diverse trajectory states. BC Failure data is generated by a geometrically distributed

World Model Architectural Details

switching between Push-T BC and Push Away policies, with a 3:1 bias towards Push-T BC. For the continuous behavior cloning policies, we use a diffusion policy using a similar Stable Diffusion VAE [20] and ViT [92] backbone to the discrete implementation, which provides cross-attention context for a diffusion policy [70] implemented via flow matching [17]. We parameterize actions as delta-actions with respect to the current robot end-effector frame.

The continuous IQL [71] implementation similarly uses Q and V decoder tokens in the same decoder structure. Since we use an action chunking policy [95], its natural that the Q function must evaluate action chunks rather than just individual actions, and we adopt an action chunking n-step bootstrap target similar to Q-Chunking [96]. The input actions are normalized and embedded using cosine positional encoding embeddings [85, 86] to provide tokenization, and added to learned decoder query tokens, where each token represents one timestep in the chunk. These tokens are then concatenated with the state context tokens to enable Q estimation. Policy extraction is then easy, where a single AWR [72, 97] weight is applied in flow matching to the whole target action chunk. Further information on hyperparameters used in training and finetuning the VAE, Policies, and Critics can be found in Table 4 and Table 3.

	Softmax BC	Diffusion BC	Softmax IQL+AWR	Diffusion IQL+AWR
Task
Environment	FourRooms [75, 76]	Push-T [83]	FourRooms	Push-T
Action space	6 discrete	6-D cont.	6 discrete	6-D cont.
Architecture
Model dim	256	256	256	256
Encoder layers	6	12	6	12
Attention heads	8	8	8	8
FFN ratio (SwiGLU)	4	3	4	3
IQL decoder layers	–	–	6	6
Action chunk $C$	–	16	–	16
IQL + AWR
Expectile $\tau$	–	–	0.7	0.7
Discount $\gamma$	–	–	0.99	0.99
AWR inverse temp. $\beta$	–	–	0.1	0.1
AWR weight clip	–	–	[0, 100]	[0, 100]
AWR warmup (steps)	–	–	20k+5k ramp	20k+5k ramp
Target EMA $\tau_{\mathrm{EMA}}$	–	–	0.005	0.005
Optimization
Optimizer	AdamW [89]	AdamW	AdamW	AdamW
Peak LR	$1\mathrm{e}{-4}$	$2\mathrm{e}{-4}$	$1\mathrm{e}{-4}$	$2\mathrm{e}{-4}$
Weight decay	0.01	0.01	0.01	0.01
Grad clip (norm)	1.0	1.0	1.0	1.0
LR schedule		warmup [90] $\to$ cosine to 0
Train steps	30k	100k	50k	50k
Global batch (seq.)	256	256	256	256
Params (non-VAE)	19.5M	30.6M	26.9M	44.6M

Hyperparameter	Value
Initialization	sd-vae-ft-mse [20]
Finetuned scope	decoder + post-quant conv (49.5M); encoder frozen
Image resolution	$144 \times144 \times3 \to[-1, 1]$
Latent grid	$18 \times18 \times4$ (8× downsample)
Latent scaling	0.18215
Loss	MSE + LPIPS (VGG) [91]
Loss weights	1.0 / 1.0, each EMA-normalized ($\alpha= 0.99$) [26]
Latent encoding	posterior sample
Optimizer	AdamW [89]
Peak LR	$1\mathrm{e}{-5}$
LR schedule	5k warmup [90] $\to$ cosine to 0
Weight decay	0.001
Grad clip (norm)	1.0
Train steps	50k
Global batch	96 sequences
Augmentation	FourRooms: color inversion + translation; Push-T: none
Seed	42

Finally, as mentioned in § 3.5, we apply z-score normalization to the raw densities with the population statistics before using them as rewards due to their significant differences in magnitude resulting from the high observation dimensionality and the relatively low action dimensionality. Z-score normalization can be seen as baselining of the rewards and power scaling of each of the Boltzmann tilt reward terms, and renders each of the rewards dimensionless which makes hyperparameter tuning of reward weights relatively easy and hyperparameter insensitive. An additional perspective is that this normalization is a tunable choice for a more restrictive trust region of the policy update. This results in the following practical learned AIM density reward used in our experiments:

World Model Architectural Details

We use two latent diffusion models in our experiments, one each for discrete and continuous environments. We implement all models in JAX [99] and for both environments we train the world models from scratch without pretrained weights to avoid data contamination.

In Four Rooms, we train a simple DiT [77] under a flow matching objective [17] with state context generated from a Stable Diffusion VAE encoder [20] fed through a ViT [92] to tokenize the latent representation. Since the environment is fully observable, we only condition on the current state observation image through cross attention. Actions are provided by concatenating a simple per-action learned entity embeddings [100] to the cross attention context.

For the continuous and partially observable Push-T environment, we must incorporate means of temporal stability through historical context. For that, we leverage a state-of-the-art latent Diffusion Forcing [18] model trained similarly with a flow matching objective [17] and a similar architecture to DreamerV4 [26]. Image observations are encoded with a Stable Diffusion VAE [20], patchified with linear projection, and visualized with Stable Diffusion VAE decoders finetuned with in domain data.

Both its action and diffusion timestep tokenization are completed using positional encodings [85, 86] fed through an MLP. The causal transformer itself enforces a sequence causal self-attention to allow for inference time autoregressive rollouts using a VideoRoPE [98] positional encoding. The pre-norm transformer is outfitted with modern transformer improvements such as QKNorm [87] and SwiGLU [88]. See Figure 7 for details. Full information on the hyperparameters used for finetuning the VAE and training the world models can be found in Table 4 and Table 5 respectively.

	Latent Diffusion WM [77]	Diffusion Forcing WM [18, 26]
Setting
Domain	FourRooms [75, 76]	Push-T [83]
Prediction	single-transition next-frame	sequence-level + AR rollout
Architecture
Model dim	512	512
Layers	12 enc. + 12 dec.	12 (9 spatial / 3 temporal) [26]
Attention heads	8	8
FFN ratio (SwiGLU)	3	3
Position encoding	2D sin–cos [85, 86]	VideoRoPE [98]
Inference
Sampler	Euler	Euler
Denoising steps	128	128
Optimization
Optimizer	AdamW [89]	AdamW
Peak LR	2e−4	2e−4
Weight decay	0.01	0.01
Grad clip (norm)	5.0	5.0
LR schedule	10k warmup [90] → cosine to 0	10k warmup → cosine to 0
Train steps	100k	100k
Global batch (seq.)	256	128
Max sequence length	48	64
Params (non-VAE)	97.8M	44.5M

Experimental Setup

The objective of our experiments is to evaluate data-generating policies that improve the controllability of world models, rather than to compare specific world models or architectures. As such, all world models on a given environment are identical and trained with exactly the same schedule except for the data they are trained on.

In Four Rooms, we generate expert “source" data with a uniform random policy and a number of PPO [78] policy variants: Local Field of View/Partial Observability (denoted POMDP PPO), an early stopped POMDP PPO (Early Stop PPO), and a sim-state-privileged variant (Oracle PPO). To evaluate performance, we sample $n=5$ random layouts, and then exhaustively evaluate every state-action pair, and match the predicted image with the maximum-likelihood state through exhaustive comparison to provide an accurate measure of simulator controllability and fidelity.

Expanded Four Rooms Results

For Push-T, such exhaustive evaluation is not feasible or informative, so instead we evaluate on rollouts from three policy evaluation transfer targets: a weak behavior cloned (BC) diffusion policy [70] trained on the original replay buffer, a “Push Away” PPO [78] policy with a dense reward encouraging diverse trajectories pushing the tee away from the traditional goal, and a BC Failure hybrid between the two to represent moderately weak learners. This is meant to evaluate two primary target policy evaluation downstream targets: improving a BC policy with an on-policy world model, transferring a policy to a new task with an off-policy world model, and improving a weak learner which regularly makes non-expert mistakes. To isolate the controllability of the world model itself, we evaluate each frame by the end-effector masked per-pixel mean absolute error recall (pixel threshold 10% of range so each color is unique) with a simulator-privileged mask to isolate only end-effector control sensitivity, penalizing both poor position and poor reconstruction as a proxy for rollout stability.

Given the cost of training multiple seeds across so many models, we only train one model per dataset/scale/resolution. For both $H = 10$ and $H = 33$, we evaluate each model with three inference seeds for each of 100 policy rollouts per evaluation distribution, using the same evaluator. We then perform bootstrap resampling of trajectories and inference seeds ($n = 10000$) to capture the variance across evaluations within a model. Our human arena A/B tests show two model-predicted rollouts alongside the corresponding ground-truth rollout to compare two models under a Bradley–Terry preference ELO rating with the prompt “Which prediction more accurately represents the behavior of the robot in the ground truth video?” (1545 comparisons, 20 users) [80].

Push-T is a continuous state space where simple Brownian motion provides limited coverage, so for fair comparison we implement both uniform random and a random goal-reaching policy, in addition to a PPO expert and a recovery baseline that alternates between a random-walk policy and the PPO expert with geometrically distributed switching times, providing a synthetic RaC-style analogue [5]. To compare against the various kinds of intrinsic reward exploration methods, we implement DIAYN [54], DADS [44], ICM [50], and METRA [55]. These methods represent the families of prior unsupervised exploration methods in RL, including mutual information skill discovery (DIAYN, DADS), curiosity (ICM), and metric-aware skill learning.

We use a common batch-online protocol to train all of these methods: each agent is first pretrained on the same prior dataset of 3005 trajectories (around $1.1 \times10^{5}$ transitions) generated by an expert PPO policy, and then learns to explore using its own mechanism. We perform $2 \times10^{5}$ gradient updates before collecting 100 new trajectories using the learned exploratory policies. We repeat this process for 5 rounds (500 online trajectories total). All these baselines share a single off-policy SAC [101] backbone: a tanh-squashed Gaussian actor and twin critics with [512, 256] hidden units over the 31-dimensional Push-T state, and differ only in the definition of the intrinsic rewards. Specifically, DIAYN [54] maximizes the discriminability of 50 discrete skills under a state discriminator. DADS [44] learns a skill-conditioned mixture-of-Gaussians dynamics model ($z \in[-1, 1]^{8}$, four mixture components) and rewards the mutual information between skills and next-state transitions. ICM [50] rewards forward-model prediction error in a learned feature space (curiosity scale $\eta= 50$, forward/inverse weight $\beta= 0.3$). METRA [55] learns four continuous unit-norm skills under a temporal-distance-preserving encoder with a dual Lipschitz constraint (reward scale 10). For DADS [44] and METRA [55], we restrict the intrinsic reward to the manipulation-relevant simulator state dimensions (the end-effector and tee poses) and for ICM [50] and DIAYN [54] to the tee pose alone, so that skills and curiosity target object motion rather than proprioception. The hyperparameters for all four methods are summarized in Table 6.

Method	Intrinsic reward	Key hyperparameters	learning rates
DIAYN [54]	skill discriminability	50 discrete skills; disc. over T-block pose	$3 \times10^{-4}$
DADS [44]	skill–dynamics MI	$z \in[-1,1]^8$, 4 mixtures, $K=100$ prior samples	$3 \times10^{-4}$
ICM [50]	forward-model error	$\eta= 50$, $\beta= 0.3$, feature dim 256	$3 \times10^{-4}$
METRA [55]	temporal-metric skills	4 continuous skills, reward scale 10, dual reg.	$1 \times10^{-4}$

Expanded Four Rooms Results

In addition to the results of § 4.2, we further investigate three questions about the Four Rooms environment: Q1: What happens to world model controllability in the low data limit? Q2: What does data coverage look like when comparing Random vs AIM data collection? Q3: What is the effect of batch online data collection on other measures of sim fidelity?

World Models Can Collapse to Their Data Generating Policy in the Low Data Limit

In Figure 8, we explore the upper and lower limits of the quantity of expert data for training world models. This figure shows an exhaustive evaluation of a Four Rooms world model trained on 150 trajectories compared to one on 300k trajectories, visualizing the expressive reproduction of action controllability in the 100% data case, compared to the near non-existent action controllability of the low data regime on the left, where the world model transitions reproduce the underlying modes of the data generating policy near exactly under any action conditioning. This proposes an intriguing hypothesis: the moderate action controllability of today’s largest world models trained on seemingly endless internet video may actually be in the low data scaling regime of action-conditioned generative world models. This further motivates our AIM intervention of generating more informative data to improve data scaling efficiency.

AIM vs Random Fixed Start Data Coverage Ratios

One likely reason for the disagreement in optimal exploration between the low dimensional discrete Four Rooms and the higher dimensional Push-T task is that uniform random actions suffer from the curse of dimensionality as state-action dimensionality go up, making uniform state visitation support unlikely in a reasonable time horizon. In contrast, AIM can explicitly perform goal-directed behavior to provide more uniform state visitation in aggregate. To test this hypothesis, we compare state visitations in Four Rooms between rollouts generated by either successive AIM iteration policies or from a random policy, with all rollout episodes fixed to a predetermined initial state. This removes the influence of uniform state sampling which Figure 5 shows is critical to the low rollout budget efficiency of uniform random actions in Four Rooms. In Figure 9, we visualize the ratio of state visitation between these two data collection policies. The heatmaps show rollouts collected with successive AIM iteration policies are more likely to explore states farther from the start point, improving coverage and supporting the perspective that the policy actively explores the environment to search for novel state transitions that enhance the informativeness of the existing dataset. Also of note is that AIM iterations cover largely non-overlapping sets of state visitation between iterations, showing that multiple batch iterations are useful to seek under-explored subsets of the state space and that the learned policy will largely avoid revisiting transitions it has already observed, precisely the desired behavior of a informative data acquisition policy.

Effect of Batch Online Data Collection on Other Four Rooms Metrics

In Figure 10, we explore the effect of batch online data collection on other metrics. One interesting detail is the strong correlation of pixel RMSE with sim fidelity, which is in contrast to broader expectations that generative models should be evaluated with measures of distributional similarity (e.g. FID [32], FVD [33], etc.). Instead, we find that, at least in this deterministic environment, RMSE pixel error is a strong proxy for state-extractive sim fidelity metrics. Our hypothesis is that this emerges in these simple deterministic environments because the only place where a model will make a generative mistake is where the interesting critical simulation state is most salient. Thus, the pixel RMSE is largely focused on the same pixels as the state-extractive sim fidelity metric.

Additionally, we see that measured world model action influence grows proportionately with sim realism, reinforcing the relationship observed in Figure 3 between world model controllability and our statistical action influence measure. Another interesting observation is that observed action influence saturates at roughly the same rate as sim fidelity, indicating it may be a strong measure for simulator fidelity. If explicitly optimized with a model’s own generations (e.g. as a post-training RL objective), the action influence metric can easily be gamed by inventing new transitions or states which do not correspond with reality but satisfy the learned reward model. Outside this pathological scenario and in conjunction with Figure 3, these indicate that action influence may serve as a useful proxy for world model sim fidelity on real transition datasets when a high quality sim realism evaluation is not available.

Expanded Push-T Results

Similarly to Section F, we expand on our investigation from § 4.2 of the Push-T environment with two further questions: Q1: Why do we need trust regions for density-based intrinsic motivation? Q2: What is the effect of batch online data collection on other measures of sim fidelity?

Trust Regions Suppress Out-of-Data Distribution Reward Estimation

In Section A.2, we show that AWR-style conservative KL constrained policy improvement [72, 97, 102] is theoretically motivated for dataset action influence policy improvement. Given prior mutual information-based skill diversity and exploration work [44, 54, 55] has used similar intrinsic motivation rewards and struggled to generalize to pixel observations and high dimensional action spaces, it seems worth investigating why this seemingly simple policy conservatism creates more focused exploration. In Figure 11, we evaluate the learned AIM reward on a grid of reachable states in the Push-T environment to visualize the underlying reward estimation from a fixed initial state condition. We see that the learned AIM intrinsic motivation reward provides significant reward for regions outside of the nominal action workspace, and unbounded policy optimization would encourage unfocused state coverage, similar to prior work, toward regions that are less interesting for interaction but are otherwise highly rewarded for their diversity. One possible explanation is that directly optimizing the AIM reward may result in the same diverse skills but relatively little environmental interaction pathology as prior work [44, 54, 55], and that KL constrained policy optimization like AWR [72, 97] restricts the reward maximizing policy to regions near existing behavior priors, providing informative transitions which minimally deviate from prior expert data, focusing exploration to the most interesting, behavior-visiting regions first before exploring outward. Additionally, another strength of evaluating our reward only in an offline RL [71, 103] setting is that our learned density reward has reduced exposure to poorly calibrated out of distribution likelihood estimation which can occur in diffusion-style density modeling [104], and that our distilled likelihood map [19] will never be queried out of distribution making its predictions lower error.

Effect of Batch Online Data Collection on Other Push-T Metrics

In Figure 12 and Figure 13, we investigate the effects of batch online data collection on other measures of Push-T sim fidelity for both the Behavior Cloning and “Push Away” policy evaluation distributions. In particular, our controllability metric evaluates the controllability of the robot arm

through end-effector pixel recall, which isn’t necessarily the same as complete simulator fidelity in all cases. We evaluate the low data regime scaling behavior of the tee pixel recall masked in the same way as the end-effector controllability measure. It’s clear that all data collection methods see similar performance improvements with additional off-policy data, with DIAYN [54] being the lone underperformer. Additionally, some of the intrinsic reward methods look particularly compelling on this metric, but this is likely due to their access to simulator state of the tee pose, as we found them unable to explore productively in the Push-T environment with only pixel observations. However, qualitatively, even the highest data volume models struggle with tee fidelity on both evaluation distributions, and the differences are more stark in the off-policy “push away" evaluation distribution in Figure 13 where active data collection scaling is noticeably less efficient compared to the near expert behavior cloning data. We hypothesize that better extrapolation to novel tee states would likely only happen with a rich prior data distribution to improve generalization, which we eschew in this work to prevent data contamination for precisely this reason. Overall, it seems pathological evaluation behaviors like “push away" Push-T would remain difficult for world models to extrapolate to without noticeably more active data collection or some on-policy expert demonstrations to guide AIM exploration.

Additionally, similar to in Section F, we find that full image pixel RMSE and recall is a surprisingly good realism metric in a deterministic world modeling setting, even when full observability is relaxed. However, the disagreement between tee pixel recall ordering, end-effector controllability in Figure 5, and the full image pixel metrics begin to show full image pixel RMSE’s lack of sensitivity to errors in semantically important regions of the image, motivating our targeted sim fidelity metrics and further work on effective evaluation of robotic world models.