betaFeedback

WorksComputer Science

Vision-Language Agents Can Learn in Their Own Latent Worlds

Abstract

Learning from experience is essential for building the next generation of agents that continually expand their capabilities. Pretrained vision-language models (VLMs) already internalize rich world knowledge, yet this knowledge is rarely leveraged to drive policy improvement through online interaction. Realizing this potential efficiently remains challenging, as policy updates shift the agent's interaction distribution, requiring the world model to adapt accordingly, while continually updating large pretrained models is computationally expensive. To address this challenge, we introduce Dwell, a model-based reinforcement learning (RL) framework that reframes a VLM as a recurrent latent transition operator, endowing it with the ability to predict, adjust, and act in latent space. Leveraging the VLM's own representations and its dual role as both world model and policy, the agent continually learns from real interactions to anticipate and evaluate action outcomes, and from latent imagination to improve its behavior. Across five interactive visual environments, Dwell achieves the strongest overall performance among trainable methods while substantially reducing the compute required for imagination, showing that effective world models for VLM agents need not reconstruct the future; they need to predict futures that are useful for action. Our code is publicly available at https://github.com/Tom111-lhx/Dwell

Introduction

AI agents [32, 41] have acquired broad capabilities through large-scale pretraining on human-generated data [9, 21] and post-training with human demonstrations and preference feedback [47, 33]. Extending this progress beyond the limits of human-provided data [43] motivates agents that continually learn from experience generated through their own interactions with the environment [38]. Recent work [44, 28] has taken steps in this direction by using reinforcement learning (RL) to improve agents’ multi-step decision-making with environmental feedback. However, the substantial environment interaction often required by model-free approaches can make experience collection a bottleneck to scaling [24, 11]. This raises a central question: how can agents achieve greater policy improvement from limited real experience?

World models offer a way to learn more from real experience by predicting action outcomes and generating imagined trajectories for policy learning. For vision-language model (VLM) agents, models that synthesize observations have already served as training environments across embodied and web tasks [46, 15]. In contrast, imagination in a compact latent space avoids reconstructing observations at every predicted step [18, 19], while predictive representation learning can emphasize predictable structure over incidental visual detail [2, 30]. Pretrained VLMs already encode rich world knowledge, suggesting that their own representations could support latent imagination for policy learning.

Accurate prediction alone does not guarantee compatibility: a world model’s latent states may not be readily usable by a VLM policy, even when they faithfully capture action outcomes [3, 12]. A further difficulty arises as policy updates shift the agent’s interaction distribution toward states and actions poorly covered by the world model’s training data [24]. In these regions, errors in predicted transitions and rewards can bias subsequent policy updates, so gains in imagination may not translate into better behavior in the environment [29]. Adapting to this evolving distribution can also be computationally expensive when it requires repeated updates to a large pretrained VLM. Together, these challenges motivate a shared representational basis for prediction and decision-making, coupled with efficient, continual correction of transition and reward predictions from real interaction.

We introduce DWELL, a model-based RL framework that enables VLMs to predict, adjust, and act in their own latent space. By recasting the VLM as a recurrent latent transition operator, we equip it to reason about the state of the world from interaction history and improve its behavior through latent imagination. Following the joint-embedding principle [2], our method draws on the VLM’s rich world knowledge to actively infer a state representation in an abstract and compact space, rather than focusing on every detail of the observation. Exploiting the VLM’s dual role as a world model and policy, we instantiate both components using separate lightweight LoRA adapters [22] on the same VLM backbone, allowing independent, parameter-efficient adaptation while sharing a common representation interface. Together, recurrence and duality provide a unified framework for modeling and acting in dynamic environments through online learning.

We conduct comprehensive experiments across five interactive visual environments spanning spatial planning, embodied navigation, manipulation, and long-horizon household tasks. Under the reported training protocol, DWELL achieves the superior performance among trainable methods, while substantially reducing the computational cost of imagination relative to a world-model baseline that reconstructs observations. In summary, the main contributions of the paper are as follows:

  • Learning and acting in a shared latent world. We recast a pretrained VLM as a recurrent latent transition operator, unifying world modeling and policy learning through a common belief interface.

  • Online policy improvement through latent imagination. We develop a learning loop in which real interactions continually refine the world and reward models, while imagined trajectories guide policy improvement.

  • Empirical gains in performance and efficiency. Our proposed method achieves strong overall performance among the evaluated trainable baselines under the reported training settings, while reducing imagination cost relative to a world model that synthesizes observations.

Method

A self-evolving agent should anticipate the consequences of its actions and use feedback from interaction to continually refine its predictions and behavior. Although VLMs have acquired rich world knowledge, they tend to be brittle when exposed to dynamic environments or when learning from experience. Our method, DWELL, anchors a VLM as a recurrent latent transition operator, morphing a static perceiver into an active predictor. Given streams of observations and actions from online interaction, it infers the most explainable states in the VLM’s own latent space, which can in turn be used for model-based planning in the absence of feedback. This cycle of interaction, refinement, and improvement enables the agent to continuously expand its understanding of the environment while adapting its actions accordingly.

Interaction Setting. We model this online interaction as a partially observable Markov decision process (POMDP) [4, 26], defined by a tuple (S,A,O,T,R,Z,γ)(\mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{T}, R, Z, \gamma), where S\mathcal{S}, A\mathcal{A}, and O\mathcal{O} correspond to the state, action, and observation spaces. At each step tt, the agent takes an action at∼πθ(at∣bt)a_t \sim\pi_\theta(a_t \mid b_t) based on its belief btb_t, a sufficient statistic computed from the history ht=(o0,a0,…,ot−1,at−1,ot)h_t = (o_0, a_0, \ldots, o_{t-1}, a_{t-1}, o_t). The environment then transitions to a hidden state st+1∼T(st+1∣st,at)s_{t+1} \sim\mathcal{T}(s_{t+1} \mid s_t, a_t) and emits a visual observation ot+1∼Z(ot+1∣st+1,at)o_{t+1} \sim Z(o_{t+1} \mid s_{t+1}, a_t) and reward rt+1=R(st,at,st+1)r_{t+1} = R(s_t, a_t, s_{t+1}). Since the state is unobserved, the VLM agent must form a belief over it and learns a policy πθ\pi_\theta to maximize the expected cumulative discounted return:

J(θ)=Eπθ,T,Z[∑t=0∞γtR(st,at,st+1)],(1)J(\theta) = \mathbb{E}_{\pi_\theta,T,Z}\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t,s_{t+1})\right], \tag*{(1)}

where γ∈[0,1)\gamma\in[0,1) is the discount factor. The belief btb_t upon which the agent acts is updated deterministically as bt=τ(bt−1,at−1,ot)b_t = \tau(b_{t-1},a_{t-1},o_t) using a belief update function τ\tau given an initial belief b0b_0. Depending on the model specification, the belief can take either a probabilistic or deterministic form.

Recurrence and Duality: A Common Interface

Recurrent Latent Model. Learning from experience requires connecting two sources of state information: what the agent observes and what it can anticipate from its actions. Let ν(ot)\nu(o_t) denote the ordered sequence of visual tokens extracted by the VLM from observation oto_t, and zt∈RK×Dz_t \in\mathbb{R}^{K \times D} represents the agent’s latent state, comprising KK slots of dimension DD. Given the previous state zt−1z_{t-1} and action at−1a_{t-1}, we define two recurrent updates:

zt=Σϕ(zt−1,at−1,ν(ot)),z^t=Ψϕ(zt−1,at−1).(2)\begin{aligned} z_t &= \Sigma_\phi(z_{t-1}, a_{t-1}, \nu(o_t)),\\ \hat{z}_t &= \Psi_\phi(z_{t-1}, a_{t-1}). \tag*{(2)} \end{aligned}

The observation-conditioned update Σϕ\Sigma_\phi incorporates the realized outcome of at−1a_{t-1} into the agent’s evolving belief, whereas the action-conditioned predictor Ψϕ\Psi_\phi anticipates that outcome without access to oto_t. We refer to their outputs as the posterior state ztz_t and the prior state z^t\hat{z}_t, respectively. Here, these two terms distinguish the information available to each update: both states are deterministic latent representations rather than probability distributions.

Conceptually, if we follow a joint embedding predictive architecture (JEPA) [2], our formulation generalizes it by allowing the incorporation of historical information. This can be seen by removing zt−1z_{t-1} and at−1a_{t-1} from the posterior and setting Σϕ=I\Sigma_\phi= I. When the observation alone is not a sufficient statistic, our formulation induces better representations while improving the separability of the latent states by anchoring their representations to the observation. Compared with reconstruction-based methods, ours operates in the latent space without focusing on every detail of the observation, thereby improving computational efficiency.

VLM as World Model and Policy. We instantiate the world model and policy using separate LoRA adapters [22] on a shared, frozen pretrained VLM backbone. Within the world model, posterior inference Σϕ\Sigma_\phi and prior prediction Ψϕ\Psi_\phi share the same trainable parameters ϕ\phi, whereas the policy πθ\pi_\theta uses a distinct set of adapter parameters θ\theta. This separation supports independent, parameter-efficient adaptation of prediction and decision-making while keeping the pretrained backbone fixed.

The latent slots retain their ordering across updates, providing a common interface for transition prediction, reward evaluation, and action selection. The policy πθ\pi_\theta conditions on posterior states during real interaction and on predicted states during latent imagination. Thus, predicted states can be passed directly to the next state transition and decision without decoding them into the raw observation and re-encoding them. Initialization and parameterization are detailed in Appendix § 7.7.

Dynamics, Representation, and Reward Learning

Learning a world model improves sample efficiency, enables planning and foresight, and enhances adaptability to environmental changes. The central questions are: (1) what state representation a good world model requires; and (2) how to make reliable predictions into the future to guide action selection. When faced with complex or high-dimensional observations, a naive approach is to optimize for reconstruction fidelity in the original space. However, this can be problematic, as it focuses too much on information that is redundant or irrelevant to solving the task, while also being computationally inefficient. VLMs can internalize rich knowledge in their latent space, providing us with a good starting point to build upon. We can leverage this internalized knowledge by operating in a compact latent space while attending only to the information most relevant to decision-making.

Dynamics Learning. When explicit feedback is absent, the world model needs to make reliable predictions into the future for planning and foresight. The posterior, in contrast, infers the states that best explain the current observation given the history, providing a natural learning target for the prior and thereby equipping the VLM with the ability to predict and adjust.

Overview of DWELL

Figure 1. Overview of DWELL. Left: The VLM learns a recurrent latent world by matching action-conditioned priors to observation-conditioned posteriors, while grounding the latent states with a frozen V-JEPA2 encoder. Right: The same VLM backbone serves as both world model and policy, alternating online interaction with latent planning for policy improvement.

Given sequences τ=(at,ot+1,at+1,ot+2,…,oT+1)\tau= (a_t, o_{t+1}, a_{t+1}, o_{t+2}, \ldots, o_{T+1}) from a replay buffer D\mathcal{D}, we learn the prior by:

Ldyn(ϕ)=Eτ∼D[∑k=1T∥z^t+k−sg⁡(zt+k)∥F2](3)\mathcal{L}_{\mathrm{dyn}}(\phi) = \mathbb{E}_{\tau\sim\mathcal{D}}\left[\sum_{k=1}^{T} \left\lVert\widehat{z}_{t+k} - \operatorname{sg}(z_{t+k}) \right\rVert_{\mathrm{F}}^2\right] \tag*{(3)}

where sg denotes stop-gradient.

Representation Learning. However, agreement between the posterior and prior alone does not establish their connection to observations; they can freely minimize the error while capturing no useful information for decision-making. While we can anchor the posterior to the VLM’s visual encoder, a representation that is well-suited for supervised learning may not readily transfer to online learning. Therefore, we turn to frozen pretrained visual encoders et=E(ot)e_t = E(o_t) such as V-JEPA2 through a learned slot-wise projection Γϕ\Gamma_\phi for posterior learning:

Lrep(ϕ)=Eτ∼D[∑k=1Td(Γϕ(zt+k),sg⁡(et+k))](4)\mathcal{L}_{\mathrm{rep}}(\phi) = \mathbb{E}_{\tau\sim\mathcal{D}}\left[\sum_{k=1}^{T} d\left(\Gamma_\phi(z_{t+k}), \operatorname{sg}(e_{t+k})\right)\right] \tag*{(4)}

where, for d, we use the averaged cosine distance over corresponding slots. Notably, despite the projection, learning remains in a compact representation space rather than the raw observation space, akin to JEPA, thereby avoiding expensive reconstruction.

Reward Learning. Although it is straightforward to learn the reward from the posterior, planning in the latent space with the prior presents a gap, which widens as the planning horizon increases due to compounding errors. For this reason, we propose to learn the reward from a pure prior lookahead given the future actions at:t+La_{t:t+L}, starting from a posterior ztz_t. We denote this compositional mapping as Π(zt,at:t+L;L)\Pi(z_t, a_{t:t+L}; L), which results in a trailing prior z^t+L+1\widehat{z}_{t+L+1}. This prior combines two ends: the history encoded in the posterior and the future actions, from which we predict the reward r^t+L+1=Rω(z^t+L+1)\widehat{r}_{t+L+1} = R_\omega(\widehat{z}_{t+L+1}). Over the trajectory, we sample a lookahead horizon up to the maximum length TT and apply Π\Pi at tk:=t+[k−L−1]t_k := t + [k - L - 1] for a given trailing step t+kt + k:

Lrew(ω)=Eτ∼D,L∼[T][∑k=1Tℓr(Rω(Π(ztk,atk:tk+L;L)),rt+k)](5)\mathcal{L}_{\mathrm{rew}}(\omega) = \mathbb{E}_{\tau\sim\mathcal{D}, L \sim[T]}\left[\sum_{k=1}^{T} \ell_r\left(R_\omega\left(\Pi(z_{t_k}, a_{t_k:t_k+L}; L)\right), r_{t+k}\right)\right] \tag*{(5)}

where ℓr\ell_r is the mean squared error for continuous rewards and the cross-entropy loss for discrete ones. This approach balances long-horizon reward prediction in the latent space with the use of historical information, providing a spectrum of stable learning for model-based planning. Notably, when L=0L = 0, it reduces to per-step prediction, which accounts for only one action at a time; when L≥TL \ge T, the prior is decoupled from the posterior and makes predictions on its own, ignoring any historical information.

Putting it all together, with an auxiliary regularizer [30] to discourage representation collapse, the objective for learning the world model is:

JWM=Ldyn+αLrep+βLrew+ηLreg,(6)\mathcal{J}_{\mathrm{WM}} = \mathcal{L}_{\mathrm{dyn}} + \alpha\mathcal{L}_{\mathrm{rep}} + \beta\mathcal{L}_{\mathrm{rew}} + \eta\mathcal{L}_{\mathrm{reg}}, \tag*{(6)}

where α\alpha, β\beta, and η\eta indicate the relative strength of each term.

Model-based Latent Planning.

A world model is a mirror of reality, allowing us to simulate without constant interaction with the environment. Planning with a world model requires anticipating future rewards to identify better actions, which can be compactly represented by a value function. However, the interleaved nature of the world model and value function necessitates a reliable initial value estimate to be backpropagated during planning, especially at the beginning of training.

Bridge the Gap. Learning to predict a state does not yet establish how much future reward it yields. With finite imagination, early policy updates also depend on value estimates that have not yet been fitted to task returns. We therefore warm up the critic on real experience using the nn-step return Gt(n)G_t^{(n)} before allowing it to guide policy improvement, while keeping the actor and world model fixed.

Planning with World Model. Starting from a posterior state z^0=zt\hat{z}_0 = z_t obtained by unrolling Eq. (2), we generate a batch B\mathcal{B} of imagined experiences {ah−1,z^h,r^h}h=1H\{a_{h-1}, \hat{z}_h, \hat{r}_h\}_{h=1}^{H} of length HH using the prior and the current policy πθ\pi_\theta:

ah−1∼πθ(⋅∣z^h−1),z^h=Ψϕ(z^h−1,ah−1),r^h=Rω(z^h),∀h∈[H].(7)a_{h-1} \sim\pi_\theta(\cdot\mid\hat{z}_{h-1}), \quad\hat{z}_h = \Psi_\phi(\hat{z}_{h-1}, a_{h-1}), \quad\hat{r}_h = R_\omega(\hat{z}_h), \quad\forall h \in[H]. \tag*{(7)}

Given the imagined batch B\mathcal{B}, in principle, we can use any actor-critic algorithm to learn both the value function and the policy. To improve sample efficiency and reduce variance, we adopt PPO [36] to update both.

Close the Loop: Iterative Online Learning. The learned policy returns to the real environment, reflecting the world model’s predictions, to collect new experiences and revise its beliefs. This cycle of alternating between interaction and learning, which subsumes the interplay between the world model and the policy and is centered around the VLM, enables continual adaptation and learning from experience at the boundary between the known and the unknown, the history and the future. Through this process, the agent improves its behavior iteratively through trial and error, as summarized in Algorithm 1.

Experiments

We evaluate DWELL across diverse interactive visual environments to examine whether latent imagination improves task performance and learning efficiency while remaining computationally practical.

Experimental Setup

Environments. We evaluate on five interactive visual environments spanning complementary decision-making regimes: Sokoban, FrozenLake, Navigation, PrimitiveSkill, and EB-ALFRED. Sokoban (Schrader, 2018) and FrozenLake [42] test spatial reasoning and multi-step planning in discrete visual environments; Navigation [45, 27] evaluates embodied navigation from first-person observations; PrimitiveSkill [40, 31] tests visually grounded manipulation; and EB-ALFRED [45] extends the evaluation to long-horizon compositional embodied tasks. Together, these environments cover increasingly complex forms of partial observability, action consequence prediction, and long-horizon decision making. We use task success rate as the primary metric and, unless otherwise specified, report the mean and standard deviation over three independent training seeds.

Baselines. We use Qwen2.5-VL-3B [6] as the unified VLM backbone for all trainable agent methods. We compare DWELL against model-free online RL with PPO [36] and GRPO [17], which optimize the policy directly from environmental feedback without imagined experience. For model-based RL, we adapt LEWORLDMODEL to our online agent setting by connecting its learned latent dynamics to the same VLM policy and online interaction pipeline; the adaptation is detailed in Appendix § 7.§ 2. We additionally report strong off-the-shelf VLMs as reference points without online adaptation.

Implementation Details. We keep the VLM backbone frozen and equip the policy and world model with separate trainable LoRA adapters. We use a frozen V-JEPA2 ([3]) encoder as the default visual target for representation grounding. We use LoRA rank r=32r = 32 with scaling factor α=64\alpha= 64. Full hyperparameters and environment-specific configurations are provided in Appendix § 7.§ 4.

Overall Performance

Results. As shown in Table 1, DWELL achieves the best overall performance among trainable methods, substantially improving the same Qwen2.5-VL-3B initialization. The gains are most evident on Sokoban, FrozenLake, and the long-horizon EB-ALFRED tasks, where successful decisions depend strongly on anticipating how current actions alter future states. This pattern supports our central hypothesis that action-conditioned latent imagination can turn limited real experience into useful additional training experience.

Model/MethodSokobanFrozenLakeNavigation BaseNavigation CommonNavigation AveragePrimitiveSkill PlacePrimitiveSkill StackPrimitiveSkill DrawerPrimitiveSkill AlignPrimitiveSkill AverageEB-ALFREDOverall
Open-Source VLMs
Qwen2.5-VL-72B ([6])0.200.440.700.770.741.000.500.001.000.630.390.48
Qwen2.5-VL-7B ([6])0.140.140.330.380.350.000.000.000.750.190.470.26
Qwen2.5-VL-3B ([6])0.130.140.200.260.230.000.000.000.000.000.010.10
VLM-R1-3B ([37])0.160.150.330.340.340.000.000.000.000.000.020.13
Proprietary VLMs
GPT-4o ([23])0.430.540.750.690.720.500.630.000.880.500.560.55
Claude 4.5 Sonnet Anthropic [1]0.310.800.670.670.670.630.500.001.000.530.710.60
Model-Free Online RL (Backbone: Qwen2.5-VL-3B)
PPO ([36])0.29±0.1250.29 \pm0.1250.22±0.0360.22 \pm0.0360.28±0.0720.28 \pm0.0720.27±0.0260.27 \pm0.0260.280000.29±0.5080.29 \pm0.5080.070.06±0.0340.06 \pm0.0340.18
GRPO ([17])0.21±0.0400.21 \pm0.0400.19±0.0520.19 \pm0.0520.74±0.1000.74 \pm0.1000.71±0.0530.71 \pm0.0530.730000.67±0.5780.67 \pm0.5780.170.10±0.0260.10 \pm0.0260.28
Model-Based RL (Backbone: Qwen2.5-VL-3B)
LeWorldModel ([30])0.44±0.0210.44 \pm0.0210.29±0.0610.29 \pm0.0610.30±0.0370.30 \pm0.0370.28±0.04930.28 \pm0.04930.2900.17±0.2890.17 \pm0.289000.040.13±0.0300.13 \pm0.0300.24
DWELL0.53±0.0120.53 \pm0.0120.51±0.0210.51 \pm0.0210.37±0.0360.37 \pm0.0360.40±0.02080.40 \pm0.02080.390.33±0.2880.33 \pm0.2880.21±0.2590.21 \pm0.25900.29±0.5080.29 \pm0.5080.280.18±0.0480.18 \pm0.0480.38

Table 1. Overall performance across five interactive visual environments. All trainable methods use Qwen2.5-VL-3B as the VLM backbone and are trained under the same environmental feedback and interaction budget. Off-the-shelf VLMs are evaluated without online adaptation. Bold indicates the best performance among trainable methods.

The benefit is task dependent rather than uniform. Model-free GRPO remains stronger on Navigation and slightly better on PrimitiveSkill. We attribute this difference to two distinct regimes: in navigation, immediate visual observations often provide strong action cues, reducing the marginal benefit of explicit future-state prediction; in contact-rich manipulation, useful imagination requires finer-grained dynamics that are harder to preserve in a compact latent rollout. In contrast, DWELL consistently improves over the adapted LeWorldModel baseline, suggesting that effective model-based learning depends not only on learning latent dynamics, but also on making predicted states directly usable by the VLM policy. We examine the efficiency of these gains and the role of each component below.

Learning and Computational Efficiency

We next study whether the gains of DWELL can be obtained efficiently, both in terms of learning from online interaction and the computation required to support imagination. Figure 2 reports learning curves under the same training protocol. For the model-based comparison, we additionally include AdaWorld ([16]), an adaptable generative world model based on latent actions, as a representative visual-generation-based world model. We adapt its official implementation to our evaluation setting; implementation details are provided in Appendix § 7.§ 2.

Figure 2. Learning Efficiency. Success rate throughout training on SOKOBAN and FROZENLAKE. Left: comparison with model-based approaches. Right: comparison with model-free approaches.

Learning Efficiency. Across both SOKOBAN and FROZENLAKE, DWELL reaches higher success rates than the alternative model-based approaches. The advantage is particularly clear on FROZENLAKE, where the competing methods improve only gradually while DWELL continues to benefit from interaction. These results suggest that recurrent latent imagination provides useful additional learning signal from experience already collected, rather than relying solely on new environment rollouts.

Computational Efficiency. We further measure the end-to-end GPU-hours required to reach the final training step used in each environment, including all GPU computation incurred by online training, such as world-model updates, imagined rollouts, and policy/value optimization. As shown in Figure 3, LeWorldModel remains the least expensive approach due to its compact latent dynamics. Nevertheless, DWELL consistently requires substantially less computation than AdaWorld, reducing GPU-hours by approximately 38–58% across the four evaluated environments. It also incurs lower total compute than PPO and GRPO under the same training schedule. Taken together with the performance results in Table 1, these findings indicate that DWELL achieves a favorable performance–compute trade-off: it retains the efficiency of latent-space imagination while avoiding the considerably higher cost of generating future visual observations.

Figure 3. Computational Efficiency. End-to-end GPU-hours required to reach the reported training step on FROZENLAKE, SOKOBAN, NAVIGATION, and EB-ALFRED. GPU-hours include all computation incurred during online training, including world-model updates, imagined rollouts, and policy/value optimization. All methods are evaluated under the same environment-specific training schedule and hardware configuration.

Ablation Studies

We ablate two aspects of DWELL that are central to learning from latent imagination: the supervision used to construct predictive latent states and the continual adaptation of the world model during online policy learning.

Role of the World Model Objectives. As shown in Figure 4, removing either predictive dynamics learning or representation grounding degrades performance on both SOKOBAN and NAVIGATION. The degradation is substantially larger without Ldyn\mathcal{L}_{\mathrm{dyn}}, indicating that direct supervision of action-conditioned future states is essential for producing useful imagined trajectories. Removing Lrep\mathcal{L}_{\mathrm{rep}} yields a smaller but consistent drop, showing that predictive consistency alone is insufficient and benefits from anchoring the latent state to perceptually meaningful visual representations. Together, the two objectives play complementary roles: Ldyn\mathcal{L}_{\mathrm{dyn}} learns how the state evolves under actions, whereas Lrep\mathcal{L}_{\mathrm{rep}} constrains what information the state should preserve.

Figure 4. Ablation Studies. Left: effect of removing the representation-grounding objective Lrep\mathcal{L}_{\mathrm{rep}} or the predictive dynamics objective Ldyn\mathcal{L}_{\mathrm{dyn}}. Right: effect of freezing the world model after initialization instead of continually updating it with newly collected experience. Predictive dynamics provide the primary supervision for useful latent imagination, while representation grounding and continual world-model adaptation further improve policy learning.

Effect of Online World Model Adaptation. We further freeze the world model after its initial fitting and continue policy optimization with the resulting fixed dynamics. The frozen model remains competitive early in training, but its performance quickly saturates on both SOKOBAN and FROZENLAKE, whereas the full DWELL continues to improve as additional interaction is collected. This behavior is consistent with a policy-induced distribution shift: as the policy changes, it visits state–action regions that are increasingly different from those used to fit the initial model. Continually refreshing the world model with newly collected experience therefore keeps imagined trajectories aligned with the evolving policy distribution and enables sustained policy improvement.

Related Work

Latent World Models for Behavior Learning. Model-based RL has long used learned dynamics to augment real experience with prediction and planning. Dreamer learns policies from trajectories imagined in a compact latent state space, while TD-MPC and its successors optimize behavior with decoder-free latent world models [18, 19, 20]. More recent predictive-representation approaches replace observation reconstruction with latent prediction, enabling action-conditioned planning or policy learning directly in representation space [3, 5]. These methods establish that control need not operate on reconstructed observations. Our focus is complementary: we study how such a latent learning interface can be built into a VLM agent, so that the representations used for real decisions also form the state space of imagined policy learning.

World Models for Agent Learning. Generative world models have recently become practical training environments for large-model agents. In embodied settings, video world models synthesize action-conditioned futures for VLA policy optimization [46, 25], while learned web simulators generate realistic agent interaction experience for self-improvement in browser agents [15, 14]. A common design is to adapt the simulator to policy behavior and then hold it fixed while optimizing the policy. This separation makes model-generated experience reusable, but the simulator cannot incorporate new evidence arising during the subsequent policy updates. World-VLA-Loop [29] takes a step toward closing this loop by iteratively adding new policy rollouts to world-model training; however, policy learning still proceeds through autoregressively generated visual observations and inherits their long-horizon drift. DWELL instead uses the VLM’s predicted beliefs themselves as the interaction states: policy learning requires neither future-image generation nor re-encoding, and the latent world is refreshed from newly collected real experience throughout training.

World Modeling within VLM Agents. A parallel line of work develops world-modeling capabilities inside the agent itself. VLM representations can provide semantically informed state features for control, and explicit state estimation and transition reasoning can improve multi-turn visual decision-making [13, 44]. Other approaches incorporate future prediction more directly into action learning: VLA-JEPA uses latent state prediction to learn action-relevant representations, and unified world–action models jointly predict observations and actions [39, 10]. In these methods, world modeling primarily shapes the representation, reasoning process, or action predictor. DWELL gives the predicted state a different operational role: it is the state on which the policy continues to act and learn. A single frozen VLM backbone therefore supports two trainable roles through separate adapters: one learns action-conditioned latent dynamics, while the other learns the policy, and real interaction continually supervises both the predicted dynamics and their task outcomes.

Limitation and Discussion

When Does Latent Imagination Help? The benefit of latent imagination is not uniform across the evaluated tasks. While DWELL shows clear gains on SOKOBAN, FROZENLAKE, and EB-ALFRED, its advantage is smaller on NAVIGATION and PRIMITIVESKILL. This pattern is consistent with the hypothesis that imagination is most useful when predicted future states provide information beyond what is already available from the current observation, whereas its benefit may diminish when action selection is largely observation-driven or depends on dynamics that are difficult to capture compactly. A natural next step is therefore to make imagination selective, allowing the agent to decide when and how far to imagine rather than applying a fixed rollout horizon uniformly across states and tasks.

What Must a Latent World Preserve? Operating directly in a compact latent space avoids the cost of reconstructing future observations, but introduces a fundamental trade-off between abstraction and decision sufficiency. Representations that discard visually incidental details may also remove fine-grained dynamics that later become relevant for control, particularly in contact-rich interaction. Moreover, errors in latent transitions and reward prediction can accumulate during planning, making long-horizon rollouts increasingly unreliable. Our current deterministic latent dynamics do not explicitly represent such uncertainty, leaving the policy unable to distinguish a confident prediction from an ambiguous one. These limitations suggest that stronger latent world models may need to learn not only what is predictable, but what must be preserved for downstream decisions, together with uncertainty estimates that determine how far imagined experience should be trusted.

Scope and Scalability. Our current evidence is limited to a single VLM backbone and five simulated interactive visual environments, and therefore does not yet establish how the same design scales with model capacity, richer action spaces, or real-world interaction. In addition, although the latent world is represented within the VLM, its representation is grounded using a frozen pretrained visual encoder; whether equally useful predictive structure can emerge solely from interaction remains an open question. Latent imagination is also not computationally free: while DWELL substantially reduces end-to-end GPU-hours relative to a generative world model, simpler latent dynamics remain cheaper, revealing a performance–compute trade-off rather than an absolute efficiency advantage. Extending this framework to larger VLMs, transferable world representations, and persistent learning across tasks and environments would be important steps toward agents that continually refine a shared model of their experience.

Conclusion

We presented DWELL, a model-based RL framework that enables VLM agents to learn from experience in their own latent worlds. Rather than treating the world model as a separate simulator that reconstructs future observations, DWELL recasts the VLM as a recurrent latent transition operator and uses a shared representational interface for prediction, value estimation, and action. By interleaving real interaction with latent imagination, the agent can reuse limited environmental experience to improve its policy without generating future observations explicitly. Across diverse interactive visual environments, DWELL achieves strong overall performance under controlled interaction budgets while substantially reducing the computational cost of imagination relative to generative world models. Taken together, these results suggest that a useful world model need not reproduce the world itself; it needs to predict a future that the agent can directly act and learn upon.

Appendix

Algorithm

Algorithm 1 DWELL

Require: posterior Σϕ\Sigma_{\phi} and prior Ψϕ\Psi_{\phi}; reward model RωR_{\omega}; actor πθ\pi_{\theta}; critic VψV_{\psi}; planning horizon HH; world model updates NN; latent imaginations LL; PPO epochs EE

  1. Prefill D\mathcal{D} with a random policy

  2. Pretrain world model using (6)

  3. Warm up VψV_{\psi} using nn-step return Gt(n)G_{t}^{(n)} as in (9)

  4. while not converged do

  5. ▷ Online Interaction

  6. for time step t=1,…,Tt = 1,\ldots,T do

  7. Observe oto_t and infer posterior state Σϕ(zt−1,at−1,ν(ot))\Sigma_{\phi}(z_{t-1}, a_{t-1}, \nu(o_t))

  8. Execute πθ(at∣zt)\pi_{\theta}(a_t|z_t) in the environment, receive ot+1o_{t+1}, reward rt+1r_{t+1}, and termination dt+1d_{t+1}

  9. Tokenize the transition (ot,at,ot+1,rt+1,dt+1)(o_t, a_t, o_{t+1}, r_{t+1}, d_{t+1}) and append it to the replay buffer D\mathcal{D}

  10. end for

  11. ▷ World Model Learning

  12. for iteration n=1,2,…,Nn = 1,2,\ldots,N do

  13. Compute loss function as in (6)

JWM=Ldyn+αLrep+βLrew+ηLreg.\mathcal{J}_{\mathrm{WM}}=\mathcal{L}_{\mathrm{dyn}}+\alpha\mathcal{L}_{\mathrm{rep}}+\beta\mathcal{L}_{\mathrm{rew}}+\eta\mathcal{L}_{\mathrm{reg}}.
  1. Update world model (ϕ,ω)(\phi,\omega) with ∇ϕ,ωJWM(ϕ,ω)\nabla_{\phi,\omega}\mathcal{J}_{\mathrm{WM}}(\phi,\omega)

  2. end for

  3. ▷ Model-based Latent Planning

  4. Freeze (ϕ,ω)(\phi,\omega), snapshot θold←θ\theta_{\mathrm{old}}\leftarrow\theta, and initialize imagined batch B←∅\mathcal{B}\leftarrow\emptyset

  5. for imagination l=1,…,Ll = 1,\ldots,L do

  6. Initialize a batch of B\mathcal{B} posteriors z0z_0 from τ∼D\tau\sim\mathcal{D}, ∣τ∣>H|\tau|>H via t∼Unif⁡{0,…,∣τ∣−H−1}t\sim\operatorname{Unif}\{0,\ldots,|\tau|-H-1\} posterior steps

  7. Unroll the prior model for HH steps over the batch following (7) and store them in B\mathcal{B}

  8. end for

  9. ▷ Actor-Critic Learning

  10. Compute advantages A^h\widehat{A}_{h} and λ\lambda-returns G^hλ\widehat{G}_{h}^{\lambda} for B\mathcal{B} using (10)

  11. for epoch e=1,…,Ee = 1,\ldots,E do

  12. Shuffle B\mathcal{B} into minibatches

  13. for each minibatch do

  14. Jointly update actor and critic via ∇θ,ψJ(θ,ψ)\nabla_{\theta,\psi}\mathcal{J}(\theta,\psi) following (12)

  15. end for

  16. end for

  17. end while

Baseline

All trainable methods use the same Qwen2.5-VL-3B initialization, environment wrappers, action vocabulary, train/evaluation splits, real-interaction budget, and evaluation seeds. PPO and GRPO optimize the environment reward directly. For LEWORLDMODEL, we retain its latent-dynamics objective but adapt its predicted latent state to the same actor–critic interface used by DWELL through a learned state mapping. This mapping only reconciles representation shape and action indexing; it does not expose observations, rewards, or labels that are unavailable to the other methods. Before policy optimization, LEWORLDMODEL uses the same actor/critic initialization stages as DWELL: behavior cloning is applied only in environments for which it is listed in Table 5, and the critic is warmed up on the same real-replay return targets. Reward scales, action mappings, termination rules, PPO clipping and GAE conventions, update budgets, and evaluation intervals are matched environment by environment. The methods differ only in the learned world-model representation and the mechanism used to produce imagined transitions. Hyperparameters are selected without access to the evaluation levels, and no evaluation trajectory is used for training, replay, warm-up, or model selection. ADAWORLD is adapted under the same interaction, optimization, and evaluation budgets while retaining its native generative world-model objective.

SettingSokobanFrozenLakeNavigationEB-ALFRED
Backbone and latent state
VLM backboneQwen2.5-VL-3B-Instruct (frozen)
LoRA rank / α\alpha / dropout32 / 64 / 0.05
Latent slots KK / width DD36 / 2048
Visual teacher Evjepa2-vitg-fpc64-384-ssv2 (frozen)
World-model fitting
Offline episodes5,0005,0005,000300
Collection policymixedmixedmixedmixed
Steps / batch2,000 / 45,000 / 4
Learning rate10−510^{-5}10−410^{-4}
Truncated-BPTT window28
Open-loop horizon / weight–4 / 0.25
Prior / posterior / delta weight0.5 / 0.1 / 0.25
Regularizer LregL_{\mathrm{reg}} weight1
Feedback head
ArchitecturelinearlinearMLP (128)linear
Logged-action horizons HrH_r1–81–4
Batch / learning rate256 / 10−410^{-4}
Actor and critic initialization
Behavior-cloning steps00500
Critic learning rate3×10−53 \times10^{-5}10−410^{-4}
Critic updates (min / max)20 / 10050 / 200200 / 1,000
Imagined PPO
Imagination horizon HH415
Start states ×\times sequences32 ×\times 1632 ×\times 168 ×\times 6464 ×\times 1
Epochs / minibatch2 / 642 / 16
Actor learning rate2×10−62 \times10^{-6}2×10−62 \times10^{-6}10−610^{-6}2×10−62 \times10^{-6}
Critic learning rate3×10−53 \times10^{-5}10−410^{-4}
Clip ϵ\epsilon / target KL0.1 / 0.010.1 / 0.020.2 / 0.02
Entropy weight η\eta0.0050.01
Behavior-cloning weight00.1
Discount γ\gamma / GAE λ\lambda0.99 / 0.95
Acceptance: top-1 / pairwise / tolerance0.60 / 0.60 / 0.050.60 / 0.60 / 0.050.60 / 0.60 / 0.05
Online schedule
World model during PPOrefreshed every 50refreshed every 20frozen
Refresh batch / learning rate16 / 10−610^{-6}1 / 10−610^{-6}–
New real episodes1 per update4 per 4 updatesnone
Replay: offline / online cap800 / 800800 / 800–
Accepted actor updates400400200100
Evaluation interval1020
GPU / GPUs per runH20 / 8

Table 5. Key hyperparameters of DWELL.

Additional Ablation Study Results

Shared latent interface. A central design choice in DWELL is that the world model and policy operate on the same latent state interface: predicted states can therefore be consumed directly by the policy without an additional representation conversion. To isolate the effect of this design, we construct a bridged variant in which the world model retains its own latent representation and a lightweight learned adapter maps predicted states into a policy-specific latent space before action selection. All other components, including the VLM backbone, world-model objectives, policy optimization, imagination horizon, and online training schedule, are kept unchanged.

As shown in Table 2, separating the prediction and decision spaces consistently degrades performance across all three evaluated environments. The drop is most pronounced on SOKOBAN, while similar degradations are observed on FROZENLAKE and NAVIGATION. These results provide evidence that the benefit of DWELL does not arise solely from learning latent dynamics: making imagined states directly compatible with the policy’s decision representation also facilitates learning from imagined experience.

VariantSokobanFrozenLakeNavigation
DWELL (Shared)0.53±0.0120.53 \pm0.0120.51±0.0210.51 \pm0.0210.390.39
Bridged Latent Interface0.38±0.1380.38 \pm0.1380.42±0.0970.42 \pm0.0970.31±0.0170.31 \pm0.017

Table 2. Effect of the shared latent interface. We compare the default DWELL, where predicted latent states are consumed directly by the policy, with a bridged variant that maps world-model states into a separate policy latent space through a learned adapter. Results are success rates averaged over three training seeds.

Environment Settings

Table 3 reports one environment decision per step. A PrimitiveSkill language turn may contain at most two sequential primitive commands; we therefore report both turns and the corresponding maximum number of executed primitives. PrimitiveSkill comprises four tasks (Place, Stack, Drawer, and Align), with 16 held-out episodes per task.

EnvironmentObservationActionsHorizonEval. episodes
Sokobantop-down RGB, 6×66 \times6, one box4 moves25/25256
FrozenLaketop-down RGB, 4×44 \times44 moves25/25128
Navigationfirst-person RGB, instruction8 motions30/10120
PrimitiveSkillRGB, instruction3 parameterized skills3/3 turns64
EB-ALFREDfirst-person RGB, instruction162 skills30/3020

Table 3. Environment settings. Horizon is the maximum number of training / evaluation decisions. PrimitiveSkill observations also include task-state text; each episode has three language turns with at most two primitive commands per turn (at most six in total), and its 64 evaluation episodes cover four tasks (16 each).

Reward assignment

Rewards enter training in two ways. Model-free baselines optimize the real task reward directly. DWELL does not insert the real environment reward into imagined trajectories: real transitions supervise the feedback head, and imagined rewards are read from that head on predicted latent states. The calibrated threshold or floor is selected on a training-only calibration split and frozen before evaluation. For EB-ALFRED, the final ranked reward checkpoint gives

EnvironmentSuccessStepLabelRewardEpisode end
Sokoban+10+10−0.1-0.1binary1.11[p≥τ]−0.011.1\mathbf{1}[p \ge\tau] - 0.01p≥τp \ge\tau
FrozenLake+10+10003-class0.11[pgoal≥0.5]0.1\mathbf{1}[p_{\mathrm{goal}} \ge0.5]goal or hole
Navigation+10+1000binary1.1p−0.11.1p - 0.1cutoff at HH
PrimitiveSkill+1+10.1+Δs0.1 + \Delta sbinary, stage1.1p−0.11.1p - 0.1success or HH
EB-ALFRED+1+1−0.1-0.1binary1.1p~1[t=H]−0.11.1\tilde{p}\mathbf{1}[t = H] - 0.1cutoff at HH

Table 4. Reward assignment. pp is the predicted success probability, τ\tau a calibrated threshold, p~\tilde{p} a damped version of pp, and tt the imagination step. For PrimitiveSkill, the step reward is a 0.1 format reward plus Δs\Delta s, the increase in the highest completed task stage, and the feedback head also predicts the stage.

c=0.3357132077,p~=0.1p+0.9clip⁡(p−c1−c,0,1).c = 0.3357132077,\qquad\tilde{p} = 0.1p + 0.9\operatorname{clip}\left(\frac{p-c}{1-c},0,1\right).

For Sokoban, the calibrated termination threshold is τ=0.4968768060\tau= 0.4968768060 (reported as 0.497 in the main text).

Offline replay sampling. For every environment, the offline episode set is fixed before optimization, uses training-only seeds, and is disjoint from all evaluation levels and seeds. Episodes are sampled uniformly at random without replacement when the fixed offline subset is formed, and replay episodes and valid within-episode windows are sampled uniformly during training; no reward-, success-, or difficulty-prioritized sampling is used. Collection policy and replay sampling are distinct: Sokoban and FrozenLake use random collection, whereas Navigation uses goal-directed, noisy goal-directed, and random collection in a 3:2:1 ratio and EB-ALFRED uses scripted, noisy-scripted, and random collection in a 4:1:1 ratio.

PrimitiveSkill protocol. The recommended complete PrimitiveSkill setting is: 20,000 offline episodes (5,000 per task), random collection, 5,000 world-model steps with batch size 4 and learning rate 10−410^{-4}, truncated-BPTT window 3, open-loop horizon 3 with weight 0.25, λep/λvar=0.05/0.005\lambda_{\mathrm{ep}}/\lambda_{\mathrm{var}} = 0.05/0.005, an MLP(128) feedback head trained on horizons 1–3 with batch size 256 and learning rate 10−410^{-4}, no behavior-cloning warm-up, critic learning rate 10−410^{-4} with 40/1,000 minimum/maximum warm-up updates, sampled imagination with 64×164 \times1 start-state/sequence batches and horizon 3, two PPO epochs with minibatch size 16, actor learning rate 10−610^{-6}, clip/target-KL 0.2/0.02, entropy weight 0.01, discount/GAE 0.99/0.95, world-model refresh every 20 actor updates using batch size 1 and learning rate 10−610^{-6}, four new real episodes every four updates, replay caps 20,000/10,000, 200 accepted actor updates, evaluation every 20 updates, and one NVIDIA H20 (96 GB) per run.

Key Hyperparameters

Table 5 lists the settings that define DWELL’s training pipeline in each environment. Three protocol details matter for reading the results. (i) Imagined actions are sampled from the current policy throughout latent rollouts. (ii) In Sokoban and FrozenLake, the training-step axis counts accepted actor updates. A candidate update is rolled back (actor, critic, and optimizer state) if action-ranking accuracy on held-out real transitions falls below the acceptance threshold. (iii) EB-ALFRED initializes the actor by behavior cloning on successful scripted demonstrations and retains a behavior-cloning term during PPO.

Offline data for Navigation mixes goal-directed, noisy goal-directed, and random episodes; for EB-ALFRED, scripted, noisy scripted, and random episodes in the ratio 4:1:1.

World Model Learning

Regularization

To prevent representational collapse, we apply an auxiliary regularizer L◊L_{\Diamond} to all valid posterior representations. Following SIGReg-style regularization [7], we encourage random one-dimensional projections of the latent distribution to match a standard Gaussian using the Epps–Pulley statistic, together with a VICReg-style variance floor [8]. Specifically,

Lreg=λep1M∑m=1M∫e−s2∣ϕ^m(s)−e−s2/2∣2 ds+λvar1D∑d=1Dmax⁡(0,1−std⁡d),(8)\mathcal{L}_{\mathrm{reg}} = \lambda_{\mathrm{ep}} \frac{1}{M} \sum_{m=1}^{M} \int e^{-s^{2}} \left|\widehat{\phi}_{m}(s) - e^{-s^{2}/2}\right|^{2} \,\mathrm{d}s + \lambda_{\mathrm{var}} \frac{1}{D} \sum_{d=1}^{D} \max(0,1-\operatorname{std}_{d}), \tag*{(8)}

where ϕ^m\widehat{\phi}_{m} denotes the empirical characteristic function along the mm-th random projection, and std⁡d\operatorname{std}_{d} is the empirical standard deviation of the dd-th latent dimension over valid posterior slots. We use λep=0.05\lambda_{\mathrm{ep}} = 0.05 and λvar=0.005\lambda_{\mathrm{var}} = 0.005.

Reward and Termination Learning

Both the reward model and the terminal predictor are lightweight MLPs that take the latent state as input, using mean pooling to aggregate the latent slots. The output layer of the reward model depends on the type of task reward: for continuous rewards, we model the reward using a Gaussian distribution with constant variance, whereas for sparse rewards or rewards that can be represented as discrete events, we instead use a categorical distribution. The terminal predictor outputs a Bernoulli distribution. Once the termination is predicted, we define the continuation as ch=1−d^hc_h = 1 - \widehat{d}_h.

Actor-Critic Learning

Critic Warm-up

Consider a real trajectory segment τ={at−1,ot,rt,dt}t=1T\tau= \{a_{t-1}, o_t, r_t, d_t\}_{t=1}^{T}, we infer states z1:Tz_{1:T} from the posterior and compute nn-step return Gt(n)G_t^{(n)} as:

GT(n)=Vψ(zT),Gt(n)=rt+1+γ(1−dt+1)Gt+1(n),t=T−1,…,0,(9)\begin{aligned} G_T^{(n)} &= V_{\psi}(z_T),\\ G_t^{(n)} &= r_{t+1} + \gamma(1-d_{t+1})G_{t+1}^{(n)}, \qquad t = T-1,\ldots,0, \tag*{(9)} \end{aligned}

which are used as targets for critic warm-up to provide a good starting point for policy improvement.

Model-based Latent Planning

Although any actor-critic algorithm can be used for model-based planning, we adopt PPO (Schulman et al. [36]) to train both the actor and critic, as it allows the same batch to be reused multiple times, thereby improving sample efficiency.

Critic Learning Given a batch of imagined trajectories B\mathcal{B}, we use generalized advantage estimation (Schulman et al. [35]) to estimate both advantage and λ\lambda-return simultaneously, reducing variance in both policy gradient and TD targets for stable learning. Set A^H=0\widehat{A}_H = 0 and compute backwards over h=H−1,…,0h = H-1,\ldots,0:

δh=r^h+1+γch+1Vψ(z^h+1)−Vψ(z^h),A^h=δh+γλch+1A^h+1,Ghλ=A^h+Vψ(z^h).(10)\begin{aligned} \delta_h &= \widehat{r}_{h+1} + \gamma c_{h+1}V_{\psi}(\widehat{z}_{h+1}) - V_{\psi}(\widehat{z}_h),\\ \widehat{A}_h &= \delta_h + \gamma\lambda c_{h+1}\widehat{A}_{h+1},\\ G_h^{\lambda} &= \widehat{A}_h + V_{\psi}(\widehat{z}_h). \tag*{(10)} \end{aligned}

We then fit the critic to these targets to capture long-term rewards alongside the policy gradient.

Actor Learning We snapshot the actor πθ\pi_{\theta} as the sampling policy for PPO and denote it as πold\pi_{\mathrm{old}}, from which we can define the importance ratio:

ρh(θ)=πθ(ah∣z^h)πold(ah∣z^h).(11)\rho_h(\theta) = \frac{\pi_{\theta}(a_h \mid\widehat{z}_h)}{\pi_{\mathrm{old}}(a_h \mid\widehat{z}_h)}. \tag*{(11)}

In summary, we have the overall objective for policy improvement:

J(θ,ψ)=EB[−min⁡(ρh(θ)A^h,clip⁡(ρh(θ),1−ϵ,1+ϵ)A^h)+β(Vψ(z^h)−Ghλ)2−ηH(πθ(⋅∣z^h))].(12)\mathcal{J}(\theta,\psi) = \mathbb{E}_{\mathcal{B}}\left[-\min\left(\rho_h(\theta)\widehat{A}_h,\operatorname{clip}(\rho_h(\theta),1-\epsilon,1+\epsilon)\widehat{A}_h\right) + \beta\left(V_{\psi}(\widehat{z}_h)-G_h^{\lambda}\right)^2-\eta\mathcal{H}\left(\pi_{\theta}(\cdot\mid\widehat{z}_h)\right)\right]. \tag*{(12)}

Here, ϵ\epsilon is the clipping parameter, β\beta the value loss coefficient, and η\eta the entropy coefficient.

References

  1. [1]Anthropic. Introducing claude sonnet 4.5, 2025.
  2. [2]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15619–15629. IEEE, 2023.arxiv.org/abs/2301.08243
  3. [3]Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.
  4. [4]Karl Johan Åström. Optimal control of markov processes with incomplete state information i. Journal of mathematical analysis and applications, 10:174–205, 1965.
  5. [5]Marco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric, and Andrea Tirinzoni. Td-jepa: Latent-predictive representations for zero-shot reinforcement learning. In International Conference on Learning Representations, volume 2026, pp. 36211–36244, 2026.arxiv.org/abs/2510.00739
  6. [6]Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.
  7. [7]Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025.
  8. [8]Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.
  9. [9]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
  10. [10]Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, et al. Worldvla: Towards autoregressive action world model. arXiv preprint arXiv:2506.21539, 2025.arxiv.org/abs/2506.21539
  11. [11]Akshay L Chandra, Iman Nematollahi, Chenguang Huang, Tim Welschehold, Wolfram Burgard, and Abhinav Valada. Diwa: Diffusion policy adaptation with world models. arXiv preprint arXiv:2508.03645, 2025.arxiv.org/abs/2508.03645
  12. [12]Delong Chen, Mustafa Shukor, Theo Moutakanni, Willy Chung, Lei Yu, Tejaswi Kasarla, Allen Bolourchi, Yann LeCun, and Pascale Fung. Vl-jepa: Joint embedding predictive architecture for vision-language. In International Conference on Learning Representations, volume 2026, pp. 89791–89807, 2026.arxiv.org/abs/2512.10942
  13. [13]William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision-language models provide promptable representations for reinforcement learning. arXiv preprint arXiv:2402.02651, 2024.arxiv.org/abs/2402.02651
  14. [14]Hang Ding, Peidong Liu, Junqiao Wang, Ziwei Ji, Meng Cao, Rongzhao Zhang, Lynn Ai, Eric Yang, Tianyu Shi, and Lei Yu. Dynaweb: Model-based reinforcement learning of web agents. arXiv preprint arXiv:2601.22149, 2026.arxiv.org/abs/2601.22149
  15. [15]Tiangting Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. Webvolver: Enhancing web agent self-improvement with co-evolving world model. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 8970–8986, 2025.
  16. [16]Shenyuаn Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503.18938, 2025.arxiv.org/abs/2503.18938
  17. [17]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
  18. [18]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.arxiv.org/abs/1912.01603
  19. [19]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.arxiv.org/abs/2301.04104
  20. [20]Nick Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. In International Conference on Learning Representations, volume 2024, pp. 47376–47405, 2024.arxiv.org/abs/2310.16828
  21. [21]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.arxiv.org/abs/2203.15556
  22. [22]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
  23. [23]Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.arxiv.org/abs/2410.21276
  24. [24]Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1906.08253
  25. [25]Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi, Vitor Guizilini, Jiageng Mao, and Yue Wang. Dreamplan: Efficient reinforcement fine-tuning of vision-language planners via video world models. arXiv preprint arXiv:2603.16860, 2026.arxiv.org/abs/2603.16860
  26. [26]Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998.
  27. [27]Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.arxiv.org/abs/1712.05474
  28. [28]Haoxi Li, Qinglin Hou, Jianfei Ma, Jinxiang Lai, Tao Han, Sikai Bai, Jingcai Guo, Jie Zhang, and Song Guo. What you think is what you see: Driving exploration in vlm agents via visual-linguistic curiosity. arXiv preprint arXiv:2605.03782, 2026.arxiv.org/abs/2605.03782
  29. [29]Xiaokang Liu, Zechen Bai, Hai Ci, Kevin Yuchen Ma, and Mike Zheng Shou. World-vla-loop: Closed-loop learning of video world model and vla policy. arXiv preprint arXiv:2602.06508, 2026.arxiv.org/abs/2602.06508
  30. [30]Lucas Maes, Quentin Le Lidéc, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworld-model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026.arxiv.org/abs/2603.19312
  31. [31]Soroush Nasiriany, Huihan Liu, and Yuke Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In 2022 International Conference on Robotics and Automation (ICRA), pp. 7477–7484. IEEE, 2022.arxiv.org/abs/2110.03655
  32. [32]OpenAI. GPT-6 Astra System Card. September 2026. URL https://deploymentsafety.openai.com/gpt-6-astra/gpt-6-astra.pdf.
  33. [33]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.arxiv.org/abs/2203.02155
  34. [35]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.arxiv.org/abs/1506.02438
  35. [36]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  36. [37]Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025.
  37. [38]David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 1(11):6, 2025.
  38. [39]Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zhezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, and Zhibo Chen. Vla-jepa: Enhancing vision-language-action model with latent world model. In European Conference on Computer Vision, pp. 478–497. Springer, 2026.arxiv.org/abs/2602.10098
  39. [40]Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, et al. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2410.00425, 2024.arxiv.org/abs/2410.00425
  40. [41]Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y Charles, et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.
  41. [42]Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024.arxiv.org/abs/2407.17032
  42. [43]Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of llm scaling based on human-generated data. In Forty-first International Conference on Machine Learning, 2024.
  43. [44]Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Yiping Lu, Zhengyuan Yang, Lijuan Wang, et al. Vagen: Reinforcing world model reasoning for multi-turn vlm agents. Advances in Neural Information Processing Systems, 38:172871–172933, 2026.arxiv.org/abs/2510.16907
  44. [45]Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. In International Conference on Machine Learning, pp. 70576–70631. PMLR, 2025.arxiv.org/abs/2502.09560
  45. [46]Fangqi Zhu, Zhengyang Yan, Zicong Hong, Quanxin Shou, Xiao Ma, and Song Guo. Wmpo: World model-based policy optimization for vision-language-action models. In International Conference on Learning Representations, volume 2026, pp. 62486–62502, 2026.arxiv.org/abs/2511.09515
  46. [47]Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.arxiv.org/abs/1909.08593

Paper details

vlnc:W_zh54cxx7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents