WorksComputer Science
Learning Latent Goal Costs for Visual World-Model Planning with an Expanding Curriculum
Abstract
Image-goal planning with a frozen visual world model depends on how predicted observations are scored against the goal. Mean-squared error (MSE) between embeddings from a pretrained visual encoder can provide a useful signal near a goal, yet favor states across an obstacle that are visually similar to the goal and difficult to reach. We fit a small residual adapter on top of frozen I-JEPA embeddings to improve this latent goal cost. Starting near each goal, a planner collects trajectories from progressively wider regions of initial states. The adapter learns from the order of observations along successful trajectories, with additional comparisons inferred from failed rollouts. No task-specific progress labels are used for fitting, so the procedure can be applied to other goal-reaching tasks. DotWall and PointMaze are used in this work. In paired 50-case evaluations with identical starts, goals, world models, and planner settings, success increases from 54% to 98% in DotWall and from 82% to 96% in PointMaze. On 16 unseen DotWall layout-goal cases, the adapter fitted on one fixed layout improves agreement between latent-cost ordering and shortest legal-path ordering from 71.9% to 85.8%. The results indicate that planning experience can improve a frozen visual representation's goal cost without retraining the encoder or the world model.
Project website
Introduction
World models have long been useful for decision making and control because they let an agent predict the consequences of its actions before acting [1], [2]. Visual latent world models make these predictions in a learned latent space: they predict the next latent state from previous latent states and actions. Once the model has learned the environment’s dynamics, a planner can roll out candidate action sequences and score their predicted outcomes against a goal. In closed-loop model predictive control, the agent executes the first action of the selected plan, receives a new observation, and replans. Predicting in latent space avoids expensive pixel reconstructions and allows many action sequences to be evaluated efficiently.
Planning this way still presents two challenges. Errors accumulate when a world model is rolled out for many steps, limiting how far ahead it can predict reliably. Separately, the latent cost between a predicted state and the goal need not reflect how difficult it is to reach the goal from that state. This is particularly relevant when a world model uses a pretrained, task-agnostic visual encoder, as in DINO-WM, which uses DINOv2 to encode image frames [3], [4]. The resulting embeddings can capture visual similarity without encoding the travel required by a particular task. Consider DotWall in
We study this problem with a world model built on a pretrained I-JEPA encoder [6]. Pretrained visual encoders have useful spatial and geometric features that extend beyond one navigation task [6], [4]. We keep the I-JEPA encoder and world model fixed, and fit a small adapter to the encoder’s latent representations so that their distance to the goal guides planning better than the raw latent cost. In DotWall and PointMaze, we know what an ideal measure of progress looks like: geodesic distance decreases along a shortest legal route to the goal. We could supervise the adapter with that distance, but doing so would build our knowledge of these particular environments into the method. For other tasks, the right measure of progress may be difficult to define—for example, when a robot must arrange scattered blocks into a target shape. We therefore propose a task-agnostic training procedure that aims to discover a useful measure of progress from experience, rather than fitting an adapter to targets specified using our expert knowledge of each task.
Our approach builds on an observation: close to the goal, the raw latent cost often has enough structure to guide the planner successfully. We first collect trajectories from starting positions near the goal. A successful trajectory eventually reaches the goal, so the order of its observations provides a weak signal of progress: observations visited later tend to be closer to completing the task than those visited earlier. We fit the adapter so that later observations have lower adapted latent cost to the goal, while recognizing that this ordering need not hold at every step. We then collect trajectories from progressively farther starting positions and repeat, using each updated adapter to guide the next stage.
In matched 50-case planning evaluations, the adapted latent cost increases success from 54% to 98% in DotWall and from 82% to 96% in PointMaze. We also evaluate the DotWall adapter on 16 cases from unseen environment layouts, where the wall and doorway have moved from their positions during training. Its latent cost agrees more closely with geodesic distance on these layouts, improving legal-path pair ordering from 71.9% to 85.8%.
Our contributions are:
We show how raw I-JEPA latent cost can misguide a CEM-based planner and measure the resulting planning failures.
We propose a task-agnostic curriculum for fitting a small, task-specific latent adapter from collected trajectories, without supervision on expert targets.
We show that the adapter improves planning in two environments. Moreover, the adapted latent cost in DotWall generalizes to unseen environment layouts: it agrees closely with the geodesic distance that describes progress in this task, even when the wall and doorway change position.
Related Work
Early visual world models learned representations and dynamics to support control. World Models trained a visual autoencoder and a recurrent dynamics model, then optimized a small controller for the task reward [7]. PlaNET learned a latent dynamics model together with observation and reward models, and used the cross-entropy method (CEM) to plan online [1]. Dreamer took a different approach: it learned an actor and a value function from trajectories imagined by its world model, rather than searching over action sequences at every decision [2]. DreamerV2 extended this approach with discrete latent states, while DreamerV3 demonstrated its effectiveness across a much wider range of tasks [8], [9]. TD-MPC is closer to online planning: it combines short-horizon planning in a learned latent space with a value estimate for outcomes beyond that horizon [10]. These methods show that latent world models can support effective control. Their reward predictors, value functions, or policies are nevertheless tied to the task objective. Changing the goal does not necessarily require relearning the environment dynamics, but it may require adapting those goal-dependent components or the planning objective.
Image-goal planning offers another way to specify what the agent should reach. DINO-WM and V-JEPA 2-AC predict future visual representations and compare them with the representation of a goal image when evaluating actions [3], [11]. This makes it possible to change the goal image without training a new reward predictor for each goal. It also makes the distance in visual latent space an important part of the planner: a low latent cost is only helpful if it indicates progress that the agent can actually make.

Figure 1 (PDF p. 2). Rendered observations from the two environments. The red ball starts at its shown position and must reach the green goal. In DotWall, the legal route passes through a doorway; in PointMaze, it goes around an obstacle.
Other goal-reaching methods have learned such information from experience. Search on the Replay Buffer (SoRB) learns a goal-conditioned measure of reachability and uses it to find routes through previously visited observations [12]. Contrastive RL learns representations whose similarity corresponds to a goal-conditioned value function [13]. These works use different policies and planning procedures from ours, but they show why visual similarity alone need not be the right measure of distance to a goal. [5] explores the failure of latent planning in more detail.
Temporal Straightening addresses the latent planning cost more directly by training an encoder and predictor so that feasible trajectories become straighter in latent space [14]. We address the same difficulty while keeping the pretrained I-JEPA encoder and the learned world model fixed. Instead of training a new encoder or supervising an adapter with shortest-path distances, we fit a small adapter using comparisons obtained from collected trajectories. In DotWall and PointMaze, geodesic distance lets us evaluate the resulting latent cost, but it is not supplied as a training target.
World Model and Latent Planning
World-model training
Our world model follows a common pattern in recent visual world modeling: a visual encoder produces a latent representation of an observation, and an action-conditioned predictor estimates how that representation changes. Related architectures are used in DINO-WM and V-JEPA 2-AC, although their encoders and details might differ. [3], [11]. Details about the internal components of our world model can be found in A.1
Let be an image observation, the ball’s proprioceptive state, and an action. A frozen I-JEPA ViT-H/14 encoder splits each image into patches and produces one -dimensional latent embedding per patch. The transformer-based world model receives these embeddings , , and , then predicts the next latent and proprioceptive state, . To train it, we encode the observed next image with the same frozen encoder and compare the predicted embeddings with those targets. We also compare its predicted proprioceptive state with the observed next state. Figure 2 summarizes this training setup. Details about the construction of the datasets used for training the world model can be found in A.2.

Figure 2. World-model training. Frozen I-JEPA encodes the current and next images. The action-conditioned world model predicts the next visual embeddings and proprioceptive state, which are compared with the observed targets.
Planning toward an image goal
Once trained, the world model predicts the outcome of actions in latent space. The current observation and goal image are encoded by I-JEPA, with . At each decision, the cross-entropy method (CEM) searches over candidate action sequences. The world model rolls out each sequence in latent space, and the predicted final embeddings are scored against . The planner executes the first action of the selected sequence, observes the new state, and plans again. This is closed-loop model-predictive control.
The raw latent cost is the mean squared error (MSE) between corresponding spatial embeddings and their channels:
where and for I-JEPA. Thus every patch embedding contributes to one scalar latent cost. Figure 3 shows how the world model is rolled out repeatedly in imagination, while the goal image enters through the terminal latent cost used to select a plan. The encoder and world model remain fixed throughout our experiments.

Figure 3. Image-goal planning with the learned world model. CEM proposes action sequences, the model predicts their outcomes in latent space, and the MSE between a predicted terminal latent and the encoded goal scores each sequence. Only the first selected action is executed before replanning.
When raw latent cost misleads planning
We study two image-goal navigation environments, shown in Figure 1. DotWall has a ball, a separating wall, and a doorway; its standard layout has wall coordinate and doorway coordinate . PointMaze is a top-down U-shaped maze. In both tasks the ball must reach the goal position, but the legal route can initially move away from the goal in Euclidean distance or visual similarity.
Figure 1. The ball can appear close to the goal across the wall, even though it has to travel through the doorway to reach the other room. A planner using raw latent cost may therefore favor a state that looks promising, although in reality it is difficult to reach the goal from it [5].
The planning procedure depends on the latent cost giving useful guidance. To inspect that cost independently of world-model prediction errors, we render an observation with the ball at each legal position and compare its encoded representation with that of a fixed goal image. Each cell in Figure 4 shows the MSE between the latent representations of these two images; the star marks the goal. To examine whether the problem extends beyond one pretrained encoder, we compute the heatmaps with both I-JEPA ViT-H/14 and DINOv2 ViT-S/14. Their latent-cost scales differ, so we compare the ordering of positions within each map rather than color magnitudes across encoders.
Figure 4. Raw latent-cost heatmaps for one goal in each environment, compared with shortest legal-path distance. Darker colors indicate lower values; each panel has its own color scale. The costs are computed from rendered observations, not imagined world-model rollouts. Both encoders show departures from legal-path ordering, although their relative quality varies by goal.
The maps show that neither pretrained encoder consistently orders states according to the travel required to reach the goal. The raw latent cost is often informative close to the goal, but farther away it can favor a visually similar position across a wall or fail to reward movement toward the only legal passage. The two encoders differ in how strongly they exhibit this problem for individual goals; raw visual similarity does not reliably encode progress along a legal route.
This creates a difficulty for a search-based planner with a finite horizon. To reach the goal, an action sequence may first have to pass through a doorway or around an obstacle, often temporarily increasing the raw latent cost. If the planning horizon ends before the sequence reaches a state with lower cost, its necessary first actions may look worse than actions that make no useful progress. A longer horizon can expose the benefit of the detour, but it also enlarges the action-sequence search space and compounds world-model prediction errors. We therefore seek a latent cost that provides useful guidance over more of the state space, without requiring the planner to discover every detour through a long imagined rollout.
Method
Fitting an adapter for latent goal scoring
The useful visual features produced by the pretrained I-JEPA encoder enable our world model to accurately predict dynamics in latent space. The difficulty lies in how the planner scores those predictions. We therefore keep the encoder and world model frozen and fit a small adapter that transforms their latent representations before the planner compares a predicted terminal latent with the goal (Figure 5). The adapter is applied to both representations, and their mean-squared error (MSE) becomes the adapted latent cost used to score candidate action sequences.

Figure 5. The encoder and world model remain frozen. The adapter transforms predicted and goal embeddings only on the latent goal-cost path used by CEM.
In DotWall and PointMaze, an obvious solution would be to calculate geodesic distance to the goal and train the adapter to reproduce it. This relies on knowledge specific to these navigation tasks. For other tasks, an appropriate measure of progress may be difficult to specify by hand. We therefore neither fit the adapter on geodesic targets nor use geodesic improvement to select training examples. Instead, we aim to adapt the latent cost from experience without expert guidance about which individual states constitute progress. Geodesic distance serves only as an evaluation reference in these two environments.
We apply a shared residual multilayer perceptron independently to each spatial embedding. It has a 64-dimensional bottleneck and is initialized as the identity map. Its output replaces the raw embeddings only in the goal-cost computation:
The world model continues to receive unmodified encoder embeddings, and both and remain frozen. Identity initialization makes the initial planner equivalent to raw latent-MSE planning. The same adapter is applied to predicted candidate embeddings and the goal embeddings at evaluation time.
Collection phase
We observe that raw I-JEPA latent cost often provides a useful planning signal close to the goal and we want to take advantage of this property and gradually diffuse it further away. We begin with planning tasks whose starting positions lie in a goal-centered square region, which we call Band 1. Initially, the agent plans with raw latent MSE using the original planning architecture (Figure 3). During execution, we record the visited observations and whether the agent reaches the goal.
Successful trajectories provide the main training signal. An agent that reaches the goal has, over the course of that trajectory, made progress toward it. We therefore use the order of visited observations to form comparisons: an observation visited later should generally receive a lower adapted latent cost to the goal than one visited earlier on the same route. This is an imperfect assumption, because a successful route can include detours or loops. However in practice most successful trajectories went directly to the goal and our method performed well despite the noisy temporal comparisons introduced by the detours.
Failed trajectories can also provide useful training signal because some make partial progress. We rank sampled observations from a failed episode by their latent-cost proximity to successful routes for the same goal, and form additional pairs from that ordering. This approach is explained in more detail in the appendix A.3.
Fitting phase and expanding bands
We fit the adapter on pairs collected from successful and failed trajectories. Let and be observations that a pair ranks as worse and better, respectively, for goal . We penalize the adapter when does not have a lower adapted latent cost than by a margin :
where:
For successful trajectories, and were visited earlier and later; for failed trajectories, their order comes from successful-route matching.
Within each band, we alternate collection and fitting. The adapter remains frozen during collection. After each round, we add new pairs to cumulative replay and fit the adapter; its updated latent cost guides the next round. Once Band 1 is complete, we add starting positions farther from the goal in Band 2 and continue expanding (Figure 6) until the chosen start grid covers the state space.

Figure 6. The first three DotWall start bands around one goal. Each stage adds starts farther from the goal. Bands determine starting positions, not the region an episode must remain within.
Filtering collected experience
Temporal order does not always describe progress: an agent may leave a region and return to it before eventually succeeding. We remove the intervening loop when a trajectory returns near an earlier visited position. We then form successful-route comparisons only from consecutive recorded observations whose endpoints remain after filtering. We also discard unchanged observations. The bands constrain starting positions; observations need not remain inside the current band. These rules clean the trajectory data without checking whether a step reduces shortest legal-path distance. The method uses a goal-success signal and a way to generate starting states around a goal, but it does not use an oracle progress measure to label the collected pairs.
Experiments
We evaluate the adapted latent cost in DotWall and PointMaze. In both environments, we measure closed-loop planning success and compare it with raw I-JEPA latent based planning on matched start–goal cases, then inspect whether the latent cost orders legal positions in a way that helps the agent reach the goal.
Evaluation protocol
Each raw-versus-adapted planning comparison uses the same frozen I-JEPA encoder, world model, CEM settings, start, goal, and planner seed, with a limit of 100 simulator steps. The 50 cases in each environment use 10 goals held out from adapter fitting. Figure 9 gives the paired success rates; the fractions above its bars give the number of successful cases. The subset categories overlap, so their counts should not be added to the overall total.
Figure 9. Matched closed-loop planning success with raw and adapted I-JEPA latent costs. (a) DotWall; directly blocked cases are a subset of opposite-room cases. (b) PointMaze; opposite-arm cases are a subset of detours. Counts above bars are successes over evaluated cases.
To inspect the latent cost independently of world-model prediction error, we render an observation at each legal ball position and compare it with a fixed goal image. Figure 7 shows one goal per environment. The shortest legal-path distance is also shown for comparison. The columns use different numerical scales; their structure is what we are interested in rather than the exact magnitudes.

Figure 7. Latent-cost fields for one goal in each environment. Columns show shortest legal-path distance, raw I-JEPA latent MSE, and adapted latent MSE. Each position represents the cost of a rendered observation with the ball at that position relative to the goal image (star). DotWall goal was held out from adapter fitting; Each panel has an independent color scale.
DotWall
The adapter was fitted on the fixed DotWall layout with the wall at and doorway centered at . Collection used 12 training goals in both rooms. Legal starting positions were drawn from a two-unit lattice in eight goal-centered square bands; their half-widths increased from 6 to 48 units in steps of 6 (Figure 6). We used three collection–fitting rounds per band. Across all rounds, 6,121 rollouts produced 5,721 successful episodes. In the outermost band, collection succeeded from 104 of 116 starts, including 82 of 94 starts opposite the goal room. These are collection outcomes under successive adapter checkpoints, distinct from the matched test below.
On the 50 held-out planning cases, success rises from 27/50 (54%) with raw I-JEPA latent MSE to 49/50 (98%)
with the adapted cost (Figure 9a). Same-room success rises from 21/25 to 25/25, while opposite-room success rises from 6/25 to 24/25. When the straight path to the goal is blocked by the wall, success rises from 4/23 to 22/23. All 27 raw successes remain successes after adaptation; the adapter solves 22 additional cases.
The first row of Figure 7 explains the difficulty behind the opposite-room results. Raw I-JEPA MSE is low around the goal, but its field away from the goal does not consistently reward the route through the doorway. The adapted cost has a smoother decrease along legal approaches. In the matched example of Figure 8, raw-cost planning remains in the starting room until timeout, whereas adapted-cost planning crosses the doorway and reaches the goal using the same world model and CEM settings.
Figure 8. Matched DotWall example: raw-cost planning times out in the left room; adapted-cost planning crosses the doorway and reaches the goal.
PointMaze
We use the same collection and fitting rule in a different environment, the U-shaped PointMaze. Training used four goals and 632 collection episodes from legal starts across seven bands, with three collection–fitting rounds per band. The final adapter is the checkpoint after Band 7, Round 3; the encoder and PointMaze world model remain frozen.
On 50 matched cases using 10 goals held out from fitting, raw-cost planning succeeds in 41/50 (82%) and adapted-cost planning in 48/50 (96%) (Figure 9b). Both costs solve all 25 local cases. The gain is in detours around the central wall: success increases from 16/25 to 23/25, including an increase from 13/21 to 19/21 for the opposite-arm subset.
In the second row of Figure 7, the raw latent MSE is irregular along the corridor and weakly distinguishes positions that require a long detour from those near the goal. After adaptation, cost increases more consistently away from the goal along the legal route. Figure 10 shows one held-out detour: raw-cost planning remains in the lower arm until timeout; adapted-cost planning travels around the obstacle and reaches the upper-arm goal.
Figure 10. Matched PointMaze detour: raw-cost planning remains in the lower arm until timeout, while adapted-cost planning goes around the central wall and reaches the goal.
Generalization to unseen DotWall layouts
We next test whether the DotWall adapter trained with a fixed wall and doorway gives useful latent costs when those obstacles move. We freeze the adapter fitted only on the standard layout and evaluate four unseen layouts: (26, 22), (26, 38), (38, 22), and (38, 38). Four goals held out from adapter fitting are tested on each layout, yielding 16 layout–goal cases. For each case, we render 688 legal ball positions and compare raw and adapted latent costs with shortest legal-path distance.
Table 1 reports four aspects of this agreement. Map-wide pair ordering is the fraction of position pairs for which the position with shorter legal-path distance has lower latent cost, restricting pairs to a distance difference of at least 0.5 units. Opposite-room ordering applies the same test to positions across the wall from the goal. Spearman compares the rankings of latent cost and legal-path distance across all positions. Doorway-local uphill steps are steps near the doorway that shorten the legal path while increasing latent cost, a failure mode for a greedy cost signal. Higher values are better for the first three rows, while lower is better for the last. The first three rows average case-level scores; the final row pools doorway-local steps across cases.
| Measure | Raw I-JEPA | Adapted |
| Map-wide pair ordering | 71.9% | 85.8% |
| Opposite-room pair ordering | 48.5% | 77.1% |
| Mean Spearman | 0.594 | 0.864 |
| Doorway-local uphill steps | 39.2% | 15.6% |
Table 1. Latent-cost geometry on 16 unseen DotWall layout–goal cases.
The adapted cost improves map-wide pair ordering on all 16 unseen cases. Its mean ordering rises from 71.9% to 85.8%, while opposite-room ordering rises from 48.5% to 77.1% (Table I). Figure 11 shows two unseen layouts: the adapted fields retain a route-shaped ordering that broadly follows the legal-path reference despite the changed wall and doorway. This is a transfer test of the latent-cost field.

Figure 11. Two unseen DotWall layouts (rows), with shortest legal-path distance and adapted latent cost (columns). The adapter was fitted only on the standard (32, 30) layout. Each panel uses its own color scale.
Discussion and Limitations
The paired planning gains point to a concrete weakness of raw visual similarity as a goal objective in obstacle-rich settings. Fitting an adapter from gradually broader planning experience can improve the objective while retaining the pretrained encoder and world model. The unseen-layout diagnostic suggests that the learned adapter transfers its benefits to unseen layouts when the obstacle geometry changes.
The method still requires a goal-success signal and enough coverage in each band. Later bands require more starting positions to cover each region. In state spaces with more dimensions than the two ball coordinates, the starting positions required increase rapidly with respect to the number of dimensions. This curse of dimensionality may make the curriculum expensive or sparse. Temporal rankings are only weak labels: even successful routes can detour, and failure-route matching used to rank observations in failed trajectories relies on the current adapter.
Conclusion
Raw pretrained visual embeddings can produce a locally useful but globally misleading image-goal cost. We showed that a small adapter, fitted through expanding collection and temporal trajectory rankings, improves matched planning success in DotWall and PointMaze while keeping the encoder and world model frozen. The adapter also improves latent-cost ordering on unseen DotWall layouts, transferring its benefits when the map’s geometry changes. Our method uses no hand-crafted supervision targets such as geodesic distance which would tie it to the specific navigation tasks. Instead it uses success signal and collected experience which is cleaned up with minimal filtering. We leave the deployment of our method in higher dimensionality tasks for future work.
Appendix
World-model architecture and training
Figure 12 shows the world-model training architecture. A frozen, pretrained I-JEPA ViT-H/14 encoder maps each observation to a grid of latent embeddings, each with 1280 dimensions. The model also receives the ball’s proprioceptive state: position in DotWall, and position and velocity in PointMaze.

Figure 12. Detailed world-model training architecture. A frozen I-JEPA encoder maps the current and next observations to visual embeddings. The model combines the current embeddings and proprioceptive state with a macro-action comprising five successive 2D actions. An action-conditioned transformer predicts the next visual embeddings and proprioceptive state; both predictions are trained against their observed targets using MSE.
One world-model step corresponds to five successive two-dimensional primitive actions. We concatenate these actions into a single 10-dimensional macro-action. Separate MLPs embed the macro-action and proprioceptive state; their outputs are combined with each visual embedding and passed through a transformer with spatial and temporal position embeddings. A visual head predicts the next grid of embeddings, while a proprioceptive head predicts the state after the five actions have been executed.
For training, the frozen I-JEPA encoder supplies the target embeddings from the observed next frame. We minimize the MSE between predicted and target visual embeddings, together with a weighted MSE between the predicted and observed next proprioceptive states. The proprioceptive states are normalized for this loss. The training target is therefore the observed outcome of each macro-action; no goal distance or geodesic target is used to train the world model.
Environments and dataset generation
We study two image-goal navigation environments (Fig. 1). In DotWall, a ball moves between two rooms separated by a wall with a doorway. The world model used for the main planning experiments was trained on a fixed layout, with the wall at and the doorway at . In PointMaze, the ball must navigate a U-shaped corridor, sometimes travelling away from the goal before it can approach it along a legal route. DotWall provides the ball position as proprioception; PointMaze provides position and velocity .
We generated a separate dataset of 2,000 simulator episodes for each world model, split into 1,600 training, 200 validation, and 200 test episodes. DotWall episodes contain 50 primitive actions. Their actions follow a random policy with correlated headings and varying action lengths, producing motion and wall-contact examples without prescribing a route to a goal. PointMaze episodes contain 100 primitive actions sampled from the environment’s action bounds. In the PointMaze setting used here, each primitive action advances five physics steps. We record rendered RGB observations, actions, and proprioceptive states. The frozen I-JEPA encoder supplies the visual embeddings; five successive 2D actions are concatenated into each 10-dimensional world-model macro-action.
The adapter-fitting experience is collected separately, after world-model training. For DotWall, planning rollouts start in progressively wider bands around 12 goals on the fixed layout. PointMaze uses the same collection principle around four training goals. Successful and failed rollouts supply the ranking comparisons described in the main text; the held-out goals used for paired planning evaluation are excluded from adapter fitting.
Ranking observations from failed episodes
A failed episode need not be devoid of progress: it may approach a route that has previously reached the same goal before moving away. We sample up to five distinct observations along each failed episode and compare each with the successful routes stored for that goal. For a sampled observation and successful route , we find the observation on that route with the lowest latent cost to under the adapter frozen for the current collection round. We then average the three lowest such costs from different successful routes. Thus, a failed observation that is close in latent space to several successful routes receives a lower score and is ranked ahead of one that is far from those routes. Near-ties and goals with too few successful reference routes are excluded from this source of pairs.
Figure 13 illustrates the calculation with two sampled observations, and , on a single failed route. For each successful route, a dashed line reaches the nearest observation in latent cost, rather than necessarily the nearest one in physical distance. The three lowest route costs are highlighted for ; their average is 0.33, compared with 1.35 for . We therefore form the weak preference , meaning that should receive a lower adapted latent cost to the goal. The numbers in the schematic are illustrative, not experimental measurements. This rule uses successful experience as a reference without supplying geodesic progress labels to the adapter.

Figure 13. Schematic of the failed-episode signal in DotWall. Green routes – reach the goal; the red route fails. Dots are visited observations and circles mark two sampled failed observations. Dashed links represent latent comparisons with each successful route, not traversable paths. Highlighted blue links are the three lowest costs for . All displayed values are illustrative.
References
- [1]D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning latent dynamics for planning from pixels,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 97, 2019, pp. 2555–2565.arxiv.org/abs/1811.04551
- [2]D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” in International Conference on Learning Representations, 2020.arxiv.org/abs/1912.01603
- [3]G. Zhou, H. Pan, Y. LeCun, and L. Pinto, “DINO-WM: World models on pre-trained visual features enable zero-shot planning,” in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267, 2025, pp. 79 115–79 135.arxiv.org/abs/2411.04983
- [4]M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024.
- [5]A. Alrasheed, B. Azam, and N. Akhtar, “The planning limits of latent world models,” 2026. [Online]. Available: https://arxiv.org/abs/2609.39235
- [6]M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas, “Self-supervised learning from images with a joint-embedding predictive architecture,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15619–15629.arxiv.org/abs/2301.08243
- [7]D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” in Advances in Neural Information Processing Systems, vol. 31, 2018. [Online]. Available: https://proceedings.neurips.cc/paper/2018/hash/2de5d16662c3c35007e4e9292f1aba-Abstract.htmlarxiv.org/abs/1809.01999
- [8]D. Hafner, T. Lillicrap, J. Ba, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=0a6awyZbOUarxiv.org/abs/2010.02193
- [9]D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse control tasks through world models,” Nature, vol. 640, pp. 647–653, 2025.DOI
- [10]N. A. Hansen, H. Su, and X. Wang, “Temporal difference learning for model predictive control,” in Proceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 162, 2022, pp. 8387–8406. [Online]. Available: https://proceedings.mlr.press/v162/hansen22a.htmlarxiv.org/abs/2203.04955
- [11]M. Assran, A. Bardes, D. Fan et al., “V-JEPA 2: Self-supervised video models enable understanding, prediction and planning,” 2025. [Online]. Available: https://arxiv.org/abs/2506.09985
- [12]B. Eysenbach, R. Salakhutdinov, and S. Levine, “Search on the replay buffer: Bridging planning and reinforcement learning,” in Advances in Neural Information Processing Systems, vol. 32, 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/hash/5c48ff18e0a47baaf81d8b8ea51eec92-Abstract.htmlarxiv.org/abs/1906.05253
- [13]B. Eysenbach, T. Zhang, S. Levine, and R. Salakhutdinov, “Contrastive learning as goal-conditioned reinforcement learning,” in Advances in Neural Information Processing Systems, vol. 35, 2022. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/hash/e7663e974c4ee7a2b475a4775101ce1f-Abstract-Conference.htmlarxiv.org/abs/2206.07568
- [14]Y. Wang, O. Bounou, G. Zhou, R. Balestriero, T. G. J. Rudner, Y. LeCun, and M. Ren, “Temporal straightening for latent planning,” in Proceedings of the 43rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 306, 2026, pp. 125 168–125 196. [Online]. Available: https://proceedings.mlr.press/v306/wang26.htmlarxiv.org/abs/2603.12231



