WorksComputer Science
Knowing When to Stop Thinking: A Dissection of Expected Free Energy as a Test-Time Compute Controller for Latent World-Model Planning
Abstract
A planner using a learned world model can improve its predicted outcome while choosing a worse action. We study when expected free energy (EFE) can help decide how much computation to spend on a plan. An EFE-inspired objective combines goal distance with disagreement among stochastic predictions. We test candidate scoring, stopping, and online predictor updates separately. Stopping when elite cost stalls reduces refinement on tasks whose success plateaus with depth, with mean success close to fixed-depth references. At the larger tested weights, rewarding disagreement lowers success with a frozen model. Penalizing it helps at some task and weight settings but harms others. When training data omit a region needed to reach the goal, online updates recover part of the resulting loss. The tested cost-stall rule can fail when further optimization exploits model errors, because predicted cost keeps falling after task success peaks. Cost progress can guide computation on a plateau, but protecting performance after a peak requires information about the consequences of the chosen actions.
Introduction
A robot planning from images can improve its next action by repeatedly imagining and comparing possible futures. Each refinement step costs computation. Stopping too early may leave a useful action undiscovered. Continuing too long can favor an action that exploits errors in the learned dynamics. The planner must decide whether further optimization will improve the action it executes.
Test-time computation has improved iterative reasoning and model-based control. Recurrent language models refine internal representations before producing an answer ([5, 21, 11]). World-model agents predict future states to evaluate candidate actions ([8, 1]). A larger optimization budget gives the planner more opportunities to improve a plan. Its objective, however, remains a prediction of task performance. A fixed budget cannot respond to differences between instances, and convergence of the objective does not establish that the predicted consequences are correct.
Expected free energy (EFE) connects goal achievement and information seeking ([4, 17]). Its pragmatic component favors preferred outcomes, and its epistemic component values information. These components suggest ways to control planning. Goal progress may indicate when refinement has stalled. Prediction uncertainty may change which plan the agent selects. New observations may improve the model before the next decision. Disagreement-based exploration methods collect information during learning ([19]), whereas uncertainty penalties discourage plans that rely on unreliable predictions ([22]). Whether disagreement helps immediate task execution with a frozen predictor remains unclear. Internal progress may also fail to reveal when more planning becomes harmful.
We test these decisions with a compact latent world model and a sampling planner. In Figure 1, candidate trajectories provide goal cost and disagreement among stochastic predictions. We vary the disagreement weight and stopping statistic with the model frozen. We then test stopping in additional planner families and remove training data from a required region to study online learning. These comparisons separate the effects of reducing refinement, changing action selection, and updating the predictor.

Figure 1. Planning and measured effects of refinement. Top, the planning workflow refines candidate actions with a frozen world model until cost progress stalls or the budget is exhausted. Bottom, DINO-WM results from five seeds with 50 starts per seed. Success and model cost are plotted on separate axes. Each cost curve is divided by its first-iteration value within each seed before averaging. PushT success reaches a plateau. Wall success peaks at iteration 11 at 57.6% and falls to 35.6% at iteration 30 as model cost continues to decrease. All plotted curves come from recorded evaluations.
Cost-stall rules reduce refinement on tasks whose success plateaus with depth, with mean success close to fixed-depth references. A best-elite baseline is competitive and often stops earlier. On tasks whose success declines after a peak, falling cost sustains harmful refinement. Rewarding disagreement reduces frozen-model planning success, whereas penalties help at selected task and weight settings. Online updates improve success when the agent must enter a region missing from its training data.
Our experiments address the stopping decision and the uses of prediction uncertainty.
We separate candidate scoring, stopping, and online adaptation in a common world-model planning protocol.
We measure when cost-stall controllers reduce planning effort and when model exploitation defeats the tested stopping signals.
We measure the task-dependent effects of disagreement penalties and test online learning by removing training data from a required region.
Related Work
Active inference formulates perception and action through free-energy objectives [4, 17]. Deep implementations combine learned representations with approximate inference [20, 3, 14, 15]. The informationseeking term requires assumptions beyond free-energy minimization [16], and Bellman-optimality results hold under specified conditions [2]. We use this framework to motivate a cost combining goal distance and stochastic prediction disagreement. Disagreement provides an uncertainty proxy. It is not a calibrated estimate of information gain.
Learned latent dynamics support planning from images [7, 10, 1]. We build on LeWM, a compact joint-embedding predictive architecture that predicts future image representations [13]. Dreamer improves policies through imagined returns [8], whereas Plan2Explore uses disagreement to gather experience during learning [19]. Our frozen-model experiments test disagreement for immediate goal-directed execution. The adaptation experiment tests what changes when observed transitions update the predictor.
Adaptive computation allocates different amounts of work to different inputs [6]. Recurrent and recursive reasoning apply this idea to internal model computation [5, 21, 11]. We study stopping inside sampling-based model predictive control and compare statistics from the same optimization process. Pessimistic offline reinforcement learning addresses unreliable estimates through model uncertainty penalties in MOPO [22] and conservative value estimates in CQL [12]. We compare a disagreement penalty with a disagreement reward. The coverage experiment then tests a setting where the goal requires entering a region absent from training, so avoiding uncertain dynamics cannot solve the task.
Method
Figure 2 shows the planning loop. A fixed world model predicts candidate futures, the candidate cost determines which actions refine the proposal, and a stopping rule determines when to execute an action. The lower branch updates the predictor from observations in a separate experiment. We vary scoring, stopping, and adaptation separately.

Figure 2. Planning and online adaptation. The upper loop encodes images, samples actions, predicts and scores futures, selects elites, and refits the proposal. The stopping rule either continues refinement or executes the selected action. The lower branch updates the predictor using collected transitions, replay, and a parameter anchor. Shuffled-target updates and a frozen predictor provide controls.
Candidate Scoring
The planner compares possible actions before executing them. Let denote the current image, its encoded representation, and the encoded goal image. A latent predictor advances the state representation under a candidate action sequence . The cross-entropy method (CEM) [18] samples action sequences from a Gaussian distribution, retains the lowest-cost candidates as an elite set, and refits the distribution to that set. Each refinement iteration repeats this sample, predict, and refit procedure.
To expose both goal progress and predictive variation, we evaluate each candidate with stochastic predictor rollouts. The terminal prediction in rollout is and their mean is . Candidate scoring uses
where for , and for . The goal term sums squared terminal-coordinate errors. The disagreement term sums coordinatewise variance with divisor . Positive rewards disagreement and negative penalizes it. This sign change tests two uses of the same measurable signal without changing the predictor.
The objective uses goal distance and disagreement to study the pragmatic and epistemic roles of EFE. Disagreement measures stochastic prediction spread. It does not estimate posterior information gain. The weight experiment enables predictor dropout at every weight, including , so the comparisons share the same stochastic scoring procedure. The deterministic reference disables dropout and uses a single rollout. At it recovers terminal goal-distance scoring. We also evaluate magnitude-normalized weights because and can have different magnitudes.
Terminal spread is large when stochastic predictions disagree about where a candidate action sequence will end. A reward favors uncertain predicted outcomes, and a penalty favors more consistent predictions. Changing the sign leaves the goal representation and feasible action space unchanged.
It tests how disagreement affects action selection without assuming that greater spread means more useful information.
Stopping Rule
A fixed iteration budget spends the same maximum effort on every solve. We instead monitor how much the elite cost improves. Let denote the elite set at iteration , and let be its mean candidate cost. The controller monitors an exponential moving average of and computes relative descent
It stops at the first eligible iteration for which throughout a window of consecutive iterations. Eligibility requires both the minimum depth and a complete patience window. The floor in the denominator handles costs near zero. If no stall occurs, refinement ends at the cap .
The patience window prevents an isolated small improvement from ending the solve. Smoothing reduces sensitivity to changes in the sampled elite set, and the minimum depth gives the optimizer time to establish a candidate distribution. A cost increase also satisfies the stall inequality because relative descent becomes negative. The rule detects insufficient recent improvement. It does not prove convergence of the sampling distribution. We compare alternative cost statistics under the same patience and minimum-depth settings.
Write for the selected stopping iteration. After stopping, the agent executes the first action block, observes the next image, and plans again. Depth measures computation within a control decision. The primary controller uses with shared settings across the LeWM tasks, so its descent signal measures improvement in predicted goal distance. We compare it with best-elite cost stall, mean-change, variance-stall, and elite-set stability. Unsmoothed and disagreement-aware variants test changes to the monitored statistic. Patience longer than the available window recovers fixed-depth refinement.
Online Adaptation
Stopping and candidate scoring operate within the current model. Online adaptation changes that model. To test its value, we remove training episodes that enter a specified corner of TwoRoom and evaluate goals inside that region. This creates a coverage hole that goal-directed behavior must encounter. A volume-matched control removes the same amount of data at random. A separate checkpoint matches the hole model’s frozen success, distinguishing low initial performance from the location of missing training data.
During evaluation, the adapter updates the latent predictor with the original training objective on collected transitions. Replay and a parameter anchor constrain the update. A frozen baseline performs no updates. A sham baseline shuffles prediction targets and keeps the rest of the update procedure unchanged. Comparing online and sham updates tests the contribution of correct target pairing. All conditions follow the same sequential evaluation protocol.
Experimental Setup
The primary study uses LeWM on TwoRoom navigation, PushT manipulation, OGBench-Cube manipulation, and DMC-Reacher control. We use Cube and Reacher as short names below. TwoRoom, PushT, and Cube use expert-data world models. Reacher uses random-policy training data. Checkpoint comparisons vary training progress within an environment. The coverage experiment changes which transitions are available during training. Table 1 gives the primary configuration.
| Quantity | Setting | Quantity | Setting |
| Model parameters | 18M | Image resolution | |
| Latent dimension | 192 | CEM samples / elites | 300 / 30 |
| Horizon / action block | 5 / 5 | Refinement cap | 30 |
| Stall threshold | Patience / minimum depth | 2 / 3 | |
| EMA coefficient | 0.5 | Episodes per seed | 50 |
| Initial controller seeds | 3 to 6 | Dose-response seeds per cell | 100 |
Table 1. Primary LeWM configuration. The same cost-stall settings are used across the four tasks. Dose-response and transfer experiments have their own stated seed counts and budgets. The normalized-weight grid uses one seed per setting.
Fixed-depth CEM measures success as refinement increases. The primary stopping rule and its alternatives share the planner configuration. We select the fixed reference from the evaluated depth grid, which means it is not independently tuned. The disagreement experiment varies with the model and rollout procedure fixed. TD-MPC2 [9] and DINO-WM [23] test stopping with a common threshold within each family. TD-MPC2 samples actions using learned latent values. DINO-WM plans with dynamics on fixed visual features.
Evaluations sample start and goal pairs from the available datasets. These samples may overlap with model-training data. Matched seed comparisons reuse start and goal sets, although predictor-dropout randomness is not fully paired. We report mean performance and the seed counts for each comparison. Controller comparisons with unequal seed sets are descriptive. Appendix A gives the sampling and pairing procedures.
Success measures goal completion on LeWM and DINO-WM tasks. TD-MPC2 uses return relative to each task’s best fixed-depth reference. Mean depth measures the refinement iterations used. With candidate count, horizon, and stochastic rollout count fixed, imagined transition evaluations decrease in proportion to depth. Iteration savings do not measure end-to-end runtime. Ensemble-size comparisons change the work per iteration, so we report them separately from the stopping comparison.
For adaptation, we report the fraction of the coverage deficit recovered. The deficit is the frozen-success gap between the volume-matched reference and the hole model. We divide the online improvement by that gap under the same region-goal evaluation. The success-matched checkpoint tests the effect of initial performance and is not the denominator. The recovered fraction can exceed one or become negative because it is measured relative to the chosen reference.
Results
Stopping Performance
Cost-stall rules reduce refinement on tasks where additional depth has little effect on success. Table 2 compares the primary rule with the best fixed-depth reference and best-elite stopping. On TwoRoom, PushT, and Reacher, the primary rule uses less depth than the cap and reaches mean success close to the fixed references. Many solves improve rapidly before stalling. Cases that continue to improve run toward the budget limit.
| Environment | Best fixed success @ depth | Primary success @ depth | Best-elite success @ depth |
| TwoRoom | 90.0 @ 30 | 88.0 @ 22.0 | 89.3 @ 9.3 |
| PushT | 88.4 @ 30 | 87.2 @ 28.3 | 87.3 @ 15.3 |
| Cube | 67.3 @ 10 | 63.3 @ 28.1 | 63.3 @ 15.8 |
| Reacher | 72.0 @ 30 | 74.7 @ 22.3 | 74.0 @ 9.3 |
Table 2. Controller success (%, mean across seeds) and mean depth. The primary rule saves 6 to 27% of iterations on TwoRoom, PushT, and Reacher relative to the cap. Each seed contains 50 episodes. Seed counts for fixed, primary, and best-elite are (3, 5, 3) on TwoRoom, (5, 5, 3) on PushT, and (3, 3, 3) on Cube and Reacher. Fixed depth is selected from the evaluated grid. The seed sets differ between methods.
Best-elite cost stall often uses fewer iterations than the primary elite-mean rule, with similar mean success. Removing smoothing or relaxing the threshold also reduces depth. Variance stall is unstable on PushT, and elite-set stability changes abruptly with its threshold. Mean-cost descent is therefore one useful stopping statistic among several. Appendix § 14 gives the complete comparisons and sensitivity curves.
Instances whose cost stalls quickly stop early. Continued cost improvement uses more of the budget. On PushT, higher initial elite cost is associated with greater allocated depth in the displayed traces. A shared rule can therefore allocate different amounts of computation without a difficulty classifier. This allocation still depends on the model cost, including when further cost reduction harms task success.
Disagreement Weights
Rewarding disagreement lowers frozen-model success at the larger tested weights in every environment. Table 3 summarizes the reductions and selected penalty gains. A disagreement reward changes the actions selected, but new observations cannot improve a predictor whose weights remain fixed. This may explain the loss in these short-horizon tasks.
| Setting | Environment | Success change (pp) | |
| Reward | All four | +10 | −3.5 to −54.7 |
| Reward | All four | +50 | −13.0 to −78.1 |
| Penalty | Reacher | −20, −50 | +2.38, +3.34 |
| Penalty | PushT | −3 | +1.42 |
| Penalty | Cube | −20, −10 | +1.44, +1.88 |
| Strong penalty | TwoRoom | −20, −50 | −4.7, −13.4 |
| Strong penalty | PushT | −50 | −4.5 |
Table 3. Success changes from the baseline. Each setting uses 100 matched seeds and 50 episodes per seed with stochastic rollouts. Differences are percentage points (pp). The reward rows give the range across tasks, and the penalty rows show selected positive and negative changes. Table 16 gives all weight comparisons.
Penalizing disagreement gives small gains at selected weights on Reacher, PushT, and Cube. Stronger penalties harm TwoRoom and PushT. The effect depends on both the weight and the task.
This cross-task comparison changes the environment and training data together, so it cannot isolate training coverage as the cause.
Checkpoint comparisons test whether weaker prediction alone explains the penalty benefit. Large changes in one-step prediction error on TwoRoom do not yield a consistent penalty advantage. Reacher shows gains at several checkpoints and reversals at others. The disagreement-aware stopping variant also changes across checkpoints. Scalar prediction error alone does not explain these patterns. Training time and model state remain coupled in this comparison. Appendix § 9 gives the checkpoint results.
Learning Missing Dynamics
Online predictor updates recover part of the deficit caused by a coverage hole. Table 4 compares frozen, online, and shuffled-target updates for goals inside the removed region. Online learning improves both hole checkpoints and outperforms shuffled-target updates. This advantage suggests that correct target pairing contributes to the gain. It does not recur in the success-matched and full-data controls.
| Update | Hole e3 | Hole e5 | Matched | Full e2 | Released |
| Seeds | 5 | 5 | 5 | 5 | 3 |
| Frozen | 37.2 | 44.0 | 37.2 | 70.8 | 75.3 |
| Online | 56.0 | 62.8 | 34.8 | 76.4 | 74.0 |
| Sham | 46.4 | 55.6 | 38.0 | 80.8 | n/a |
| Online − sham (pp) | +9.6 | +7.2 | −3.2 | −4.4 | n/a |
| Deficit recovered | 37.6% | 48.0% | n/a | n/a | n/a |
Table 4. Coverage intervention on a second independently trained sample. Success is the mean percentage over 50 episodes per seed. Checkpoints e2, e3, and e5 denote training epochs. The matched control is a random-drop e2 checkpoint matched to hole e3 on frozen success. Deficit recovery uses volume-matched e3/e5 references with success 87.2%/83.2%. The update conditions in each column share a seed set.
Random removal preserves access to the region and matches training-data volume. The success-matched checkpoint tests whether poor initial performance alone explains the benefit of adaptation. Together, these controls support learning dynamics needed by the evaluated goals. Shuffled-target updates are an imperfect control for target pairing because shuffling can also change gradient magnitudes and preserve information about visited states.
Lower prediction loss on collected transitions does not consistently improve task success. Reacher adaptation lacks a replicated benefit across its broad-coverage checkpoints. The frozen disagreement penalty also fails to repair the TwoRoom coverage hole consistently across training samples. Action selection under an existing model and learning missing dynamics therefore need separate evaluation. Appendix § 13 gives the adaptation protocol and replications.
Model Exploitation
Cost-stall rules can continue refining after task success has peaked. Cube reaches its best fixed-depth result before the cap, but the primary controller continues and has lower success. On DINO-WM Wall, success also peaks before the full budget as predicted cost continues to fall. Cost-only halting runs close to the cap and misses the earlier peak (Table 5).
| Planner setting | Success | Mean depth |
| Fixed , curve peak | 0.576 | 11 |
| Fixed , full budget | 0.356 | 30 |
| Cost-only halt, | 0.364 | 29.04 |
| Fixed | 0.508 | 30 |
Table 5. DINO-WM Wall with a cap of 30 iterations. Success is the mean over five seeds with 50 starts per seed. The peak is selected from the common-depth mean curve. The fixed-depth setting uses more computation per iteration.
The tested disagreement guards do not reliably distinguish harmful optimization from productive refinement. Tighter guards sometimes improve Cube but stop prematurely on plateau-shaped tasks. Appendix § 14.§ 1 reports the guard and detector comparisons. A fixed ensemble improves Wall success by changing candidate scoring and computation, without choosing an earlier stopping point.
Extra iterations on a plateau waste computation. After a success peak, they can also worsen the chosen action. Appendix § 12 formalizes the difference using an idealized success curve and constructs identical cost traces with incompatible optimal depths. A stopping guarantee needs a link between internal progress and external success.
Transfer Across Planners
The stall statistic also reduces refinement outside the primary CEM planner. TD-MPC2 retains most of the best-grid return across the evaluated DMControl tasks at a small mean depth. DINO-WM PushT reaches the same observed mean success as its fixed-cap reference with fewer iterations (Table 6). The threshold is shared within each family. Threshold selection was not evaluated on a separate held-out set.
| Family and task | Performance | Mean depth |
| TD-MPC2, 19 tasks | 98.9% mean retained return | 3.8 |
| TD-MPC2, lowest retention | 96.1% retained return | n/a |
| DINO-WM PushT, fixed | 0.852 success | 30 |
| DINO-WM PushT, halt | 0.852 success | 26.44 |
Table 6. Cross-family stopping results. TD-MPC2 averages tasks equally and normalizes return by each task’s best fixed-depth mean on the same evaluation grid. DINO-WM uses five seeds and cap 30. Detailed task results and configurations appear in Table 8 and Table 9.
Table 8: TD-MPC2 evaluation on nineteen DMControl tasks. Each task uses one released checkpoint and four seeds , with 30 episodes per seed. Returns are means across seeds. The fixed reference maximizes mean return over on the evaluation seeds. Halting uses . Its cap is 8 on dog-walk, humanoid-run, and humanoid-walk, and 6 on the remaining tasks. Relative return is 100 × halt mean/fixed mean, so values above 100% indicate greater return than the fixed reference. Depth is averaged over seeds and summary rows weight tasks equally. Calibration evaluations are excluded. The fixed reference is selected retrospectively.
| Task | Fixed return | Halt return | Relative return | Depth | |
| acrobot-swingup | 6 | 661.2 | 661.8 | 100.1% | 3.1 |
| cheetah-run | 9 | 861.1 | 828.1 | 96.2% | 3.5 |
| cup-catch | 9 | 981.1 | 979.1 | 99.8% | 3.0 |
| dog-walk | 9 | 799.2 | 783.8 | 98.1% | 5.3 |
| finger-spin | 6 | 988.3 | 985.7 | 99.7% | 3.8 |
| finger-turn-hard | 3 | 972.7 | 967.0 | 99.4% | 3.0 |
| fish-swim | 12 | 787.5 | 775.7 | 98.5% | 3.2 |
| hopper-hop | 12 | 570.1 | 568.0 | 99.6% | 5.9 |
| hopper-stand | 6 | 961.3 | 952.9 | 99.1% | 3.0 |
| humanoid-run | 12 | 680.9 | 659.6 | 96.9% | 7.4 |
| humanoid-walk | 12 | 926.5 | 890.4 | 96.1% | 5.3 |
| pendulum-swingup | 12 | 846.1 | 846.0 | 100.0% | 3.0 |
| quadruped-run | 6 | 957.8 | 953.3 | 99.5% | 3.7 |
| quadruped-walk | 9 | 979.8 | 976.0 | 99.6% | 3.1 |
| reacher-easy | 3 | 977.9 | 978.7 | 100.1% | 3.0 |
| reacher-hard | 6 | 981.9 | 981.8 | 100.0% | 3.0 |
| walker-run | 9 | 891.2 | 867.9 | 97.4% | 3.6 |
| walker-stand | 6 | 994.8 | 994.3 | 99.9% | 3.0 |
| walker-walk | 9 | 985.3 | 984.1 | 99.9% | 3.0 |
| Mean over 19 tasks | 98.9% | 3.8 | |||
| Worst retained return | 96.1% | . |
Table 8.
| Configuration | PushT success @ depth | Wall success @ depth |
| Fixed | 0.852 @ 30 | 0.356 @ 30 |
| Halt | 0.852 @ 26.44 | 0.364 @ 29.04 |
| Halt | 0.852 @ 28.71 | 0.356 @ 29.79 |
| Fixed | · | 0.508 @ 30 |
Table 9. DINO-WM evaluation with five seeds (1 to 5), 50 starts per seed, and cap 30. Success is the mean across seeds. Depth counts the iterations used by each trajectory. The configuration uses fixed-depth ensemble scoring.
DINO-WM contains both PushT, where stopping saves computation, and Wall, where further optimization exploits model errors. Planner identity alone does not predict whether cost-stall stopping will help. The comparison requires measuring task success across depths as well as the computation saved.
Ablation Studies
Stopping earlier reduces refinement with the candidate objective unchanged. Increasing the stochastic rollout count changes both candidate scoring and computation per iteration. On the evaluated LeWM tasks, averaging multiple imagined futures is not necessary for the observed plateau performance. On Wall, a fixed ensemble changes success substantially. Depth comparisons therefore need to account for the rollout count.
Goal distance and terminal prediction variance have different magnitudes. We normalize these magnitudes within the candidate batch before applying a dimensionless weight. Larger positive weights still harm performance. This result suggests that the reward failure is not confined to the original raw weighting. The normalized grid uses fewer seeds than the uniform weight experiment, so the result remains exploratory. Appendix I gives the normalization rule, seed count, and complete values.
For a value-based planner, a large common offset can dominate predicted value magnitudes without changing candidate rankings. Normalizing disagreement by that magnitude can overwhelm the differences used to select actions. Candidate-value spread measures the ranking scale more directly. The partial TD-MPC2 weight grid shows this problem. Its shared-threshold stopping experiment instead measures relative progress in the original objective.
Under the same coverage-hole condition, we compare fixed and adaptive allocation with frozen and online predictors. Online updates improve success under both allocation rules, and adaptive stopping lowers success with either predictor. The mean interaction is small, but these evaluations do not establish independent effects of stopping and learning. Table 10 gives the factorial results.
| Env. | Best | EFE | Coverage | |
| TwoRoom | 90.0@30 | 88.0 | +2.0 | · |
| PushT | 88.4@30 | 87.2 | +1.2 | · |
| Reacher | 72.0@30 | 74.7 | −2.7 | · |
| Cube | 67.3@10 | 63.3 | +4.0 | · |
| hole (e3) | 50.0 → 31.2 (TTT +18.8, 38%) | 31.2 | ||
| hole (e5) | 39.2 → 20.4 (TTT +18.8, 48%) | 20.4 | ||
| Fixed | Adaptive | |||
| Frozen | 37.2 | 33.2 | ||
| Online (TTT) | 55.2 | 51.6 | ||
| TTT effect | +18.2 | |||
| Adaptive effect | −3.8 | |||
| Interaction | +0.4 |
Table 10. Stopping and online adaptation. Above, success relative to fixed-depth and coverage references. Below, the TwoRoom hole-e3 factorial uses seeds and 50 episodes per seed. Cells report mean success. The mean interaction is +0.4 pp, with within-seed values pp. The factorial and Table 4 are separate evaluations of the same labeled condition.
Success Across Depths
A stopping rule observes optimization progress to decide how long to refine an action. Its purpose is to preserve task performance. Consider an idealized episode that uses a single depth at every control step. Let be its success probability at depth for instance , and let be the earliest depth with maximal success. The relevant error is the difference between success at that optimum and success at the selected depth. Final planning cost measures a different quantity.
Suppose success is nondecreasing before and nonincreasing afterward. Early stopping loses progress available before the peak. Late stopping loses any success that declines after it. On a plateau, the latter loss is zero even when the planner spends unnecessary iterations. On a declining curve, cost can keep improving as success falls. A conservative stopping rule can therefore save computation on a plateau and still fail to protect performance after a peak.
Appendix § 12 proves this separation and constructs instances with identical cost traces but different optimal depths. A rule that reads only those traces must make the same decision on both instances, so at least one incurs a success loss. The statement concerns an idealized curve with a single depth throughout an episode. Actual receding-horizon execution selects depth repeatedly. Stopping in that setting may require observations that connect model predictions with their realized consequences.
Conclusion
Cost stalling reduces refinement when success plateaus with depth. Disagreement penalties help at selected task and weight settings, and online predictor updates recover part of a deficit caused by missing training coverage. These gains come from different changes to planning. When continued optimization exploits model errors, falling cost no longer indicates a better action. Stopping safely after a success peak requires information beyond the cost trajectory.
Evaluation Details
Pairing and Compute
Each paired comparison reuses the starts and goals sampled by an evaluation seed. If and are success rates for seed under configurations and , the mean difference is
Each experiment retains its seed count and evaluation protocol. Repeating episodes under a fixed seed does not reproduce variation across independently sampled seeds. Repeated evaluations also leave training variation unchanged.
Depth has a similarly specific interpretation. If each refinement iteration evaluates action candidates with stochastic rollouts over horizon , a simple count of imagined transition evaluations is proportional to
The second expression is the iteration saving relative to a fixed cap with the same per-iteration configuration. It is not an end-to-end runtime measurement. Encoding, action execution, batched stopping, and any uncertainty or online-update overhead contribute additional cost. Increasing the rollout count can consume more computation even if the chosen depth falls. Our principal compute comparison therefore reports used iterations alongside success and keeps ensemble-size changes in the ablations.
Regret and Coverage
The following success curves describe the success-rate tasks. Let denote a sampled start and goal pair, and let denote planner randomness. Fixed-depth episode curves use the same depth at every control step. In the idealization, selects one depth for the episode. Let denote success probability at integer depth . Define as an allowed depth that maximizes success. The adaptive stopping regret is
This uniform-depth oracle does not measure our controller’s sequence of per-step decisions. Identifying curve-adaptive success with observed policy success requires conditional independence of selected depth and outcomes given the instance. The common-fixed-depth population gap lower-bounds the curve-based oracle gap only when both use the same adaptive term. Selecting the fixed-depth reference after evaluation can bias the measured gap upward. TD-MPC2 returns use a different scale and are excluded from this formalism.
A separate descriptive gap concerns training coverage (§ 5.3, including the distinct matched controls). Let be success of the volume-matched frozen reference, success of the hole model, and success after adaptation under the same region-goal protocol. Then
The ratio depends on the adaptation budget and the chosen reference. It may be negative or exceed unity, and becomes unstable when the reference gap is close to zero. It describes the recovered success gap without bounding other sources of model error. The factorial and coverage comparisons use separate online evaluations under the same condition label. We retain the seed pairing within each comparison.
Appendix § 12 constructs identical observable cost traces with incompatible optimal depths, giving positive worst-case regret for deterministic trace-only rules and a separate randomized bound. Exact equality is a construction assumption, not established for Cube or Wall. The tested detectors also have limited success. Their results leave open whether other stopping signals can identify model exploitation.

Figure 3. Fixed-depth success and adaptive halting plotted against mean refinement depth. Figure 2 shows the planning procedure.
Post-peak Loss
The success curve supplies a precise distinction between harmless extra computation and damaging over-optimization. Under the appendix’s weakly unimodal shape assumption (nondecreasing before the earliest peak and nonincreasing afterward), let be the earliest maximizing depth. Define the pre-peak shortfall and post-peak decline by
Both are nonnegative, but they describe different errors. An early stop can forfeit progress before the optimum. A late stop can hurt only to the extent that success declines afterward. Monotonicity on each side of the earliest peak gives the pointwise inequality
Taking expectations gives Appendix F’s early/late bound. A conservative rule can waste iterations on a plateau yet preserve success. The same delay can incur loss on a declining curve. This explanation assumes a success curve for each instance. The measured aggregate curves do not determine the episode regret of a controller that changes depth at every step.
Scope and limitations
The normalized-weight and detector comparisons include small-seed and incomplete grids. They describe the tested configurations. They do not establish a calibrated uncertainty measure or replicate every contrast across planner families.
The experiments use near-fully-observed tasks and short receding-horizon control with a moderate-scale world model. In partially observed tasks, information can improve the belief state without updating predictor parameters. That benefit remains untested here. Online adaptation uses a particular update procedure and coverage manipulation. Behavioral evaluation samples may overlap with training data. The sham control preserves anchor supervision but can change gradient norms. We share thresholds within each planner family without a separate held-out selection protocol. These choices limit transfer to other data distributions, update rules, and planning horizons.
Supplementary Results
These figures supplement the analyses of compute allocation, disagreement weighting, and over-optimization. Captions specify the evaluation protocol for each experiment.
Checkpoint Comparisons
Cross-environment contrasts conflate training data with task differences. We therefore compare checkpoints within a training trajectory for TwoRoom and Reacher. The environment and planner
Figure 4. Elite-cost trajectories and allocated depth. Left, example trajectories show rapid early descent and a slower-descending case that receives more iterations. Red markers indicate halting. Right, initial elite cost correlates with depth on PushT (). Initial cost is a model-based difficulty proxy. The correlation does not use an independently measured difficulty label.

Figure 5. Disagreement-weight sweep with 100 seeds per cell and 50 episodes per seed. (a) Mean success against on a symlog axis. Larger positive weights reduce success in every environment. Some negative weights give small gains on Reacher, PushT, and Cube. Strong penalties harm TwoRoom and PushT. (b) Mean paired differences from (Table 16). The gains depend on the environment and weight. Coverage is not independently varied.

Figure 6. DINO-WM depth curves with five seeds and a cap of 30. Wall’s mean success peaks at 0.576 at iteration 11. It falls to 0.356 at iteration 30, a 38.2% relative decline. The peak is selected from the mean curve at a common depth across seeds. Cost-only halting with attains 0.364 success at mean depth 29.04. The fixed-depth ensemble attains 0.508. The ensemble uses more stochastic rollouts at a fixed depth.

Figure 8. Candidate preference and compute allocation. Positive favors terminal rollout spread and negative penalizes it. Highlighted bundles show candidate selection. Penalties can help or hurt task success. Cost-only halting () controls refinement depth. Disagreement is an uncalibrated surrogate. Schematic.
protocol remain fixed. Each checkpoint has a held-out one-step prediction error and evaluations of fixed-depth and adaptive planning (Table 7). Training time and model state vary together. Coverage is not independently manipulated.
Table 7: Within-environment checkpoint comparison. Per-epoch LeWM checkpoints, 50 episodes per cell. Probe MSE, / success, and adaptive halting. See Figure 7 for visualization.
| Env | Ckpt | Probe MSE | adaCost | adaEFE | ||
| TwoRoom | e1 | 2.161 | 48.0 | 48.8 | 45.3 | 42.7 |
| TwoRoom | e2 | 0.232 | 88.0 | 88.0 | 87.3 | 89.3 |
| TwoRoom | e3 | 0.157 | 90.0 | 88.7 | 90.7 | 92.0 |
| TwoRoom | e5 | 0.158 | 88.0 | 90.0 | 90.0 | 90.7 |
| TwoRoom | e8 | 0.079 | 89.3 | 88.7 | 89.3 | 89.3 |
| TwoRoom | e12 | 0.037 | 87.3 | 84.7 | 85.3 | 87.3 |
| TwoRoom | e18 | 0.014 | 84.7 | 84.0 | 84.7 | 86.0 |
| TwoRoom | e25 | 0.006 | 82.0 | 84.0 | 86.0 | 86.7 |
| TwoRoom | e30 | 0.004 | 86.0 | 84.0 | 86.0 | 87.3 |
| Reacher | e1 | 0.137 | 35.2 | 39.6 | 44.0 | 38.0 |
| Reacher | e2 | 0.055 | 50.8 | 52.8 | 46.7 | 44.0 |
| Reacher | e3 | 0.194 | 42.8 | 49.6 | 42.0 | 44.0 |
| Reacher | e5 | 0.052 | 62.8 | 64.4 | 62.0 | 60.7 |
| Reacher | e8 | 0.028 | 78.4 | 75.6 | 70.7 | 76.0 |
| Reacher | e12 | 0.023 | 80.8 | 85.6 | 85.3 | 78.7 |
| Reacher | e18 | 0.020 | 84.4 | 85.6 | 80.7 | 79.3 |
| Reacher | e25 | 0.019 | 82.0 | 87.6 | 82.7 | 80.0 |
| Reacher | e30 | 0.018 | 81.2 | 85.2 | 82.7 | 81.3 |
Table 7.

Figure 7. Checkpoint quality and uncertainty penalties. Fixed-depth success at and is plotted against held-out one-step MSE on a log axis. Means are listed in Table 7. TwoRoom’s curves remain close across the measured range. Reacher has positive contrasts at several checkpoints and a negative contrast at e8. Coverage is not independently varied. Stars mark released-model anchors with fewer seeds than the separate weight sweep.
The penalty’s effect changes with the checkpoint. TwoRoom’s fixed-depth success curves stay close as probe error falls, and their difference changes sign. Reacher gains at several checkpoints but loses success at e8. Its largest gain coincides with the e3 increase in probe error. Training time and model state change together, so this association does not identify either probe error or coverage as the cause.
The relative performance of the stopping rules also changes during training. At Reacher e8, epistemic-aware halting exceeds cost-only halting but remains below fixed-depth planning. At e12, cost-only halting has the higher mean. These reversals argue against treating epistemic-aware halting as a consistent improvement.
Candidate Selection
The diagram shows how disagreement changes candidate selection and how a separate stopping rule controls refinement.
Full Results Tables
The tables report checkpoint, transfer, and adaptation experiments. Each experiment retains its own seed count and evaluation protocol. Trace measurements and checkpoint sweeps use separate evaluations from the disagreement-weight comparison.
Regret Analysis
Formal setup
The analysis uses the uniform-depth episode idealization of Appendix § 7.§ 2. A single depth is applied at every control step. The evaluated controller instead chooses a sequence of depths. The bounds therefore describe success curves rather than its episode-policy regret.
All depths are integer-valued. Fix a refinement budget and minimum depth . A fixed-depth policy uses iterations for every instance. An adaptive rule chooses for instance and planner randomness , with . We suppress in the notation below. Expectations and probabilities include this randomness unless the rule is deterministic. Let denote success probability at fixed depth . Let denote the elite planning cost at iteration . The controller observes cost trajectories but cannot observe at deployment.
Per-instance stopping regret and its average over a test distribution are
where is the smallest maximizer of over allowed integer depths.
Relation to adaptive execution. The selected depth can depend on the solver’s random trajectory. Under conditional independence of selected depth and potential outcomes given , the adaptive success probability equals . Without that condition, selecting depth can also select outcomes. This effect is distinct from allocating different depths to different instances. Instance-dependent allocation alone can outperform a common fixed depth. The curve-based regret is non-negative by definition. The column in Table 10 instead compares marginal means with a common fixed-depth reference. That descriptive difference can be negative.
Unimodal depth structure. Say is weakly unimodal with onset if it is nondecreasing on and nonincreasing on . Selecting the earliest maximizer also defines early and late stopping when the optimum is flat. Then . Define the pre-plateau shortfall and post-plateau decline
for a flat post-peak plateau. A positive permits success loss from late stopping.
Proposition 1. Stopping regret under unimodal structure. Let and be the early- and late-halt probabilities of an adaptive rule. If is weakly unimodal for every , then
Proof. Condition on the halt event. If the regret is zero. If , monotonicity on gives , so . If , monotonicity on gives , so . Taking expectations and applying the law of total expectation yields Eq. (9), with a zero-probability event contributing zero (equivalently use expectations of and times their event indicators). □
The bound separates loss from early stopping and post-peak decline. If a rule never stops early and for every instance, its curve-based regret is zero. Otherwise the bound depends on the stopping probabilities and conditional shortfalls. These quantities depend on the instance distribution.
We do not estimate or , which require per-instance oracle depths. The measured grids use a small set of fixed depths, with some cells evaluated at a single seed. They show aggregate plateaus or declines but do not establish unimodality for every instance. In particular, a best-fixed-minus-adaptive success gap does not estimate . The Cube and Wall curves illustrate declining aggregate success after a peak. They do not supply the per-instance quantities needed to evaluate the bound.
Cost-based Stopping Limit
Proposition 1 bounds regret in terms of , , which depend on the rule’s ability to locate . But the rule sees only the cost trajectory. Call a halting rule cost-internal if is a measurable function of alone, where is the pragmatic elite cost minimized by the primary controller at . Our EFE rule and its patience and threshold variants are cost-internal. The disagreement guards in Appendix § 14.§ 1 use additional information.
Write when for all . Such instances are indistinguishable to a cost-internal rule. The following construction gives a worst-case lower bound within this class.
Proposition 1. Positive worst-case regret for cost-internal halting. Fix and choose . Fix depths . There exists a pair of weakly unimodal instances with
(any ) such that every deterministic cost-internal rule satisfies
Proof. A cost-internal rule reads only , which is identical on and . It therefore chooses the same depth for both. The earliest oracle depths are and . If then on the rule attains against an oracle value , so . If then on it attains against oracle , so . No avoids both. □
Proposition 2 gives a worst-case construction rather than a lower bound for every instance. Flat success curves can have zero regret at every depth. The construction shows that cost traces alone cannot guarantee optimal depth without a link between cost and external success.
Exact equality of cost traces is an assumption of the construction. The Cube and Wall experiments do not establish such an indistinguishable pair. Similar traces also need not yield similar decisions because threshold rules can be discontinuous. An approximate version would require a bound on observational distinguishability. For randomized rules, the corresponding worst-case expected regret is at least . This follows by balancing the probabilities of stopping before and after .
Disagreement is an additional observation, so Equation (10) does not cover guards that use it. The fixed-depth ensemble exceeds cost-only halting in four of five Wall seeds. That comparison changes candidate scoring and computation, rather than testing disagreement-based stopping. Appendix § 14.§ 1 evaluates such stopping signals separately.
Reference-based coverage deficit
Let be a model trained with a spatial coverage hole. Its coverage deficit compares success with a volume-matched reference under the same region-goal protocol.
Two controls. The volume-matched model removes the same number of transitions at random. It is evaluated at the same training epoch as the hole model. This comparison holds dataset size fixed and changes spatial coverage. The success-matched model instead uses an earlier random-drop checkpoint. Its frozen success equals the hole e3 model’s success (Table 4). Online adaptation changes its success by −2.4 points. Thus low initial success alone does not predict an adaptation benefit in this control. The deficit below uses the volume-matched reference.
Writing for frozen success on region goals,
where is the frozen-to-online improvement on the hole model. At the e3 checkpoint pts and pts, leaving pts that test-time training does not recover. At e5 the deficit is 39.2 and the gain is 18.8, leaving 20.4 (Table 10).
The reference is one volume-matched model. (11) therefore measures a descriptive success gap. It does not bound the performance of other models or depth allocations.
Stopping and Adaptation
The factorial experiment tests whether stopping changes the effect of online adaptation. Write the cell means of the factorial as for allocation and training . The interaction contrast is
and an additive effect on success corresponds to . The mean interaction is +0.4 points. Within-seed interactions are points. The interaction changes sign across seeds. The small mean does not establish independent effects. Adaptive stopping lowers success in both training conditions. Online adaptation improves success under both allocation rules.
Per-solve cost regret
The logged elite-cost trajectory shows whether refinement has converged under the model objective. For one optimizer call with halt iteration , define
where is the recorded elite-mean cost. This measures deterioration of that summary after its best recorded value. It is neither episode-success regret nor the evaluated cost of the returned mean action sequence. A monotonically decreasing cost trace has , even when external success decreases. The identical-cost construction in Proposition 2 permits this case.
The calculation uses the primary controller configuration. Every optimizer call with a complete cost trajectory is included, regardless of episode success. Calls with only endpoint costs cannot provide the trajectory minimum. Table 11 counts these calls as missing.
| TwoRoom | Reacher | PushT | Cube | |
| Seeds with full traces | 1, 2, 3, 4 | 1, 2 | 3, 4 | 1, 2 |
| Calls with full traces | 241 | 156 | 124 | 142 |
| Calls without full traces | 62 | 81 | 196 | 65 |
| Mean | 6.395 | 2.723 | 0.075 | 0.042 |
| Calls with positive denominator | 239 | 151 | 124 | 141 |
| Mean | 0.178 | 0.247 | 0.007 | 0.015 |
| Zero-denominator calls | 2 | 5 | 0 | 1 |
Table 11. Primary-controller elite-summary regret. Only complete trajectories in the listed seed subsets contribute to the cost averages. All calls are counted regardless of episode outcome. The normalized row uses iteration 3 as its reference and excludes eight zero denominators, seven with positive raw regret. The trace subset differs from the full-seed episode-success comparison.
For scale context we also compute
when the denominator is positive, using the controller’s minimum allowed depth. Calls with zero denominator remain in the raw average and are excluded from the normalized average. Most excluded calls have positive raw regret, so the normalized average describes a different subset of calls. The ratio can exceed one. Calculations use the logger’s four-decimal cost records and give equal weight to each eligible optimizer call.
Cost scales, seed subsets, and optimizer-call counts differ across environments. The table describes internal convergence on the available complete trajectories. It does not estimate a causal effect of stopping or a paired relationship with episode success.
Online Adaptation
Online adaptation improves the hole model in independently trained samples, but the frozen disagreement penalty changes sign between them. Table 4 reports the second training sample. Table 12 adds the first sample and the remaining controls. The hole-model gain is positive at every evaluation seed in the first sample. The volume-matched and full-data models gain less.
| TwoRoom, sample 1 | Frozen | Online | Change |
| Coverage hole | 54.7 | 69.3 | +14.7 |
| Volume control | 66.0 | 70.7 | +4.7 |
| Full data | 74.0 | 75.3 | +1.3 |
| Frozen penalty contrast | Sample 1 | Sample 2 | |
| Coverage hole | +20.0 | −4.0 | |
| Reacher | Online change | Interaction | |
| Sample 1 | e3 | −6.8 | +3.2 |
| Sample 1 | e5 | +2.4 | −3.2 |
| Sample 2 | e3 | −4.0 | · |
| Sample 2 | e5 | +0.3 | · |
| Controls without a hole | Frozen success | Change | |
| Full data | e1 | · | +9.6 |
| Random drop | e1 | · | +6.8 |
| Random drop | e2 | · | −2.4 |
| Full data | e2 | 70.8 | +5.6 |
Table 12. Additional online-adaptation results. Success and changes are percentages and percentage points, respectively. Paired changes are reported separately from rounded success means. Sample 1 uses three paired seeds per TwoRoom condition. Its hole-model gain is positive in all three. Sample 1 Reacher uses five paired seeds, with four negative changes at e3. The separate sample 2 volume-matched references have frozen success 87.2 at e3 and 83.2 at e5. On collected Reacher transitions, prediction error decreases from approximately 1.1 to 0.63 under adaptation, and anchor error stays flat.
The success-matched reference uses an earlier random-drop checkpoint with the same frozen success as the hole model. Its success falls after adaptation. The volume-matched references instead use the same training epochs as the hole models and have higher frozen success. The factorial experiment evaluates the hole condition separately from these coverage comparisons.
Reacher gives mixed results. Online adaptation lowers success at the earlier checkpoint in both training samples and gives a small gain at the later checkpoint. Its interaction with disagreement seeking also changes sign. Prediction error on collected transitions falls, but this improvement does not ensure better control.
Adaptation also helps some models trained without a coverage hole. Its benefit therefore depends on more than the presence of the hole. The comparison with shuffled-target updates tests whether pairing each collected transition with its correct target matters.
Larger penalty gains coincide with increases in probe error during each Reacher training run (Table 13). The forward test is weaker. Its mean difference is positive, but the gain does not hold across seeds and it misses the prespecified criterion. The forward test therefore does not confirm that probe error predicts when the penalty helps.
| Training sample | Checkpoint | Probe MSE | Penalty change |
| 1 | e3 | 0.194 | +6.8 |
| 2 | e5 | 0.122 | +8.0 |
| 2 | e8 | 0.035 | +2.0 |
| 2 | e2, forward test | 0.131 | +3.2 |
Table 13. Reacher probe error and frozen-penalty changes in success. Differences are percentage points. The sample 2 forward test at e2 has two positive seeds out of five and misses its prespecified criterion.
Stopping Rules

Figure 9. Success and refinement depth for the stopping rules. Each signal is evaluated across its threshold grid. The gray line shows fixed-depth success and the black star marks the primary controller. Cost-based rules follow the plateau with different depth allocations. Geometric rules are sensitive to thresholds. No evaluated rule reliably recovers Cube’s earlier success peak.
The primary controller uses seeds on TwoRoom and PushT. It uses on Cube and Reacher. Fixed-reference seeds are on TwoRoom, on PushT, and elsewhere. Best-elite, variance, and mean-change rules use . Their shared thresholds are , , and , respectively. No-EMA uses on TwoRoom. Epistemic-aware halting uses on Reacher and the elite-stability seed elsewhere. Elite stability uses the single seed listed in Table 14.
| Halting signal | TwoRoom | PushT | Cube | Reacher |
| EFE EMA-mean (primary) | 88.0<br>@ 22.0 | 87.2<br>@ 28.3 | 63.3<br>@ 28.1 | 74.7<br>@ 22.3 |
| no-EMA (v1) | 87.3<br>@ 16.4 | ··· | · | · |
| epistemic-aware (adaEFE) | 90.0<br>@ 24.9 | 96.0<br>@ 29.8 | 72.0<br>@ 29.2 | 78.5<br>@ 26.7 |
| best-elite cost stall | 89.3<br>@ 9.3 | 87.3<br>@ 15.3 | 63.3<br>@ 15.8 | 74.0<br>@ 9.3 |
| elite-variance stall | 88.0<br>@ 8.5 | 77.3<br>@ 20.9 | 66.0<br>@ 10.3 | 76.7<br>@ 19.0 |
| mean-change stall | 88.7<br>@ 22.5 | 89.3<br>@ 17.9 | 65.3<br>@ 24.7 | 76.0<br>@ 26.2 |
| elite-stability (fixed radius) | depth 30.0 at thresholds (seed 42) | |||
| elite stability, | 86.0<br>@ 14.8 | 90.0<br>@ 23.7 | 70.0<br>@ 21.1 | 64.0<br>@ 21.2 |
| best fixed depth | 90.0<br> | 88.4<br> | 67.3<br> | 72.0<br> |
Table 14. Stopping-rule comparison. Success is the mean percentage across seeds. The second line in each adaptive cell gives mean depth. Each seed uses 50 episodes and cap 30. Seed sets and thresholds are specified below. Mean differences are descriptive where seed sets differ. Missing entries indicate unevaluated configurations.
All rules use the same solver, patience , minimum depth , and candidate counts. They differ in the statistic used to detect stalling. Variance and mean-change rules monitor the CEM sampling distribution. Best-elite stall monitors the minimum elite cost. Elite stability measures the fraction of current elites within an action-space radius of the previous elite set. This radius-based rule never triggers at the tested thresholds.
A self-calibrated alternative compares cross-set nearest-neighbor distance with within-set spacing. It stops when the median cross-set distance falls below times the median within-set distance. The single-seed grid changes abruptly with . At the rule reaches the cap, whereas at it stops at the three-iteration minimum. The intermediate setting gives the results in Table 14. These measurements do not establish a stable geometric stopping signal. Best-elite stall uses less depth than the primary rule, with nearby success means. The evaluated rules do not reliably recover Cube’s earlier success peak.
Disagreement guards and detector results
A disagreement guard stops when the elite disagreement exceeds times its running minimum. We add this guard to the primary controller and vary on Cube. A shared setting is also evaluated on the three plateau tasks (Table 15). The tightest Cube guard stops earlier and recovers part of the lost success. No tested setting reaches the earlier fixed-depth peak. The shared guard also stops prematurely on PushT, where success declines substantially. Thus the tested guard does not improve allocation consistently across tasks.
| Rule | Cube | TwoRoom | PushT | Reacher |
| fixed-10 (Cube peak) | 71.3 | · | · | · |
| fixed-30 | 61.0 | · | · | · |
| primary, unguarded | 67.0 @ 28.3 | · | · | · |
| guard | 68.7 @ 18.1 | · | · | · |
| guard | 64.0 @ 20.5 | 82.0 @ 12.4 | 69.3 @ 12.1 | 78.7 @ 21.4 |
| guard | 66.7 @ 22.3 | · | · | · |
Table 15. Disagreement guards. Mean success is followed by mean depth. Each cell uses three seeds and 50 episodes per seed. The guard stops when disagreement exceeds times its running minimum. It supplements the primary controller. The plateau tasks use . At these seeds, the unguarded adaptive reference reaches 86 on TwoRoom and 92 to 98 on PushT. Fixed-30 reaches 76 on Reacher. Missing entries indicate unevaluated configurations.
Four additional detectors use the recorded per-instance trajectories. Two monitor disagreement rise. Another compares imagined terminal cost with the realized outcome. The fourth compares disagreement with its early-iteration level. Their statistics and firing rates overlap across environments. These detectors do not reliably separate Cube’s over-optimization from the plateau tasks. The result limits this detector family without ruling out externally informed stopping signals.
Normalized Weights and Sensitivity
Negative disagreement weights have different effects across tasks (Table 16). Reacher, PushT, and Cube gain at some weights, whereas every tested penalty lowers mean success on TwoRoom. Larger positive weights lower success on all tasks. The weight comparison reuses sampled starts and goals across seeds. The dropout generator is not explicitly seeded, so imagined trajectories are not fully paired.
| Env | Success (%) | (pp) | |
| TwoRoom | -50 | 71.06 | |
| TwoRoom | -20 | 79.82 | |
| TwoRoom | -10 | 82.36 | |
| TwoRoom | -3 | 83.12 | |
| TwoRoom | 0 | 84.50 | |
| TwoRoom | +3 | 76.42 | |
| TwoRoom | +10 | 63.88 | |
| TwoRoom | +50 | 48.40 | |
| PushT | -50 | 81.72 | |
| PushT | -20 | 86.40 | |
| PushT | -10 | 87.42 | |
| PushT | -3 | 87.66 | |
| PushT | 0 | 86.24 | |
| PushT | +3 | 70.12 | |
| PushT | +10 | 31.58 | |
| PushT | +50 | 8.18 | |
| Reacher | -50 | 81.48 | |
| Reacher | -20 | 80.52 | |
| Reacher | -10 | 79.00 | |
| Reacher | -3 | 77.62 | |
| Reacher | 0 | 78.14 | |
| Reacher | +3 | 76.58 | |
| Reacher | +10 | 72.40 | |
| Reacher | +50 | 48.88 | |
| Cube | -50 | 68.96 | |
| Cube | -20 | 69.08 | |
| Cube | -10 | 69.52 | |
| Cube | -3 | 68.66 | |
| Cube | 0 | 67.64 | |
| Cube | +3 | 67.58 | |
| Cube | +10 | 64.16 | |
| Cube | +50 | 54.66 |
Table 16. Disagreement-weight comparison with 100 paired seeds and 50 episodes per run. Success is the mean percentage across seeds. is the mean difference from the same-seed reference. Every configuration uses 30 CEM iterations and stochastic rollouts.
Reacher’s gains at stronger penalties are smaller in the larger seed sample than in the earlier evaluation. Its weaker penalties give both positive and negative differences. The results therefore do not support a uniformly beneficial negative-weight range.
Repeated evaluations also change the measured success of a fixed configuration. Reacher’s initial evaluation was higher than every repetition. These repeated runs use the same evaluation seed and selected configurations, so they do not describe variation across training runs or all evaluation seeds.
We also scale disagreement by the current candidate-batch ratio of pragmatic and disagreement magnitudes. The dimensionless weight is (Figure 11). The

Figure 11. Scale-matched dose response by environment. Seven normalized weights and 28 values use a common scale from 0 to 100%. Baseline annotations give mean success. Nonzero-weight cells use a single seed. Disagreement magnitude is matched to the current candidate batch. The intermediate positive weight reduces success by 20 to 52% relative on three environments. The intermediate negative weight improves Reacher success by approximately 11% relative.

Figure 10. Disagreement-weight results with 100 seeds per cell. (a) Mean success. (b) Mean paired differences from . (c) Ratio of mean success at to , reaching 10.54× on PushT. Each seed uses 50 episodes, giving 160,000 episode evaluations across the weight grid. Reacher’s differences are pp, compared with pp at each weight in the earlier 20-seed sample. In a separate set of repeated evaluations, Reacher success was 90% initially and 80 to 82% in five repetitions.
intermediate positive weight lowers success on most environments. The largest positive weight gives zero observed success on PushT. The intermediate negative weight improves mean success on Reacher. Nonzero-weight cells use a single seed. The sweep suggests that the positive-weight loss can persist after magnitude matching, but does not establish robustness across seeds.
Changing the ensemble size has little effect on Reacher success at the selected negative weight (Table 17). The largest tested dropout rate lowers success. The cells share an evaluation seed, but the separate multi-seed baseline does not give a paired estimate of the gain. Dropout changes the imagined dynamics as well as the disagreement signal.
| Success | 80 | 80 | 82 | 76 | 80 | 64 |
Table 17. Proxy sensitivity on Reacher at . The ensemble-size sweep fixes dropout at . The dropout sweep fixes . Each cell uses seed 42. The separate multi-seed baseline is 72.0.
Candidate values in TD-MPC2 can share a large common offset. Their magnitude can therefore exceed the variation that determines candidate ranking. That variation also contracts as MPPI converges (Table 18). Magnitude matching can consequently overweight disagreement for either sign of . We instead normalize by the per-task spread of candidate values. The available columns show smaller return losses for negative than positive weights. Their task subsets differ, so they do not provide a paired comparison across doses. Candidate-value spread is a useful scale to consider when transporting the weight between planner families.
| (spread-normalized) | ||||
| Return, % of | 99.2% | 96.5% | 32.9% | 5.9% |
| tasks complete | 2 | 10 | 11 | 10 |
Table 18. Spread-normalized weights on TD-MPC2. Raw equals times per-task candidate-value spread divided by disagreement scale. Depth is fixed at 6. Return is a percentage of the baseline. Each column covers of 19 tasks, with different task subsets. Value magnitude is approximately candidate spread. The spread contracts approximately over six MPPI iterations. Dog-walk reaches 117.2% at in one 10-episode evaluation.
References
- [1]Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.
- [2]Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl Friston, and Ryan Smith. Reward maximization through discrete active inference. Neural Computation, 35(5):807–852, 2023.
- [3]Zafeirios Fountas, Noor Sajid, Pedro Mediano, and Karl Friston. Deep active inference agents using Monte-Carlo methods. In NeurIPS, 2020.arxiv.org/abs/2006.04176
- [4]Karl Friston. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11 (2):127–138, 2010.DOI
- [5]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025.arxiv.org/abs/2502.05171
- [6]Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016.arxiv.org/abs/1603.08983
- [7]Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, 2019.arxiv.org/abs/1811.04551
- [8]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.arxiv.org/abs/2301.04104
- [9]Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In ICLR, 2024.arxiv.org/abs/2310.16828
- [10]Nicklas A. Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In ICML, 2022.arxiv.org/abs/2203.04955
- [11]Alexia Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks. arXiv preprint arXiv:2510.04871, 2025.
- [12]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In NeurIPS, 2020.arxiv.org/abs/2006.04779
- [13]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. LeWorld-Model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026.arxiv.org/abs/2603.19312
- [14]Pietro Mazzaglia, Tim Verbelen, and Bart Dhoedt. Contrastive active inference. In NeurIPS, 2021.arxiv.org/abs/2110.10083
- [15]Pietro Mazzaglia, Tim Verbelen, Ozan Çatal, and Bart Dhoedt. The free energy principle for perception and action: A deep learning perspective. Entropy, 24(2):301, 2022.DOI
- [16]Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Whence the expected free energy? Neural Computation, 33(2):447–482, 2021.DOI
- [17]Thomas Parr, Giovanni Pezzulo, and Karl J Friston. Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. MIT Press, 2022.
- [18]Reuven Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1:127–190, 1999.DOI
- [19]Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML, 2020.arxiv.org/abs/2005.05960
- [20]Kai Ueltzhöffer. Deep active inference. Biological Cybernetics, 112(6):547–573, 2018.DOI
- [21]Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734, 2025.arxiv.org/abs/2506.21734
- [22]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In NeurIPS, 2020.arxiv.org/abs/2005.13239
- [23]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983, 2024.arxiv.org/abs/2411.04983
