betaFeedback

WorksComputer Science

Knowing When to Stop Thinking: A Dissection of Expected Free Energy as a Test-Time Compute Controller for Latent World-Model Planning

Abstract

A planner using a learned world model can improve its predicted outcome while choosing a worse action. We study when expected free energy (EFE) can help decide how much computation to spend on a plan. An EFE-inspired objective combines goal distance with disagreement among stochastic predictions. We test candidate scoring, stopping, and online predictor updates separately. Stopping when elite cost stalls reduces refinement on tasks whose success plateaus with depth, with mean success close to fixed-depth references. At the larger tested weights, rewarding disagreement lowers success with a frozen model. Penalizing it helps at some task and weight settings but harms others. When training data omit a region needed to reach the goal, online updates recover part of the resulting loss. The tested cost-stall rule can fail when further optimization exploits model errors, because predicted cost keeps falling after task success peaks. Cost progress can guide computation on a plateau, but protecting performance after a peak requires information about the consequences of the chosen actions.

Introduction

A robot planning from images can improve its next action by repeatedly imagining and comparing possible futures. Each refinement step costs computation. Stopping too early may leave a useful action undiscovered. Continuing too long can favor an action that exploits errors in the learned dynamics. The planner must decide whether further optimization will improve the action it executes.

Test-time computation has improved iterative reasoning and model-based control. Recurrent language models refine internal representations before producing an answer ([5, 21, 11]). World-model agents predict future states to evaluate candidate actions ([8, 1]). A larger optimization budget gives the planner more opportunities to improve a plan. Its objective, however, remains a prediction of task performance. A fixed budget cannot respond to differences between instances, and convergence of the objective does not establish that the predicted consequences are correct.

Expected free energy (EFE) connects goal achievement and information seeking ([4, 17]). Its pragmatic component favors preferred outcomes, and its epistemic component values information. These components suggest ways to control planning. Goal progress may indicate when refinement has stalled. Prediction uncertainty may change which plan the agent selects. New observations may improve the model before the next decision. Disagreement-based exploration methods collect information during learning ([19]), whereas uncertainty penalties discourage plans that rely on unreliable predictions ([22]). Whether disagreement helps immediate task execution with a frozen predictor remains unclear. Internal progress may also fail to reveal when more planning becomes harmful.

We test these decisions with a compact latent world model and a sampling planner. In Figure 1, candidate trajectories provide goal cost and disagreement among stochastic predictions. We vary the disagreement weight and stopping statistic with the model frozen. We then test stopping in additional planner families and remove training data from a required region to study online learning. These comparisons separate the effects of reducing refinement, changing action selection, and updating the predictor.

Planning workflow and measured effects of refinement

Figure 1. Planning and measured effects of refinement. Top, the planning workflow refines candidate actions with a frozen world model until cost progress stalls or the budget is exhausted. Bottom, DINO-WM results from five seeds with 50 starts per seed. Success and model cost are plotted on separate axes. Each cost curve is divided by its first-iteration value within each seed before averaging. PushT success reaches a plateau. Wall success peaks at iteration 11 at 57.6% and falls to 35.6% at iteration 30 as model cost continues to decrease. All plotted curves come from recorded evaluations.

Cost-stall rules reduce refinement on tasks whose success plateaus with depth, with mean success close to fixed-depth references. A best-elite baseline is competitive and often stops earlier. On tasks whose success declines after a peak, falling cost sustains harmful refinement. Rewarding disagreement reduces frozen-model planning success, whereas penalties help at selected task and weight settings. Online updates improve success when the agent must enter a region missing from its training data.

Our experiments address the stopping decision and the uses of prediction uncertainty.

  • We separate candidate scoring, stopping, and online adaptation in a common world-model planning protocol.

  • We measure when cost-stall controllers reduce planning effort and when model exploitation defeats the tested stopping signals.

  • We measure the task-dependent effects of disagreement penalties and test online learning by removing training data from a required region.

Related Work

Active inference formulates perception and action through free-energy objectives [4, 17]. Deep implementations combine learned representations with approximate inference [20, 3, 14, 15]. The informationseeking term requires assumptions beyond free-energy minimization [16], and Bellman-optimality results hold under specified conditions [2]. We use this framework to motivate a cost combining goal distance and stochastic prediction disagreement. Disagreement provides an uncertainty proxy. It is not a calibrated estimate of information gain.

Learned latent dynamics support planning from images [7, 10, 1]. We build on LeWM, a compact joint-embedding predictive architecture that predicts future image representations [13]. Dreamer improves policies through imagined returns [8], whereas Plan2Explore uses disagreement to gather experience during learning [19]. Our frozen-model experiments test disagreement for immediate goal-directed execution. The adaptation experiment tests what changes when observed transitions update the predictor.

Adaptive computation allocates different amounts of work to different inputs [6]. Recurrent and recursive reasoning apply this idea to internal model computation [5, 21, 11]. We study stopping inside sampling-based model predictive control and compare statistics from the same optimization process. Pessimistic offline reinforcement learning addresses unreliable estimates through model uncertainty penalties in MOPO [22] and conservative value estimates in CQL [12]. We compare a disagreement penalty with a disagreement reward. The coverage experiment then tests a setting where the goal requires entering a region absent from training, so avoiding uncertain dynamics cannot solve the task.

Method

Figure 2 shows the planning loop. A fixed world model predicts candidate futures, the candidate cost determines which actions refine the proposal, and a stopping rule determines when to execute an action. The lower branch updates the predictor from observations in a separate experiment. We vary scoring, stopping, and adaptation separately.

Planning and online adaptation

Figure 2. Planning and online adaptation. The upper loop encodes images, samples actions, predicts and scores futures, selects elites, and refits the proposal. The stopping rule either continues refinement or executes the selected action. The lower branch updates the predictor using collected transitions, replay, and a parameter anchor. Shuffled-target updates and a frozen predictor provide controls.

Candidate Scoring

The planner compares possible actions before executing them. Let oto_t denote the current image, ztz_t its encoded representation, and zgz^g the encoded goal image. A latent predictor fθf_\theta advances the state representation under a candidate action sequence aa. The cross-entropy method (CEM) [18] samples action sequences from a Gaussian distribution, retains the lowest-cost candidates as an elite set, and refits the distribution to that set. Each refinement iteration repeats this sample, predict, and refit procedure.

To expose both goal progress and predictive variation, we evaluate each candidate with KK stochastic predictor rollouts. The terminal prediction in rollout kk is zT(k)(a)z_T^{(k)}(a) and their mean is zˉT(a)=K−1∑k=1KzT(k)(a)\bar{z}_T(a) = K^{-1}\sum_{k=1}^{K} z_T^{(k)}(a). Candidate scoring uses

Cβ(a)=P(a)−βU(a),P(a)=∥zˉT(a)−zg∥2,(1)\mathcal{C}_{\beta}(a) = P(a) - \beta U(a), \qquad P(a) = \left\lVert\bar{z}_T(a) - z^g \right\rVert^2, \tag*{(1)}

where U(a)=K−1∑k=1K∥zT(k)(a)−zˉT(a)∥2U(a) = K^{-1}\sum_{k=1}^{K}\left\lVert z_T^{(k)}(a) - \bar{z}_T(a)\right\rVert^2 for K>1K > 1, and U(a)=0U(a) = 0 for K=1K = 1. The goal term PP sums squared terminal-coordinate errors. The disagreement term UU sums coordinatewise variance with divisor KK. Positive β\beta rewards disagreement and negative β\beta penalizes it. This sign change tests two uses of the same measurable signal without changing the predictor.

The objective uses goal distance and disagreement to study the pragmatic and epistemic roles of EFE. Disagreement measures stochastic prediction spread. It does not estimate posterior information gain. The weight experiment enables predictor dropout at every weight, including β=0\beta= 0, so the comparisons share the same stochastic scoring procedure. The deterministic reference disables dropout and uses a single rollout. At β=0\beta= 0 it recovers terminal goal-distance scoring. We also evaluate magnitude-normalized weights because PP and UU can have different magnitudes.

Terminal spread is large when stochastic predictions disagree about where a candidate action sequence will end. A reward favors uncertain predicted outcomes, and a penalty favors more consistent predictions. Changing the sign leaves the goal representation and feasible action space unchanged.

It tests how disagreement affects action selection without assuming that greater spread means more useful information.

Stopping Rule

A fixed iteration budget spends the same maximum effort on every solve. We instead monitor how much the elite cost improves. Let EnE_n denote the elite set at iteration nn, and let ene_n be its mean candidate cost. The controller monitors an exponential moving average E‾n\overline{E}_n of ene_n and computes relative descent

qn=E‾n−1−E‾nmax⁡(∣E‾n−1∣,10−8).(2)q_n = \frac{\overline{E}_{n-1} - \overline{E}_n}{\max\left(\lvert\overline{E}_{n-1}\rvert, 10^{-8}\right)}. \tag*{(2)}

It stops at the first eligible iteration for which qn<εq_n < \varepsilon throughout a window of pp consecutive iterations. Eligibility requires both the minimum depth nmin⁡n_{\min} and a complete patience window. The floor in the denominator handles costs near zero. If no stall occurs, refinement ends at the cap NN.

The patience window prevents an isolated small improvement from ending the solve. Smoothing reduces sensitivity to changes in the sampled elite set, and the minimum depth gives the optimizer time to establish a candidate distribution. A cost increase also satisfies the stall inequality because relative descent becomes negative. The rule detects insufficient recent improvement. It does not prove convergence of the sampling distribution. We compare alternative cost statistics under the same patience and minimum-depth settings.

Write n^\hat{n} for the selected stopping iteration. After stopping, the agent executes the first action block, observes the next image, and plans again. Depth measures computation within a control decision. The primary controller uses β=0\beta= 0 with shared settings across the LeWM tasks, so its descent signal measures improvement in predicted goal distance. We compare it with best-elite cost stall, mean-change, variance-stall, and elite-set stability. Unsmoothed and disagreement-aware variants test changes to the monitored statistic. Patience longer than the available window recovers fixed-depth refinement.

Online Adaptation

Stopping and candidate scoring operate within the current model. Online adaptation changes that model. To test its value, we remove training episodes that enter a specified corner of TwoRoom and evaluate goals inside that region. This creates a coverage hole that goal-directed behavior must encounter. A volume-matched control removes the same amount of data at random. A separate checkpoint matches the hole model’s frozen success, distinguishing low initial performance from the location of missing training data.

During evaluation, the adapter updates the latent predictor with the original training objective on collected transitions. Replay and a parameter anchor constrain the update. A frozen baseline performs no updates. A sham baseline shuffles prediction targets and keeps the rest of the update procedure unchanged. Comparing online and sham updates tests the contribution of correct target pairing. All conditions follow the same sequential evaluation protocol.

Experimental Setup

The primary study uses LeWM on TwoRoom navigation, PushT manipulation, OGBench-Cube manipulation, and DMC-Reacher control. We use Cube and Reacher as short names below. TwoRoom, PushT, and Cube use expert-data world models. Reacher uses random-policy training data. Checkpoint comparisons vary training progress within an environment. The coverage experiment changes which transitions are available during training. Table 1 gives the primary configuration.

QuantitySettingQuantitySetting
Model parameters18MImage resolution224×224224 \times224
Latent dimension192CEM samples / elites300 / 30
Horizon / action block5 / 5Refinement cap30
Stall threshold10−210^{-2}Patience / minimum depth2 / 3
EMA coefficient0.5Episodes per seed50
Initial controller seeds3 to 6Dose-response seeds per cell100

Table 1. Primary LeWM configuration. The same cost-stall settings are used across the four tasks. Dose-response and transfer experiments have their own stated seed counts and budgets. The normalized-weight grid uses one seed per setting.

Fixed-depth CEM measures success as refinement increases. The primary stopping rule and its alternatives share the planner configuration. We select the fixed reference from the evaluated depth grid, which means it is not independently tuned. The disagreement experiment varies β\beta with the model and rollout procedure fixed. TD-MPC2 [9] and DINO-WM [23] test stopping with a common threshold within each family. TD-MPC2 samples actions using learned latent values. DINO-WM plans with dynamics on fixed visual features.

Evaluations sample start and goal pairs from the available datasets. These samples may overlap with model-training data. Matched seed comparisons reuse start and goal sets, although predictor-dropout randomness is not fully paired. We report mean performance and the seed counts for each comparison. Controller comparisons with unequal seed sets are descriptive. Appendix A gives the sampling and pairing procedures.

Success measures goal completion on LeWM and DINO-WM tasks. TD-MPC2 uses return relative to each task’s best fixed-depth reference. Mean depth measures the refinement iterations used. With candidate count, horizon, and stochastic rollout count fixed, imagined transition evaluations decrease in proportion to depth. Iteration savings do not measure end-to-end runtime. Ensemble-size comparisons change the work per iteration, so we report them separately from the stopping comparison.

For adaptation, we report the fraction of the coverage deficit recovered. The deficit is the frozen-success gap between the volume-matched reference and the hole model. We divide the online improvement by that gap under the same region-goal evaluation. The success-matched checkpoint tests the effect of initial performance and is not the denominator. The recovered fraction can exceed one or become negative because it is measured relative to the chosen reference.

Results

Stopping Performance

Cost-stall rules reduce refinement on tasks where additional depth has little effect on success. Table 2 compares the primary rule with the best fixed-depth reference and best-elite stopping. On TwoRoom, PushT, and Reacher, the primary rule uses less depth than the cap and reaches mean success close to the fixed references. Many solves improve rapidly before stalling. Cases that continue to improve run toward the budget limit.

EnvironmentBest fixed success @ depthPrimary success @ depthBest-elite success @ depth
TwoRoom90.0 @ 3088.0 @ 22.089.3 @ 9.3
PushT88.4 @ 3087.2 @ 28.387.3 @ 15.3
Cube67.3 @ 1063.3 @ 28.163.3 @ 15.8
Reacher72.0 @ 3074.7 @ 22.374.0 @ 9.3

Table 2. Controller success (%, mean across seeds) and mean depth. The primary rule saves 6 to 27% of iterations on TwoRoom, PushT, and Reacher relative to the cap. Each seed contains 50 episodes. Seed counts for fixed, primary, and best-elite are (3, 5, 3) on TwoRoom, (5, 5, 3) on PushT, and (3, 3, 3) on Cube and Reacher. Fixed depth is selected from the evaluated grid. The seed sets differ between methods.

Best-elite cost stall often uses fewer iterations than the primary elite-mean rule, with similar mean success. Removing smoothing or relaxing the threshold also reduces depth. Variance stall is unstable on PushT, and elite-set stability changes abruptly with its threshold. Mean-cost descent is therefore one useful stopping statistic among several. Appendix § 14 gives the complete comparisons and sensitivity curves.

Instances whose cost stalls quickly stop early. Continued cost improvement uses more of the budget. On PushT, higher initial elite cost is associated with greater allocated depth in the displayed traces. A shared rule can therefore allocate different amounts of computation without a difficulty classifier. This allocation still depends on the model cost, including when further cost reduction harms task success.

Disagreement Weights

Rewarding disagreement lowers frozen-model success at the larger tested weights in every environment. Table 3 summarizes the reductions and selected penalty gains. A disagreement reward changes the actions selected, but new observations cannot improve a predictor whose weights remain fixed. This may explain the loss in these short-horizon tasks.

SettingEnvironmentβ\betaSuccess change (pp)
RewardAll four+10−3.5 to −54.7
RewardAll four+50−13.0 to −78.1
PenaltyReacher−20, −50+2.38, +3.34
PenaltyPushT−3+1.42
PenaltyCube−20, −10+1.44, +1.88
Strong penaltyTwoRoom−20, −50−4.7, −13.4
Strong penaltyPushT−50−4.5

Table 3. Success changes from the β=0\beta= 0 baseline. Each setting uses 100 matched seeds and 50 episodes per seed with K=4K = 4 stochastic rollouts. Differences are percentage points (pp). The reward rows give the range across tasks, and the penalty rows show selected positive and negative changes. Table 16 gives all weight comparisons.

Penalizing disagreement gives small gains at selected weights on Reacher, PushT, and Cube. Stronger penalties harm TwoRoom and PushT. The effect depends on both the weight and the task.

This cross-task comparison changes the environment and training data together, so it cannot isolate training coverage as the cause.

Checkpoint comparisons test whether weaker prediction alone explains the penalty benefit. Large changes in one-step prediction error on TwoRoom do not yield a consistent penalty advantage. Reacher shows gains at several checkpoints and reversals at others. The disagreement-aware stopping variant also changes across checkpoints. Scalar prediction error alone does not explain these patterns. Training time and model state remain coupled in this comparison. Appendix § 9 gives the checkpoint results.

Learning Missing Dynamics

Online predictor updates recover part of the deficit caused by a coverage hole. Table 4 compares frozen, online, and shuffled-target updates for goals inside the removed region. Online learning improves both hole checkpoints and outperforms shuffled-target updates. This advantage suggests that correct target pairing contributes to the gain. It does not recur in the success-matched and full-data controls.

UpdateHole e3Hole e5MatchedFull e2Released
Seeds55553
Frozen37.244.037.270.875.3
Online56.062.834.876.474.0
Sham46.455.638.080.8n/a
Online − sham (pp)+9.6+7.2−3.2−4.4n/a
Deficit recovered37.6%48.0%n/an/an/a

Table 4. Coverage intervention on a second independently trained sample. Success is the mean percentage over 50 episodes per seed. Checkpoints e2, e3, and e5 denote training epochs. The matched control is a random-drop e2 checkpoint matched to hole e3 on frozen success. Deficit recovery uses volume-matched e3/e5 references with success 87.2%/83.2%. The update conditions in each column share a seed set.

Random removal preserves access to the region and matches training-data volume. The success-matched checkpoint tests whether poor initial performance alone explains the benefit of adaptation. Together, these controls support learning dynamics needed by the evaluated goals. Shuffled-target updates are an imperfect control for target pairing because shuffling can also change gradient magnitudes and preserve information about visited states.

Lower prediction loss on collected transitions does not consistently improve task success. Reacher adaptation lacks a replicated benefit across its broad-coverage checkpoints. The frozen disagreement penalty also fails to repair the TwoRoom coverage hole consistently across training samples. Action selection under an existing model and learning missing dynamics therefore need separate evaluation. Appendix § 13 gives the adaptation protocol and replications.

Model Exploitation

Cost-stall rules can continue refining after task success has peaked. Cube reaches its best fixed-depth result before the cap, but the primary controller continues and has lower success. On DINO-WM Wall, success also peaks before the full budget as predicted cost continues to fall. Cost-only halting runs close to the cap and misses the earlier peak (Table 5).

Planner settingSuccessMean depth
Fixed K=1K = 1, curve peak0.57611
Fixed K=1K = 1, full budget0.35630
Cost-only halt, K=1K = 10.36429.04
Fixed K=5K = 50.50830

Table 5. DINO-WM Wall with a cap of 30 iterations. Success is the mean over five seeds with 50 starts per seed. The peak is selected from the common-depth mean curve. The fixed-depth K=5K = 5 setting uses more computation per iteration.

The tested disagreement guards do not reliably distinguish harmful optimization from productive refinement. Tighter guards sometimes improve Cube but stop prematurely on plateau-shaped tasks. Appendix § 14.§ 1 reports the guard and detector comparisons. A fixed ensemble improves Wall success by changing candidate scoring and computation, without choosing an earlier stopping point.

Extra iterations on a plateau waste computation. After a success peak, they can also worsen the chosen action. Appendix § 12 formalizes the difference using an idealized success curve and constructs identical cost traces with incompatible optimal depths. A stopping guarantee needs a link between internal progress and external success.

Transfer Across Planners

The stall statistic also reduces refinement outside the primary CEM planner. TD-MPC2 retains most of the best-grid return across the evaluated DMControl tasks at a small mean depth. DINO-WM PushT reaches the same observed mean success as its fixed-cap reference with fewer iterations (Table 6). The threshold is shared within each family. Threshold selection was not evaluated on a separate held-out set.

Family and taskPerformanceMean depth
TD-MPC2, 19 tasks98.9% mean retained return3.8
TD-MPC2, lowest retention96.1% retained returnn/a
DINO-WM PushT, fixed0.852 success30
DINO-WM PushT, halt0.852 success26.44

Table 6. Cross-family stopping results. TD-MPC2 averages tasks equally and normalizes return by each task’s best fixed-depth mean on the same evaluation grid. DINO-WM uses five seeds and cap 30. Detailed task results and configurations appear in Table 8 and Table 9.

Table 8: TD-MPC2 evaluation on nineteen DMControl tasks. Each task uses one released checkpoint and four seeds {2,3,4,5}\{2, 3, 4, 5\}, with 30 episodes per seed. Returns are means across seeds. The fixed reference maximizes mean return over N∈{3,6,9,12}N \in\{3, 6, 9, 12\} on the evaluation seeds. Halting uses ε=3×10−3\varepsilon= 3 \times10^{-3}. Its cap is 8 on dog-walk, humanoid-run, and humanoid-walk, and 6 on the remaining tasks. Relative return is 100 × halt mean/fixed mean, so values above 100% indicate greater return than the fixed reference. Depth is averaged over seeds and summary rows weight tasks equally. Calibration evaluations are excluded. The fixed reference is selected retrospectively.

TaskN∗N^*Fixed returnHalt returnRelative returnDepth
acrobot-swingup6661.2661.8100.1%3.1
cheetah-run9861.1828.196.2%3.5
cup-catch9981.1979.199.8%3.0
dog-walk9799.2783.898.1%5.3
finger-spin6988.3985.799.7%3.8
finger-turn-hard3972.7967.099.4%3.0
fish-swim12787.5775.798.5%3.2
hopper-hop12570.1568.099.6%5.9
hopper-stand6961.3952.999.1%3.0
humanoid-run12680.9659.696.9%7.4
humanoid-walk12926.5890.496.1%5.3
pendulum-swingup12846.1846.0100.0%3.0
quadruped-run6957.8953.399.5%3.7
quadruped-walk9979.8976.099.6%3.1
reacher-easy3977.9978.7100.1%3.0
reacher-hard6981.9981.8100.0%3.0
walker-run9891.2867.997.4%3.6
walker-stand6994.8994.399.9%3.0
walker-walk9985.3984.199.9%3.0
Mean over 19 tasks98.9%3.8
Worst retained return96.1%.

Table 8.

ConfigurationPushT success @ depthWall success @ depth
Fixed K=1K = 10.852 @ 300.356 @ 30
Halt ε=10−2\varepsilon= 10^{-2}0.852 @ 26.440.364 @ 29.04
Halt ε=3×10−3\varepsilon= 3 \times10^{-3}0.852 @ 28.710.356 @ 29.79
Fixed K=5K = 5·0.508 @ 30

Table 9. DINO-WM evaluation with five seeds (1 to 5), 50 starts per seed, and cap 30. Success is the mean across seeds. Depth counts the iterations used by each trajectory. The K=5K = 5 configuration uses fixed-depth ensemble scoring.

DINO-WM contains both PushT, where stopping saves computation, and Wall, where further optimization exploits model errors. Planner identity alone does not predict whether cost-stall stopping will help. The comparison requires measuring task success across depths as well as the computation saved.

Ablation Studies

Stopping earlier reduces refinement with the candidate objective unchanged. Increasing the stochastic rollout count changes both candidate scoring and computation per iteration. On the evaluated LeWM tasks, averaging multiple imagined futures is not necessary for the observed plateau performance. On Wall, a fixed ensemble changes success substantially. Depth comparisons therefore need to account for the rollout count.

Goal distance and terminal prediction variance have different magnitudes. We normalize these magnitudes within the candidate batch before applying a dimensionless weight. Larger positive weights still harm performance. This result suggests that the reward failure is not confined to the original raw weighting. The normalized grid uses fewer seeds than the uniform weight experiment, so the result remains exploratory. Appendix I gives the normalization rule, seed count, and complete values.

For a value-based planner, a large common offset can dominate predicted value magnitudes without changing candidate rankings. Normalizing disagreement by that magnitude can overwhelm the differences used to select actions. Candidate-value spread measures the ranking scale more directly. The partial TD-MPC2 weight grid shows this problem. Its shared-threshold stopping experiment instead measures relative progress in the original objective.

Under the same coverage-hole condition, we compare fixed and adaptive allocation with frozen and online predictors. Online updates improve success under both allocation rules, and adaptive stopping lowers success with either predictor. The mean interaction is small, but these evaluations do not establish independent effects of stopping and learning. Table 10 gives the factorial results.

Env.Best ddΔ\DeltaEFEΔ\DeltaCoverage RmodelpostR_{\mathrm{model}}^{\mathrm{post}}
TwoRoom90.0@3088.0+2.0·
PushT88.4@3087.2+1.2·
Reacher72.0@3074.7−2.7·
Cube67.3@1063.3+4.0·
hole (e3)50.0 → 31.2 (TTT +18.8, 38%)31.2
hole (e5)39.2 → 20.4 (TTT +18.8, 48%)20.4
FixedAdaptive
Frozen37.233.2
Online (TTT)55.251.6
TTT effect+18.2
Adaptive effect−3.8
Interaction+0.4

Table 10. Stopping and online adaptation. Above, success relative to fixed-depth and coverage references. Below, the TwoRoom hole-e3 factorial uses seeds {42,7,123,5,99}\{42, 7, 123, 5, 99\} and 50 episodes per seed. Cells report mean success. The mean interaction is +0.4 pp, with within-seed values {−6,−4,+4,0,+8}\{-6, -4, +4, 0, +8\} pp. The factorial and Table 4 are separate evaluations of the same labeled condition.

Success Across Depths

A stopping rule observes optimization progress to decide how long to refine an action. Its purpose is to preserve task performance. Consider an idealized episode that uses a single depth at every control step. Let s(d,x)s(d, x) be its success probability at depth dd for instance xx, and let dp(x)d_p(x) be the earliest depth with maximal success. The relevant error is the difference between success at that optimum and success at the selected depth. Final planning cost measures a different quantity.

Suppose success is nondecreasing before dp(x)d_p(x) and nonincreasing afterward. Early stopping loses progress available before the peak. Late stopping loses any success that declines after it. On a plateau, the latter loss is zero even when the planner spends unnecessary iterations. On a declining curve, cost can keep improving as success falls. A conservative stopping rule can therefore save computation on a plateau and still fail to protect performance after a peak.

Appendix § 12 proves this separation and constructs instances with identical cost traces but different optimal depths. A rule that reads only those traces must make the same decision on both instances, so at least one incurs a success loss. The statement concerns an idealized curve with a single depth throughout an episode. Actual receding-horizon execution selects depth repeatedly. Stopping in that setting may require observations that connect model predictions with their realized consequences.

Conclusion

Cost stalling reduces refinement when success plateaus with depth. Disagreement penalties help at selected task and weight settings, and online predictor updates recover part of a deficit caused by missing training coverage. These gains come from different changes to planning. When continued optimization exploits model errors, falling cost no longer indicates a better action. Stopping safely after a success peak requires information beyond the cost trajectory.

Evaluation Details

Pairing and Compute

Each paired comparison reuses the starts and goals sampled by an evaluation seed. If Sr,AS_{r,A} and Sr,BS_{r,B} are success rates for seed rr under configurations AA and BB, the mean difference is

Δ^A−B=1R∑r=1R(Sr,A−Sr,B).(3)\widehat{\Delta}_{A-B} = \frac{1}{R}\sum_{r=1}^{R}(S_{r,A} - S_{r,B}). \tag*{(3)}

Each experiment retains its seed count and evaluation protocol. Repeating episodes under a fixed seed does not reproduce variation across independently sampled seeds. Repeated evaluations also leave training variation unchanged.

Depth has a similarly specific interpretation. If each refinement iteration evaluates SS action candidates with KK stochastic rollouts over horizon HH, a simple count of imagined transition evaluations is proportional to

Bimag(x)=SKHn^(x),ηiter=1−Ex[n^(x)]N.(4)B_{\mathrm{imag}}(x) = SKH\hat{n}(x), \qquad\eta_{\mathrm{iter}} = 1 - \frac{\mathbb{E}_{x}[\hat{n}(x)]}{N}. \tag*{(4)}

The second expression is the iteration saving relative to a fixed cap with the same per-iteration configuration. It is not an end-to-end runtime measurement. Encoding, action execution, batched stopping, and any uncertainty or online-update overhead contribute additional cost. Increasing the rollout count can consume more computation even if the chosen depth falls. Our principal compute comparison therefore reports used iterations alongside success and keeps ensemble-size changes in the ablations.

Regret and Coverage

The following success curves describe the success-rate tasks. Let xx denote a sampled start and goal pair, and let ω\omega denote planner randomness. Fixed-depth episode curves use the same depth at every control step. In the idealization, d~(x,ω)∈{nmin⁡,…,N}\tilde{d}(x,\omega) \in\{n_{\min}, \ldots, N\} selects one depth for the episode. Let s(d,x)∈[0,1]s(d,x) \in[0,1] denote success probability at integer depth dd. Define d∗(x)∈arg⁡max⁡d∈{nmin⁡,…,N}s(d,x)d^*(x) \in\arg\max_{d\in\{n_{\min},\ldots,N\}} s(d,x) as an allowed depth that maximizes success. The adaptive stopping regret is

rstop(x,ω)=s(d∗(x),x)−s(d~(x,ω),x),Rstop=Ex,ω[rstop(x,ω)].(5)r_{\mathrm{stop}}(x,\omega) = s(d^*(x),x) - s(\tilde{d}(x,\omega),x), \qquad R_{\mathrm{stop}} = \mathbb{E}_{x,\omega}[r_{\mathrm{stop}}(x,\omega)]. \tag*{(5)}

This uniform-depth oracle does not measure our controller’s sequence of per-step decisions. Identifying curve-adaptive success with observed policy success requires conditional independence of selected depth and outcomes given the instance. The common-fixed-depth population gap lower-bounds the curve-based oracle gap only when both use the same adaptive term. Selecting the fixed-depth reference after evaluation can bias the measured gap upward. TD-MPC2 returns use a different scale and are excluded from this formalism.

A separate descriptive gap concerns training coverage (§ 5.3, including the distinct matched controls). Let SmatchS_{\mathrm{match}} be success of the volume-matched frozen reference, SholeS_{\mathrm{hole}} success of the hole model, and SonlineS_{\mathrm{online}} success after adaptation under the same region-goal protocol. Then

Ghole=Smatch−Shole,ρrecovered=Sonline−SholeGhole.(6)G_{\mathrm{hole}} = S_{\mathrm{match}} - S_{\mathrm{hole}}, \qquad\rho_{\mathrm{recovered}} = \frac{S_{\mathrm{online}} - S_{\mathrm{hole}}}{G_{\mathrm{hole}}}. \tag*{(6)}

The ratio depends on the adaptation budget and the chosen reference. It may be negative or exceed unity, and becomes unstable when the reference gap is close to zero. It describes the recovered success gap without bounding other sources of model error. The factorial and coverage comparisons use separate online evaluations under the same condition label. We retain the seed pairing within each comparison.

Appendix § 12 constructs identical observable cost traces with incompatible optimal depths, giving positive worst-case regret for deterministic trace-only rules and a separate randomized bound. Exact equality is a construction assumption, not established for Cube or Wall. The tested detectors also have limited success. Their results leave open whether other stopping signals can identify model exploitation.

Fixed-depth success and adaptive halting plotted against mean refinement depth for TwoRoom, Push1, Cube, and Reacher, with fixed-depth and adaptive methods

Figure 3. Fixed-depth success and adaptive halting plotted against mean refinement depth. Figure 2 shows the planning procedure.

Post-peak Loss

The success curve supplies a precise distinction between harmless extra computation and damaging over-optimization. Under the appendix’s weakly unimodal shape assumption (nondecreasing before the earliest peak and nonincreasing afterward), let dp(x)d_p(x) be the earliest maximizing depth. Define the pre-peak shortfall and post-peak decline by

G(x)=s(dp(x),x)−s(nmin⁡,x),D(x)=s(dp(x),x)−s(N,x).(7)G(x) = s(d_p(x), x) - s(n_{\min}, x), \qquad D(x) = s(d_p(x), x) - s(N, x). \tag*{(7)}

Both are nonnegative, but they describe different errors. An early stop can forfeit progress before the optimum. A late stop can hurt only to the extent that success declines afterward. Monotonicity on each side of the earliest peak gives the pointwise inequality

rstop(x,ω)≤G(x)1[d~(x,ω)<dp(x)]+D(x)1[d~(x,ω)>dp(x)].(8)r_{\mathrm{stop}}(x,\omega) \le G(x)\mathbf{1}[\tilde{d}(x,\omega) < d_p(x)] + D(x)\mathbf{1}[\tilde{d}(x,\omega) > d_p(x)]. \tag*{(8)}

Taking expectations gives Appendix F’s early/late bound. A conservative rule can waste iterations on a plateau yet preserve success. The same delay can incur loss on a declining curve. This explanation assumes a success curve for each instance. The measured aggregate curves do not determine the episode regret of a controller that changes depth at every step.

Scope and limitations

The normalized-weight and detector comparisons include small-seed and incomplete grids. They describe the tested configurations. They do not establish a calibrated uncertainty measure or replicate every contrast across planner families.

The experiments use near-fully-observed tasks and short receding-horizon control with a moderate-scale world model. In partially observed tasks, information can improve the belief state without updating predictor parameters. That benefit remains untested here. Online adaptation uses a particular update procedure and coverage manipulation. Behavioral evaluation samples may overlap with training data. The sham control preserves anchor supervision but can change gradient norms. We share thresholds within each planner family without a separate held-out selection protocol. These choices limit transfer to other data distributions, update rules, and planning horizons.

Supplementary Results

These figures supplement the analyses of compute allocation, disagreement weighting, and over-optimization. Captions specify the evaluation protocol for each experiment.

Checkpoint Comparisons

Cross-environment contrasts conflate training data with task differences. We therefore compare checkpoints within a training trajectory for TwoRoom and Reacher. The environment and planner

Figure 4. Elite-cost trajectories and allocated depth. Left, example trajectories show rapid early descent and a slower-descending case that receives more iterations. Red markers indicate halting. Right, initial elite cost correlates with depth on PushT (r≈0.3r\approx0.3). Initial cost is a model-based difficulty proxy. The correlation does not use an independently measured difficulty label.

Disagreement-weight sweep

Figure 5. Disagreement-weight sweep with 100 seeds per cell and 50 episodes per seed. (a) Mean success against β\beta on a symlog axis. Larger positive weights reduce success in every environment. Some negative weights give small gains on Reacher, PushT, and Cube. Strong penalties harm TwoRoom and PushT. (b) Mean paired differences from β=0\beta=0 (Table 16). The gains depend on the environment and weight. Coverage is not independently varied.

DINO-WM depth curves

Figure 6. DINO-WM depth curves with five seeds and a cap of 30. Wall’s mean success peaks at 0.576 at iteration 11. It falls to 0.356 at iteration 30, a 38.2% relative decline. The peak is selected from the mean curve at a common depth across seeds. Cost-only halting with ε=10−2\varepsilon= 10^{-2} attains 0.364 success at mean depth 29.04. The fixed-depth K=5K = 5 ensemble attains 0.508. The ensemble uses more stochastic rollouts at a fixed depth.

Candidate preference and compute allocation

Figure 8. Candidate preference and compute allocation. Positive β\beta favors terminal rollout spread and negative β\beta penalizes it. Highlighted bundles show candidate selection. Penalties can help or hurt task success. Cost-only halting (β=0\beta=0) controls refinement depth. Disagreement is an uncalibrated surrogate. Schematic.

protocol remain fixed. Each checkpoint has a held-out one-step prediction error and evaluations of fixed-depth and adaptive planning (Table 7). Training time and model state vary together. Coverage is not independently manipulated.

Table 7: Within-environment checkpoint comparison. Per-epoch LeWM checkpoints, 50 episodes per cell. Probe MSE, β=0\beta=0/β=−10\beta=-10 success, and adaptive halting. See Figure 7 for visualization.

EnvCkptProbe MSEβ=0\beta=0β=−10\beta=-10adaCostadaEFE
TwoRoome12.16148.048.845.342.7
TwoRoome20.23288.088.087.389.3
TwoRoome30.15790.088.790.792.0
TwoRoome50.15888.090.090.090.7
TwoRoome80.07989.388.789.389.3
TwoRoome120.03787.384.785.387.3
TwoRoome180.01484.784.084.786.0
TwoRoome250.00682.084.086.086.7
TwoRoome300.00486.084.086.087.3
Reachere10.13735.239.644.038.0
Reachere20.05550.852.846.744.0
Reachere30.19442.849.642.044.0
Reachere50.05262.864.462.060.7
Reachere80.02878.475.670.776.0
Reachere120.02380.885.685.378.7
Reachere180.02084.485.680.779.3
Reachere250.01982.087.682.780.0
Reachere300.01881.285.282.781.3

Table 7.

Checkpoint quality and uncertainty penalties

Figure 7. Checkpoint quality and uncertainty penalties. Fixed-depth success at β=0\beta=0 and β=−10\beta=-10 is plotted against held-out one-step MSE on a log axis. Means are listed in Table 7. TwoRoom’s curves remain close across the measured range. Reacher has positive contrasts at several checkpoints and a negative contrast at e8. Coverage is not independently varied. Stars mark released-model anchors with fewer seeds than the separate n=100n=100 weight sweep.

The penalty’s effect changes with the checkpoint. TwoRoom’s fixed-depth success curves stay close as probe error falls, and their difference changes sign. Reacher gains at several checkpoints but loses success at e8. Its largest gain coincides with the e3 increase in probe error. Training time and model state change together, so this association does not identify either probe error or coverage as the cause.

The relative performance of the stopping rules also changes during training. At Reacher e8, epistemic-aware halting exceeds cost-only halting but remains below fixed-depth planning. At e12, cost-only halting has the higher mean. These reversals argue against treating epistemic-aware halting as a consistent improvement.

Candidate Selection

The diagram shows how disagreement changes candidate selection and how a separate stopping rule controls refinement.

Full Results Tables

The tables report checkpoint, transfer, and adaptation experiments. Each experiment retains its own seed count and evaluation protocol. Trace measurements and checkpoint sweeps use separate evaluations from the disagreement-weight comparison.

Regret Analysis

Formal setup

The analysis uses the uniform-depth episode idealization of Appendix § 7.§ 2. A single depth is applied at every control step. The evaluated controller instead chooses a sequence of depths. The bounds therefore describe success curves rather than its episode-policy regret.

All depths are integer-valued. Fix a refinement budget NN and minimum depth nmin⁡n_{\min}. A fixed-depth policy uses dd iterations for every instance. An adaptive rule chooses d^(x,ω)\hat{d}(x,\omega) for instance xx and planner randomness ω\omega, with nmin⁡≤d^≤Nn_{\min} \le\hat{d} \le N. We suppress ω\omega in the notation below. Expectations and probabilities include this randomness unless the rule is deterministic. Let s(d,x)∈[0,1]s(d,x) \in[0,1] denote success probability at fixed depth dd. Let ct(x)c_t(x) denote the elite planning cost at iteration tt. The controller observes cost trajectories but cannot observe ss at deployment.

Per-instance stopping regret and its average over a test distribution D\mathcal{D} are

rstop(x)=s(d∗(x),x)−s(d^(x),x),Rstop=Ex∼D,ω[rstop(x)],r_{\mathrm{stop}}(x) = s(d^*(x),x) - s(\hat{d}(x),x), \qquad\mathcal{R}_{\mathrm{stop}} = \mathbb{E}_{x\sim\mathcal{D},\omega}[r_{\mathrm{stop}}(x)],

where d∗(x)d^*(x) is the smallest maximizer of s(d,x)s(d,x) over allowed integer depths.

Relation to adaptive execution. The selected depth can depend on the solver’s random trajectory. Under conditional independence of selected depth and potential outcomes given xx, the adaptive success probability equals Ex,ω[s(d^(x),x)]\mathbb{E}_{x,\omega}[s(\hat{d}(x),x)]. Without that condition, selecting depth can also select outcomes. This effect is distinct from allocating different depths to different instances. Instance-dependent allocation alone can outperform a common fixed depth. The curve-based regret is non-negative by definition. The Δ\Delta column in Table 10 instead compares marginal means with a common fixed-depth reference. That descriptive difference can be negative.

Unimodal depth structure. Say s(⋅,x)s(\cdot,x) is weakly unimodal with onset dp(x)=min⁡{d∈[nmin⁡,N]:s(d,x)=max⁡d′∈[nmin⁡,N]∩Zs(d′,x)}d_p(x) = \min\{d \in[n_{\min},N] : s(d,x) = \max_{d' \in[n_{\min},N]\cap\mathbb{Z}} s(d',x)\} if it is nondecreasing on [nmin⁡,dp(x)][n_{\min},d_p(x)] and nonincreasing on [dp(x),N][d_p(x),N]. Selecting the earliest maximizer also defines early and late stopping when the optimum is flat. Then d∗(x)=dp(x)d^*(x) = d_p(x). Define the pre-plateau shortfall and post-plateau decline

G(x)=s(dp(x),x)−s(nmin⁡,x)≥0,D(x)=s(dp(x),x)−s(N,x)≥0.G(x) = s(d_p(x),x) - s(n_{\min},x) \ge0, \qquad D(x) = s(d_p(x),x) - s(N,x) \ge0.

D(x)=0D(x) = 0 for a flat post-peak plateau. A positive D(x)D(x) permits success loss from late stopping.

Proposition 1. Stopping regret under unimodal structure. Let qe=Pr⁡x,ω[d^(x)<dp(x)]q_e = \Pr_{x,\omega}[\hat{d}(x) < d_p(x)] and qℓ=Pr⁡x,ω[d^(x)>dp(x)]q_\ell= \Pr_{x,\omega}[\hat{d}(x) > d_p(x)] be the early- and late-halt probabilities of an adaptive rule. If s(⋅,x)s(\cdot,x) is weakly unimodal for every xx, then

Rstop≤qe⋅E[G(x)∣d^<dp]+qℓ⋅E[D(x)∣d^>dp].(9)\mathcal{R}_{\mathrm{stop}} \le q_e \cdot\mathbb{E}\left[G(x)\mid\hat{d}<d_p\right] + q_\ell\cdot\mathbb{E}\left[D(x)\mid\hat{d}>d_p\right]. \tag*{(9)}

Proof. Condition on the halt event. If d^(x)=dp(x)\hat{d}(x) = d_p(x) the regret is zero. If d^(x)<dp(x)\hat{d}(x) < d_p(x), monotonicity on [nmin⁡,dp][n_{\min},d_p] gives s(d^,x)≥s(nmin⁡,x)s(\hat{d},x) \ge s(n_{\min},x), so rstop(x)≤s(dp,x)−s(nmin⁡,x)=G(x)r_{\mathrm{stop}}(x) \le s(d_p,x) - s(n_{\min},x) = G(x). If d^(x)>dp(x)\hat{d}(x) > d_p(x), monotonicity on [dp,N][d_p,N] gives s(d^,x)≥s(N,x)s(\hat{d},x) \ge s(N,x), so rstop(x)≤D(x)r_{\mathrm{stop}}(x) \le D(x). Taking expectations and applying the law of total expectation yields Eq. (9), with a zero-probability event contributing zero (equivalently use expectations of GG and DD times their event indicators). □

The bound separates loss from early stopping and post-peak decline. If a rule never stops early and D(x)=0D(x)=0 for every instance, its curve-based regret is zero. Otherwise the bound depends on the stopping probabilities and conditional shortfalls. These quantities depend on the instance distribution.

We do not estimate qeq_e or qℓq_\ell, which require per-instance oracle depths. The measured grids use a small set of fixed depths, with some cells evaluated at a single seed. They show aggregate plateaus or declines but do not establish unimodality for every instance. In particular, a best-fixed-minus-adaptive success gap does not estimate D(x)D(x). The Cube and Wall curves illustrate declining aggregate success after a peak. They do not supply the per-instance quantities needed to evaluate the bound.

Cost-based Stopping Limit

Proposition 1 bounds regret in terms of qeq_e, qℓq_\ell, which depend on the rule’s ability to locate dp(x)d_p(x). But the rule sees only the cost trajectory. Call a halting rule cost-internal if d^(x)\hat{d}(x) is a measurable function of {ct(x)}t≤N\{c_t(x)\}_{t\leq N} alone, where ctc_t is the pragmatic elite cost minimized by the primary controller at β=0\beta=0. Our Δ\DeltaEFE rule and its patience and threshold variants are cost-internal. The disagreement guards in Appendix § 14.§ 1 use additional information.

Write x∼x′x\sim x' when ct(x)=ct(x′)c_t(x)=c_t(x') for all t≤Nt\leq N. Such instances are indistinguishable to a cost-internal rule. The following construction gives a worst-case lower bound within this class.

Proposition 1. Positive worst-case regret for cost-internal halting. Fix δ,δ′∈(0,1]\delta,\delta'\in(0,1] and choose v∈[0,1−max⁡{δ,δ′}]v\in[0,1-\max\{\delta,\delta'\}]. Fix depths nmin⁡≤a<b≤Nn_{\min}\leq a<b\leq N. There exists a pair x∼x′x\sim x' of weakly unimodal instances with

s(d,x)={vd<av+δa≤d<bv+δ−δ0d≥bs(d,x′)={vd<bv+δ′d≥bs(d,x)= \begin{cases} v & d<a \\ v+\delta& a\leq d<b \\ v+\delta-\delta_0 & d\geq b \end{cases} \qquad s(d,x')= \begin{cases} v & d<b \\ v+\delta' & d\geq b \end{cases}

(any 0<δ0≤δ0<\delta_0\leq\delta) such that every deterministic cost-internal rule satisfies

max⁡{rstop(x),rstop(x′)}≥min⁡{δ0,δ′}>0.(10)\max\{r_{\mathrm{stop}}(x),r_{\mathrm{stop}}(x')\}\geq\min\{\delta_0,\delta'\}>0. \tag*{(10)}

Proof. A cost-internal rule reads only {ct}\{c_t\}, which is identical on xx and x′x'. It therefore chooses the same depth d^\hat{d} for both. The earliest oracle depths are d∗(x)=ad^*(x)=a and d∗(x′)=bd^*(x')=b. If d^<b\hat{d}<b then on x′x' the rule attains vv against an oracle value v+δ′v+\delta', so rstop(x′)≥δ′r_{\mathrm{stop}}(x')\geq\delta'. If d^≥b\hat{d}\geq b then on xx it attains v+δ−δ0v+\delta-\delta_0 against oracle v+δv+\delta, so rstop(x)≥δ0r_{\mathrm{stop}}(x)\geq\delta_0. No d^\hat{d} avoids both. □

Proposition 2 gives a worst-case construction rather than a lower bound for every instance. Flat success curves can have zero regret at every depth. The construction shows that cost traces alone cannot guarantee optimal depth without a link between cost and external success.

Exact equality of cost traces is an assumption of the construction. The Cube and Wall experiments do not establish such an indistinguishable pair. Similar traces also need not yield similar decisions because threshold rules can be discontinuous. An approximate version would require a bound on observational distinguishability. For randomized rules, the corresponding worst-case expected regret is at least δ0δ′/(δ0+δ′)\delta_0\delta'/(\delta_0+\delta'). This follows by balancing the probabilities of stopping before and after bb.

Disagreement is an additional observation, so Equation (10) does not cover guards that use it. The fixed-depth ensemble exceeds cost-only halting in four of five Wall seeds. That comparison changes candidate scoring and computation, rather than testing disagreement-based stopping. Appendix § 14.§ 1 evaluates such stopping signals separately.

Reference-based coverage deficit

Let θH\theta_H be a model trained with a spatial coverage hole. Its coverage deficit compares success with a volume-matched reference under the same region-goal protocol.

Two controls. The volume-matched model θV\theta_V removes the same number of transitions at random. It is evaluated at the same training epoch as the hole model. This comparison holds dataset size fixed and changes spatial coverage. The success-matched model instead uses an earlier random-drop checkpoint. Its frozen success equals the hole e3 model’s success (Table 4). Online adaptation changes its success by −2.4 points. Thus low initial success alone does not predict an adaptation benefit in this control. The deficit below uses the volume-matched reference.

Writing sfz(θ)s^{\mathrm{fz}}(\theta) for frozen success on region goals,

Rmodel=sfz(θV)−sfz(θH),Rmodelpost=Rmodel−ΔTTT,(11)\mathcal{R}_{\mathrm{model}} = s^{\mathrm{fz}}(\theta_V) - s^{\mathrm{fz}}(\theta_H), \qquad\mathcal{R}_{\mathrm{model}}^{\mathrm{post}} = \mathcal{R}_{\mathrm{model}} - \Delta_{\mathrm{TTT}}, \tag*{(11)}

where ΔTTT\Delta_{\mathrm{TTT}} is the frozen-to-online improvement on the hole model. At the e3 checkpoint Rmodel=50.0\mathcal{R}_{\mathrm{model}} = 50.0 pts and ΔTTT=18.8\Delta_{\mathrm{TTT}} = 18.8 pts, leaving Rmodelpost=31.2\mathcal{R}_{\mathrm{model}}^{\mathrm{post}} = 31.2 pts that test-time training does not recover. At e5 the deficit is 39.2 and the gain is 18.8, leaving 20.4 (Table 10).

The reference is one volume-matched model. (11) therefore measures a descriptive success gap. It does not bound the performance of other models or depth allocations.

Stopping and Adaptation

The factorial experiment tests whether stopping changes the effect of online adaptation. Write the cell means of the 2×22 \times2 factorial as μaθ\mu_{a\theta} for allocation a∈{fixed,adaptive}a \in\{\mathrm{fixed}, \mathrm{adaptive}\} and training θ∈{frozen,TTT}\theta\in\{\mathrm{frozen}, \mathrm{TTT}\}. The interaction contrast is

ι=(μada,TTT−μfix,TTT)−(μada,fz−μfix,fz),\iota= (\mu_{\mathrm{ada,TTT}} - \mu_{\mathrm{fix,TTT}}) - (\mu_{\mathrm{ada,fz}} - \mu_{\mathrm{fix,fz}}),

and an additive effect on success corresponds to ι=0\iota= 0. The mean interaction is +0.4 points. Within-seed interactions are {−6,−4,+4,0,+8}\{-6, -4, +4, 0, +8\} points. The interaction changes sign across seeds. The small mean does not establish independent effects. Adaptive stopping lowers success in both training conditions. Online adaptation improves success under both allocation rules.

Per-solve cost regret

The logged elite-cost trajectory shows whether refinement has converged under the model objective. For one optimizer call with halt iteration n^\hat{n}, define

rps=c^n^−min⁡1≤t≤n^ct≥0,r_{\mathrm{ps}} = \hat{c}_{\hat{n}} - \min_{1 \le t \le\hat{n}} c_t \ge0,

where ctc_t is the recorded elite-mean cost. This measures deterioration of that summary after its best recorded value. It is neither episode-success regret nor the evaluated cost of the returned mean action sequence. A monotonically decreasing cost trace has rps=0r_{\mathrm{ps}} = 0, even when external success decreases. The identical-cost construction in Proposition 2 permits this case.

The calculation uses the primary controller configuration. Every optimizer call with a complete cost trajectory is included, regardless of episode success. Calls with only endpoint costs cannot provide the trajectory minimum. Table 11 counts these calls as missing.

TwoRoomReacherPushTCube
Seeds with full traces1, 2, 3, 41, 23, 41, 2
Calls with full traces241156124142
Calls without full traces628119665
Mean rpsr_{\mathrm{ps}}6.3952.7230.0750.042
Calls with positive denominator239151124141
Mean r~ps\widetilde{r}_{\mathrm{ps}}0.1780.2470.0070.015
Zero-denominator calls2501

Table 11. Primary-controller elite-summary regret. Only complete trajectories in the listed seed subsets contribute to the cost averages. All calls are counted regardless of episode outcome. The normalized row uses iteration 3 as its reference and excludes eight zero denominators, seven with positive raw regret. The trace subset differs from the full-seed episode-success comparison.

For scale context we also compute

r~ps=rpsc3−min⁡1≤t≤n^ct\widetilde{r}_{\mathrm{ps}} = \frac{r_{\mathrm{ps}}}{c_3 - \min_{1 \leq t \leq\hat{n}} c_t}

when the denominator is positive, using the controller’s minimum allowed depth. Calls with zero denominator remain in the raw average and are excluded from the normalized average. Most excluded calls have positive raw regret, so the normalized average describes a different subset of calls. The ratio can exceed one. Calculations use the logger’s four-decimal cost records and give equal weight to each eligible optimizer call.

Cost scales, seed subsets, and optimizer-call counts differ across environments. The table describes internal convergence on the available complete trajectories. It does not estimate a causal effect of stopping or a paired relationship with episode success.

Online Adaptation

Online adaptation improves the hole model in independently trained samples, but the frozen disagreement penalty changes sign between them. Table 4 reports the second training sample. Table 12 adds the first sample and the remaining controls. The hole-model gain is positive at every evaluation seed in the first sample. The volume-matched and full-data models gain less.

TwoRoom, sample 1FrozenOnlineChange
Coverage hole54.769.3+14.7
Volume control66.070.7+4.7
Full data74.075.3+1.3
Frozen penalty contrastSample 1Sample 2
Coverage hole+20.0−4.0
ReacherOnline changeInteraction
Sample 1e3−6.8+3.2
Sample 1e5+2.4−3.2
Sample 2e3−4.0·
Sample 2e5+0.3·
Controls without a holeFrozen successChange
Full datae1·+9.6
Random drope1·+6.8
Random drope2·−2.4
Full datae270.8+5.6

Table 12. Additional online-adaptation results. Success and changes are percentages and percentage points, respectively. Paired changes are reported separately from rounded success means. Sample 1 uses three paired seeds per TwoRoom condition. Its hole-model gain is positive in all three. Sample 1 Reacher uses five paired seeds, with four negative changes at e3. The separate sample 2 volume-matched references have frozen success 87.2 at e3 and 83.2 at e5. On collected Reacher transitions, prediction error decreases from approximately 1.1 to 0.63 under adaptation, and anchor error stays flat.

The success-matched reference uses an earlier random-drop checkpoint with the same frozen success as the hole model. Its success falls after adaptation. The volume-matched references instead use the same training epochs as the hole models and have higher frozen success. The factorial experiment evaluates the hole condition separately from these coverage comparisons.

Reacher gives mixed results. Online adaptation lowers success at the earlier checkpoint in both training samples and gives a small gain at the later checkpoint. Its interaction with disagreement seeking also changes sign. Prediction error on collected transitions falls, but this improvement does not ensure better control.

Adaptation also helps some models trained without a coverage hole. Its benefit therefore depends on more than the presence of the hole. The comparison with shuffled-target updates tests whether pairing each collected transition with its correct target matters.

Larger penalty gains coincide with increases in probe error during each Reacher training run (Table 13). The forward test is weaker. Its mean difference is positive, but the gain does not hold across seeds and it misses the prespecified criterion. The forward test therefore does not confirm that probe error predicts when the penalty helps.

Training sampleCheckpointProbe MSEPenalty change
1e30.194+6.8
2e50.122+8.0
2e80.035+2.0
2e2, forward test0.131+3.2

Table 13. Reacher probe error and frozen-penalty changes in success. Differences are percentage points. The sample 2 forward test at e2 has two positive seeds out of five and misses its prespecified criterion.

Stopping Rules

Success and refinement depth for the stopping rules across TwoRoom, PushT, Cube, and Reacher

Figure 9. Success and refinement depth for the stopping rules. Each signal is evaluated across its threshold grid. The gray line shows fixed-depth success and the black star marks the primary controller. Cost-based rules follow the plateau with different depth allocations. Geometric rules are sensitive to thresholds. No evaluated rule reliably recovers Cube’s earlier success peak.

The primary controller uses seeds {42,1,2,3,4}\{42, 1, 2, 3, 4\} on TwoRoom and PushT. It uses {42,1,2}\{42, 1, 2\} on Cube and Reacher. Fixed-reference seeds are {1,2,3}\{1, 2, 3\} on TwoRoom, {42,1,2,3,4}\{42, 1, 2, 3, 4\} on PushT, and {42,1,2}\{42, 1, 2\} elsewhere. Best-elite, variance, and mean-change rules use {42,1,2}\{42, 1, 2\}. Their shared thresholds are 10−210^{-2}, 3×10−23 \times10^{-2}, and 10−110^{-1}, respectively. No-EMA uses {42,1,2}\{42, 1, 2\} on TwoRoom. Epistemic-aware halting uses {42,1,2,3}\{42, 1, 2, 3\} on Reacher and the elite-stability seed elsewhere. Elite stability uses the single seed listed in Table 14.

Halting signalTwoRoomPushTCubeReacher
Δ\DeltaEFE EMA-mean (primary)88.0<br>@ 22.087.2<br>@ 28.363.3<br>@ 28.174.7<br>@ 22.3
no-EMA (v1)87.3<br>@ 16.4·····
epistemic-aware (adaEFE)90.0<br>@ 24.996.0<br>@ 29.872.0<br>@ 29.278.5<br>@ 26.7
best-elite cost stall89.3<br>@ 9.387.3<br>@ 15.363.3<br>@ 15.874.0<br>@ 9.3
elite-variance stall88.0<br>@ 8.577.3<br>@ 20.966.0<br>@ 10.376.7<br>@ 19.0
mean-change stall88.7<br>@ 22.589.3<br>@ 17.965.3<br>@ 24.776.0<br>@ 26.2
elite-stability (fixed radius)depth 30.0 at thresholds {0.6,0.8,0.9}\{0.6, 0.8, 0.9\} (seed 42)
elite stability, t=1.0t = 1.086.0<br>@ 14.890.0<br>@ 23.770.0<br>@ 21.164.0<br>@ 21.2
best fixed depth90.0<br>N=30N = 3088.4<br>N=30N = 3067.3<br>N=10N = 1072.0<br>N=30N = 30

Table 14. Stopping-rule comparison. Success is the mean percentage across seeds. The second line in each adaptive cell gives mean depth. Each seed uses 50 episodes and cap 30. Seed sets and thresholds are specified below. Mean differences are descriptive where seed sets differ. Missing entries indicate unevaluated configurations.

All rules use the same solver, patience p=2p = 2, minimum depth nmin⁡=3n_{\min} = 3, and candidate counts. They differ in the statistic used to detect stalling. Variance and mean-change rules monitor the CEM sampling distribution. Best-elite stall monitors the minimum elite cost. Elite stability measures the fraction of current elites within an action-space radius 0.5σˉd0.5\bar{\sigma}\sqrt{d} of the previous elite set. This radius-based rule never triggers at the tested thresholds.

A self-calibrated alternative compares cross-set nearest-neighbor distance with within-set spacing. It stops when the median cross-set distance falls below tt times the median within-set distance. The single-seed grid changes abruptly with tt. At t=0.8t = 0.8 the rule reaches the cap, whereas at t=1.2t = 1.2 it stops at the three-iteration minimum. The intermediate setting t=1.0t = 1.0 gives the results in Table 14. These measurements do not establish a stable geometric stopping signal. Best-elite stall uses less depth than the primary rule, with nearby success means. The evaluated rules do not reliably recover Cube’s earlier success peak.

Disagreement guards and detector results

A disagreement guard stops when the elite disagreement exceeds γ\gamma times its running minimum. We add this guard to the primary controller and vary γ\gamma on Cube. A shared setting is also evaluated on the three plateau tasks (Table 15). The tightest Cube guard stops earlier and recovers part of the lost success. No tested setting reaches the earlier fixed-depth peak. The shared guard also stops prematurely on PushT, where success declines substantially. Thus the tested guard does not improve allocation consistently across tasks.

RuleCubeTwoRoomPushTReacher
fixed-10 (Cube peak)71.3···
fixed-3061.0···
primary, unguarded67.0 @ 28.3···
guard γ=1.15\gamma=1.1568.7 @ 18.1···
guard γ=1.3\gamma=1.364.0 @ 20.582.0 @ 12.469.3 @ 12.178.7 @ 21.4
guard γ=1.5\gamma=1.566.7 @ 22.3···

Table 15. Disagreement guards. Mean success is followed by mean depth. Each cell uses three seeds and 50 episodes per seed. The guard stops when disagreement exceeds γ\gamma times its running minimum. It supplements the primary controller. The plateau tasks use γ=1.3\gamma= 1.3. At these seeds, the unguarded adaptive reference reaches 86 on TwoRoom and 92 to 98 on PushT. Fixed-30 reaches 76 on Reacher. Missing entries indicate unevaluated configurations.

Four additional detectors use the recorded per-instance trajectories. Two monitor disagreement rise. Another compares imagined terminal cost with the realized outcome. The fourth compares disagreement with its early-iteration level. Their statistics and firing rates overlap across environments. These detectors do not reliably separate Cube’s over-optimization from the plateau tasks. The result limits this detector family without ruling out externally informed stopping signals.

Normalized Weights and Sensitivity

Negative disagreement weights have different effects across tasks (Table 16). Reacher, PushT, and Cube gain at some weights, whereas every tested penalty lowers mean success on TwoRoom. Larger positive weights lower success on all tasks. The weight comparison reuses sampled starts and goals across seeds. The dropout generator is not explicitly seeded, so imagined trajectories are not fully paired.

Envβ\betaSuccess (%)Δ\Delta (pp)
TwoRoom-5071.06−13.44-13.44
TwoRoom-2079.82−4.68-4.68
TwoRoom-1082.36−2.14-2.14
TwoRoom-383.12−1.38-1.38
TwoRoom084.50⋅\cdot
TwoRoom+376.42−8.08-8.08
TwoRoom+1063.88−20.62-20.62
TwoRoom+5048.40−36.10-36.10
PushT-5081.72−4.52-4.52
PushT-2086.40+0.16+0.16
PushT-1087.42+1.18+1.18
PushT-387.66+1.42+1.42
PushT086.24⋅\cdot
PushT+370.12−16.12-16.12
PushT+1031.58−54.66-54.66
PushT+508.18−78.06-78.06
Reacher-5081.48+3.34+3.34
Reacher-2080.52+2.38+2.38
Reacher-1079.00+0.86+0.86
Reacher-377.62−0.52-0.52
Reacher078.14⋅\cdot
Reacher+376.58−1.56-1.56
Reacher+1072.40−5.74-5.74
Reacher+5048.88−29.26-29.26
Cube-5068.96+1.32+1.32
Cube-2069.08+1.44+1.44
Cube-1069.52+1.88+1.88
Cube-368.66+1.02+1.02
Cube067.64⋅\cdot
Cube+367.58−0.06-0.06
Cube+1064.16−3.48-3.48
Cube+5054.66−12.98-12.98

Table 16. Disagreement-weight comparison with 100 paired seeds and 50 episodes per run. Success is the mean percentage across seeds. Δ\Delta is the mean difference from the same-seed β=0\beta= 0 reference. Every configuration uses 30 CEM iterations and K=4K = 4 stochastic rollouts.

Reacher’s gains at stronger penalties are smaller in the larger seed sample than in the earlier evaluation. Its weaker penalties give both positive and negative differences. The results therefore do not support a uniformly beneficial negative-weight range.

Repeated evaluations also change the measured success of a fixed configuration. Reacher’s initial evaluation was higher than every repetition. These repeated runs use the same evaluation seed and selected configurations, so they do not describe variation across training runs or all evaluation seeds.

We also scale disagreement by the current candidate-batch ratio of pragmatic and disagreement magnitudes. The dimensionless weight is β^∈{−1,−0.3,−0.1,+0.1,+0.3,+1}\hat{\beta} \in\{-1, -0.3, -0.1, +0.1, +0.3, +1\} (Figure 11). The

Scale-matched dose response by environment

Figure 11. Scale-matched dose response by environment. Seven normalized weights and 28 values use a common scale from 0 to 100%. Baseline annotations give mean success. Nonzero-weight cells use a single seed. Disagreement magnitude is matched to the current candidate batch. The intermediate positive weight reduces success by 20 to 52% relative on three environments. The intermediate negative weight improves Reacher success by approximately 11% relative.

Disagreement-weight results across four environments

Figure 10. Disagreement-weight results with 100 seeds per cell. (a) Mean success. (b) Mean paired differences from β=0\beta= 0. (c) Ratio of mean success at β=0\beta= 0 to +50+50, reaching 10.54× on PushT. Each seed uses 50 episodes, giving 160,000 episode evaluations across the weight grid. Reacher’s β=−20,−50\beta= -20, -50 differences are +2.38,+3.34+2.38, +3.34 pp, compared with +4.4+4.4 pp at each weight in the earlier 20-seed sample. In a separate set of repeated evaluations, Reacher success was 90% initially and 80 to 82% in five repetitions.

intermediate positive weight lowers success on most environments. The largest positive weight gives zero observed success on PushT. The intermediate negative weight improves mean success on Reacher. Nonzero-weight cells use a single seed. The sweep suggests that the positive-weight loss can persist after magnitude matching, but does not establish robustness across seeds.

Changing the ensemble size has little effect on Reacher success at the selected negative weight (Table 17). The largest tested dropout rate lowers success. The cells share an evaluation seed, but the separate multi-seed baseline does not give a paired estimate of the gain. Dropout changes the imagined dynamics as well as the disagreement signal.

K=2K=2K=4K=4K=8K=8p=0.05p=0.05p=0.1p=0.1p=0.2p=0.2
Success808082768064

Table 17. Proxy sensitivity on Reacher at β^=−0.3\hat{\beta}=-0.3. The ensemble-size sweep fixes dropout at p=0.1p=0.1. The dropout sweep fixes K=4K=4. Each cell uses seed 42. The separate multi-seed β=0\beta=0 baseline is 72.0.

Candidate values in TD-MPC2 can share a large common offset. Their magnitude can therefore exceed the variation that determines candidate ranking. That variation also contracts as MPPI converges (Table 18). Magnitude matching can consequently overweight disagreement for either sign of β^\hat{\beta}. We instead normalize by the per-task spread of candidate values. The available columns show smaller return losses for negative than positive weights. Their task subsets differ, so they do not provide a paired comparison across doses. Candidate-value spread is a useful scale to consider when transporting the weight between planner families.

β^\hat{\beta} (spread-normalized)−0.3-0.3−1-1+0.3+0.3+1+1
Return, % of β=0\beta=099.2%96.5%32.9%5.9%
nn tasks complete2101110

Table 18. Spread-normalized weights on TD-MPC2. Raw β\beta equals β^\hat{\beta} times per-task candidate-value spread divided by disagreement scale. Depth is fixed at 6. Return is a percentage of the β=0\beta=0 baseline. Each column covers nn of 19 tasks, with different task subsets. Value magnitude is approximately 90×90\times candidate spread. The spread contracts approximately 70×70\times over six MPPI iterations. Dog-walk reaches 117.2% at β^=−1\hat{\beta}=-1 in one 10-episode evaluation.

References

  1. [1]Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.
  2. [2]Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl Friston, and Ryan Smith. Reward maximization through discrete active inference. Neural Computation, 35(5):807–852, 2023.
  3. [3]Zafeirios Fountas, Noor Sajid, Pedro Mediano, and Karl Friston. Deep active inference agents using Monte-Carlo methods. In NeurIPS, 2020.arxiv.org/abs/2006.04176
  4. [4]Karl Friston. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11 (2):127–138, 2010.DOI
  5. [5]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025.arxiv.org/abs/2502.05171
  6. [6]Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016.arxiv.org/abs/1603.08983
  7. [7]Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, 2019.arxiv.org/abs/1811.04551
  8. [8]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.arxiv.org/abs/2301.04104
  9. [9]Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In ICLR, 2024.arxiv.org/abs/2310.16828
  10. [10]Nicklas A. Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In ICML, 2022.arxiv.org/abs/2203.04955
  11. [11]Alexia Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks. arXiv preprint arXiv:2510.04871, 2025.
  12. [12]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In NeurIPS, 2020.arxiv.org/abs/2006.04779
  13. [13]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. LeWorld-Model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026.arxiv.org/abs/2603.19312
  14. [14]Pietro Mazzaglia, Tim Verbelen, and Bart Dhoedt. Contrastive active inference. In NeurIPS, 2021.arxiv.org/abs/2110.10083
  15. [15]Pietro Mazzaglia, Tim Verbelen, Ozan Çatal, and Bart Dhoedt. The free energy principle for perception and action: A deep learning perspective. Entropy, 24(2):301, 2022.DOI
  16. [16]Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Whence the expected free energy? Neural Computation, 33(2):447–482, 2021.DOI
  17. [17]Thomas Parr, Giovanni Pezzulo, and Karl J Friston. Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. MIT Press, 2022.
  18. [18]Reuven Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1:127–190, 1999.DOI
  19. [19]Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML, 2020.arxiv.org/abs/2005.05960
  20. [20]Kai Ueltzhöffer. Deep active inference. Biological Cybernetics, 112(6):547–573, 2018.DOI
  21. [21]Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734, 2025.arxiv.org/abs/2506.21734
  22. [22]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In NeurIPS, 2020.arxiv.org/abs/2005.13239
  23. [23]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983, 2024.arxiv.org/abs/2411.04983

Paper details

vlnc:W_4azh8s6w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents