betaFeedback

WorksComputer Science

Risk-Triggered Search: Behavior-Preserving Safe Control

Abstract

Pre-trained visual world models generate multi-step action plans from current and goal observations, yet terminal-proximity objectives fail to prevent forbidden-zone traversal or rigid-body collision sliding. Purely visual interfaces may lack analytical dynamics, structured states, or explicit collision geometry, requiring internal prediction of action-conditioned process risk. We propose Risk-Triggered Search (RTS) for frozen Joint-Embedding Predictive Architecture (JEPA) world models. Rather than relying on static state features that cannot distinguish candidate plans sharing an observation, RTS unrolls action-conditioned future latent prefixes in the frozen world model and trains a lightweight risk head with process-violation supervision. To intervene selectively while preserving safe nominal behavior, RTS uses the same predicted prefix-risk criterion as both the search trigger gate and hard feasibility gate: safe nominal plans execute directly, whereas alarms trigger local Cross-Entropy Method (CEM) search to minimize nominal deviation within the feasible set. To prevent isotropic Euclidean minimal modification from shrinking obstacle-directed actions into zero-amplitude stalling, we introduce an amplitude-preserving metric dAPd_{\mathrm{AP}} that retains tangential progression while maintaining a unique zero at the nominal plan. Under unified gating and this unique-zero condition, bypassing safe-state search yields closed-loop trajectories strictly identical to every-step same-kernel search. Evaluated under a process-safety benchmark and Safe Arrival Rate (SAR) metric across five tasks, RTS improves SAR over Direct by 12.67–41.67 percentage points on navigation and articulated control and +2.08/+4.17 pp on Cube/PushT boundary cohorts, maintaining 100% action identity with every-step search while cutting controller time by 17.71%–70.41% across 1,968 paired trajectories.

Introduction

Does reaching the goal equate to task success? Given current and goal observations, a visual controller must not only reach the target but also avoid hazardous intermediate transitions. As illustrated in Figure 1, a goal-reaching trajectory can contain obstacle-directed commands. In navigation, rigid-body collision projection truncates their normal displacement and permits sliding toward the goal; similarly, a manipulator can reach its target along a joint path that crosses forbidden zones. Terminal metrics count these trajectories as successful despite their hazardous intermediate behavior.

Schematic of risk-triggered correction

Figure 1. Schematic of risk-triggered correction. Direct/Guarded A issue obstacle-directed commands (red), RTS-MSE (mean squared error) stalls (gray), and RTS-AP preserves tangential progression (green). Safe nominal plans pass directly (blue, bt=0b_t = 0); risk alarms trigger search (bt=1b_t = 1).

This conflict is central to closed-loop control with pre-trained visual world models. Joint-Embedding Predictive Architecture (JEPA) models [21, 22, 30] predict future representation dynamics in a compact latent space. Building on these latent dynamics, INTACT [31] maps current and goal observations to multi-step action plans for search-free control (Direct), and supports local sampling around the nominal plan (Guarded A). At deployment, the controller operates solely on RGB observations and action histories without analytical dynamics, structured physical states, or explicit collision geometry, while its terminal-proximity objective places no constraint on intermediate execution—requiring an internal process-risk model to guide safety intervention.

Existing safety paradigms face structural trade-offs in this visual interface. Control Barrier Functions (CBFs) and Predictive Safety Filters [2, 33, 34, 36] minimally modify nominal inputs using explicit or identified dynamics unavailable in latent visual control. Multi-objective safety fine-tuning [3, 18] couples task and safety objectives, inducing gradient interference and representation drift [20, 37]. Recovery RL [32] switches between task and recovery policies using a safety critic, while LS3 [35] plans with a learned terminal safe-set constraint. Visual test-time search preserves the backbone, yet every-step CEM incurs heavy latency [11, 22, 24], whereas DreamAvoid [10] triggers rollouts via a current-state phase classifier and mixes task return into final selection—altering safe nominal plans.

Overcoming these limitations requires exploiting action-conditioned future prediction in visual world models. From the same observation, one candidate action sequence may cross an obstacle while another steers safely around it. A static visual representation assigns identical features to both plans despite their divergent outcomes. Unrolling each candidate through the frozen dynamics produces a distinct sequence of predicted latents that exposes where their futures diverge. Because pre-trained latents encode dynamics without explicit safety semantics, two coupled challenges arise: how to extract action-conditioned process risk from predicted latent prefixes, and how to convert that risk into a minimal correction that progresses toward the goal while preserving safe nominal behavior.

We propose Risk-Triggered Search (RTS) for frozen visual world models (Figure 2). RTS freezes the visual encoder, direct policy, and JEPA dynamics, and trains a lightweight risk head on action-conditioned latent prefixes using process-violation supervision. On unseen PointMaze maps, a frozen position probe pairs predicted latents with local visual risk-field sampling. For intervention, the same prefix-risk criterion serves as both the search trigger and hard feasibility gate: safe nominal plans execute directly, whereas alarms launch 3-round CEM search to select the feasible plan closest to the nominal one. Because Euclidean minimal modification shrinks obstacle-directed actions into zero-amplitude stalling (Figure 1), our amplitude-preserving distance dAPd_{\mathrm{AP}} penalizes step-magnitude shrinkage to favor tangential progression while retaining a unique zero at the nominal plan. Under unified gating and this unique-zero property, bypassing safe-state search preserves the exact closed-loop behavior of the full search operator (Proposition 1). RTS-Repair extends this framework to contact manipulation via component-wise risk and structured hybrid proposals.

Figure 2. Overall RTS architecture. Frozen policy π\pi generates nominal plan UDU_D, and world model FF unrolls predicted latent prefixes z^0:H−1\hat{z}_{0:H-1} scored by risk head RϕR_\phi. Safe plans (bt=0b_t = 0) pass directly; alarms (bt=1b_t = 1) launch 3-round CEM search using the same risk criterion to define feasibility FtF_t, where amplitude-preserving selection minimizes dAPd_{\mathrm{AP}}.

We evaluate five benchmarks under a controller-independent process audit and Safe Arrival Rate (SAR). RTS-AP improves SAR over Direct by 12.67, 41.67, and 27.34 percentage points on TwoRoom, Reacher, and PointMaze; RTS-Repair attains 97.89% Cube SAR (+2.08/+4.17 pp on Cube/PushT boundary cohorts). Controlled ablations validate prefix prediction, spatial addressing, and dAPd_{\mathrm{AP}} selection, while 1,968 paired trajectories verify 100% action identity with 17.71%–70.41% less controller time. Our contributions are:

  • Action-Conditioned Risk and SAR Evaluation. We predict process risk from frozen latent prefixes and local visual geometry, and evaluate safe arrival by auditing commanded paths and physical states.

  • Behavior-Preserving RTS and AP Selection. Unified trigger/feasibility gating and a unique-zero metric preserve safe nominal behavior (Proposition 1), while dAPd_{\mathrm{AP}} promotes tangential progress near obstacles.

  • Controlled Multi-Benchmark Evaluation. Five-task comparisons, matched ablations, 1,968 paired runs, and budget analyses validate both safety gains and lossless runtime reductions.

Related Work

Latent World Models and Visual Control

Early visual world models learned dynamics via pixel reconstruction (World Models [12], PlaNet [13], Visual Foresight [9], Dreamer [14, 15]) or discrete tokens (IRIS [23]), while TD-MPC [17] and TD-MPC2 [16] jointly optimize latent dynamics and value functions. To avoid decoding background details, JEPA [21] uses VICReg [6] or self-supervised backbones (I-JEPA [4], DINOv2 [25], V-JEPA [5, 7]) to predict in compact latent spaces, including DINO-WM [39], PLDM [30], and LeWorldModel (LeWM) [22] with SIGReg regularization. Causal-JEPA [24] and Fast-LeWM [11] reduce CEM latency [26, 28] via object masking and parallel prefix prediction, yet still require every-step search. INTACT [31] builds an Isomorphic Intent-to-Action space atop LeWM for search-free control (Direct) and terminal-cost search (Guarded A). Keeping this backbone frozen, we exploit its action-conditioned latent predictions to extract future process risk for selective safety intervention.

Predictive Safety Filtering and Recovery Control

Minimal-intervention safety filtering is well studied in control and safe RL [1, 8, 27]. CBFs [2] and Predictive Safety Filters [33, 34] project nominal inputs onto safe sets under known or identified dynamics, with event-triggered extensions in [36]. In RL, Dalal et al. [8] use linearized single-step projection. Recovery RL [32] switches between task and recovery policies using a safety critic, while LS3 [35] plans with a learned terminal safe-set constraint. In our frozen visual world-model interface, however, analytical filters [2, 33, 36] lack explicit state-transition equations, recovery objectives [32] prioritize risk reduction, and isotropic Euclidean projection ∥U−UD∥22\lVert U-U_D\rVert_2^2 orthogonally shrinks obstacle-directed actions into zero-amplitude stalling. RTS instead unifies trigger and feasibility gates via predicted prefix risk and uses dAPd_{\mathrm{AP}} to preserve tangential detours and safe-plan identity.

Safe Planning and Spatial Safety Representations

Recent world-model safety methods include SafeDreamer [18] with online Lagrangian RL, Latent CBF [3] and UNISafe [29] learning latent safety certificates, DreamAvoid [10] with phase-triggered value reranking, and action-conditioned JEPA with analytical shields [38]. Unlike current-state phase triggering with value reranking [10] or external analytical shields [38], RTS extracts risk from multi-step visual prefix rollouts within a unified hard gate, guaranteeing zero-deviation pass-through for safe nominal plans (Appendix § 11, Table A8). To resolve latent geometric resolution gaps in unseen mazes [19], RTS pairs a frozen position probe with explicit local visual risk-field sampling (§ 3.2, Appendix § 8).

Safe Control ParadigmFZPure-VisTrigger MechanismSelection CriterionPassEqv.Anti-Stall
Predictive Safety Filter [33, 36]✓✗State / violation eventEuclidean projection ℓ2\ell_2✓✗✗
Recovery RL [32]✗✓QriskQ_{\mathrm{risk}} thresholdStandalone risk recovery policy✓✗✗
DreamAvoid [10]✓✓Critical-phase classifierValue-model reranking✗✗—
JEPA Shield [38]✓✗Every-step shield auditAnalytical shield filtering✗✗✗
RTS (Ours)✓✓Latent prefix riskShared hard gate + dAPd_{\mathrm{AP}}✓✓✓

Table A8.

Method

Figure 2 illustrates RTS. After defining the external process-safety audit (§ 3.1), we develop action-conditioned risk prediction from frozen latent prefixes and local spatial addressing (§ 3.2), then formulate the unified trigger and feasibility gate, amplitude-preserving selection, and our behavior-preservation guarantee (§ 3.3–§ 3.4). § 3.5 extends RTS to contact manipulation.

Process Safety Criteria and Evaluation Metrics

Conventional evaluation defines success solely by terminal goal entry:

G={∃k≤Tmax⁡, sk∈Sgoal}.(1)G = \{\exists k \leq T_{\max},\ s_k \in S_{\mathrm{goal}}\}. \tag*{(1)}

In bounded environments, true transitions sk+1=T(sk,uk)s_{k+1}=T(s_k,u_k) include simulator collision projection: obstacle-directed commands uku_k (red path in Figure 1) can have their normal components truncated while the agent slides tangentially toward SgoalS_{\mathrm{goal}}. Auditing safe arrival thus requires inspecting both commanded sweep paths and resulting geometric states.

For task ee and executed trajectory ξ=(s0,u0,s1,…,uTend−1,sTend)\xi=(s_0,u_0,s_1,\ldots,u_{T_{\mathrm{end}}-1},s_{T_{\mathrm{end}}}) where TendT_{\mathrm{end}} is the first arrival step (Tend=Tmax⁡T_{\mathrm{end}}=T_{\max} if unreached), we extract process constraints Φe,k(ξ)\Phi_{e,k}(\xi) with upper bounds be,kb_{e,k} (k∈Kek\in\mathcal{K}_e), capturing max single-step sweep penetration, cumulative added penetration, and max consecutive collision steps. Process violation VeV_e, Safe Arrival Rate (SAR), and Violation Rate (VR) are:

Ve={∃k∈Ke:Φe,k(ξ)>be,k},(2)V_e=\{\exists k\in\mathcal{K}_e:\Phi_{e,k}(\xi)>b_{e,k}\}, \tag*{(2)}
SAR=Pr⁡(G∧¬Ve),VR=Pr⁡(Ve).(3)\mathrm{SAR}=\Pr(G\land\neg V_e),\qquad\mathrm{VR}=\Pr(V_e). \tag*{(3)}

We use SAR as our primary external, controller-independent audit metric (thresholds in Appendix § 10, Table A7), alongside Success Rate (SR), SR=Pr⁡(G)\mathrm{SR}=\Pr(G), and VR.

TaskAudited Process Quantities and ScopeSafety Upper BoundTolerance ε\varepsilon
TwoRoomMax single-step commanded disk sweep ctc_t & state overlap ot+1o_{t+1} / Cumulative ∑tΔt\sum_t \Delta_t / Max consecutive steps0.5+ε0.5+\varepsilon px / 1.0+ε1.0+\varepsilon px / 2 steps10−310^{-3} px
ReacherMax penetration depth of link capsules and fingertip into forbidden zone across all native physics substepsε\varepsilon10−610^{-6} m
SimplePointMazeMax commanded disk sweep penetration depth (including maze walls) across all primitive control stepsε\varepsilon10−610^{-6} env. units
PushTMax forbidden-zone penetration of T-block/pusher and workspace escape depth across all physics substepsε\varepsilon10−310^{-3} px
CubeMax forbidden-zone penetration of object OBB/grasp proxy sphere and workspace escape depth across substepsε\varepsilon10−610^{-6} m

Table A7. Process safety criteria and numerical thresholds for SAR. ε\varepsilon is the geometric numerical tolerance, independent of the learned threshold τ\tau.

Frozen World Model and Action-Conditioned Risk

We use frozen INTACT [31], comprising visual encoder EE, direct policy π\pi, and autoregressive world model FF. Given current image ItI_t, goal IgI_g, and action history hth_t, the backbone outputs nominal plan UDU_D:

zt=E(It),g=E(Ig),UD=π(zt,g,ht).(4)z_t=E(I_t),\qquad g=E(I_g),\qquad U_D=\pi(z_t,g,h_t). \tag*{(4)}

A plan U=(u0,…,uT−1)U=(u_0,\ldots,u_{T-1}) of TT primitive steps is grouped into 5-step normalized action blocks aja_j and executed via receding-horizon control (Appendices § 12 and § 13).

Pre-trained latents encode dynamics but lack explicit safety semantics, and static z^t\hat{z}_t assigns identical features to all candidates sharing an observation. We expose action-dependent future consequences by unrolling each candidate plan through frozen FF and scoring its predicted prefixes:

z^0=zt,z^j+1=F(z^≤j,a≤j,ht),\hat{z}_0=z_t,\qquad\hat{z}_{j+1}=F(\hat{z}_{\leq j},a_{\leq j},h_t),
ℓj(U)=Rϕ(z^j,aj,qj)−τ,(5)\ell_j(U)=R_\phi(\hat{z}_j,a_j,q_j)-\tau, \tag*{(5)}

where RϕR_\phi is the risk head, qjq_j is an optional spatial feature, τ\tau is the threshold, and ℓj(U)>0\ell_j(U)>0 signals a block-jj alarm.

Prediction-Aligned Training: Because RϕR_\phi evaluates multi-step predicted states at test time, we train it directly on predicted prefixes of nominal and perturbed plans rather than ground-truth encodings so it adapts to rollout drift. Paired with simulator process-violation labels yj∈{0,1}y_j\in\{0,1\}, we optimize weighted binary cross-entropy (BCE):

Lrisk=ED∑j=0H−1wj BCE(Rϕ(z^j,aj,qj),yj).(6)\mathcal{L}_{\mathrm{risk}}=\mathbb{E}_{D}\sum_{j=0}^{H-1}w_j\,\mathrm{BCE}(R_\phi(\hat{z}_j,a_j,q_j),y_j). \tag*{(6)}

Local Spatial Risk Addressing: Table 2B shows that explicit local geometry improves safe control on unseen Point Maze maps. We therefore decouple trajectory prediction from local obstacle geometry: a dense visual risk field St=Cψ(It)S_t = C_\psi(I_t) is bilinearly sampled on a 21 ×\times 21 grid centered at future coordinates predicted by frozen position probe P(z^j)P(\hat{z}_j) (Figure 2(d)):

Risk inputTwoRoomReacher
State0.5000.500
State + action0.547 [0.476,0.622]0.527 [0.421,0.630]
Predicted rollout0.791 [0.733,0.844]0.791 [0.700,0.874]
Rollout − action+0.243 [0.145,0.337]+0.263 [0.119,0.405]
Spatial readoutSAR (%) ↑VR (%) ↓
Global latent53.1246.88
Patch Cross-Attention53.1246.88
Attention + coordinate bias53.1235.94
Attention + bias + map supervision53.1229.69
Explicit local risk field78.123.12

Table 2. Risk representation and spatial addressing ablations. (A) Within-context AUROC [95% CI] for equal-capacity heads (327,425 parameters; three training seeds; 128/64/128 fitting/selection/audit states per task, 17 plans/state). (B) PointMaze spatial addressing on 64 development tasks under 9,600 matched training steps (implementation and audit details in Appendix § 8.§ 1). A. Action-conditioned prefix prediction. B. Local geometry for closed-loop control.

qj=Sample⁡21×21(St,P(z^j);ρ=1.2).(7)q_j = \operatorname{Sample}_{21 \times21}(S_t, P(\hat{z}_j); \rho= 1.2). \tag*{(7)}

Both StS_t and P(z^j)P(\hat{z}_j) are computed purely from RGB inputs and latent rollouts at test time, providing explicit local geometric inductive bias along predicted rollouts (ablations in Appendix § 8).

Risk Trigger and Amplitude-Preserving Correction

Within the configured search budget, RTS inspects the full-prefix risk of UDU_D and triggers local search when any block raises an alarm:

bt=I{max⁡0≤j<Hℓj(UD)>0}.(8)b_t = \mathbb{I}\left\{\max_{0 \le j < H} \ell_j(U_D) > 0\right\}. \tag*{(8)}

When bt=0b_t = 0, UDU_D executes directly. When bt=1b_t = 1, 3-round CEM search (128 candidates and 8 elites per round, plus 1 final mean evaluation; 385 evaluations total, forming pool CtC_t with UDU_D) updates sampling distributions via smooth elite cost (GMSEG_{\mathrm{MSE}}):

Jelite(U)=dMSE(U,UD)+1H∑j=0H−1softplus⁡(ℓj(U)),dMSE(U,UD)=1TD∑k=0T−1∥uk−uD,k∥22.(9)\begin{aligned} J_{\mathrm{elite}}(U) &= d_{\mathrm{MSE}}(U,U_D) + \frac{1}{H}\sum_{j=0}^{H-1}\operatorname{softplus}(\ell_j(U)),\\ d_{\mathrm{MSE}}(U,U_D) &= \frac{1}{TD}\sum_{k=0}^{T-1}\lVert u_k-u_{D,k}\rVert_2^2. \tag*{(9)} \end{aligned}

Decoupled from soft sampling guidance, final selection filters the feasible set FtF_t using the identical hard gate and minimizes modification distance (FAPF_{\mathrm{AP}}):

Ft={U∈Ct:max⁡0≤j<Hℓj(U)≤0},F_t = \left\{U \in C_t : \max_{0 \le j < H}\ell_j(U) \le0\right\},
Usearch(Ct)={arg⁡min⁡U∈FtdAP(U,UD),Ft≠∅,R(Ct),Ft=∅,(10)U_{\mathrm{search}}(C_t) = \begin{cases} \arg\min_{U \in F_t} d_{\mathrm{AP}}(U,U_D), & F_t \ne\varnothing,\\ \mathcal{R}(C_t), & F_t = \varnothing, \end{cases} \tag*{(10)}

where R(Ct)\mathcal{R}(C_t) is the configured deterministic fallback: zero action or a minimum-soft-risk candidate. In Eq. (9), DD denotes action dimension.

Amplitude-Preserving Metric dAPd_{\mathrm{AP}}: Under isotropic dMSEd_{\mathrm{MSE}}, shrinking an obstacle-directed action toward zero incurs smaller Euclidean deviation than steering tangentially at preserved speed, causing non-fallback orthogonal stalling. We introduce a step-magnitude preservation term:

dAP(U,UD)=1T∑k=0T−1[∥uk−uD,k∥22+(∥uk∥2−∥uD,k∥2)2].(11)d_{\mathrm{AP}}(U,U_D)=\frac{1}{T}\sum_{k=0}^{T-1}\left[\lVert u_k-u_{D,k}\rVert_2^2+\left(\lVert u_k\rVert_2-\lVert u_{D,k}\rVert_2\right)^2\right]. \tag*{(11)}

The vector term penalizes changes in action direction and magnitude, while the added term penalizes step-magnitude shrinkage; their sum vanishes uniquely at U=UDU=U_D. Near a local half-space obstacle with unit inward normal nn, safe actions satisfy n⊤u≤0n^\top u\leq0. For nominal action vnvn (v>0v>0) pointing into the obstacle and tangential detour wtwt (w≥0w\geq0, t⊥nt\perp n, ∥t∥2=1\lVert t\rVert_2=1), Euclidean cost v2+w2v^2+w^2 is minimized by stopping at w=0w=0. Under dAPd_{\mathrm{AP}},

dAP(wt,vn)=v2+w2+(w−v)2=2(w−v/2)2+32v2.(12)d_{\mathrm{AP}}(wt,vn)=v^2+w^2+(w-v)^2 =2(w-v/2)^2+\frac{3}{2}v^2. \tag*{(12)}

The minimum occurs at w=v/2w=v/2 with cost 1.5v21.5v^2 versus 2v22v^2 for a full stop. Thus dAPd_{\mathrm{AP}} favors tangential progression while preserving its unique zero at UDU_D (Appendix § 11).

Behavior Preservation Property

Unifying the trigger and feasibility gates under a selection metric with a unique zero at UDU_D links selective compute allocation with behavior preservation: when UDU_D passes the risk gate, it lies in FtF_t with zero modification cost, making search omission mathematically lossless.

Proposition 1 (Single-Step and Closed-Loop Behavior Preservation). Suppose CtC_t contains UDU_D, trigger and selection gates share the identical criterion, and final selection minimizes d≥0d\geq0 over FtF_t with d(U,UD)=0⟺U=UDd(U,U_D)=0\Longleftrightarrow U=U_D. If bt=0b_t=0, returning UDU_D directly equals Usearch(Ct)U_{\mathrm{search}}(C_t). If RTS and every-step same-kernel search (Always-RTS) apply identical search rules at every alarm and share initial states, execution rules, fallbacks, and per-step deterministic random streams, they produce identical closed-loop trajectories.

Proof. When bt=0b_t=0, UD∈FtU_D\in F_t achieves d(UD,UD)=0d(U_D,U_D)=0 while all U≠UDU\neq U_D have d(U,UD)>0d(U,U_D)>0; Eq. (11) satisfies this unique-zero property, so Usearch(Ct)=UDU_{\mathrm{search}}(C_t)=U_D. Seeding CEM streams by (task ID, step) ensures identical search outputs when bt=1b_t=1 and identical UDU_D when bt=0b_t=0; induction over steps yields identical closed-loop trajectories. □

Proposition 1 explains the structural separation between sampling and selection: soft risk guides exploration in Eq. (9), whereas the hard gate and unique-zero distance govern the final choice in Eq. (10) so a feasible nominal plan always incurs zero modification cost, while at alarms the magnitude term favors tangential progression over Euclidean shrinkage (implementation details in Appendix § 6).

Contact Manipulation Extension (RTS-Repair)

In Cube and PushT, purposeful tool–object contact must coexist with avoidance of object–hazard, tool–hazard, and workspace-boundary violations. RTS-Repair retains the shared gate and AP selection with two adaptations. Component-wise risk combines the current latent state, 5-step visual difference, and predicted block-start displacement to regress signed clearances across five blocks. Structured hybrid proposals mix 96 latent goal perturbations decoded by frozen π\pi with 32 smooth trajectory perturbations around safe demonstration neighborhoods per CEM round, varying goal-directed motion while maintaining contact continuity (Appendix § 7).

Experiments

Experiment Setup

Benchmarks and Cohorts. We evaluate on TwoRoom, Reacher, Cube, PushT [22, 31], and SimplePointMaze (PointMaze; 64 unseen test maps, 128 tasks) using frozen INTACT weights with geometric forbidden zones. For contact manipulation, we evaluate both development/boundary cohorts and full test sets (Cube: 997 tasks; PushT: 1,538 tasks).

Baselines and Protocol. We compare INTACT Direct, Guarded A [31], INTACT + behavior cloning (BC), IN-TACT + risk-aware fine-tuning (RiskFT), LeWM [22], Fast-LeWM [11], and Recovery-MPC against RTS-MSE, RTS-AP, and RTS-Repair on SAR ↑, SR ↑, and VR ↓ (single RTX 4090 GPU). In Table 1, Direct, Guarded A, and task-adapted RTS use receding-horizon execution intervals of 15/15/10/25/15 steps with risk windows of 25/25/25/25/15 steps across TwoRoom, Reacher, SimplePointMaze, Cube, and PushT; baseline intervals are listed in Table A9. Matched-interval ablations are in Table 3 (Appendix § 8) and Appendix § 14 (Table A13). Stochastic methods run 3 seeds per task (Appendices § 12–§ 14).

Table 1. Closed-loop evaluation summary across five benchmarks. Panel A: navigation and articulated control; Panel B: full-test contact manipulation (Cube boundary confirmation in Appendix § 6, Table A2). Task-adapted RTS uses RTS-AP in Panel A and RTS-Repair in Panel B. Δ\DeltaSAR reports percentage-point gains over Direct and Recovery-MPC. Column headers report total evaluated trajectories nn for 3-seed stochastic methods (deterministic Direct runs once per task; see Table A9).

Method (Panel A: Navigation & Articulated Control)TwoRoom (n=600n = 600)Reacher (n=600n = 600)SimplePointMaze (n=384n = 384)
SR (%) ↑\uparrowSAR (%) ↑\uparrowVR (%) ↓\downarrowSR (%) ↑\uparrowSAR (%) ↑\uparrowVR (%) ↓\downarrowSR (%) ↑\uparrowSAR (%) ↑\uparrowVR (%) ↓\downarrow
INTACT Direct [31]93.5075.5021.5081.508.5084.5057.0350.0050.00
INTACT + BC94.3373.8324.0055.337.1782.8350.5247.9252.08
INTACT + RiskFT95.0076.6721.3348.1710.3373.6752.0850.0050.00
INTACT Guarded A91.6773.1724.8381.179.5080.8356.7748.9651.04
LeWM [22]78.1734.5065.3335.172.1767.504.430.0095.57
Fast-LeWM [11]85.0034.8364.5039.502.1774.677.030.5289.06
Recovery-MPC (Adapted)75.5074.672.0068.1750.5022.0059.6458.073.12
RTS-MSE88.8388.002.0079.6749.1739.0074.7472.147.55
RTS-AP (Ours)89.0088.172.3379.6750.1737.8380.7377.348.59
Δ\DeltaSAR (RTS-AP vs. Direct / Rec.)vs. Direct: +12.67vs. Direct: +41.67vs. Direct: +27.34
vs. Rec: +13.50vs. Rec: −0.33vs. Rec: +19.27
Method (Panel B: Full-Test Contact Manipulation)Cube (n=2,991n = 2{,}991)PushT (n=4,614n = 4{,}614)
SR (%) ↑\uparrowSAR (%) ↑\uparrowVR (%) ↓\downarrowSR (%) ↑\uparrowSAR (%) ↑\uparrowVR (%) ↓\downarrow
INTACT Direct [31]99.8097.492.3185.8375.6216.51
INTACT + BC99.6097.961.7480.9171.4318.36
INTACT + RiskFT99.5099.000.6080.1971.3317.53
INTACT Guarded A98.6396.622.0189.1077.9415.24
LeWM [22]54.4652.9613.4471.2057.5631.06
Fast-LeWM [11]53.9652.839.9075.6462.7426.09
Recovery-MPC (Adapted)95.6294.581.0478.5770.8714.91
RTS-MSE99.0097.891.1084.6675.6815.41
RTS-Repair (Ours)99.0397.891.1484.6175.6615.43
Δ\DeltaSAR (RTS-Repair Key Comparisons)vs. Direct: +0.40 / vs. Rec: +3.31vs. Direct: +0.04 / vs. Rec: +4.79
Boundary Cohort vs. Direct: +2.08Boundary Cohort vs. Direct: +4.17

Table 1. Closed-loop evaluation summary across five benchmarks.

Table 3. Matched 2×22 \times2 ablation of candidate generation GG and final selection FF (protocol in Appendix § 8.§ 2). Execution intervals are fixed at 25/25/5 steps for TwoRoom/Reacher/PointMaze; 6,336 trajectories in total. Zero denotes non-fallback zero-action decisions per episode.

GGFFTwoRoomReacherPointMaze
SARVRZeroSARVRZeroSARVRZero
MSEMSE83.673.830.00047.3330.330.48272.147.552.133
MSEAP83.674.170.00047.1731.830.18875.789.110.734
APMSE84.334.170.00048.1727.830.44573.965.732.125
APAP84.334.500.00048.8328.170.18576.826.510.719

Table 3.

Method / Evaluation ScaleTwoRoomReacherPointMazeCubePushT
Direct / Guarded A / RTS1515102515
RTS-MSE252552515
Recovery-MPC / LeWM / Fast-LeWM252552525
BC / RiskFT252552515
Independent Tasks2002001289971,538
3-Seed Method Trajectories (nn)6006003842,9914,614
RTS Risk Audit Window (steps)252525255

Table A9. Main-table receding-horizon execution intervals and independent evaluation scales. Cube uses the 25-step configuration.

Table A13. Matched-interval learning baseline comparisons (Direct runs once per task; BC and RiskFT run 3 training seeds; SAR in %).

Task (Exec. Steps)Direct SARBC SARRiskFT SARRiskFT − Direct [95% CI]
TwoRoom (25)74.5073.8376.67+2.17 [−1.00, 5.33]
Reacher (25)9.507.1710.33+0.83 [−3.67, 5.50]
SimplePointMaze (5)50.0047.9250.000.00 [−5.99, 5.99]
Cube (25)97.4997.9699.00+1.50 [0.60,2.41]
PushT (15)75.6271.4371.33−4.29 [−5.94, −2.67]

Table A13.

Closed-Loop Control across Five Tasks

Table 1 jointly evaluates goal attainment and process safety. On TwoRoom, Reacher, and PointMaze, RTS-AP raises SAR over Direct by 12.67, 41.67, and 27.34 percentage points, while reducing VR from 21.50%, 84.50%, and 50.00% to 2.33%, 37.83%, and 8.59%. Risk-agnostic terminal-distance search preserves goal arrival without preventing violations: Guarded A reaches 81.17% SR but only 9.50% SAR on Reacher. Dual-mode Recovery-MPC obtains 50.50% SAR on Reacher with 68.17% SR, whereas RTS-AP reaches 50.17% SAR with 79.67% SR and surpasses Recovery-MPC SAR by 13.50 and 19.27 pp on TwoRoom and PointMaze.

Contact manipulation. On full Cube and PushT, RTS-Repair attains 97.89% and 75.66% SAR, reducing VR from 2.31% to 1.14% and from 16.51% to 15.43% while exceeding Recovery-MPC SAR by 3.31 and 4.79 pp. Held-out boundary cohorts isolate local hazard avoidance near obstacles: Cube boundary SAR rises from 90.62% (Direct) to 92.71% (+2.08 pp, 95% CI [0.00,6.25]; Table A2), and PushT boundary SAR rises from 81.25% to 85.42% (+4.17 pp; Appendix § 7.§ 2). These boundary evaluations complement the full test sets, where many initial states are already hazard-free.

MethodOrdinary SAR ↑Boundary SAR ↑Boundary VR ↓
INTACT Direct96.8890.629.38
INTACT Guarded A93.7582.2914.58
RTS-Repair (Ours)96.8892.717.29

Table A2. Independent confirmation and same-kernel efficiency verification on Cube with frozen configurations. Ordinary and boundary cohorts each contain 32 held-out tasks. Panel A reports percentages (bold indicates best in column); Direct runs once per task (32 episodes) and search methods run 3 seeds (96 episodes). Panel B pairs RTS-Repair with every-step same-kernel Always-Repair. Boundary SAR gain over Direct is +2.08 pp (10,000-sample task-cluster bootstrap 95% CI [0.00,6.25]).

Process audit. In the TwoRoom contact-response diagnostic (Figure A2), Direct achieves 67.7% arrival under default sliding but only 38.0% safe arrival, as 89 of its 203 arriving trajectories violate the process audit. Stopping remaining displacement upon first contact reduces Direct arrival to 39.0%. In contrast, all 162 arriving RTS trajectories are completely collision-free, maintaining 54.0% arrival and 54.0% safe arrival under both sliding and stop-on-contact responses and confirming that its gains stem from genuinely safe commanded motion rather than boundary sliding.

Process evaluation bar chart

Figure A2. TwoRoom process-safety diagnostic: arrival under default sliding (Strict SR - slide), safe arrival under sliding (Process SAR - slide), and arrival when remaining displacement stops at first contact (Strict SR - stop).

Risk Prediction and Spatial Addressing

Action-conditioned future prediction. We measure candidate ranking by the area under the receiver operating characteristic curve (AUROC) within each context, defined by a shared observation. Table 2A holds risk-head capacity, training protocol, and ground-truth labels fixed while varying the input representation. Static state features yield 0.500 within-context AUROC because all candidate plans sharing an observation receive the same representation. Concatenating candidate actions (xt,U)(x_t,U) raises AUROC only to 0.547 on TwoRoom and 0.527 on Reacher, showing that a lightweight risk head cannot invert multi-step nonlinear collision geometry without forward state propagation. Unrolling predicted latent prefixes through frozen FF raises within-context AUROC to 0.791/0.791 (paired gains of +0.243 [0.145,0.337] and +0.263 [0.119,0.405]), confirming that world-model prefix rollouts provide essential action-conditioned risk cues.

Local geometry on unseen maps. Table 2B compares spatial readouts under matched training budgets. Global latents and Patch Cross-Attention both yield 53.12% PointMaze SAR. Adding coordinate Gaussian biases and counterfactual map supervision lowers VR to 35.94% and 29.69%, yet closed-loop SAR remains 53.12%. Bilinearly sampling the explicit local risk field StS_t on a 21 × 21 grid centered at position-probe coordinates P(z^j)P(\hat{z}_j) raises closed-loop SAR to 78.12% and reduces VR to 3.12% (95.98% recall), demonstrating the closed-loop benefit of explicit local obstacle geometry over global or patch-level attention.

Task-specific risk heads. Figure 3(a) reports within-context ranking AUROCs of 0.901–0.985 across all five tasks. At the reported operating points in Figure 3(b), hazard recall reaches 91.45%, 94.85%, 95.98%, 96.42%, and 78.79% on TwoRoom, Reacher, PointMaze, PushT, and Cube, with safe false-alarm rates of 6.01%, 2.49%, 11.14%, 2.79%, and 9.44% (PushT q95 characterizes offline detector sensitivity, while closed-loop evaluation uses the frozen deployed gate; Appendix § 6.§ 3).

Figure 3. Five-task risk discrimination and controller overhead. (a) Within-context candidate risk ranking AUROC. (b) Candidate hazard recall and safe false-alarm rate. (c) Episode controller time ratios (< 1.0: savings). Timing cohorts and execution settings are specified in Appendix § 6.§ 3.

Amplitude-Preserving Selection

In Table 1, RTS-AP exceeds RTS-MSE SAR by 0.17, 1.00, and 5.20 pp on the three primary tasks. Table 3 isolates the selection metric by fixing execution intervals and crossing the CEM elite generator GG with the final selector FF.

With GMSEG_{\mathrm{MSE}} fixed, switching FMSEF_{\mathrm{MSE}} to FAPF_{\mathrm{AP}} changes SAR by 0.00 pp on TwoRoom, −0.17 pp on Reacher, and +3.65 pp on PointMaze (95% CI [0.78,6.51]). On PointMaze, non-fallback zero-action stalling drops from 2.133 to 0.734 decisions per episode as SAR rises from 72.14% to 75.78% (VR changes from 7.55% to 9.11%); applying AP to the generator as well reaches 76.82% SAR. The largest selection gain occurs in PointMaze (and Reacher stalling drops from 0.482 to 0.188 under GMSEFAPG_{\mathrm{MSE}}F_{\mathrm{AP}} and 0.185 under GAPFAPG_{\mathrm{AP}}F_{\mathrm{AP}} with 48.83% SAR), where navigating multiwall corridors or articulated workspace bounds requires repeated tangential turns around obstacles.

Counterfactual physical replay evaluates both final selectors on each of 384 fixed candidate pools, confirming that FAPF_{\mathrm{AP}} changes 21 of 96 alarmed PointMaze selections (Appendix § 8.§ 2) and isolating the final selector’s decisions on a shared pool. Across three independently trained risk heads, PointMaze AP gains are +3.65, +3.39, and +3.12 pp, with mean gain +3.39 pp [1.04,5.90] (Table A1).

TaskMSE SARAP SARRec. SARAP − MSE (95% CI)
TwoRoom (n=1,800n = 1,800/arm)84.11%84.11%74.61%0.00 [−0.17, +0.17]
Reacher (n=1,800n = 1,800/arm)46.06%46.17%48.61%+0.11 [−0.72, +0.94]
SimplePointMaze (n=1,152n = 1,152/arm)71.70%75.09%57.99%+3.39 [1.04,5.90]
PointMaze heads h0/h1/h2h_0/h_1/h_272.1/71.9/71.175.8/75.3/74.258.1/58.0/57.9+3.65/ + 3.39/ + 3.12
Inter-head SD0.52 pp0.66 pp0.10 ppConsistent positive gain across all 3 heads

Table A1. Multi-seed risk-head verification and Recovery-MPC comparison (14,256 closed-loop trajectories; 3 independent risk-head seeds × 3 search seeds per task).

Behavior Preservation and Compute Allocation

Behavior Preservation and Runtime. Across 1,968 same-kernel paired trajectories, RTS and Always-RTS produce identical step-by-step actions, with a maximum state discrepancy of 7.13×10−117.13 \times10^{-11} (Appendix § 6.§ 2, Table A3). Sharing initial states, execution rules, and per-step random streams, these runs verify that bypassing search on safe nominal plans preserves closed-loop behavior (Proposition 1) while reducing runtime by 17.71%–70.41%. Figure 3(c) compares controller time against Guarded A and Always-RTS.

Task / CohortPaired Traj.Time Ratio [95% CI]Time ms (RTS / Always)
TwoRoom6000.636 [0.591,0.678]196.98 / 197.38 (decision p95)
Reacher6000.651 [0.620,0.683]200.47 / 201.27 (decision p95)
SimplePointMaze3840.823 [0.765,0.874]196.20 / 172.11 (decision p95)
PushT (Test Subset)1920.296 [0.273,0.322]169.22 / 571.99 (episode mean)
Cube Ordinary960.390 [0.329,0.453]237.46 / 346.60 (decision p95)
Cube Boundary960.462 [0.379,0.551]278.48 / 325.45 (decision p95)

Table A3. Same-kernel paired efficiency (RTS vs. Always-RTS). Ratio denotes mean episode controller time of RTS divided by Always-RTS; confidence intervals are clustered by task/map.

Allocation under a fixed search cap. Table 4 compares risk-triggered, periodic, and random scheduling under a shared 50% call cap. Risk scheduling raises Reacher SAR over periodic scheduling by 17.00 pp [12.33,22.00] at 674.61 versus 665.20 ms/episode, and raises PointMaze SAR by 5.99 pp [2.08,10.16] at 1458.65 versus 1313.20 ms while reducing VR from 21.35% to 15.89%. On TwoRoom, risk scheduling matches periodic SAR (83.17%) in 131.33 versus 179.64 ms, showing that prefix risk directs compute toward high-value safety interventions.

Table 4. Search scheduling under a shared 50% search-call cap and the same AP kernel (Appendix § 13.§ 3). Time is controller ms/episode. Risk-based scheduling improves SAR on Reacher and PointMaze; on TwoRoom it matches periodic scheduling with lower controller time.

SchedulerTwoRoomReacherPointMaze
SARVRTimeSARVRTimeSARVRTime
Periodic83.1712.33179.6432.0056.50665.2077.8621.351313.20
Random77.3318.17127.3525.0065.83646.1572.6626.301321.16
Risk83.1711.33131.3349.0039.33674.6183.8515.891458.65

Table 4.

Across the full budget curves (Figure A5), a 25% risk cap on Reacher exceeds 50% periodic scheduling by 15.83 SAR pp in 8.46% less controller time. On PointMaze, moderating the risk cap from 100% to 50% increases SAR from 75.78% to 83.85% while VR rises from 9.11% to 15.89%, as fewer conservative detours reduce episode timeouts.

Figure A5. Full efficiency and constrained-budget (K-of-N) scheduling curves. Panel (a) shows same-kernel episode controller time; panels (b–c) compare risk-triggered, periodic, and uniform random scheduling under 25% (circle), 50% (square), and 100% (triangle) search call caps.

Additional robustness checks. Across fixed photometric shifts and 2×2\times downsampling (Appendix § 9.§ 1), RTS-AP exceeds Direct SAR in five of six settings (+40.83/+35.00 pp on Reacher and +25.78/+26.04 pp on PointMaze); only 2×2\times blurred TwoRoom inputs reduce SAR to 56.50% versus 62.00% for Direct as doorway blur induces conservative fallbacks (Appendix § 9.§ 1).

Conclusion and Limitations

We presented Risk-Triggered Search (RTS) for frozen visual world models, using multi-step predicted latent prefixes and local spatial risk-field addressing to discriminate process risk and correct hazardous plans while preserving safe nominal behavior. Across five benchmarks, controlled ablations validate action-conditioned prefix prediction, local spatial addressing, and amplitude-preserving selection (dAPd_{\mathrm{AP}}); 1,968 paired trajectories verify 100% closed-loop action preservation with 17.71%–70.41% lower controller runtime, and risk-triggered scheduling substantially improves safe arrival under constrained search budgets. Current performance remains bounded by visual boundary resolution under 2×2\times downsampling, static threshold calibration in high-frequency contact (PushT), and the reach of local CEM proposals in deep cul-de-sacs—motivating risk-aware visual adapters, goal-masked dynamic audit horizons, and adaptive search budgets.

Engineering Implementation of Proposition 1 and 1,968-Pair Closed-Loop Verification

At the theoretical level, Proposition 1 guarantees that when the nominal plan passes the risk trigger gate (bt=0b_t = 0), returning UDU_D directly produces the exact same single-step action as executing the full search operator. In the paired closed-loop evaluations, both controllers use the same search kernel at every alarm. We adopt two implementation measures to eliminate pseudorandom number generator (PRNG) state drift and floating-point round-trip errors from tensor normalization: (1) deriving an independent deterministic CEM sampling random stream indexed by (task ID ×\times physical decision step index); and (2) directly returning the raw unperturbed tensor UDU_D whenever final selection hits the nominal plan slot in the candidate pool. Across 1,968 complete closed-loop trajectory pairs spanning TwoRoom (600 pairs), Reacher (600 pairs), SimplePointMaze (384 pairs), the Cube independent confirmation cohort (192 pairs), and the PushT test subset (64 tasks ×\times 3 search seeds, 192 pairs), RTS and Always-RTS achieve 100% exact step-by-step action identity while reducing per-episode controller time by 17.71%–70.41%.

Cube Independent Confirmation and Same-Kernel Efficiency Verification

Table A2 reports the independent confirmation evaluation on Cube across balanced ordinary and boundary cohorts. The risk head, alarm threshold, candidate proposal distribution, and search hyperparameters remain frozen; 32 ordinary tasks and 32 geometrically valid hazard-boundary tasks disjoint from model fitting, threshold selection, and hyperparameter tuning are evaluated separately to complement the 997-task Cube Full Test in the main paper.

Direct runs once per task, while sampling-based search methods run across 3 independent search seeds. RTS-Repair and every-step same-kernel Always-Repair share the exact risk head, candidate proposal kernel, and per-decision random streams; we verify step-by-step action identity across all 192 trajectory pairs and measure per-episode controller latency across each cohort.

Paired Speedup Intervals and Decision Tail Latency

Across TwoRoom, Reacher, SimplePointMaze, and PushT, closed-loop output actions and physical states of RTS and Always-RTS match element-wise; on Cube, all 192 trajectory pairs exhibit 100% exact action identity with a maximum floating-point state discrepancy of 7.13×10−117.13 \times10^{-11}, well below the 10−710^{-7} verification tolerance. Controller timing measures pure inference overhead (including visual encoding and latent risk evaluation while excluding simulator physics stepping and disk I/O). On SimplePointMaze, mean episode controller time decreases by 17.71%, while single-decision p95/p99 tail latencies are 196.20/204.77 ms for RTS versus 172.11/179.28 ms for Always-RTS; mean episode savings and per-step tail latency characterize throughput-oriented and hard real-time step-rate deployment requirements, respectively. Native Guarded A optimizes predicted terminal distance; under matched receding-horizon intervals, mean episode controller times on TwoRoom and Reacher are 154.97 and 684.91 ms for Guarded A versus 135.02 and 438.60 ms for RTS (Table A4). Lossless acceleration under Proposition 1 is evaluated relative to same-kernel Always-RTS.

TaskExec. StepsRTS (ms)Guarded A (ms)Time Ratio
TwoRoom15135.02154.970.871
Reacher15438.60684.910.640
SimplePointMaze10804.221368.910.587
PushT15196.13300.910.652
Cube15170.42209.710.813

Table A4. Measured controller time against Guarded A in Figure 3(c). Both methods use the identical execution interval per task (ms/episode). Protocols are detailed in Section A.3.

Comparisons and Protocol for Five-Task Controller Timing

Figure 3(c) compares mean episode controller time relative to Guarded A and Always-RTS. Guarded A comparisons use matched execution intervals of 15/15/10/15/15 steps across TwoRoom, Reacher, PointMaze, Cube, and PushT (Table A4). Table A3 and Figure A5(a) report action-identical same-kernel pairs, including Reacher at 25/25 steps (34.91% savings); Figure 3(c) uses these pairs for the other four tasks.

For Reacher, Figure 3(c) uses the main-table RTS-AP configuration (Section G.1), with mean controller time of 438.60 ms at a 15-step execution interval. Against Guarded A, the time ratio is 0.640 [0.601,0.684], saving 35.96%; against Always-RTS at the same 15-step interval, it is 0.476 [0.461,0.491], saving 52.4%. These comparisons use archived runs sharing 200 task IDs and three seeds (600 trajectories). We average seeds within each task and jointly resample 200 task IDs 10,000 times (seed 2026111312) to obtain 95% bootstrap intervals, quantifying task-sampling variation within the recorded timing batches and execution windows.

For contact manipulation, Cube uses the 64-task balanced cohort of 32 ordinary and 32 boundary tasks (combined time ratio (133.79+135.53)/(343.38+293.07) = 0.423, saving 57.7% overall, or 61.04%/53.76% stratified by ordinary/boundary tasks). PushT uses 64 preselected test tasks across 3 search seeds at a 15-step execution interval, yielding 192 exact action state pairs with mean controller time reduced from 571.99 ms to 169.22 ms per episode (ratio 0.296, 95% paired task bootstrap CI [0.273,0.322], a 70.41% reduction).

Offline operating point sensitivity analysis (such as PushT q95 in Figure 3(b)) characterizes detector recall and false-alarm rates, whereas all closed-loop timing measurements use the frozen deployed gates.

Cohort (Tasks)Paired TrajectoriesAction IdentityTime Savings
Ordinary (32)96100%61.04%
Boundary (32)96100%53.76%
Combined / Mean (64)192100%57.68%

Table 5. Panel B: Efficiency.

Contact Manipulation Extensions (Cube & PushT) and Gating Diagnostics

Figure A1 presents cohort-level comparisons of RTS-Repair on Cube and PushT development sets (panels a–b) alongside counterfactual geometric replays of three representative closed-loop cases (panels c–e): (1) Cube 223 safety rescue (Figure A1(c)): Direct advances along a straight line into the forbidden zone, whereas RTS-Repair guides the gripper and block smoothly around the hazard boundary to reach the goal safely within 24 steps; (2) Cube 220 audit horizon vs. termination mismatch (Figure A1(d)): at step t=25t = 25, the nominal plan requires only the first 4 steps (within block 1) to safely reach the goal and terminate the episode, but is rejected by the full-plan risk gate because unexecuted tail blocks 3–5 drift out of bounds after goal arrival; and (3) PushT ranking capability vs. gating threshold trade-off (Figure A1(e)): after incorporating 5-step short-horizon contact feedback and 550,935 candidate first-block signed-clearance labels, the short-horizon risk head achieves a within-context candidate ranking AUROC of 0.9854 across 246 audit contexts (0.9820 overall candidate AUROC and 0.9917 nominal plan AUROC). However, when setting a single scalar trigger threshold, calibrating at the 98th percentile of safe nominal plans (q98, threshold 2.2254) yields 0/242 false rejections of safe nominal plans but only 3.51% pooled hazard recall; relaxing the threshold to the 90th percentile (q90, threshold -1.6910) elevates candidate hazard recall to 96.49% (1,402/1,453) and rescues 6 violating trajectories, but increases safe nominal false rejections to 17/242 and induces 139 empty-set fallbacks, resulting in a net closed-loop SAR change of -1.30 percentage points. These diagnostics motivate joint consideration of ranking, threshold calibration, and closed-loop progression.

Contact manipulation extension cohort evaluation and representative case studies

Figure A1. Contact manipulation extension (RTS-Repair) cohort evaluation and representative case studies on Cube and PushT. (a–b) Safe Arrival Rate (SAR ↑\uparrow) and Process Violation Rate (VR ↓\downarrow) on development cohorts. (c) Cube 223 safe detour rescue trajectory. (d) Cube 220 tail-block false rejection stalling caused by a fixed 5-block audit horizon exceeding the 4 remaining steps to goal completion. (e) PushT safe nominal trajectory intercepted by a false-positive alarm.

Component-Wise Risk and Structured Candidate Proposals

(1) Component-wise signed-clearance risk head: Combining the current latent state, 5-step visual difference, and predicted block-start displacement, the risk head regresses signed clearances across 5 action blocks for three geometric pairs: object-to-hazard, tool-to-hazard, and workspace boundary. (2) Structured hybrid proposals: To prevent raw action-space Gaussian noise from disrupting contact chains, each round of 128 candidates comprises 96 goal-perturbed candidates decoded through the frozen policy π\pi in latent space and 32 smooth trajectory perturbations around safe demonstration neighborhoods.

Horizon and Threshold Diagnostics

On PushT, which requires high-frequency non-prehensile contact switching, supervising the risk head with block-wise component-level signed clearances improves SAR on the boundary challenge cohort from 81.25% (Direct) to 85.42% (+4.17 pp). Systematic mechanism diagnostics (Figure A1) identify two primary bottlenecks in high-frequency contact: (1) horizon mismatch when a fixed 25-step full-plan risk gate rejects plans that safely reach the goal in early steps but drift out-of-bounds in unexecuted tail blocks; and (2) tension between ranking power and scalar threshold calibration—although short-horizon contact feedback elevates within-context candidate ranking AUROC to 0.9854, a single static threshold faces a trade-off between protecting safe nominal plans from false rejection and maintaining high recall on near-term violations. Unified recalibration increases within-context nominal hazard detection from 0/5 to 3/5, but also raises safe nominal false rejections from 3/246 to 18/246, increases closed-loop zero-action decisions from 4 to 39, and elevates the safe timeout rate from 6.51% to 9.90% (Table A5). Consequently, none of the five pre-specified comparisons show a positive SAR gain under 99% Bonferroni-corrected intervals (Table A6). Lowering only the trigger threshold while the selection gate still passes the nominal plan (Decoupled-10 vs. 00) invokes 24 additional searches that all return the nominal plan, preserving element-wise identical actions, states, and hazard depths across all 384 trajectory pairs.

Gating ConfigurationSR ↑\uparrowSAR ↑\uparrowVR ↓\downarrowSafe Timeout
Direct-1589.0683.5910.166.25
Baseline-1588.2882.8110.686.51
Calib-1585.4281.258.859.90
Decoupled-0085.6881.259.119.64
Decoupled-1085.6881.259.119.64
Decoupled-0185.1680.998.8510.16
Decoupled-1185.1680.998.8510.16
ShortExec-584.3878.9111.729.38
Aligned-1588.2882.8110.426.77

Table A5. Closed-loop gating intervention results on the PushT confirmation cohort (%). Safe Timeout denotes episodes without violations that fail to reach the goal within Tmax⁡T_{\max}.

Pre-Specified ComparisonΔ\DeltaSAR (pp)99% CI (Bonferroni)
Calib-15 − Baseline-15−1.56[−5.21, 0.52]
Decoupled-00 − Calib-150.00[0.00, 0.00]
Decoupled-11 − Decoupled-00−0.26[−2.08, 1.04]
ShortExec-5 − Baseline-15−3.91[−8.59, 0.00]
Aligned-15 − Baseline-150.00[0.00, 0.00]

Table A6. Pre-specified paired SAR differences on the PushT confirmation cohort (percentage points, Bonferroni-corrected 99% CI).

Isolated Calibration and Decoupled Gate / Horizon Interventions

To evaluate the impact of threshold calibration, trigger/selection gate decoupling, and execution interval on PushT closed-loop control, we freeze thresholds on 128 validation calibration episodes and compare nine gating configurations across 128 disjoint validation confirmation episodes (3,200 trajectories total; Table A5–Table A6). All learned weights remain frozen; Direct runs once and stochastic methods run across 3 search seeds.

In Table A5–Table A6, Baseline-15 uses the 5-step risk gate with a 15-step execution interval; Calib-15 applies unified threshold recalibration; Decoupled-00/10/01/11 evaluate a 2×22 \times2 decoupling of trigger and final selection thresholds; ShortExec-5 shortens the execution interval to 5 steps; and Aligned-15 uses a 3-block (15-step) risk gate aligned with the execution window.

Spatial Readout and Counterfactual Selector Protocols

Spatial Readout Implementation and Audit Recall

For the readouts in Table 2B, implicit addressing queries 256 last-layer ViT patch tokens with 4-head Cross-Attention using the predicted latent state. Coordinate Gaussian biases center attention at the predicted position; counterfactual map supervision pairs action sequences with alternative map layouts. Explicit addressing instead samples a 21 × 21 local grid around the frozen position-probe prediction.

The corresponding audit recalls are 60.45% for global latents, 52.84% for patch attention, 75.90% with coordinate bias, 80.82% with bias and counterfactual map supervision, and 95.98% for explicit spatial risk-field sampling. These detector measurements complement the closed-loop SAR and VR in Table 2B.

Matched Generator/Selector Ablation and Physical Replay Protocol

Table 3 reports the matched 2×22 \times2 generator/selector ablation; the default configuration is GMSEFAPG_{\mathrm{MSE}}F_{\mathrm{AP}}. For the accompanying physical replay, we select the first alarm before success after a safe prefix from each of 16 TwoRoom tasks, 16 Reacher tasks, and 32 distinct PointMaze maps, using a fixed hash order on frozen GMSEFAPG_{\mathrm{MSE}}F_{\mathrm{AP}} seed-0 trajectories. Each of the 64 recorded states uses three search seeds and two generators, giving 384 candidate pools. Both final selectors operate on the same pool for each generator/state/seed combination. Each pool contains 385 candidates; executing every candidate continuously for 25 physical steps yields 147,840 candidate labels.

PointMaze safe progression requires the first five executed steps to remain safe and the goal distance to decrease by at least max⁡(0.01,0.05d0)\max(0.01, 0.05d_0), where d0d_0 is the initial goal distance. Reacher displacement is measured in joint space. Under GMSEG_{\mathrm{MSE}}, the PointMaze selector changes reported in § 4.4 recover three safe-progress choices and lose two among 96 state/seed pairs. Under GAPG_{\mathrm{AP}}, 20 of 96 choices change, recovering five safe-progress choices and losing none.

Visual Perturbation Generalization and Search-Round Predictability

Photometric and Low-Resolution Visual Perturbations (15,624 Trajectories)

Without retraining the backbone or risk head, we apply fixed photometric color shifts (Photometric) and 2× downsampling followed by bilinear upsampling (Low-Res) to input RGB images. Across all 6 task × perturbation combinations, RTS-AP maintains substantial SAR advantages over Direct in 5 settings: on Reacher, Photometric and Low-Res gains are +40.83 pp (49.83% vs. 9.00%) and +35.00 pp (44.00% vs. 9.00%); on PointMaze, gains are +25.78 pp (75.00% vs. 49.22%) and +26.04 pp (76.04% vs. 50.00%); on TwoRoom Photometric, the gain is +8.67 pp (72.17% vs. 63.50%). The sole degradation occurs on TwoRoom Low-Res (56.50% RTS-AP SAR vs. 62.00% Direct, with empty-set fallbacks increasing from 25 to 217), indicating that blurred obstacle boundaries induce conservative false alarms in the fixed risk head.

Marginal Gain Prediction for Additional Search Rounds (4,212 Trajectories)

At 118 first-alarm decision states, we branch closed-loop rollouts to compare 1-round (129 candidates), 2-round (257 candidates), and 3-round (385 candidates) CEM search, training ridge regressors on 23-D search statistics or 384-D visual features to predict whether an extra round yields positive closed-loop return. On PointMaze and Reacher, audit MSEs of learned regressors (0.2086 and 0.1819) do not improve over constant-mean baselines (0.2052 and 0.1722), indicating that while forward prefix risk reliably determines when to trigger search, the marginal benefit of 1–3 rounds within triggered states is governed by non-linear stochastic sampling trajectories.

SAR Process Safety Criteria and Thresholds

Table A7 lists the exact process constraint parameters for Eqs. (2a)–(2b). For any controller, an independent evaluator applies identical geometric rules and numerical tolerance ε\varepsilon to determine goal arrival GG and process violation VεV_{\varepsilon}:

Commanded Penetration, Cumulative Penetration, and Persistence

In TwoRoom, let d(p)d(p) be the non-negative geometric penetration depth of the agent disk at position pp, ctc_t the maximum penetration along the raw commanded segment prior to collision projection, and ot+1=d(pt+1)o_{t+1}=d(p_{t+1}) the resulting state overlap. Defining δt=max⁡{ct−d(pt),0}\delta_t=\max\{c_t-d(p_t),0\} and Δt=δtI{δt>ε}\Delta_t=\delta_t\mathbb{I}\{\delta_t>\varepsilon\}, an episode is marked as violating if single-step depth >0.501>0.501 px, cumulative added penetration ∑tΔt>1.001\sum_t\Delta_t>1.001 px, or consecutive penetration reaches 3 steps. PointMaze audits continuous disk sweeps; Reacher, PushT, and Cube audit geometry after every native physics substep. Only prefixes executed prior to goal termination are audited.

Fixed Geometric Definitions for Contact Manipulation

PushT places a circular hazard of radius 28 px at (256, 256) px, auditing the true T-block polygon and 15 px circular pusher (object vertices within [7,504]2[7,504]^2 px, pusher within [22,489]2[22,489]^2 px). Cube places a spherical hazard of radius 0.04 m at (0.425, 0, 0.055) m, auditing the oriented object bounding box (within [0.22,0.63]×[−0.38,0.38]×[−0.02,0.45][0.22, 0.63]\times[-0.38, 0.38]\times[-0.02, 0.45] m) and a 0.025 m grasp-point proxy sphere. Normal pushing, grasping, and table support contacts are permitted.

Arrival Conditions and Step Budgets

Goal arrival requires distance <16<16 px in TwoRoom (<8<8 px with valid door traversal in Figure A2), <0.2<0.2 in PointMaze, <0.2inReacher,and<0.2 in Reacher, and<0.2 in PushT. Maze, joint angle errors <0.05< 0.05 rad in Reacher, object-to-goal Euclidean distance ≤0.04\le0.04 m in Cube, and native eval_state success in PushT. Step budgets Tmax⁡T_{\max} are 50 (TwoRoom, Cube, PushT), 200 (Reacher), and 50/200 (PointMaze local/route).

Mechanism Comparison and Half-Space Tangential Progression Derivation

Table A8 contrasts RTS against representative safe control paradigms across six dimensions. To formalize the anti-stalling property of dAPd_{\mathrm{AP}} in a local half-space model, let nn be a unit normal pointing into the forbidden half-space, n⊤u≤0n^\top u \le0 the safe half-space constraint, and uD=vnu_D = vn (v>0v > 0) a nominal action pointing orthogonally into the obstacle. Under dMSEd_{\mathrm{MSE}}, the unique orthogonal projection onto n⊤u≤0n^\top u \le0 is the zero-amplitude stop action u=0u=0. Under dAPd_{\mathrm{AP}}, evaluating a tangential detour action u=wtu=wt (w≥0w \ge0) along unit tangent t⊥nt \perp n yields single-step cost:

dAP(wt,vn)=v2+w2+(w−v)2=2(w−v/2)2+32v2.(13)d_{\mathrm{AP}}(wt,vn)=v^2+w^2+(w-v)^2 =2(w-v/2)^2+\frac{3}{2}v^2. \tag*{(13)}

Setting tangential step length w=v/2w=v/2 achieves the minimum cost 1.5v21.5v^2, strictly lower than full stopping (w=0w=0, cost 2v22v^2), geometrically incentivizing tangential progression over orthogonal stalling.

Evaluation Protocol and Risk Gate Operating Points

Evaluation Configuration

Table A9 lists all receding-horizon execution intervals used in Table 1. For Table 1, RTS-MSE and RTS-AP allow at most four searches per episode on TwoRoom and Reacher, then execute the frozen Direct policy for the remainder of the episode. This budget is motivated by the budget sweep (Section H.3), where moderate search caps can improve safe arrival by reducing conservative detours and timeouts.

Risk Gate Operating Points

The 997-task Cube Full Test includes 388 tasks whose initial states already satisfy the goal distance tolerance; across the remaining 609 active manipulation tasks, 25-step Direct and RTS achieve 95.89% and 96.55% SAR, respectively. On the full Cube test set, RTS improves SAR by +0.40 pp over Direct (95% CI [-0.30,1.14]), +1.27 pp over Guarded A ([0.30,2.24]), and +3.31 pp over Recovery-MPC ([2.27,4.38]).

Tables A10–A11 report detailed nominal and candidate-level detection counts corresponding to Figure 3(a,Figure 3b).

Table A8. Six-dimensional mechanism comparison between RTS and prior safe control paradigms. FZ: frozen backbone; Pure-Vis: purely visual closed-loop (no external analytical shield); Pass: exact pass-through of safe nominal plans; Eqv.: strict equivalence between bypassed search and every-step same-kernel search (Prop. 1); Anti-Stall: anti-orthogonal-shrinkage stalling.

Feedback Intervals, Risk Horizons, and Constrained Search Budgets (K-of-N)

Six-Point Feedback Scans under Fixed Risk Windows

With search available at every risk alarm, we conduct two 69,426-trajectory scans across k∈{1,2,5,10,15,25}k \in\{1, 2, 5, 10, 15, 25\}: (1) a fixed-risk-window scan (Figure A3 and Table A12); and (2) a horizon-aligned scan where executing 1/2/5/10/15/25 steps audits the corresponding 5/5/5/10/15/25-step risk prefix (Figure A4).

Table A12. Closed-loop SAR (%) of RTS across receding-horizon execution intervals kk under fixed risk windows. Cube in Table 1 uses k=25k = 25.

Taskk=1k = 1k=2k = 2k=5k = 5k=10k = 10k=15k = 15k=25k = 25
TwoRoom70.5069.3371.3386.8387.8383.67
Reacher38.3343.6739.3346.8348.6747.17
SimplePointMaze71.6174.7475.7877.3472.4069.79
Cube94.9297.0297.6997.6397.1297.89
PushT (val)79.6981.2581.2582.0382.0378.91

Table A12.

Figure A3. Six-point receding-horizon feedback scan under fixed risk audit windows. The horizontal axis denotes execution/observation interval k∈{1,2,5,10,15,25}k \in\{1, 2, 5, 10, 15, 25\}; columns show strict Safe Arrival Rate (SAR), mean episode controller time, and single-decision latency (PushT evaluated on 128 validation tasks).

Figure A4. Six-point receding-horizon feedback scan with risk audit coverage scaled to match the execution window (Horizon-Aligned Scan). Columns show SAR, mean episode controller time, and single-decision latency (PushT evaluated on 128 validation tasks).

Horizon-Aligned Risk Windows

Under the horizon-aligned scan at k=10k = 10, Cube RTS achieves 98.40% SAR versus 97.69% for Direct (+0.70 pp paired gain, 95% CI [0.23,1.27]; 97.37% vs. 96.22% across the 609 non-trivial initial tasks), while Table 1 reports the validation-locked k=25k = 25 configuration (97.89% SAR).

Full K-of-N Constrained Budget Curves

Figure A5 extends the 50% comparison in Table 4 to search-call caps of 25%, 50%, and 100% of decision slots. Risk-triggered, fixed-period, and uniform random schedules share the same amplitude-preserving search kernel. The full curves report safe arrival, violations, and measured controller time across these operating points; panel (a) separately reports the same-kernel paired timing protocol described in Section A.2.

Training-Time Fine-Tuning Alternatives and External Baseline Verification

Recovery-MPC Adaptation. Recovery-MPC adapts the predictive risk-sum planning mechanism of Recovery RL [32] to the shared frozen JEPA world model and task risk heads, which replace the original visual representation and Bellman risk critic. At each observation, frozen Direct produces a 25-step nominal plan UDU_D. Recovery search is triggered when any of its five action blocks raises an alarm:

A(UD)=I[max⁡j=0,…,4ℓj(UD)>0].(14)A(U_D)=\mathbb{I}\left[\max_{j=0,\ldots,4}\ell_j(U_D)>0\right]. \tag*{(14)}

If A(UD)=0A(U_D)=0, the nominal plan is executed unchanged. Otherwise, CEM minimizes the summed sigmoid risk:

Jrec(U)=∑j=04σ(ℓj(U)+τ),(15)J_{\mathrm{rec}}(U)=\sum_{j=0}^{4}\sigma\left(\ell_j(U)+\tau\right), \tag*{(15)}

where σ\sigma is the logistic sigmoid and adding τ\tau restores the unshifted risk-head score from Eq. (5). CEM uses JrecJ_{\mathrm{rec}} alone for elite ranking, best-candidate retention across rounds, and final selection over all scored candidates, including those that still raise an alarm.

Candidate Search. On TwoRoom, Reacher, and Point-Maze, CEM samples 50-dimensional Gaussian action sequences centered initially on UDU_D, with 8 elites per round. The initial standard deviation and its floor are 0.25/0.05 for TwoRoom and Reacher, and 0.05/0.01 for PointMaze. Actions are clipped to [−1,1][-1,1] and [−0.2,0.2][-0.2,0.2], respectively. On Cube and PushT, each round combines 96 latent-goal candidates decoded by the frozen action generator with 32 smooth demonstration-based action proposals, retaining 8 elites per family. Latent-goal sampling uses an initial standard deviation of 0.25 and a floor of 0.05; the action branch scales perturbations by the task action scale, interpolates five temporal knots to 25 steps, and clips actions to [−1,1][-1, 1].

Task / Risk TagNominal TP / Haz.Nominal Recall (%)Nominal FP / SafeNominal FPR (%)Candidate TP / Haz.
TwoRoom44 / 4695.655 / 826.10727 / 795
Reacher (continuous 25-step)18 / 18100.003 / 1102.73313 / 330
SimplePointMaze74 / 7894.871 / 502.005,771 / 6,013
Cube (signed-clearance)2 / 2100.005 / 4311.6378 / 99
PushT (Short-5, q95)4 / 4100.008 / 2423.311,401 / 1,453

Table A10. Nominal and candidate hazard detection at the Figure 3(b) operating points. Recall denotes TP / Hazardous; False-Alarm / False-Rejection Rate (FPR) denotes FP / Safe.

TaskNmixN_{\mathrm{mix}}Within-Context AUROCCand. TP / Haz.Recall (%)Cand. FP / SafeFPR (%)
TwoRoom560.901727 / 79591.4583 / 1,3816.01
Reacher190.962313 / 33094.8546 / 1,8462.49
SimplePointMaze1280.9825,771 / 6,01395.98257 / 2,30711.14
PushT (q95)210.9851,401 / 1,45396.422,600 / 93,2572.79
Cube170.92678 / 9978.79125 / 1,3249.44

Table A11. Five-task candidate risk ranking AUROC and operating point statistics. NmixN_{\mathrm{mix}} denotes audit contexts containing both safe and hazardous candidates.

Each round reserves slots for the nominal plan, zero actions, and the best candidate found so far within its 128-candidate budget. Three rounds plus one final mean evaluation yield 385 search evaluations; including the initial nominal-plan risk evaluation gives 386 plan risk evaluations per triggered decision. Each method updates its proposal distribution from its own elites. Recovery-MPC audits all 25 steps on every task and executes 25/25/5/25/25 steps on TwoRoom/Reacher/PointMaze/Cube/PushT. Table A9 lists the corresponding RTS execution intervals and risk windows, including the 5-step PushT risk window.

Matched Comparison of BC and RiskFT (Table A13). Actor fine-tuning updates only the policy π\pi while keeping the visual encoder, world model, risk head, and spatial readout frozen. BC applies action MSE anchoring, whereas RiskFT adds the softplus risk loss from the frozen risk head; both share initialization and training budgets (600 AdamW steps, batch size 32, learning rate 3×10−53 \times10^{-5}). While RiskFT yields +1.50 pp on Cube, it degrades PushT SAR by -4.29 pp and has confidence intervals spanning zero on the remaining tasks, highlighting the fragility of multi-objective fine-tuning relative to test-time intervention.

Impact of Visual Hazard Overlays (Table A14) and Native Calibration (Table A15). For LeWM [22] and Fast-LeWM [11], we compare inputs with visual hazard overlays (zone, default in Table 1) against inputs without overlays (clear, matching their pre-training appearance; 34,020 trajectories total). Removing overlays improves contact SAR by 7.48–14.44 pp by reducing visual distribution shift, but VR remains high (e.g., Fast-LeWM on clear PushT reaches 87.17% SR but 25.18% VR and 70.22% SAR). Table A15 reports our native unconstrained success rates alongside the published reference results, with the TwoRoom evaluation settings listed separately.

Table A14. Paired comparison of external baselines with (zone) and without (clear) visual hazard overlays (SAR/SR/VR in %).

TaskModelzone SARclear SARclear SRclear VRSAR Diff [95% CI]
TwoRoomLeWM / Fast34.50 / 34.8334.50 / 34.8378.17 / 85.0065.33 / 64.500.00 / 0.00
ReacherLeWM2.172.6725.6764.33+0.50 [−1.33, 2.50]
Fast-LeWM2.175.1741.5060.50+3.00 [0.50,5.67]
PushTLeWM57.5667.6482.4225.18+10.08 [8.24,11.94]
Fast-LeWM62.7470.2287.1725.18+7.48 [5.77,9.23]
CubeLeWM52.9662.6264.798.69+9.66 [7.79,11.57]
Fast-LeWM52.8367.2769.918.12+14.44 [12.40,16.52]

Table A14.

Table A15. Native unconstrained success rates: measured / published reference (%) from [11]. TwoRoom settings specify the goal interval and step budget used in our evaluations; the same published reference rate is shown for both settings.

SettingLeWMFast-LeWM
TwoRoom (25/50)88 / 8796 / 98
Reacher76 / 8686 / 88
TwoRoom (100/150)10 / 8738 / 98
PushT96 / 9698 / 96
Cube70 / 7478 / 80

Table A15.

PointMaze Adaptation. For PointMaze, we train LeWM and Fast-LeWM from random initialization using their respective released implementations. The training split contains 512 trajectories from 32 maps, and the validation split contains 128 trajectories from 8 maps; evaluation uses 64 disjoint test maps. LeWM and Fast-LeWM are trained for 100 and 10 epochs, respectively, and evaluated using the final checkpoints. Both controllers replan every five primitive steps.

Tool use. OpenAI Codex assisted with manuscript revision and \LaTeX preparation; Google Gemini with English translation and phrasing; and ChatGPT with experimental code and an initial main-figure draft. The authors reviewed and revised these outputs and take responsibility for the final work.

References

  1. [1]Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML), pages 22–31, 2017. 2arxiv.org/abs/1705.10528
  2. [2]Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 18th European Control Conference (ECC), pages 3420–3431, 2019. 2, 3arxiv.org/abs/1903.11199
  3. [3]Mehul Anand and Shishir Kolathaya. Safety certification in the latent space using control barrier functions and world models. arXiv preprint arXiv:2507.13871, 2025. 2, 3arxiv.org/abs/2507.13871
  4. [4]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629, 2023. 2arxiv.org/abs/2301.08243
  5. [5]Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. 2arxiv.org/abs/2506.09985
  6. [6]Adrien Bardes, Jean Ponce, and Yann LeCun. VI-CReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations (ICLR), 2022. 2DOI
  7. [7]Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471, 2024. 2arxiv.org/abs/2404.08471
  8. [8]Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757, 2018. 2arxiv.org/abs/1801.08757
  9. [9]Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018. 2arxiv.org/abs/1812.00568
  10. [10]Xianzhe Fan, Yuxiang Lu, Shenyuan Gao, Xiaoyang Wu, Ruihua Han, Manling Li, and Hengshuag Zhao. DreamAvoid: Critical-phase test-time dreaming to avoid failures in VLA policies. arXiv preprint arXiv:2605.11750, 2026. 2, 3, 17arxiv.org/abs/2605.11750
  11. [11]Yuntian Gao and Xiangyu Xu. Fast LeWorldModel. arXiv preprint arXiv:2606.26217, 2026. 2, 5, 6, 18, 19arxiv.org/abs/2606.26217
  12. [12]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 2arxiv.org/abs/1809.01999
  13. [13]Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning (ICML), pages 2555–2565, 2019. 2arxiv.org/abs/1811.04551
  14. [14]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020. 2arxiv.org/abs/1912.01603
  15. [15]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647–653, 2025. 2DOI
  16. [16]Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024. 2arxiv.org/abs/2310.16828
  17. [17]Nicklas A. Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning (ICML), pages 8387–8406, 2022. 2arxiv.org/abs/2203.04955
  18. [18]Weidong Huang, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. SafeDreamer: Safe reinforcement learning with world models. In International Conference on Learning Representations (ICLR), 2024. 2, 3arxiv.org/abs/2307.07176
  19. [19]Matthew Kim, Kensuke Nakamura, and Andrea Bajcsy. How well do latent world models understand partially observable safety constraints? In Conference on Robot Learning (CoRL), 2026. 3arxiv.org/abs/2510.06492
  20. [20]Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022. 2arxiv.org/abs/2202.10054
  21. [21]Yann LeCun. A path towards autonomous machine intelligence. OpenReview Preprint, 2022. 1, 2
  22. [22]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026. 1, 2, 5, 6, 18arxiv.org/abs/2603.19312
  23. [23]Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In International Conference on Learning Representations (ICLR), 2023. 2arxiv.org/abs/2209.00588
  24. [24]Heeung Nam, Quentin Le Lidec, Lucas Maes, Yann LeCun, and Randall Balestriero. Causal-JEPA: Learning world models through object-level latent masking. In International Conference on Machine Learning (ICML), pages 91808–91832, 2022. 2
  25. [25]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeddin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024. 2arxiv.org/abs/2304.07193
  26. [26]Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learning (CoRL), pages 1049–1065, 2021. 2arxiv.org/abs/2008.06389
  27. [27]Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. OpenAI technical report, 2019. 2
  28. [28]Reuven Y. Rubinstein and Dirk P. Kroese. The Cross-Entropy Method: A Unified Approach to Combinatorial Op- timization, Monte-Carlo Simulation, and Machine Learning. Springer, 2004. 2
  29. [29]Junwon Seo, Kensuke Nakamura, and Andrea Bajcsy. Uncertainty-aware latent safety filters for avoiding out-of-distribution failures. In Conference on Robot Learning (CoRL), pages 4442–4472, 2025. 3arxiv.org/abs/2505.00779
  30. [30]Vlad Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestriero, Tim G. J. Rudner, and Yann LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models. arXiv preprint arXiv:2502.14819, 2025. 1, 2arxiv.org/abs/2502.14819
  31. [31]Junhan Sun, Hao Zhao, and Guofeng Zhang. INTACT: Isomorphic intent-to-action learning for search-free world models. arXiv preprint arXiv:2607.26056, 2026. 1, 2, 3, 5, 6arxiv.org/abs/2607.26056
  32. [32]Brijen ThananJeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery RL: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3): 4915–4922, 2021. 2, 3, 17
  33. [33]Kim P. Wabersich and Melanie N. Zeilinger. A predictive safety filter for learning-based control of constrained nonlinear dynamical systems. Automatica, 129:109597, 2021. 2, 3, 17arxiv.org/abs/1812.05506
  34. [34]Kim P. Wabersich, Andrew J. Taylor, Jason J. Choi, Koushil Sreenath, Claire J. Tomlin, Aaron D. Ames, and Melanie N. Zeilinger. Data-driven safety filters: Hamilton-Jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5): 137–177, 2023. 2DOI
  35. [35]Albert Wilcox, Ashwin Balakrishna, Brijen ThananJeyan, Joseph E. Gonzalez, and Ken Goldberg. LS3: Latent space safe sets for long-horizon visuomotor control of sparse reward iterative tasks. In Conference on Robot Learning (CoRL), pages 959–969, 2022. 2, 3arxiv.org/abs/2107.04775
  36. [36]Wei Xiao, Colin Belta, and Christos G. Cassandras. Event-triggered control for safety-critical systems with unknown dynamics. IEEE Transactions on Automatic Control, 68(7): 4143–4158, 2023. 2, 3, 17
  37. [37]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 5824–5836, 2020. 2arxiv.org/abs/2001.06782
  38. [38]Kaiming Zhong, Tianhua Liu, and Yue Wang. Calibrated predictive safety for heterogeneous robots: An action-conditioned JEPA framework with model-based safety shields. arXiv preprint arXiv:2608.17496, 2026. 3, 17arxiv.org/abs/2608.17496
  39. [39]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983, 2024. 2arxiv.org/abs/2411.04983

Paper details

vlnc:W_73ww99ym
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents